From GitHub Copilot to Artemis: Optimizing AI-Generated Code

‍

Part of the Vibe to Viable series, this time we interview Fan on how he vibe coded an agent simulation project using GitHub Copilot and optimized it further in Artemis.

Introduction

From Vibe to Viable, don't miss out on how we take some vibe, coated code and evolve it using Artemis. Artemis. Our evolutionary AI platform helps you transform from the code you have to the code you need. It uses the intelligence engine including a mixture of tools such as our context management, validation tools, evolutionary algorithms, and our data science expertise who know their way around LLMs to help you squeeze the best performance out of your code.

So let's evolve.

Transcript

Anni: Hi everyone. I'm Anni and I'm a developer evangelist here at TurinTech AI, and today I'm super excited to welcome Fan who is going to showcase his awesome project that he's been working on. Hey, fan, would you like to introduce yourself first? 

Fan: Hi Anni. Thank you. My name is Fan. I'm the co-founder and chief scientist at TurinTech AI and thank you to have me. 

Anni: Thank you for joining us. So Fan, could you give us a quick overview of what your project is and what inspired you to create it? 

Fan: Sure. So I've always wanted to build agent simulation framework that allows the simulation of the interactions between different agent and agent within the environment, and also the evolution of the agent.

So I used large language models to create this project and build the whole project from scratch. However, I did find that the simulation is a little bit slow. That prompted me to use Artemis to optimize this particular project. 

Anni: That's very cool. You mentioned you built this from scratch. What kind of technologies and tools did you actually use in the project to bring it to life?

Fan: Uh, yeah, sure. So I actually used VSCode plus Copilot to build this project completely from scratch. And this project is completely vibe coded and I didn't touch any of the codes. So everything is written by AI. So the code is written in Python and it uses Python game library to simulate the agent environment.

Anni: Oh, that's very cool. We love the whole vibe coding. It's such a buzzword these days. So would you mind walking us through, how did you go from vibe coding to optimization in a demo? 

Fan: Yeah, absolutely. So this is a GitHub page of the Agent Simulation Project. This project is completely built by AI and. What I did was to import the project to Artis, which is a code optimization platform.

So after I import the project into Artemis, I used the tool to automatically analyze the source code to find out where are the bottlenecks of the performance in the project. So you can run scoring to score different functions or different files of the project to find out where are the performance bottlenecks, and then the next step you can generate different code versions or code suggestions from the tools directly. 

So for this particular project, I use a feature called Artemis Intelligence, which uses an evolutionary optimization framework to iteratively generate different versions, different recommendations of the original source code. So for example, here you can run Artemis Intelligence.

There are some advanced options that I definitely recommend. After the tool generates the different version of recommendations, you can actually compile it, test it, and benchmark it. That will allow you to validate the code changes, but more importantly, it gives some feedback to the evolutionary framework to allow it to generate next iteration of optimization based on the performance of the previous iteration.

Also, it allows you to customize the prompt of exactly what you want to achieve with the project. So after I generate a lot of different code versions from Artemis Intelligence, you can actually run an optimization process to try to find different combination of those code changes to find the best version of your software.

So here is an example of the result. So it actually runs a lot of different versions of the original software. Each version is a combination of the code recommendations from the previous page, and we are actually running and validating. Those code changes and using benchmarks to evaluate for the performance of those versions.

So after we generate a lot of the version and potentially you find the best version out of those code recommendations, you can actually generate a pull request back into the original repository. So let me show you in the original GitHub repository, I actually have a pull request that is generated from automate directly.

Now, let me show you what is the exact result of this optimization. First, I run the original version, so this is a agent simulation framework that simulates different agents, interact with the environment, and as you can see, it is running extremely slow and after the optimization. That is the result. As you can see, there is a huge difference between those two versions and to be precise, the actual difference is 35 times faster.

Anni: That is really, really cool. Thank you so much for showing this project to us then. So what would you say is the one key thing you learned whilst working on this project? Do you have any insights or advice? 

Fan: So for this particular project, what I learned that I think is important is prompting. If you use a generic prompt, you may get a very average result, but if you can provide more specific instructions of exactly what you want and how you want to you want to optimize it, the tool can actually provide better and more relevant code recommendations for your particular project, which is very helpful for achieving that 35 times faster result. 

Anni: That makes a lot of sense. Prompting is important. The more specific it is, the better. So did you face any challenges during this process or areas you think you'd like to explore further into your project?

‍Fan: Yes. So on Artemis, there is an agent framework that allows developers to describe what they want to achieve in natural language, and the agent will take that instructions and also the source code of your project and actually implement what they want to achieve by actually making changes to the source code. It will automatically take in the instructions and intelligently find out exactly where they need to make changes to the original source code and execute that plan precisely, and you will get the improved version or implemented features back directly from the agent. 

Anni: That sounds really cool. I can't wait to see it next time when you show us. So finally, what advice would you give to other developers looking to build something similar with AI or LMS and things like that? 

Fan: Yeah, I think code validation is very important, which is not provided by any other AI tools. So if you just take the code recommendations for large linkage models, there is no guarantee that it is even correct. So if you can actually validate it, which is part of the feature of Artemis, you can actually get the absolutely correct result. And also you can even benchmark it to make sure that you get the best performing code recommendations. 

And also another thing I think is important when using Large Language Models is the context. So context is definitely very important. The more information you can provide to the large English model and also the more relevant it is, it usually can give you a better result of something that is more relevant to the particular goal you are trying to achieve. 

Anni: That makes a lot of sense. So two takeaways, code validation and context is very important.

Well, thank you so much for sharing with us today, fan. We're super excited to see what you're gonna build next. 

Fan: Thank you to have me today. 

‍Anni: Bye bye.