It'd be great if this was focused on utilizing an open-source model like Llama 2 instead of GPT-4...

[rossjohnson87]

GPT-4 is in so many other projects and has a cost associated with it. This is to develop a POC to prove out what many other similar projects have tried to as well, and most of them all have GPT-4 as the backbone model. I've come to the conclusion that having a proprietary API is probably going to lead to the same result the others have encountered; ton of hype at first, people try it once and spend $10-15 on tokens and then never touch it again as soon as the very next project with a similar goal comes along.

For sustainability, while the open source models are less capable, the fact that you can run them essentially for free would lead to much, much more participation and further experimentation by the community (and add a longer lifespan to the interest of the project by the community). The common thread in these projects are how quickly token usage grows as the AI's attempts get more and more complex. While using the best model seems like it would lead to less overall token usage since it's the 'smartest' I think honestly not being constrained by tokens will lead to the most experimentation by the community, and hopefully the most productive results.

I wish you good luck and will be watching! Just a suggestion, after all.

[zvone187]

Hey @rossjohnson87, sorry for the delay. Thank you so much for the feedback and a good luck.

Yes, what you're saying definitely makes sense. We are reviewing a PR that will enable this - basically to pipe any LLM to GPT Pilot. However, I'm not sure how far will it go without the best LLM. The costs really are quite large but I wanted to see how far can AI go in building an app so using the best makes sense. There is still a lot of research that needs to be done before GPT Pilot can create a really meaningful app. I think we need more research on how to develop GPT Pilot so it can work at scale, rather than testing different LLMs to see how good can they be.

Nevertheless, I think you'll be able to play with other LLMs and GPT pilot soon.

[dalamar96]

I have to agree with @rossjohnson87 on this one. I am very interested in researching on this to help build better and larger apps. But I don't have the capital to run against GPT 4. I would love to use Llama2Code if I could running locally, of course it will be the 13b one, not the full size, and I know it is going to be sub par, but I also see this being something that is run by a lot more people on the 13b size models rather than against 40-70b ones as a lot of us can't afford it for the larger applications.

[ahmadmumtaz360]

nice

[nalbion]

@dalamar96 if you'd like to have a bit of a play with various models, Mistral is currently free through OpenRouter, or you could use LocalAI.

I've put together some pointers on what changes may be necessary to GPT Pilot here - #206 (comment)

[ahmadmumtaz360]

yep

[GoZippy]

Can I suggest looking at some of the other projects that already made this part of the code - to allow user to self-host their own local LLM. I want to be able to define agent roles (processes) that use different LLM specialized and pruned models that are unique to the given role of that agent for speeding things up, then a manager generalist LLM perhaps that uses the larger llama or groq or opeanAI chatgpt 3.5 selections... lower cost options for generalist thought (if not free) and only use the paid ones for subagents/workers/processes which actually require the expertise of that model that the agent uses... It would mean we would build a starter loop that helps the generalist pick how to setup a new subagent - define the tasks and roles and expertise needed to complete the tasks assigned, locate a current list of "best llm" for specific tasks (like the benchmarks that are well known) and select or prompt user to select which to use from the list of available best options or ones that user defines as hosted locally or elsewhere for thought processes and interpretations based on their benchmarked qualities. This would allow us to define multiple agents working with different trimmed models that are custom trained for the task they need to accomplish and not waste compute time or memory overall. No? Are there any pre-releases I can import to a local clone and play with to make some suggestions for code to implement too?

[lucas-0liveira]

Indeed, striking a balance between capabilities and accessibility is crucial, and your suggestion of favoring open source models for their cost-effectiveness and potential for wider community engagement is a valuable one.

It will be interesting to see how projects like these evolve and adapt in the future. Your feedback and insights are appreciated, and I'm sure the developers will take them into consideration. Good luck with your endeavors as well!

[dalamar96]

I am sure they will. The amount of work they have already put into this project is wonderful! I can’t wait to go through and learn all I can about it and see how I can contribute!

Thank you for the reply and good luck to you as well!

[lucas-0liveira]

thank you ❤ @dalamar96

[nalbion]

Feel free to test GPT Pilot with Llama by configuring ENDPOINT=OPENROUTER

https://github.com/Pythagora-io/gpt-pilot/blob/main/pilot/.env.example

Local LLMs may already be supported. Apparently you just need to set OPENAI_API_BASE=http://localhost:8080 and run LocalAI.

[mansyno]

can you elaborate some on this method? is it only working when using the gpt-pilot repository clone or can i make it work also in the extension?

[aaronjolson]

I don't see any way to set ENV variables in the extension, so I would assume the command line using a local clone of the project.

[senko]

That's correct - extension currently doesn't expose this, but you can work around it. Here's a more detailed writeup: Using GPT Pilot with local LLMs