GitHub Copilot Chat - How old is it's model?

[Gareth064]

Select Topic Area

Question

Body

I just asked Copilot Chat

"Do you know what the latest version of dotnet is currently"

And it responded with

"Yes, the latest version of .NET is 5.0. It was released on November 10, 2020. It includes many new features such as improved performance, support for Windows ARM64, and new APIs that allow developers to build applications with .NET that were not possible before."

Is this just using the same data used to train GPT-4's model? So its only valid up until September 2021?

[bleugreen]

This is totally a guess, but based on the speed of the chat / general level of understanding, I think its gpt-3.5-turbo

[Gareth064]

This is disappointing. Since its a coding companion, I would have expected it to be using a model which has been updated with all the latest code knowledge. A lot has changed in the dev world since 2021. At a minimum I would have thought that it would be able to look at any public code in GitHub to help answer your questions. There is a new Blazor UI library in GitHub that I wanted it to help me with but it didn't have a clue.

[dgokcin]

I just got accepted to the private beta. I asked copilot chat what it uses behind the scenes and it answered gpt-3.

Considering copilot chat is a part of copilot x, which seems to be powered by gpt-4, I expected the beta to use gpt-4. I hope that they update it to use gpt-4 since there are so many community plugins that already does the same thing as copilot chat does.

[Gareth064]

I just want a code copilot that is up to date with the latest code, docs, etc.

[bleugreen]

Yeah I agree. I can see how gpt-3.5-turbo is cost effective (especially for alpha/beta testing), but the lack of depth makes it basically unusable for anything but the simplest of boilerplate. The appeal of a editor-native chat for me is to be able to ask specific, nitty gritty questions that I can't find on google.

[Gareth064]

I get that they don't want these LLM's to contain up to date data for current affairs, the type of stuff that would let bad actors try to spread misinformation for upcoming elections, but there must be a way to target specific data to keep updating the model with. There are some low hanging trees like MS Docs, W3 Schools, MDN Web Docs, etc. And since this is a GitHub service, then update the model with the latest GitHub repos (that are public). And if there are Repo owners, who have public repo's but don't want them in the model then introduce a flag you can set on your repo to exclude it from the "crawl" etc.

[Gareth064]

As a dotnet developer. I'd love for GitHub Copilot to be kept up to date with changes for Microsoft's Open Source repos, which I believe includes the contents of MS Docs as well.

[Mmhmmmmm]

Yes, I find this in extension package of Copilot Chat

This product includes software developed by AIDungeon.

** GPT-3-Encoder 1.1.3 **

[beefster]

I have recently been let into the beta; when I prompt copilot chat it claims to be using ''OpenAI Codex"

[github-actions[bot]]

🕒 Stale Discussion Alert 🕒

This Discussion has been labeled as stale by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This stale notification will only apply to Discussions with the Question label. To learn more, see our recent stalebot announcement.

Thank you for helping bring this Discussion to a resolution! 💬

[MrBns]

After a long time, they've been switched to OpenAi Codex model.
I used it in learning golang latest version. but as much I understood it provide pretty old code..

[DeveloperChris]

The latest co-pilot appears to be GPT-4 however I am quite disappointed in the responses that quite often refer to old outdated data.

For example this morning it provided links to aws resources on github that no longer exist. the current version is 17 the provided version was 14.

You would think that a github specific LLM like github co-pilot would be much more up to date on its training dataset and regularly updated. I know the training data has to be screened to keep unwanted data (like the recent explosion of exploited repo's) out of the training data but is 6 months behind too much to ask for?

[accts4mjs]

When asked what the latest version of data is, this was copilot's response:

As an AI programming assistant, I don't have access to real-time data or any database. I generate responses based on pre-existing knowledge up to the point of my last training data, which includes information up to September 2021. I don't have the ability to access or retrieve personal data unless it has been shared with me in the course of our conversation. I am designed to respect user privacy and confidentiality.

[aunovis-seiberlich]

Hi, @DeveloperChris! Thanks for your screenshot! I've observed this (or similar) response(s) and wondered how they match up with the end-of-November announcement that Chat uses GPT-4:

[MrBns]

Hi, @DeveloperChris! Thanks for your screenshot! I've observed this (or similar) response(s) and wondered how they match up with the end-of-November announcement that Chat uses GPT-4:

they should really shamed of that.

[DeveloperChris]

Perhaps the difference is in that I am using the paid version of github copilot? Thus I get GPT4.

[mcscheer23]

@DeveloperChris nope, same with paid version of github copilot:

[raayu83]

IMHO kind of embarrassing using data that old on a premium product that needs up to date data to make correct suggestions.

[MrBns]

I used copilot for learning golang. it was useful most of the case. but a lots of time it provide me older data that is not supported. pretty older.

[ashishagarwal2023]

GitHub Copilot uses an AI model that was developed in collaboration with OpenAI, Microsoft Azure AI, and GitHub. The model offers a 13% latency improvement over the previous OpenAI Codex model, and enhanced contextual filtering that takes into account a wider range of a developer's context and usage patterns.
GitHub Copilot is powered by the OpenAI Codex, which is a modified, production version of the Generative Pre-trained Transformer 3 (GPT-3), a language model.

[DeveloperChris]

Who said we are fighting? My question was one on relevance.

As to the GPT-3/4 question it seems that github are running multiple versions. My assumption would be you get GPT4 if you are on a paid subscription GPT-3 otherwise.

As a paid subscriber I expect that the AI would have a later dataset. As a developer I want cutting edge. Therefore a 2 year out of date AI is only marginally useful for me and can be downright harmful as it suggests methods and technologies that have been superseded and may have critical issues.

[raayu83]

I agree here. Started using the professional version and if I remember correctly the latest version of Django it knows is 3.2 - the current version is 5.0.x

[aunovis-seiberlich]

Sorry I didn't write back earlier - I don't know why GitHub isn't clearer about what they're doing. When we had two Q&A sessions at a company event with GitHub's Tom Horton and Chris Reddington, they explained that Copilot Chat in some way figures out whether Codex or GPT-4 can answer your question. They weren't specific how Copilot does this - my humble guess is that the size of the included context plays an important role. The extension in VS Code is very transparent about it and shows for each answer which model generated it.

[mcscheer23]

@aunovis-seiberlich where can I find the information which model generated the answer in VSCode Extension?

[aunovis-seiberlich]

In the output window, select GitHub Copilot Chat. This happens If I ask Copilot for the latest C++ version it knows:

2024-03-28 08:58:01.761 [info] [chat fetch] url https://api.githubcopilot.com/chat/completions
2024-03-28 08:58:01.761 [info] [chat fetch] modelMaxPromptTokens 3072
2024-03-28 08:58:01.761 [info] [chat fetch] modelMaxResponseTokens 3947
2024-03-28 08:58:01.761 [info] [chat fetch] chat model gpt-4
2024-03-28 08:58:03.794 [info] [chat fetch] request.response: [https://api.githubcopilot.com/chat/completions], took 2032 ms
2024-03-28 08:58:05.868 [info] [streamMessages] message 0 returned. finish reason: [stop]
2024-03-28 08:58:05.869 [info] [streamChoices] request done: requestId: [xxxx] responseId: [xxxx] model deployment ID: [xxxx]
2024-03-28 08:58:05.933 [info] [chat fetch] url https://api.githubcopilot.com/chat/completions
2024-03-28 08:58:05.933 [info] [chat fetch] modelMaxPromptTokens 7168
2024-03-28 08:58:05.933 [info] [chat fetch] modelMaxResponseTokens 4096
2024-03-28 08:58:05.933 [info] [chat fetch] chat model gpt-3.5-turbo
2024-03-28 08:58:06.414 [info] [chat fetch] request.response: [https://api.githubcopilot.com/chat/completions], took 481 ms
2024-03-28 08:58:06.424 [info] [streamMessages] message 0 returned. finish reason: [stop]
2024-03-28 08:58:06.425 [info] [streamChoices] request done: requestId: [xxxx] responseId: [xxxx] model deployment ID: [xxxx]

[n-r-w]

I agree here. Started using the professional version and if I remember correctly the latest version of Django it knows is 3.2 - the current version is 5.0.x

Same for golang. It's still suggesting outdated code ("io/ioutil" for example).

[MrBns]

it give us a lots of outdated suggestion.. I left it to use. waiting zed release for linux.

[christianbueno1]

Interesting. ChatGPT in web with free subscription gives me an answer using knowledge until November 2023.

I do need that model in GitHub Copilot. I like GitHub Copilot and do have a Copilot Individual Plan, but don't like to have an outdated model there. Give us acces to GPT-4o with GitHub Copilot in VSCode

[NendosColl]

I would prefer Claude 3.5 Sonnet.

[raayu83]

tested Claude 3.5 Sonnet with some django related questions a few days ago, it was way worse then GPT4o