Languages section of sidebar uses funny math, meter makes no sense

[adamlui]

Select Topic Area

Question

Body

For example, https://github.com/KudoAI/duckduckgpt has a few hundred lines of Python and thousands of lines of JS, yet the meter shows

[Sworzen1]

Man, you can click on each language and you will see files with code :D

[adamlui]

But why are you telling me this?

[Sworzen1]

due your question

[adamlui]

You should try doing it, it confirms I have 2 JS files with thousands of lines of code, and 2 Python files with few hundred lines. (So JavaScript should be higher percent than Python)

[asdacosta]

It displays the Language that has most files, not most code.

[adamlui]

I doubt it but there are 2 files for each so it should be 50/50 by your interpretation. But also look at https://github.com/KudoAI/bravegpt it's like a clone repo with very similar codebase. Note that one makes JS majority instead (so a bug is clearly occurring)

[Sworzen1]

You have file .user.js not .js - probably github doesn't take duckduckgpt.user.js file into calculations

[adamlui]

It is JavaScript thuogh even shows when you click JavaScript below the meter: https://github.com/search?q=repo%3AKudoAI%2Fduckduckgpt++language%3AJavaScript&type=code
.

Also even if true, then by the same logic https://github.com/kudoai/bravegpt should have JS be small on meter (the two repos have 90% identical codebase) yet the meter is opposite

[asdacosta]

It's possible the language detection tool might be malfunctioning and can't accurately calculate the code of each. Cause I went through the first repo and truly JS is more. You can submit an issue on that to GitHub.

[adamlui]

but this forum is the method they encourage users encountering such issues to submit them on (hence why I made this post)

[asdacosta]

Ok
I doubt there is another reason but I will leave for an expertise to come help you then...

[aamijar]

Most likely Github has not recalculated the language statistics for your most recent push yet.

When you push changes to a repository on GitHub.com, a low priority background job is enqueued to analyze the default branch of your repository as explained above.

Github uses linguist to calculate the language percentages
https://github.com/github-linguist/linguist/blob/master/docs/how-linguist-works.md

The percentages are calculated based on the bytes of code for each language as reported by the List Languages API.

[adamlui]

Why do you deem recalculation of most recent push "most likely" the cause? Your assumption is I added thousands of lines of JavaScript in a single push thus propelling JS to be the sudden leader? Does that sound right to you?

[aamijar]

Yeah I think the most recent JS push hasn't been calculated yet.

[adamlui]

But why? You think my last push was over 1000 lines of JS? (This is the only condition which would change Python being leader to JS, I already explained Python is few hundred and JS is few thousand)

[adamlui]

I'm saying my last push is obviously not thousands of lines of JS, so it cannot possibly be the cause

[aamijar]

Ah yeah that is true, so I guess that's not the cause.

[asdacosta]

The guidelines over here will help you:
https://github.com/github-linguist/linguist/blob/master/docs/troubleshooting.md

[adamlui]

What do you want me to read?

[asdacosta]

Fixing a misclassified language

Most languages are detected by their file extension defined in languages.yml. For disambiguating between files with common extensions, Linguist applies some heuristics and a statistical classifier. This process can help differentiate between, for example, .h files which could be either C, C++, or Obj-C.

Misclassifications can often be solved by either adding a new filename or extension for the language or adding more samples to make the classifier smarter.

You can get the full documentation here:
https://github.com/github-linguist/linguist/blob/master/CONTRIBUTING.md
This might also be helpful:
https://github.com/github-linguist/linguist/blob/master/docs/overrides.md

[adamlui]

But you highlighted this:

Misclassifications can often be solved by either adding a new filename or extension for the language or adding more samples to make the classifier smarter.

This does not help me, I still do not know what you want me to change. What file do you speak of that requires the changing of filename or extension?

[asdacosta]

That wasn’t the only thing I highlighted from the documentation. And I provided the full documentation so if you’re confused, look it up there.

[adamlui]

The second thing you highlighted was equally unhelpful. Rather than copy/paste random stuff, perhaps tell me what you want me to do? (The full documentation is not useful if you don't read it yourself to find what will help

[adamlui]

(How can I possibly mark as answer something that just says read a manual which may or may not even contain anything useful

[adamlui]

Also I am reporting what is blatantly a bug, the doc you linked is contribution guidelines to a repo, this does not help victim of bug, only those who seek to learn how to contribute to repo. The other doc you linked is equally unhelpful for similar reason. It teaches you how to override stuff, does not relate to eliminating a bug

[asdacosta]

The documentations I provided, just like what @aamijar provided, are 100% relavant to problem with the linguist which is what your question was about. Maybe instead of expecting a straightforward answer, which you won’t get, you should go through the documentation and follow the guidelines they’re providing to help navigate your problem.

Just like I said earlier that I went through your repo to check, I also definitely went through the documentation to notice it was related before recommending. Is your question not about the language detection tool - Linguist?

instead of just reading the topic about the documentation and concluding immediately that it doesn’t contain what you need, you should rather take time to read if you want your problem solved. Because the documentation does highlight a section that talks about having issues or bugs with the Linguist and possible causes and you ate to navigate them.

[asdacosta]

The documentations I provided, just like what @aamijar provided, are 100% relavant to problem with the linguist which is what your question was about. Maybe instead of expecting a straightforward answer, which you won’t get, you should go through the documentation and follow the guidelines they’re providing to help navigate your problem.

Just like I said earlier that I went through your repo to check, I also definitely went through the documentation to notice it was related before recommending. Is your question not about the language detection tool - Linguist?

instead of just reading the topic about the documentation and concluding immediately that it doesn’t contain what you need, you should rather take time to read if you want your problem solved. Because the documentation does highlight a section that talks about having issues or bugs with the Linguist and possible causes and you ate to navigate them.

I provided more documentations related to the problem to make you understand the question you asked better.

[adamlui]

The documentations I provided, just like what @aamijar provided, are 100% relavant to problem with the linguist which is what your question was about

How does a guideline on how to contribute to a repo answer the question why is math in a sidebar wrong?

Maybe instead of expecting a straightforward answer, which you won’t get, you should go through the documentation and follow the guidelines they’re providing to help navigate your problem.

But the docs provided by you and the others provide do not help in getting GitHub to fix this bug, so it is a waste of time to link to them (they are docs that teach you how to config software or contribute to repos, not deal with software bug)

Is your question not about the language detection tool - Linguist?

If Linguist is the background tool that generates the meters, it is very tangentially related, but I do not use this tool so have no need to learn how to config it. Every other meter generates fine without config, learning to config is not the answer

instead of just reading the topic about the documentation and concluding immediately that it doesn’t contain what you need, you should rather take time to read if you want your problem solved. Because the documentation does highlight a section that talks about having issues or bugs with the Linguist and possible causes and you ate to navigate them.

I actually skimmed and found nothing about possible causes, but if you want to be selected as answer, you have to answer the question not me, so feel free to highlight what the possible cause is I can take action on (otherwise you have not answered the question)

[aamijar]

Hi @adamlui I think I have found the solution.
I was able to replicate the problem by using the following steps:

Clone your repository
Install and run github-linguist CLI tool

cd duckduckgpt
github-linguist -b


63.36%  10101      Python
18.56%  2958       SCSS
10.14%  1616       JavaScript
7.94%   1266       Shell

Shell:
.husky/_/applypatch-msg
.husky/_/commit-msg
.husky/_/h
.husky/_/post-applypatch
.husky/_/post-checkout
.husky/_/post-commit
.husky/_/post-merge
.husky/_/post-rewrite
.husky/_/pre-applypatch
.husky/_/pre-auto-gc
.husky/_/pre-commit
.husky/_/pre-push
.husky/_/pre-rebase
.husky/_/prepare-commit-msg
.husky/pre-commit

JavaScript:
eslint.config.mjs

Python:
greasemonkey/utils/remove-json-keys.py
greasemonkey/utils/translate-en-messages.py

SCSS:
styles/scss/black-rising-stars.scss
styles/scss/white-rising-stars.scss

Notice that under JavaScript the file greasemonkey/duckduckgpt.user.js does not appear which is leading to the languages calculations problem.

If you run github-linguist greasemonkey/duckduckgpt.user.js on just that one file you will get this.

greasemonkey/duckduckgpt.user.js: 1148 lines (1034 sloc)
  type:      Text
  mime type: text/javascript
  language:  JavaScript
  appears to be generated source code

To avoid this file from being ignored by github-linguist as generated source code you can add this to your .gitattributes file.

greasemonkey/duckduckgpt.user.js linguist-generated=false

After making a new commit your new language percentages should look like this.

94.01%  224647     JavaScript
4.23%   10101      Python
1.24%   2958       SCSS
0.53%   1266       Shell

[adamlui]

This is a great workaround! Note a bug is still occurring since https://github.com/KudoAI/bravegpt and https://github.com/KudoAI/googlegpt do not require hacking .gitattributes