Personal Instance with Repo Access PAT Cannot Read Yearly Commit Correctly

[ForsakenRei]

Describe the bug

Update 6/10: Current situation explained in #2793 (comment)

---

Sorry I know this have been asking multiple times recently, I have been using my own vercel instance for months and everything worked well until I synced my fork yesterday and remove count_private was inplemented. Yesterday it shows 300+ commits but today it's 200+.

Counted from my contribution activity it should be 300ish in 2023, if it is last year from June 2022 to June 2023 it will be 400ish.

Top language seems working properly since the percentage didn't change much from yesterday but status card only shows public commits now? If I add include_all_commits=true the commit number(1.3K for all years) seems contain all my commits regardless of private or public repo.

I recreated a vercel instance based on 38c8a7a with a new PAT granted repo access, but the behavior is still the same.

Am I missing any argument?

Expected behavior

Status card includes commits to private repo as well.

Screenshots / Live demo link

https://github.com/ForsakenRei/ForsakenRei

From official instance

Additional context

No response

[rickstaa]

Include_all_commits=false

@ForsakenRei I just checked your stats, and for the include_all_commits=false case it looks like you did something wrong while deploying your own Vercel instance. Please ensure that the PAT has the rights to read data from your private repositories. You can check this out yourself using the https://docs.github.com/en/graphql/overview/explorer:

query userInfo($login: String!) {
  user(login: $login) {
    name
    login
    contributionsCollection {
      totalCommitContributions
      restrictedContributionsCount
    }
  }
}

{"login": "forsakenRei"}

I don't have access to your private repositories, so I will see the public ones. You will likely see a higher number if you have private commits.

Include_all_commits=true

For the include_all_commits=true case, I just noticed that we use the Public REST API, which does not have access to your private commits:

github-readme-stats/src/fetchers/stats-fetcher.js

Lines 146 to 156 in 7a302f6

	const fetchTotalCommits = (variables, token) => {
	return axios({
	method: "get",
	url: `https://api.github.com/search/commits?q=author:${variables.login}`,
	headers: {
	"Content-Type": "application/json",
	Accept: "application/vnd.github.cloak-preview",
	Authorization: `token ${token}`,
	},
	});
	};

This indeed is a bug, and it can be fixed by merging #1691. This pull request, however, cannot be merged on the Public instance since it will lead to the GraphQL limits being triggered rapidly. It will likely be deployed under an ENV variable when the GitHub action is released (see #2179).

There however is a way to solve it now on the public instance. We can change the code above so that the PAT that is set is used (see https://docs.github.com/en/rest/overview/authenticating-to-the-rest-api?apiVersion=2022-11-28) 👍🏻. It would be great if somebody can create a pull request for this 🚀.

What caused the change

The change was caused by a bug that was present in the old code:

github-readme-stats/src/fetchers/stats-fetcher.js

Lines 233 to 236 in 0fe51cd

	if (count_private) {
	stats.totalCommits +=
	user.contributionsCollection.restrictedContributionsCount;
	}

This adds the private CONTRIBUTIONS to the total commits returned from the GitHub REST API. This was removed in #2736.

[ForsakenRei]

@rickstaa thanks for all the explanations.

Here's the output of GraphQL following your example.

{
  "data": {
    "user": {
      "name": "しぐれ",
      "login": "ForsakenRei",
      "contributionsCollection": {
        "totalCommitContributions": 429,
        "restrictedContributionsCount": 0
      }
    }
  }
}

but I signed in with my account...so I guess it's not really testing my PAT then I tried this

curl -sS -f -I -H "Authorization: {myToken}" https://api.github.com

but didn't see anything about repo. I check another one PAT with full access to my account which basically have the same output so I think I'm not doing it with the right API end point?

and my PAT has repo access only

was added to my vercel instance

scratching my head lol

[ForsakenRei]

Here's a comparison of the top language card
Official Instance

Personal Instance

So my PAT is working since it reads all my personal Python code, but just not working properly for the commit?

[rickstaa]

curl -sS -f -I -H "Authorization: {myToken}" https://api.github.com

Sorry, my last response was incorrect. It seems that we do use the PAT when fetching data from the REST API:

github-readme-stats/src/common/retryer.js

Lines 24 to 28 in 44c268e

	let response = await fetcher(
	variables,
	process.env[`PAT_${retries + 1}`],
	retries,
	);

You should therefore receive the same number of commits as would you would receive when curling the REST API directly:

export TOKEN=<YOUR_PAT>
curl -sS -f -I -H "Authorization: $TOKEN" https://api.github.com/search/commits?q=author:ForsakenRei

[rickstaa]

The following permissions should work:

[rickstaa]

Please note that the Rest API is buggy (see #1515).

[ForsakenRei]

Please note that the Rest API is buggy (see #1515).

Thanks, yeah I think the numbers difference is probably caused by the buggy API. 1.2K is actually the right number.

curl -f -H "Authorization: $TOKEN" https://api.github.com/search/commits?q=author:ForsakenRei | grep to
tal

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  "total_count": 1186,
100  262k  100  262k    0     0   483k      0 --:--:-- --:--:-- --:--:--  484k

And my PAT does have repo permission, which is the only permission it has. Here's the API response after some doc reading.

curl -f -H "Authorization: $TOKEN" https://api.github.com/search/commits?q=author:ForsakenRei+author-date:\>2022-06-09 | rg total

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  "total_count": 412,
100  273k  100  273k    0     0   451k      0 --:--:-- --:--:-- --:--:--  452k

So it's actually 412 instead of 268 if I use my personal instance.

[rickstaa]

Please note that the Rest API is buggy (see #1515).

Thanks, yeah I think the numbers difference is probably caused by the buggy API. 1.2K is actually the right number.

curl -f -H "Authorization: $TOKEN" https://api.github.com/search/commits?q=author:ForsakenRei | grep to
tal

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  "total_count": 1186,
100  262k  100  262k    0     0   483k      0 --:--:-- --:--:-- --:--:--  484k

And my PAT does have repo permission, which is the only permission it has. Here's the API response after some doc reading.

curl -f -H "Authorization: $TOKEN" https://api.github.com/search/commits?q=author:ForsakenRei+author-date:\>2022-06-09 | rg total

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  "total_count": 412,
100  273k  100  273k    0     0   451k      0 --:--:-- --:--:-- --:--:--  452k

So it's actually 412 instead of 268 if I use my personal instance.

Ah, thanks for the in-depth analysis 👍🏻.

[ForsakenRei]

I mean, the issue for private commit is not solved yet...or it just cannot be solved now?

If my PAT has the correct permission and I'm using my own instance(most used language also provided evidence) I should see 412 instead of 268 commits, but now I got exact the same number regardless I'm using the official instance or my own.

[rickstaa]

I mean, the issue for private commit is not solved yet...or it just cannot be solved now?

If my PAT has the correct permission and I'm using my own instance(most used language also provided evidence) I should see 412 instead of 268 commits, but now I got exact the same number regardless I'm using the official instance or my own.

The Rest API is buggy and is therefore only when include_all_commits=true. If you want to check whether the yearly commits are correct you should compare it with the GraphQL response when you are logged it. It's very hard for me to help you since I don't have access to your account and can not reproduce the issue on my accounts. I think the best thing to do is:

1. Fork the repository.
2. Ensure that it is up to date with the main branch of the upstream.
3. Build the GRS package locally (see https://github.com/anuraghazra/github-readme-stats/blob/master/CONTRIBUTING.md#local-development).
4. Install vscode.
5. Put debug statements before the API calls:

github-readme-stats/src/fetchers/stats-fetcher.js

Lines 146 to 156 in 44c268e

	const fetchTotalCommits = (variables, token) => {
	return axios({
	method: "get",
	url: `https://api.github.com/search/commits?q=author:${variables.login}`,
	headers: {
	"Content-Type": "application/json",
	Accept: "application/vnd.github.cloak-preview",
	Authorization: `token ${token}`,
	},
	});
	};

github-readme-stats/src/fetchers/stats-fetcher.js

Line 196 in 44c268e

let res = await statsFetcher(username);

6. Check what the the APIs return in the code.

Let me know if you need any help.

[ForsakenRei]

I mean, the issue for private commit is not solved yet...or it just cannot be solved now?
If my PAT has the correct permission and I'm using my own instance(most used language also provided evidence) I should see 412 instead of 268 commits, but now I got exact the same number regardless I'm using the official instance or my own.

The Rest API is buggy and is therefore only when include_all_commits=true. If you want to check whether the yearly commits are correct you should compare it with the GraphQL response when you are logged it. It's very hard for me to help you since I don't have access to your account and can not reproduce the issue on my accounts. I think the best thing to do is:

1. Fork the repository.
2. Ensure that it is up to date with the main branch of the upstream.
3. Build the GRS package locally (see https://github.com/anuraghazra/github-readme-stats/blob/master/CONTRIBUTING.md#local-development).
4. Install vscode.
5. Put debug statements before the API calls:

github-readme-stats/src/fetchers/stats-fetcher.js

Lines 146 to 156 in 44c268e

	const fetchTotalCommits = (variables, token) => {
	return axios({
	method: "get",
	url: `https://api.github.com/search/commits?q=author:${variables.login}`,
	headers: {
	"Content-Type": "application/json",
	Accept: "application/vnd.github.cloak-preview",
	Authorization: `token ${token}`,
	},
	});
	};

github-readme-stats/src/fetchers/stats-fetcher.js

Line 196 in 44c268e

let res = await statsFetcher(username);

6. Check what the the APIs return in the code.

Let me know if you need any help.

Thanks, I will probably try to build locally later today. Now I want to try deleting my fork and fork a new one then deploy a fresh instance, but Vercel is taking forever to import the repo, will update if it works on not once it goes through.

[rickstaa]

Since this is a duplicate of #1515, let's discuss it there further.

[ForsakenRei]

Well originally my main issue is my perosnal instance with correct permission PAT cannot read private commit correctly. The total commit difference is more like a side discovery.

The fresh repo and vercel instance give me the same result. Therefore I forked it locally and run vercel dev. If I don't have PAT setup it will post a few messages ask for PAT and the localhost API retrun is also asking for a PAT. With the correct PAT set, I got the same result as the vercel instance I'm using. But apperantly my most use languages still includes all my private repos(compared to #2793 (comment))

I didn't know much about JS so I'm not sure if I'm doing it currotly but add debugger; before line 146 seems have no effect on my local vercel dev instance...?

And it seems the API is really buggy, I tried postman just want to see what I can get and compared to GRS.

GET https://api.github.com/search/commits?q=author:forsakenrei+author-date:>2022-12-31

Without auth token "total_count": 221 and with token "total_count": 349. On my profile page it is 357 for 2023 but includes all contributions not only commits. Commits numbers are different but relatively speaking, the current year commit number I got from my own GRS instance is more like the one without PAT, i.e. not counting my private repo.

I don't know where I can try to fix it, tried hardcode my PAT to line 86 and 153 of stats-fetcher.js but nothing changed. Clearly it's working since I didn't hardcode it in top-language-fetcher.js and when refersh top language card I got PAT_* faild and Maximum retries exceeded, but refresh the stats card just works, returns the same result though.

[rickstaa]

And it seems the API is really buggy, I tried postman just want to see what I can get and compared to GRS

Yea, the REST API is sadly very buggy, the search function sometimes returns fewer results.

The fresh repo and vercel instance give me the same result. Therefore I forked it locally and run vercel dev. If I don't have PAT setup it will post a few messages ask for PAT and the localhost API retrun is also asking for a PAT.

About the PAT, you can simply add the PAT to Vercel, as it will inject it when you run vercel dev.

I didn't know much about JS so I'm not sure if I'm doing it currotly but add debugger; before line 146 seems have no effect on my local vercel dev instance...?

To debug the code, you have to ensure the JavaScript debugger is working in an example script. If the debugger works, you can use the following launch.json:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Attach by Process ID",
            "processId": "${command:PickProcess}",
            "request": "attach",
            "skipFiles": [
                "<node_internals>/**"
            ],
            "type": "node"
        }
    ]
}

This config will allow you to attach to the local Vercel instance that is started with the vercel dev command. You can then add breakpoints, on which the debugger will stop if they are triggered. For me, the process I had to attach to showed up as:

Hope you can find your problem, as I cannot reproduce it on my private Vercel instance.

[ForsakenRei]

Yes I used .env originally, just want to check if hardcode it has the same effect as the injection and the answser is yes I got the exact same result. I tried different PAT with full account access as well.

I will try JS debugger after I got back home, another thing I can try is a different github account(borrow one from my friend probably). Like you said, if we two are using the same configuration and same repo but yours has no issue at all, will there be something wrong with my Github account?

BTW could you re-open the issue as I'm not done with my investigation yet?

[rickstaa]

Yes I used .env originally, just want to check if hardcode it has the same effect as the injection and the answser is yes I got the exact same result. I tried different PAT with full account access as well.

I will try JS debugger after I got back home, another thing I can try is a different github account(borrow one from my friend probably). Like you said, if we two are using the same configuration and same repo but yours has no issue at all, will there be something wrong with my Github account?

BTW could you re-open the issue as I'm not done with my investigation yet?

Done! Maybe you can explain to me again what problem you are still experiencing. The reason that I say that I don't experience your problem is that my commits did not change compared to the last changes that could have affected the commits, and there is a clear difference between the private and public results (see below) 🤔. I, however, did not verify that the results I get for my account from GitHub are the correct ones because there is not much we can do if they are not correct 😅.

Public results

[![Anurag's GitHub stats](https://github-readme-stats.vercel.app/api?username=rickstaa)](https://github.com/anuraghazra/github-readme-stats)

[![Anurag's GitHub stats](https://github-readme-stats.vercel.app/api?username=rickstaa&include_all_commits=true)](https://github.com/anuraghazra/github-readme-stats)

Private results

[![Anurag's GitHub stats](https://github-readme-stats-rickstaa.vercel.app/api?username=rickstaa)](https://github.com/anuraghazra/github-readme-stats)

[![Anurag's GitHub stats](https://github-readme-stats-rickstaa.vercel.app/api?username=rickstaa&include_all_commits=true)](https://github.com/anuraghazra/github-readme-stats)

[ForsakenRei]

Thanks, and let me go through the current issue again(after all those debugging I'm getting lost as well lol), bear with me since it will be a little lengthy.

For total commits(all or laster year), between official instance and private instace, there should be a clear difference since only private instance can read private repo commits. Unless the user has no private commits at all.

The example of your account shows a 8.9K vs 6.8K difference of total commits, which is the expected behavior since only your own instance can read all those private commits. And that's true for my case as well after I checked with postman, 1339 vs 1186 matched the result I have earlier 1.3K vs 1.2K. (#2793 (comment))

Therefore I think we can conclude that include_total_commits works as expected though the API is still buggy.

But if we look at total commits in 2023, from your account they are both 1.5K, I don't know how many commits you have in private repos so if you barely have private commits in 2023 it's probably correct, otherwise something is wrong. And for my case, I got the exact same number which is not expected since I have about 100 commits in private repos in 2023. When I use postman to call the API

https://api.github.com/search/commits?q=author:forsakenrei+author-date:>2022-12-31

with PAT I got 350, without it's 221. And from GRS the number is 269 for 2023(I assume the date range is after 2022-12-31, correct me if I'm wrong since I didn't read all the code). So there seems to be something wrong, with the API or the fetcher or combined?

P.S. I got another Github account, will do some test and update later.

[rickstaa]

Thanks, and let me go through the current issue again(after all those debugging I'm getting lost as well lol), bear with me since it will be a little lengthy.

For total commits(all or laster year), between official instance and private instace, there should be a clear difference since only private instance can read private repo commits. Unless the user has no private commits at all.

The example of your account shows a 8.9K vs 6.8K difference of total commits, which is the expected behavior since only your own instance can read all those private commits. And that's true for my case as well after I checked with postman, 1339 vs 1186 matched the result I have earlier 1.3K vs 1.2K. (#2793 (comment))

Therefore I think we can conclude that include_total_commits works as expected though the API is still buggy.

But if we look at total commits in 2023, from your account they are both 1.5K, I don't know how many commits you have in private repos so if you barely have private commits in 2023 it's probably correct, otherwise something is wrong. And for my case, I got the exact same number which is not expected since I have about 100 commits in private repos in 2023. When I use postman to call the API

https://api.github.com/search/commits?q=author:forsakenrei+author-date:>2022-12-31

with PAT I got 350, without it's 221. And from GRS the number is 269 for 2023(I assume the date range is after 2022-12-31, correct me if I'm wrong since I didn't read all the code). So there seems to be something wrong, with the API or the fetcher or combined?

P.S. I got another Github account, will do some test and update later.

Thanks for the detailed report. Great to hear that, although buggy, the include_all_commit flag is working with the PAT.

For the other problem, since it uses GraphQL, it is effortless to check if it works by comparing the public version with the results from the GraphQL explorer when logged in to your user account. For my account, I got the results below. Strangely enough, the card doesn't show the right private commits 🤔.

GraphQL private:

{
  "data": {
    "user": {
      "name": "Rick Staa",
      "login": "rickstaa",
      "contributionsCollection": {
        "totalCommitContributions": 1832,
        "restrictedContributionsCount": 352
      }
    }
  }
}

GraphQL public

{
  "data": {
    "user": {
      "name": "Rick Staa",
      "login": "rickstaa",
      "contributionsCollection": {
        "totalCommitContributions": 1554,
        "restrictedContributionsCount": 675
      }
    }
  }
}

Private card

[![Anurag's GitHub stats](https://github-readme-stats-rickstaa.vercel.app/api?username=rickstaa)](https://github.com/anuraghazra/github-readme-stats)

Public card

[![Anurag's GitHub stats](https://github-readme-stats.vercel.app/api?username=rickstaa)](https://github.com/anuraghazra/github-readme-stats)

[ForsakenRei]

Yes, I think that's the case. PR, issue etc. all works, total commits works, but just commits in recent year didn't work properly.

My result from GraphQL basically echos your case, 412 is the total includes private commits while 269 is all the public commits. However private instance card shows the same number as the public one. A different Github/Vercel account reproduced this issue with the exact same behavior.

Debugger works but my JS knowledge is dragging me down, don't even know what I should look at lol.

[rickstaa]

@ForsakenRei No problem. I created a small script in which we can test the behaviour 👍🏻.

/**
 * Call the GitHub GraphQL API to see how many commits a user has made in the last year.
 */
import axios from "axios";

const QUERY = `
    query userInfo($login: String!) {
        user(login: $login) {
        name
        login
        contributionsCollection {
            totalCommitContributions
        }
        }
    }
`;
const VARIABLES = {
    "login": "rickstaa",
};
const PRIVATE_PAT = "<INSERT_PAT>"; // Pat that has access to your private repos and user data.
const PUBLIC_PAT = "<INSERT_PUBLIC_PAT>"; // Pat that has no access to your private repos.    

const main = async () => {

    // Fetch total public commits for a user.
    const res = await axios({
        url: "https://api.github.com/graphql",
        method: "post",
        headers: {
            Authorization: `bearer ${PRIVATE_PAT}`,
        },
        data: {
            "query": QUERY, "variables": VARIABLES
        },
    })

    // Print total private commits.
    console.log(`Total private commits: ${res.data.data.user.contributionsCollection.totalCommitContributions}`);

    // Fetch total public commits for a user.
    const res2 = await axios({
        url: "https://api.github.com/graphql",
        method: "post",
        headers: {
            Authorization: `bearer ${PUBLIC_PAT}`,
        },
        VARIABLES,
    })

    // Print total private commits.
    console.log(`Total public commits: ${res.data.data.user.contributionsCollection.totalCommitContributions}`);
}

main();

[rickstaa]

@ForsakenRei I think I have it the PAT needs the read:user permission which makes sense 😅.

[rickstaa]

My new results after giving the read:user permission to my PAT:

Public results

[![Anurag's GitHub stats](https://github-readme-stats.vercel.app/api?username=rickstaa&random=122243)](https://github.com/anuraghazra/github-readme-stats)

[![Anurag's GitHub stats](https://github-readme-stats.vercel.app/api?username=rickstaa&include_all_commits=true&random=122243)](https://github.com/anuraghazra/github-readme-stats)

Private results

[![Anurag's GitHub stats](https://github-readme-stats-rickstaa.vercel.app/api?username=rickstaa&random=122243)](https://github.com/anuraghazra/github-readme-stats)

[![Anurag's GitHub stats](https://github-readme-stats-rickstaa.vercel.app/api?username=rickstaa&include_all_commits=true&random=122423)](https://github.com/anuraghazra/github-readme-stats)

[ForsakenRei]

Works for me as well!
I think no code change needed(hopefully) but doc change will be necessary then.

[rickstaa]

Yea, will update the docs. For the language card, the repo:status permission is needed.

Language card

Public

[![Top Langs](https://github-readme-stats.vercel.app/api/top-langs/?username=rickstaa&langs_count=10&layout=compact&random=1232435)](https://github.com/anuraghazra/github-readme-stats)

Private

[![Top Langs](https://github-readme-stats-rickstaa.vercel.app//api/top-langs/?username=rickstaa&langs_count=10&layout=compact&random=1234235)](https://github.com/anuraghazra/github-readme-stats)

[ForsakenRei]

Thanks for all the help!

[rickstaa]

No problem. Thanks for bringing it to my attention. I created https://github.com/anuraghazra/github-readme-stats/pull/2805/files to prevent other people from having this problem 👍🏻.