Unable to cache most of bitmaps

[aliktab]

I have about 500 bitmaps loaded but starting from several days ego beatlist has problems to cache most of them with the error "Request error while retrieving data on beatsaver" and it's showing me the only 74 map.

clear cache and update library and scan doesn't help

beatlist version 1.2.2

"
468 beatmaps loaded.
16 playlists loaded.
0 beatmaps are cached.
1 beatmaps are invalid.
708 beatmaps inside some playlists are invalid.
Last scan done the 08/12/2020, 20:02:34
"

[Alaanor]

This is a know problem. Was working on it these few last day. Can't tell you when it will be fixed tho, but next version 👍 . Basically we're hitting the rate-limit of beatsaver.

[Penthious]

@Alaanor I noticed this issue as well, it seems like some of the keys are not registered IE if you try to type w in the search you will get that error. I looks like you might have not included all of the keycodes

[mhombach]

Is this related to the problem that songs don't show up in the Beatlist tool (windows), even though they are inside beatsaber and are appearing ingame? Is beatlist maybe not displaying songs at all, where it is not able to retrieve meta-information about from beatsaver?

[Naephilim]

I keep getting this:
"262 beatmaps loaded.
1 playlist is loaded.
71 beatmaps are cached.
2 beatmaps are invalid.
191 beatsaver cached beatmaps are invalid."

But all 262 songs are being read just fine within the game.
Is this still being fixed? I noticed an update to this program within the past couple day, but still have these messages. I apologise if this has been repeated one too many times.

[Rawk3rs]

Here I only have a couple of invalid maps, however, I do have 663 songs in my folder and BeatList only shows 486

[Alaanor]

I know that the UI I made isn't very intuitive concerning the recent change for rate-limiting. But basically this might be just that:

1. You click on the scan button
2. Beatlist will try to fetch as many song as he can, until he get rate-limited.
3. Once he get rate-limited. He skip the rest of the request to do. Close the scan popup. Display a message that you reached the rate-limit.
4. At the same time there's a trigger sent in background that tell beatlist to retry to scan once the rate-limit period has passed.
5. It will scan in the background automatically in loop until he fetch all song.

Where the UI is bad and I'll admit it. Is that I make it look like the scan has finished and all the rate-limited song goes in same category as those who were invalid. (But this list should go down over time as he will scan it later again)

Can you guys confirm that it is only a UI/misunderstanding issue rather than an actual bug ?

Also know that since 1.2.3 there's a mirror settings that you can use to change the source. This should make you avoid the rate-limit issue completely.

[Naephilim]

I have waited for the scan to finish. It calls the invalid maps. I check into the beatmap tabs, and it only shows those limited songs that's DID validate. The rest wasn't showing up. And I waited 20-30 a couple times, in case it was just delayed.

I have tried a couple directory changes where the source folder is, putting the veatnaps somewhere else. But yields the same result.

[Naephilim]

Also: I have waited for the rate-limit countdown. But nothing changes. Even tried to "update" the beatmap validation, but it says "no changes made".

[mhombach]

@Alaanor About the rate-limit: Is beatlist sending one API-request for each song? That would be very bad. What should be done is, that it scannes all files in the folder, sends ONE single API-request with an array with those IDs/hashes, gets the response and maps the response to the files.
This will also immensely benefit the database on beatsaver, since the DB is not flodded with thousands of requests and instead only has to run one single query. Can you tell us if there is some problem you encountered with this method? If there is no such bulk api endpoint on beatsaver, then the solution is to get in contact with the owner of beatsaver. This would, as already mentioned, be in their favour too.

[Alaanor]

There's no bulk as far as I know. This is indeed bad and I agree with. And I'm not sure the version of beatsaver really reflect the content of the beatsaver repo. I don't have a contact to these owner.

[Alaanor]

Tbh I personally think the solution out of all of that is to me to rewrite the scanner to not fetch song and do all locally. This has the consequences of not having some column such as vote/like etc but at least everything would work just so much faster.

[mhombach]

@Alaanor I just created a feature-request for bulk-api nedpoints on the beatsaber repo, as the official documentation shows none.
luludotdev/beatsaver-reloaded#106

I also agree, that showing metadata from the beatsaver-DB should be optional. The files should be still displayed (author and songname are in thr local JSON-file, so showng those information should not be depending on the DB information.
I think the solution we should aim for here is a mix of both:

• One global "update/sync metadata" button (or however it will be called) on the top where the user can, from time to time, click it and which will send one single api-request to beatsaver to get all metadata for all songs beatlist has found locally.

• Beatlist working even without any DB-connection, so managing songs and playlists localls works.

I am more profound in Angular, but if you need help to change this, I would be available for discussing this and participating in coding.
If interested, write me up at martin.hombach (at) gmail.com :)

[Alaanor]

Thanks for effort, appreciate it. This kinda imply a massive change so I need to give some time to think about all of that. But this is definitely something I wish to do.

[mhombach]

Got a response, the bulk endpoints are already planned, but the repo owner doesn't know when he has time to finish them :)

luludotdev/beatsaver-reloaded#106 (comment)

So i would suggest to start on the beatlist-repo to make the online-metadata optional and while that rewrite is happening (or after it) switch to the bulk endpoints, when they are ready.

[LanceFoudre]

I just wanna add, that for me, vr 1.2.3 solved the problem, before that i had the same bug happening but now after few scans all my beatmaps are loaded (500)

[mhombach]

@LanceFoudre @Alaanor Few short questions regarding this:

1. How long is the wait (currently) after getting rate-limited?
2. Is there a way, the user can see information about this (timeout-countdown, the status of how many maps were skipped, a correct percentage of scanned/skipped)?
3. Will the user see some indicator when the next scan is triggered? I don't think that there is a popup, but will the UI correctly show the newly started scan-result?

[Alaanor]

1. I don't know and this shouldn't be something I should be interested. This is a variable fixed by beatsaver request's header. And my program should handle according to these values. (But to answer, last time I saw, this was something like 1req/s, and you can skip this limit using the mirror available in options)
2. Yes and yes. In the same page there's the scan, an orange text appear showing in how much seconds the reset will happens and disappear once it is down to zero. As for the skipped one, you can check in the "?" button next to invalid beatsaver's beatmap counter. All concerned entries will be shown as rate limited, this list should gradually come to zero after few scan (being manually or automatically launched).
3. Whenever a scan is happening, the logo does change on the sidebar. Iirc, I only made to show a small text telling the change if this was triggered by the user.

On the second topic, I've been thinking a little these few past day and I forgot the most important part for the reason I did not went for a full local scan: playlists. I can't know what song is what song without fetching that online if the user doesn't have the song locally. A hybrid solution would have been the best tho, I do agree.

I am still thinking and I honestly don't know if I will put effort or not on this issue to be transparent. Got other personal projects to work on and this isn't a small issue that can be solved in less than a few weeks.