Replay Bot: Considerations and roadblocks

[HymnsForDisco]

I started and mostly finished the replay bots project a long time ago. It's been neglected since then, but some players have been asking me about it again. I'll just summarize the main issues that have impeded me from finishing this up.

Storage of replay files

I'd like to store/address the replays by a unique ID, this way each speedrun record can potentially point to it's own replay and be watched. This allows us a few nice features:

• We can store multiple top replays per map, and replays will have protection from immediately being overwritten if a cheated record is set.
• We can control replay storage on case by case, and allow donation perks like storing extra replays.

Issues

• Replays take a moderate amount of file space, and cannot be allowed to grow indefinitely. Estimates are at the scale of 10s of GB per server, if somewhat lenient limits are set.
  50 bytes per tick * 100 tick/s * 60 s/min * 30 min of replay/map * 1000 maps = 9 GB
• I've so far been working with storing the replays in a database table as binary blob (SQL has a function to read/write a blob as a file in a query, this just requires setting a cvar so that the sourcemod plugin knows the absolute file path root). This works, but would grow the overall size of the database substantially. Perhaps could be moved to it's own schema? I don't know if this is important
• Due to the assumption that creating a new replay doesn't immediately overwrite an existing one, I believe this will require some kind of "clean up" routine. I know we have similar existing things for demos, calculating player ranks etc., I'm hoping something similar can be applied here. Ex. periodically running through all speedruns replays, and clearing replays for any runs that are no longer in the top 3 and past a certain age, or perhaps setting a quota per map? Still unsure here.

General lag / instability of replays

I've looked at multiple other timers, written and re-written the replays API core as closely as possible trying to apply all the same things. Despite this, in all my testing, replay playback is always has a bit of stutter. On other timers, you'll notice this stuttering when applying playback on yourself (due to client prediction issues I assume). In my testing though, I notice this stuttering when spectating players and bots who are the puppet of the replay.
The main mechanism is as follows:
Record client's input states and player position/velocity per tick in OnPlayerRunCmd
Replay by setting client's input states and player position/velocity per tick in OnPlayerRunCmd
I'm not sure where the issue lies, maybe something to do with my test server specifically.

@hlstriker , I'm posting this here to see if you have any input and let you know the actual specifics of things, I can probably figure this all out but I'd just like your input before pushing something that has potentially big considerations and effects on the back end

[hlstriker]

I would save the replay of every new map record and we can have a script clean up the records that get beaten after a week, or whatever time is enough to verify the record wasn't cheated. The main issue I see is do we save records for every style? It might be worth looking into various compression methods to see if you can cut the blob size down.

As for the stuttering issue I'm not sure. I'd have to see a video of what's actually happening I think since I've never experienced it.

[HymnsForDisco]

Update:
I've figured out the replay save/load. Replays are encoded to hex in-memory, which is directly inserted into the query to be saved. Loading decodes the returned string from hex. (queries use SQL HEX and UNHEX functions)

@hlstriker Here are videos demonstrating the slight stuttering of the bot. It appears as if the bot is just being teleported repeatedly at a very fast rate (which of course is what my plugin is actually doing, but isn't the desired effect). It probably won't be noticeable in the web players, you'll have to download the files and watch locally to see the full bitrate and framerate.
https://drive.google.com/file/d/1R6zopitRysFqkLGteW-bBpm-pyAzoF0D/view?usp=sharing
https://www.dropbox.com/s/jjkzxczzhebq0mw/Counter-strike%20%20Global%20Offensive%202021.01.03%20-%2002.58.41.01.mp4?dl=0
From my understanding of this, I've been assuming the stuttering comes from the client prediction being incorrect when watching the bot.

This is the source for GOKZ, which has replays which appear as smooth as if you were spectating a real player.
https://bitbucket.org/kztimerglobalteam/gokz/src/master/addons/sourcemod/scripting/
I have contacted two of the main contributors of this project

This gist contains the relevant code, and some comments.
https://gist.github.com/HymnsForDisco/187d9154d3b07c981cdc83ccf8ac2dc5

[HymnsForDisco]

The stuttering issue is now fixed. I found what I overlooked in the GOKZ code:
For most ticks, you should move the bot only be setting the velocity and angles, and not the position. You can calculate the required velocity to move the bot to where it should go in the next tick. Setting the position directly is only needed on some ticks like at the start and when paused.