Optimize SQLite database periodically

[glassez]

It optimizes the database every time a session is loaded, and after every 10,000 writes/removes (can be useful for long-running instances).
Note, that 10,000 is an arbitrary number taken for initial implementation and I don't want to have long discussions about it (unless you can agree on some other value quickly enough). Some might also say that it would be nice to have an appropriate configuration option for it. I agree that it probably makes sense, but I would strongly not like to deal with it right now.

[sledgehammer999]

Personally, I am not very fond of this heuristic.
I would suggest issuing a VACUUM by periodically querying the page_count and freelist_count. And if free space is >10% then issue a VACUUM.
NOTE 1: 10% was randomly chosen
NOTE 2: There are some other heuristics discussed here. The free space one seems the most straightforward.
NOTE 3: That is my personal opinion and I don't insist on it, if you want to through with this PR.

[glassez]

Personally, I am not very fond of this heuristic.

In fact, I'm not "thrilled" with it either.
But this PR is not as spontaneous as it might seem. I've been thinking about this problem for a long time (though not very actively). And a lot of what you referred to above coincides more or less with my own conclusions.

I would suggest issuing a VACUUM by periodically querying the page_count and freelist_count. And if free space is >10% then issue a VACUUM.

Increasing the size of the database due to the number of unused/free pages is not the main problem in our case. Since qBittorrent actively uses the database, these pages will still be re-used again later.

[glassez]

Since qBittorrent usually produces records of different lengths, it is more susceptible to the problem of data/page fragmentation. Therefore, the most suitable heuristic for qBittorrent, IMO, is the one based on the number of writing/deleting made.
That's what I tried to implement. However, to do it completely, you need to save data on the number of changes made in the database itself (well, or somewhere else), otherwise those who do not use qBittorrent very actively will never reach the optimization point, since the counter will be reset too early.
For the first time, I decided to "cheat" a little, so as not to save this data, but just make another "forced" optimization at each startup. So maybe I won't merge it now, but will finish the work first.

[sledgehammer999]

You have summed my thoughts very well.

For the fragmentation part, I wonder if a good enough heuristic can be obtained by using the info in DBSTAT.

[glassez]

For the fragmentation part, I wonder if a good enough heuristic can be obtained by using the info in DBSTAT.

Hmm, it looks like it might be useful...

[glassez]

For the fragmentation part, I wonder if a good enough heuristic can be obtained by using the info in DBSTAT.

Hmm, it looks like it might be useful...

The problem is that DBSTAT is an optional feature that is enabled at compile time, so we may find ourselves in a situation where it will not be available on the target system. At least it is disabled in SQLite plugin shipped with the official Qt binaries for Windows. So at least we'll need some kind of fallback to do without it.
What do you think?

[HanabishiRecca]

Something to take into consideration: https://sqlite.org/pragma.html#pragma_auto_vacuum

[glassez]

Something to take into consideration: https://sqlite.org/pragma.html#pragma_auto_vacuum

I initially rejected this idea because of the following:

Note, however, that auto-vacuum only truncates the freelist pages from the file. Auto-vacuum does not defragment the database nor repack individual database pages the way that the VACUUM command does. In fact, because it moves pages around within the file, auto-vacuum can actually make fragmentation worse.

In fact, right now I intend to just add an optimization command to the menu so that the user can run it manually. It would be better than nothing until automatic optimization is added someday.

[HanabishiRecca]

In fact, right now I intend to just add an optimization command to the menu so that the user can run it manually. It would be better than nothing until automatic optimization is added someday.

Good idea. I think the manual option would be useful even alongside with automatic optimization.

[glassez]

I think the manual option would be useful even alongside with automatic optimization.

Sure.