Chunk ... does not exist in the storage

[Robert-M-Muench]

I'm running duplicacy via a cron job. The backup log looks good:

Storage set to b2://duplicacy-qnap-backup
download URL is: https://f001.backblazeb2.com
Last backup at revision 827 found
Indexing /share/qnap-backup
Parsing filter file /share/qnap-backup/.duplicacy/filters
Loaded 2 include/exclude pattern(s)
Discarding file attributes
Backup for /share/qnap-backup at revision 828 completed
Files: 1371131 total, 8360G bytes; 0 new, 0 bytes
File chunks: 1760415 total, 8465G bytes; 0 new, 0 bytes, 0 bytes uploaded
Metadata chunks: 157 total, 792,936K bytes; 0 new, 0 bytes, 0 bytes uploaded
All chunks: 1760572 total, 8466G bytes; 0 new, 0 bytes, 0 bytes uploaded
Total running time: 00:03:00

But pruning reports, "Chunk ... does not exist in the storage."

Storage set to b2://duplicacy-qnap-backup
download URL is: https://f001.backblazeb2.com
Keep no snapshots older than 360 days
Keep 1 snapshot every 30 day(s) if older than 180 day(s)
Keep 1 snapshot every 7 day(s) if older than 30 day(s)
Keep 1 snapshot every 1 day(s) if older than 7 day(s)
Deleting snapshot qnap-backup at revision 1
<this now goes on for many snapshots>
Deleting snapshot qnap-backup at revision 797
Deleting snapshot qnap-backup at revision 798
Chunk d3196f1738c54a862dcda212eb6ca349dafe4a93b7c3a8432536599d6fdaab8d does not exist in the storage

1. It seems the snapshots 1..798 are not deleted. Is this the case? If there is an error, is no snapshot deleted?
2. How can it happen that a chunk is missing in the storage? I assume the storage referees to Backblaze's B2 here, right?
3. What to do now? How to solve the situation?

[Robert-M-Muench]

I still don't understand what's going on and where the problems come from.

I'm now using this loop to fix the problems step-by-step and catch up to my latest snapshot.

for ((i = 0 ; i <= 10 ; i++)); do duplicacy prune -exhaustive -exclusive -threads 20; .duplicacy/scripts/post-backup; done

Which runs the clean-up and pruning in a cycle. It's super slow and seems to re-download/re-check things over and over again. However, in every cycle, it finds something like this:

download URL is: https://f001.backblazeb2.com
The chunk chunks/0a/905242261681bea446ab45b0bb22454c8771a664867b50aee787a36df72179.fsl has been resurrected
The chunk chunks/5b/e20d2f4ae0314ccd4ce0382d47c25cafbf4ab770fe60f0a156960cbdfcc3e7.fsl has been resurrected
The chunk chunks/5d/3432c9684719cccd0bb1d458ff0eda65e30c60fde57911c5f1f86c0a88d8f5.fsl has been resurrected
The chunk chunks/e4/7e43d8ea6c8921188670c11cb9ed2428cf0f0a22ccf851afc99b34734d79e4.fsl has been resurrected
The chunk chunks/ee/b87f8268d0bd9dc6e71cd02b86f93365881724a22c278aa9c06aa4cd12faac.fsl has been resurrected

Sometimes only 2, sometimes 20... I don't see any pattern.

@gilbertchen Would be great to get some feedback on this.

[stamster]

I'm having exact same issue from time to time, also using Backblaze B2 in conjunction with duplicacy CLI.

This could be either bug in the duplicacy or some sort of bit-rot case happening over at Backblaze B2.

I'm now imposing check (-chunks flag) after each revision is done.

e.g.
duplicacy -v check -r 338 -chunks

so if I don't see any errors there - then I'll assume B2 is to blame, which would be very bad thing if it proves to be true..

[Robert-M-Muench]

I don't think it's a B2 bit-rot problem. Looks more like a B2-state vs. DP-state issue.

I abandon the approach with check and prune because it took ages and used a simpler and radical way:

1. I deleted the local .duplicacy/cache directory
2. I deleted the snapshot directory in B2 because I didn't need the older ones anyway.
3. I run a backup (which will get all chunks in B2, and only uploads chunks missing). So, you get a fresh backup with revision number 1.
4. I run a duplicacy prune -exhaustive -exclusive -threads 20 which removes all chunks in B2 that are no longer needed reducing the storage costs.

After that, I had a fresh backup, reduced B2 storage, and no more error messages.