Recently installed Seq on ubuntu and have a 'The last attempt to apply retention policies failed. Retention processing may be temporarily disrupted if manual event deletion is requested.' error permanently showing up

[nblumhardt]

Recently installed Seq on ubuntu and have a 'The last attempt to apply retention policies failed. Retention processing may be temporarily disrupted if manual event deletion is requested.' error permanently showing up

The error in the logs seems to be Failed to apply retention policy {RetentionPolicyId}","@l":"Error","@x":"Flare.Ffi.Result.ValidationErrorException: Invalid query: the procedure sp_reclaim_storage requires 5 arguments.

What might be causing this?

====================

1. System

---

Seq Version : 2024.3.11547
.NET Version : 8.0.4
Operating System : Ubuntu 22.04.4 LTS

{"@t":"2024-07-14T13:55:16.1622634Z","@sp":"a1bb89e184437198","@tr":"1521565e2c1f03a91980e1fedf61b878","@mt":"Applying {Count} retention policies","Count":3,"SourceContext":"Seq.Server.Features.Retention.RetentionProcessor"}
{"@t":"2024-07-14T13:55:16.1627685Z","@sp":"c4d7cb17469005c6","@tr":"1521565e2c1f03a91980e1fedf61b878","@mt":"Removing all events before {DeletingUntilUtc:u} under policy {RetentionPolicyId}","@r":["2024-06-29 13:55:00Z"],"DeletingUntilUtc":"2024-06-29T13:55:00.0000000Z","RetentionPolicyId":"retentionpolicy-20","SourceContext":"Seq.Server.Features.Retention.RetentionProcessor"}
{"@t":"2024-07-14T13:55:16.1637036Z","@sp":"c4d7cb17469005c6","@tr":"1521565e2c1f03a91980e1fedf61b878","@mt":"Retention policy progressed in {Elapsed:0.000} ms (headway: {Headway}×), will resume from {ResumeStart:u}","@r":["0.783","2024-06-29 13:55:00Z"],"Elapsed":0.7825,"Headway":null,"ResumeStart":"2024-06-29T13:55:00.0000000Z","SourceContext":"Seq.Server.Features.Retention.RetentionProcessor","RetentionPolicyId":"retentionpolicy-20"}
{"@t":"2024-07-14T13:55:16.1670983Z","@sp":"c4d7cb17469005c6","@tr":"1521565e2c1f03a91980e1fedf61b878","@mt":"Process retention policy {RetentionPolicyId}","RetentionPolicyId":"retentionpolicy-20","@st":"2024-07-14T13:55:16.1627539Z","@ps":"a1bb89e184437198","SourceContext":"Seq.Server.Features.Retention.RetentionProcessor"}
{"@t":"2024-07-14T13:55:16.1684091Z","@sp":"f428790a837b577d","@tr":"1521565e2c1f03a91980e1fedf61b878","@mt":"Removing events before {DeletingUntilUtc:u} matching {RemovedSignalExpression} under policy {RetentionPolicyId} (deleting from {DeletingFrom:u})","@r":["2024-07-12 13:55:00Z","null"],"DeletingUntilUtc":"2024-07-12T13:55:00.0000000Z","RemovedSignalExpression":"`signal-25`","RetentionPolicyId":"retentionpolicy-43","DeletingFrom":null,"SourceContext":"Seq.Server.Features.Retention.RetentionProcessor"}
{"@t":"2024-07-14T13:55:16.1690257Z","@sp":"f428790a837b577d","@tr":"1521565e2c1f03a91980e1fedf61b878","@mt":"Process retention policy {RetentionPolicyId}","RetentionPolicyId":"retentionpolicy-43","@st":"2024-07-14T13:55:16.1673449Z","@ps":"a1bb89e184437198","SourceContext":"Seq.Server.Features.Retention.RetentionProcessor"}
{"@t":"2024-07-14T13:55:16.1691930Z","@sp":"a1bb89e184437198","@tr":"1521565e2c1f03a91980e1fedf61b878","@mt":"**Failed to apply retention policy {RetentionPolicyId}","@l":"Error","@x":"Flare.Ffi.Result.ValidationErrorException: Invalid query: the procedure `sp_reclaim_storage` requires 5 arguments.\n**   at Seq.Engine.Storage.StorageEngine.Query(Span`1 query, Object[] sharedColumnBuffer, CancellationToken cancel)\n   at Seq.Engine.Events.EventStore.Query(Span`1 query, Object[] sharedColumnBuffer, Boolean disableReadRateLimit, CancellationToken cancel)\n   at Seq.Engine.Queries.DataStore.ReclaimStorage(DateTimeRangeBounds bounds, String filter, IndexExpression indexExpression, CancellationToken cancel)\n   at Seq.Server.Features.Retention.RetentionProcessor.ApplyPolicy(RetentionPolicy retentionPolicy, RetentionPolicyBookmark bookmark, IReadOnlyDictionary`2 associatedSignals, CancellationToken cancel)\n   at Seq.Server.Features.Retention.RetentionProcessor.Apply(CancellationToken cancel)","RetentionPolicyId":"retentionpolicy-43","SourceContext":"Seq.Server.Features.Retention.RetentionProcessor"}

Originally posted by @iJeax in #2232

[nblumhardt]

From @KodrAus:

This is a bug we spotted on our main branch where if you use a signal in a retention policy that also has indexing disabled then the policy won't be able to run.

... you should be able to get everything back to working again by checking the signals used in retention policies on the Data > Indexing page and re-enabling indexes for the ones that are missing it.