You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello there, I am implementing Graceful shutdown, and I want to stop my reminders if I get a SIGTERM from os (right now k8s)
My application will wait for 75 after a SIGTERM to close all the in-flight and in-progress requests.
but 90% of the time I want to deploy my new pods, I will get these sort of exception.
{"@timestamp":"2024-05-19T17:47:12.5330983+03:30","level":"Error","messageTemplate":"{Name} failed to stop due to errors at stage {Stage}","message":"\"LocalReminderService\" failed to stop due to errors at stage \"Active (20000)\"","exceptions":[{"Depth":0,"ClassName":"System.Threading.Tasks.TaskCanceledException","Message":"Stopping ReminderService failed because the task was cancelled","Source":"Orleans.Core","StackTraceString":" at Orleans.Internal.OrleansTaskExtentions.WithCancellation(Task taskToComplete, String message, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 148\n at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStop(CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Lifecycle/SiloLifecycleSubject.cs:line 163","RemoteStackTraceString":null,"RemoteStackIndex":0,"HResult":-2146233029,"HelpURL":null},{"Depth":1,"ClassName":"System.Threading.Tasks.TaskCanceledException","Message":"A task was canceled.","Source":"System.Private.CoreLib","StackTraceString":" at Orleans.Internal.OrleansTaskExtentions.WithCancellation(Task taskToComplete, String message, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 144","RemoteStackTraceString":null,"RemoteStackIndex":0,"HResult":-2146233029,"HelpURL":null}],"Name":"LocalReminderService","Stage":"Active (20000)","EventId":{"Id":100450},"SourceContext":"Orleans.Runtime.SiloLifecycleSubject","MachineName":"opay-0","ApplicationName":"OPay.API","EnvironmentName":"Development"}
{"@timestamp":"2024-05-19T17:47:12.5431698+03:30","level":"Error","messageTemplate":"Stopping lifecycle encountered an error at stage {Stage}. Continuing to stop.","message":"Stopping lifecycle encountered an error at stage 20000. Continuing to stop.","exceptions":[{"Depth":0,"ClassName":"System.Threading.Tasks.TaskCanceledException","Message":"Stopping ReminderService failed because the task was cancelled","Source":"Orleans.Core","StackTraceString":" at Orleans.Internal.OrleansTaskExtentions.WithCancellation(Task taskToComplete, String message, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 148\n at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStop(CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Lifecycle/SiloLifecycleSubject.cs:line 163\n at Orleans.LifecycleSubject.OnStop(CancellationToken cancellationToken) in /_/src/Orleans.Core/Lifecycle/LifecycleSubject.cs:line 194","RemoteStackTraceString":null,"RemoteStackIndex":0,"HResult":-2146233029,"HelpURL":null},{"Depth":1,"ClassName":"System.Threading.Tasks.TaskCanceledException","Message":"A task was canceled.","Source":"System.Private.CoreLib","StackTraceString":" at Orleans.Internal.OrleansTaskExtentions.WithCancellation(Task taskToComplete, String message, CancellationToken cancellationToken) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 144","RemoteStackTraceString":null,"RemoteStackIndex":0,"HResult":-2146233029,"HelpURL":null}],"Stage":20000,"EventId":{"Id":100451},"SourceContext":"Orleans.Runtime.SiloLifecycleSubject","MachineName":"opay-0","ApplicationName":"OPay.API","EnvironmentName":"Development"}
{"@timestamp":"2024-05-19T17:47:12.5643788+03:30","level":"Error","messageTemplate":"{Name} failed to stop due to errors at stage {Stage}","message":"\"ActivationCollector\" failed to stop due to errors at stage \"RuntimeServices (4000)\"","exceptions":[{"Depth":0,"ClassName":"System.Threading.Tasks.TaskCanceledException","Message":"A task was canceled.","Source":"System.Private.CoreLib","StackTraceString":" at Orleans.Runtime.ActivationCollector.Stop(CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Catalog/ActivationCollector.cs:line 434\n at Orleans.Runtime.ActivationCollector.<Orleans.ILifecycleParticipant<Orleans.Runtime.ISiloLifecycle>.Participate>b__37_1(CancellationToken cancellation) in /_/src/Orleans.Runtime/Catalog/ActivationCollector.cs:line 444\n at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStop(CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Lifecycle/SiloLifecycleSubject.cs:line 163","RemoteStackTraceString":null,"RemoteStackIndex":0,"HResult":-2146233029,"HelpURL":null}],"Name":"ActivationCollector","Stage":"RuntimeServices (4000)","EventId":{"Id":100450},"SourceContext":"Orleans.Runtime.SiloLifecycleSubject","MachineName":"opay-0","ApplicationName":"OPay.API","EnvironmentName":"Development"}
{"@timestamp":"2024-05-19T17:47:12.5656335+03:30","level":"Error","messageTemplate":"Stopping lifecycle encountered an error at stage {Stage}. Continuing to stop.","message":"Stopping lifecycle encountered an error at stage 4000. Continuing to stop.","exceptions":[{"Depth":0,"ClassName":"System.Threading.Tasks.TaskCanceledException","Message":"A task was canceled.","Source":"System.Private.CoreLib","StackTraceString":" at Orleans.Runtime.ActivationCollector.Stop(CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Catalog/ActivationCollector.cs:line 434\n at Orleans.Runtime.ActivationCollector.<Orleans.ILifecycleParticipant<Orleans.Runtime.ISiloLifecycle>.Participate>b__37_1(CancellationToken cancellation) in /_/src/Orleans.Runtime/Catalog/ActivationCollector.cs:line 444\n at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStop(CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Lifecycle/SiloLifecycleSubject.cs:line 163\n at Orleans.LifecycleSubject.OnStop(CancellationToken cancellationToken) in /_/src/Orleans.Core/Lifecycle/LifecycleSubject.cs:line 194","RemoteStackTraceString":null,"RemoteStackIndex":0,"HResult":-2146233029,"HelpURL":null}],"Stage":4000,"EventId":{"Id":100451},"SourceContext":"Orleans.Runtime.SiloLifecycleSubject","MachineName":"opay-0","ApplicationName":"OPay.API","EnvironmentName":"Development"}
my tasks is in this reminders are not atomic and they are just idempotent things, should I worry about these exceptions?
what should I do In my code to prevent these errors?
How can I prevent reminders from starting after SIGTERM has been sent to my pod? (I mean in that 75-second window.)
The text was updated successfully, but these errors were encountered:
Hello there, I am implementing Graceful shutdown, and I want to stop my reminders if I get a SIGTERM from os (right now k8s)
My application will wait for 75 after a SIGTERM to close all the in-flight and in-progress requests.
but 90% of the time I want to deploy my new pods, I will get these sort of exception.
my tasks is in this reminders are not atomic and they are just idempotent things, should I worry about these exceptions?
what should I do In my code to prevent these errors?
How can I prevent reminders from starting after SIGTERM has been sent to my pod? (I mean in that 75-second window.)
The text was updated successfully, but these errors were encountered: