uWSGI / Django - Catching long requests harakiri

[TheBlusky]

I am using Sentry on a Django application served by uWSGI with the follwoing configuration:

[uwsgi]
[...]
harakiri = 20
[...]

That means that if a request last more than 20s, uWSGI will kill the worker.
As the worker is killed, Sentry (configured for Django) is unable to send any data about the request.

I saw some options to retrieve Stacktrace (such as harakiri-verbose, py-tracebacker, uwsgi.after_req_hook). However, nothing (at least recent) regarding implementing is with Sentry.

Is it possible for Sentry to catch the Harakiri signal, and send an Exception or a Message before being killed ?

[smeubank]

hi @TheBlusky

sorry for the late reply here

I have not heard this come up before and not sure that the SDK can manage this out of the box.

Looking around it does seem this has come up for others though. We don't officially support the libraries or methods below, but it may be helpful. Maybe a feature request if more users are interested though! Thanks for writing in.

https://blog.theodo.com/2022/02/logging-uwsgi-alarms-errors-sentry/

https://github.com/futurecolors/raven-harakiri

[TheBlusky]

Thanks for the answer.

I also already saw those posts, and nothing came out useful.

However, we came with another approch :

• we added a django middleware that create a thread, and kill the thread when the query is finished
• the thread wait harakiri time - 2 seconds then capture an exception / error message "Query is too long" and finish the transaction (to send to the performance Sentry database)
• 2s later, the request will be harakiried

With this, if a query last less than harakiri time - 2 seconds, the middleware do nothing. If not, the query will be sent to sentry.

It's not the cleanest way to handle this issue, but so far, it's working fine, and we've been able to pinpoint where queries were stucked / looped when harakiri occured.

[smeubank]

getsentry/sentry-docs#6428

opening a docs issue. Worth discussing with team if we would want to add something like this there

Thank you! And if you have any specific code snippet examples, it could make it easier to discuss with the team.

[TheBlusky]

The following snippet should work with Django and uWSGI :

timeout_delay = None
try:
    import uwsgi

    if uwsgi_harakiri := uwsgi.opt.get("harakiri", False):
        timeout_delay = int(uwsgi_harakiri) - 2
except ImportError:
    pass


def log_before_harakiri(thread_hub, transaction):
    with thread_hub:
        capture_message("Long request")
    if transaction is not None:
        transaction.finish()
    thread_hub.flush()


class TimeoutMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        if timeout_delay is None:
            return self.get_response(request)
        thread_hub = sentry_sdk.Hub(sentry_sdk.Hub.current)
        transaction = sentry_sdk.Hub.current.scope.transaction
        harakiri_timer = threading.Timer(
            timeout_delay, log_before_harakiri, args=[thread_hub, transaction]
        )
        harakiri_timer.start()
        try:
            return self.get_response(request)
        finally:
            harakiri_timer.cancel()

However, as-is, there is a drawback as request during between harakiri - 2s and harakiris won't be fully sent to Sentry.

[antonpirker]

Yes, we could add this to the docs. Maybe we should create a "cook book" section, where we put recipies for common use cases?

[smeubank]

I like it :D

would just like to follow something similar to other frameworks, that could also be transparently used in the includes across uWSGI

also I will put on PM hat: uWSGI Framework Features section might have more SEO bump

[antonpirker]

There is a (kind of) related issue for Gunicorn: #1447
When Gunicorn times out Sentry catches the logging message that is sent by Gunicorn and sends it to Sentry.
A similar thing could be done for uwsgi harakiri.

[TheBlusky]

uWSGI can now gracefully kill before Harakiri: https://uwsgi-docs.readthedocs.io/en/latest/Changelog-2.0.22.html

[TheBlusky]

I was able to create a Django middleware able to catch Harakiri, and send transaction to sentry before being killed using new gracefull killing feature :

class HarakiriLoggerMiddleware:
    transactions = {}

    @classmethod
    def handle_signal(cls, *args):
        hubs = []
        for (hub, transaction) in HarakiriLoggerMiddleware.transactions.values():
            with hub:
                capture_message("LBH02 - Exiting")
            if transaction is not None:
                transaction.finish()
            if hub not in hubs:
                hubs.append(hub)
        for hub in hubs:
            hub.flush()

    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        hub = sentry_sdk.Hub(sentry_sdk.Hub.current)
        transaction = sentry_sdk.Hub.current.scope.transaction
        t_id = str(uuid.uuid4())
        HarakiriLoggerMiddleware.transactions[t_id] = (hub, transaction)
        try:
            return self.get_response(request)
        finally:
            HarakiriLoggerMiddleware.transactions.pop(t_id)

Adding signal handle in the app config:

class MyappConfig(AppConfig):
    def ready(self):
        signal.signal(
            signal.SIGSYS, HarakiriLoggerMiddleware.handle_signal
        )

And adding the following in uWSGI config :

harakiri = 20
lazy-apps = true
harakiri-graceful-timeout = 5
harakiri-graceful-signal = 31
py-call-osafterfork = true

It might need further testing, but imho this is a good way to do it.