Consider using Django 2.0 execute_wrapper()

[shaib]

Silk currently uses monkeypatching to install its wrapper around SQLCompiler.execute_sql. Django 2.0 provides a new API for installing wrappers around SQL executions, and it may be preferable to use this API when available.

Points arising:

• The new API works at a lower level, that of the connection rather than SQLCompiler. I'm not sure what the implications are.
• The new API installs the wrapper only in the current thread and in a context-manager which removes it on exit, these seem to be improvements over the current monkey-patching.
• There is an intention to make the new API available for older versions of Django, via a 3rd-party package. So, when checking for its availability, the right check is not a Django version check, but rather hasattr(connection, 'execute_wrapper').

[avelis]

@shaib If you have the time I encourage you to make a PR with this enhancement. I can take a look and merge it in.

[SebCorbin]

Coming from #630

Issue

I've tracked down the issue to SQLInsertCompiler not being patched to log queries. IMHO patching compilers is dangerous, because all have a different behaviors, and some call super().execute_sql(), some don't (like SQLInsertCompiler). Hence I looked into our possibilities:

Method 1: Keep patching

Keep patching the compilers, I've tried to patch all 5 compilers:

• SQLCompiler
• SQLInsertCompiler
• SQLDeleteCompiler
• SQLUpdateCompiler
• SQLAggregateCompiler

but this leads to either recursion errors (because of a super() call), or some dirty patching, along with possibly altering the behavior of the user's compilers (reminder: compiler depends on DATABASES['ENGINE'] which can be 3rd-party, e.g. Oracle or Postgis).
Also, I suspect our way of patching being the root cause of these issues:

• Fails with django-tenants-schemas #598
• It'll raise a django.db.utils.ProgrammingError when using different databases such as Mysql and SQLite #545
• Incompatible with django-channels #216
• Problem with silk in child threads #169

Method 2: Wrap the cursor, not the compiler

I took a look at how django-debug-toolbar logs requests, this is less intrusive than patching compilers. We loose the access to query and query.model to differentiate Silky queries from the rest, but it can be replaced by simple string search on tables.

Method 3: Add a db wrapper on the connection for the request

As @shaib mentioned, it is possible to wrap cleanly (since Django 2.0) SQL queries, through a context manager, in the middleware for example:

class SilkyMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        self.process_request(request)

        query_logger = SilkyQueryLogger()
        with connection.execute_wrapper(qquery_logger):
            response = self.get_response(request)

        return self.process_response(request, response)

This is the cleanest way to log queries, and supported through official API. As in method 2, we don't have access to query model context.

Method 4: Add a permanent db wrapper on the connection in app.ready()

This is slightly different from method 3, because here the wrapper is added on app.ready(), to be made permanent, giving an advantage we might want to solve some feature requests like:

• Profile Commands? #593
• Use with django channels #383
• Detect manual sql queries #80
• Allowing profiling outside of the request/response cycle #3

This should work:

class SilkAppConfig(AppConfig):
    default_auto_field = "django.db.models.AutoField"
    name = "silk"


    def ready(self):
        # Add wrapper to db connection
        if not any(isinstance(wrapper, SilkQueryWrapper) for wrapper in connection.execute_wrappers):
            connection.execute_wrappers.append(SilkQueryWrapper())

I've prepared a pull request for that last method, @jazzband/django-silk what's your take on this?

[OscarVanL]

This would be awesome. I have several celery jobs, management commands, and other non-API contexts where I'd love to get the data I get from django-silk.

As a temporary workaround, I found this GitHub gist that you can use to run django-silk against arbitrary functions not wrapped in APIs. It would be fantastic to see something supported by django-silk in the future.