Improve performance by splitting query details off into seperate table #481
+175
−34
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@auvipy could you hit the button to approve the workflow run? |
auvipy
reviewed
Jul 2, 2021
| @@ -0,0 +1,38 @@ | |||
| # Generated by Django 2.2.24 on 2021-06-24 09:54 | |||
|
|
|||
There was a problem hiding this comment.
is this migration handwritten? or auto generated
There was a problem hiding this comment.
The code inside the migration was handwritten, the file was autogenerated using ./manage.py makemigrations silk --empty --name fill_query_details
Codecov Report
@@ Coverage Diff @@
## master #481 +/- ##
==========================================
+ Coverage 82.83% 84.32% +1.48%
==========================================
Files 50 53 +3
Lines 2051 2086 +35
==========================================
+ Hits 1699 1759 +60
+ Misses 352 327 -25
Continue to review full report at Codecov.
|
|
@ErwinJunge This is a candidate for merge if tox tests can be passed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR aims to improve the performance of silk by splitting the query details off into a seperate database table.
Some background information: The current system stores the details of the queries directly inside the query table itself. This leads to a very wide table and consequently few tuples per page, which makes doing aggregates over large tables very expensive in IO since the database has to load all the pages into memory.
What this PR does to fix that is pull the details that aren't used in the overview pages out of the main table and into a seperate table. This decreases the width of the main table, leading to more tuples per page and therefore less IO when generating overviews.
Specifically, in our production environment the sqlquery table has >3M rows. This lead to the overview page taking more than a minute to load and therefore timeout. With these changes, the overview page takes ~9s. Still long, but at least it doesn't timeout anymore.