Skip to content
This repository has been archived by the owner on Sep 30, 2024. It is now read-only.

Commit

Permalink
[Backport 5.1] internal/event_logs: optimize ping cte (#55714)
Browse files Browse the repository at this point in the history 
The aggregatedCodyUsageEventsQuery is timing out for dotcom. These
changes are cte optimization steps.

## Test plan

Extract query from most expensive ping-supporting query,
[aggregatedCodyUsageEventsQuery](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/internal/database/event_logs.go?L1594),
analyze original query:
```
GroupAggregate  (cost=143042.53..143382.23 rows=270 width=158) (actual time=492.653..608.924 rows=87 loops=1)
  Group Key: event_logs.name, (date_trunc('month'::text, '2023-08-01 17:00:00-07'::timestamp with time zone)), ((date_trunc('week'::text, ('2023-08-01 17:00:00-07'::timestamp with time zone + '1 day'::interval)) - '1 day'::interval)), (date_trunc('day'::text, '2023-08-01 17:00:00-07'::timestamp with time zone))
  CTE code_generation_keys
    ->  Function Scan on unnest key  (cost=0.00..0.09 rows=9 width=32) (actual time=0.005..0.007 rows=9 loops=1)
  CTE explanation_keys
    ->  Function Scan on unnest key_1  (cost=0.00..0.05 rows=5 width=32) (actual time=0.012..0.013 rows=5 loops=1)
  ->  Sort  (cost=143042.38..143046.21 rows=1530 width=62) (actual time=492.526..496.803 rows=33511 loops=1)
        Sort Key: event_logs.name
        Sort Method: external merge  Disk: 3056kB
        ->  Gather  (cost=1000.16..142961.45 rows=1530 width=62) (actual time=12.381..418.370 rows=33511 loops=1)
              Workers Planned: 2
              Workers Launched: 2
              ->  Parallel Seq Scan on event_logs  (cost=0.17..141808.45 rows=638 width=62) (actual time=10.540..417.177 rows=11170 loops=3)
                    Filter: ((name !~~ '%completion:suggested%'::text) AND (name !~~ '%completion:started%'::text) AND (name !~~ '%CTA%'::text) AND (name !~~ '%Cta%'::text) AND (NOT (hashed SubPlan 9)) AND (lower(name) ~~ '%cody%'::text) AND ("timestamp" >= (date_trunc('month'::text, '2023-08-01 17:00:00-07'::timestamp with time zone) - '1 mon'::interval)))
                    Rows Removed by Filter: 382174
                    SubPlan 9
                      ->  Function Scan on unnest  (cost=0.00..0.13 rows=13 width=32) (actual time=0.007..0.008 rows=13 loops=3)
  SubPlan 8
    ->  CTE Scan on explanation_keys explanation_keys_3  (cost=0.00..0.10 rows=5 width=32) (actual time=0.000..0.001 rows=5 loops=1)
  SubPlan 7
    ->  CTE Scan on explanation_keys explanation_keys_2  (cost=0.00..0.10 rows=5 width=32) (actual time=0.014..0.016 rows=5 loops=1)
  SubPlan 6
    ->  CTE Scan on explanation_keys explanation_keys_1  (cost=0.00..0.10 rows=5 width=32) (actual time=0.000..0.001 rows=5 loops=1)
  SubPlan 5
    ->  CTE Scan on code_generation_keys code_generation_keys_1  (cost=0.00..0.18 rows=9 width=32) (actual time=0.000..0.002 rows=9 loops=1)
  SubPlan 4
    ->  CTE Scan on explanation_keys  (cost=0.00..0.10 rows=5 width=32) (actual time=0.000..0.001 rows=5 loops=1)
  SubPlan 3
    ->  CTE Scan on code_generation_keys  (cost=0.00..0.18 rows=9 width=32) (actual time=0.007..0.010 rows=9 loops=1)
Planning Time: 1.556 ms
Execution Time: 610.143 ms
```

Add indexes and analyze again in comments. <br> Backport
41e5cf3 from #55538

---------

Co-authored-by: Nathan Downs <[email protected]>
  • Loading branch information
@github-actions @nathan-downs
github-actions[bot] and nathan-downs authored Aug 10, 2023
1 parent 47f426c commit 72d8422
Show file tree
Hide file tree
Showing 10 changed files with 244 additions and 81 deletions.
3 changes: 3 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,14 @@ All notable changes to Sourcegraph are documented in this file.
- Pressing `Mod-f` will always select the input value in the file view search [#55546](https://github.com/sourcegraph/sourcegraph/pull/55546)
- When using OpenAI or Azure OpenAI for Cody completions, code completions will be disabled - chat will continue to work. This is because we currently don't support code completions with OpenAI. [#55624](https://github.com/sourcegraph/sourcegraph/pull/55624)
- The commit message defined in a batch spec will now be passed to `git commit` on stdin using `--file=-` instead of being included inline with `git commit -m` to improve how the message is interpreted by `git` in certain edge cases, such as when the commit message begins with a dash, and to prevent extra quotes being added to the message. This may mean that previous escaping strategies will behave differently.
- Improved performance of ping aggregations by adding functional indexes. [#55538](https://github.com/sourcegraph/sourcegraph/pull/55538)

### Fixed

- Fixed a bug where user account requests could not be approved even though the license would permit user creation otherwise. [#55482[(https://github.com/sourcegraph/sourcegraph/pull/55482)
- Fixed a bug where the background scheduler for embedding jobs based on policies would not schedule jobs for private repositories. [#55698](https://github.com/sourcegraph/sourcegraph/pull/55698)
- Fixed a bug where user account requests could not be approved even though the license would permit user creation otherwise. [#55482](https://github.com/sourcegraph/sourcegraph/pull/55482)
- Fixed a bug where date truncations were incorrectly converted to UTC, causing some dates to be off by one day. [#55367](https://github.com/sourcegraph/sourcegraph/pull/55367)

### Removed

Expand Down
69 changes: 13 additions & 56 deletions internal/database/event_logs.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1565,27 +1565,11 @@ func (l *eventLogStore) aggregatedSearchEvents(ctx context.Context, queryString
return events, nil
}

// List of events that don't meet the criteria of "active" usage of Cody.
var nonActiveCodyEvents = []string{
"CodyVSCodeExtension:CodySavedLogin:executed",
"web:codyChat:tryOnPublicCode",
"web:codyEditorWidget:viewed",
"web:codyChat:pageViewed",
"CodyConfigurationPageViewed",
"ClickedOnTryCodySearchCTA",
"TryCodyWebOnboardingDisplayed",
"AboutGetCodyPopover",
"TryCodyWeb",
"CodySurveyToastViewed",
"SiteAdminCodyPageViewed",
"CodyUninstalled",
"SpeakToACodyEngineerCTA",
}

var aggregatedCodyUsageEventsQuery = `
WITH events AS (
WITH
events AS (
SELECT
name AS key,
name,
` + aggregatedUserIDQueryFragment + ` AS user_id,
` + makeDateTruncExpression("month", "timestamp") + ` as month,
` + makeDateTruncExpression("week", "timestamp") + ` as week,
Expand All @@ -1595,36 +1579,11 @@ WITH events AS (
` + makeDateTruncExpression("day", "%s::timestamp") + ` as current_day
FROM event_logs
WHERE
timestamp >= ` + makeDateTruncExpression("month", "%s::timestamp") + `
AND lower(name) like '%%cody%%'
AND name not like '%%CTA%%'
AND name not like '%%Cta%%'
AND (name NOT IN ('` + strings.Join(nonActiveCodyEvents, "','") + `'))
),
code_generation_keys AS (
SELECT * FROM unnest(ARRAY[
'CodyVSCodeExtension:recipe:rewrite-to-functional:executed',
'CodyVSCodeExtension:recipe:improve-variable-names:executed',
'CodyVSCodeExtension:recipe:replace:executed',
'CodyVSCodeExtension:recipe:generate-docstring:executed',
'CodyVSCodeExtension:recipe:generate-unit-test:executed',
'CodyVSCodeExtension:recipe:rewrite-functional:executed',
'CodyVSCodeExtension:recipe:code-refactor:executed',
'CodyVSCodeExtension:recipe:fixup:executed',
'CodyVSCodeExtension:recipe:translate-to-language:executed'
]) AS key
),
explanation_keys AS (
SELECT * FROM unnest(ARRAY[
'CodyVSCodeExtension:recipe:explain-code-high-level:executed',
'CodyVSCodeExtension:recipe:explain-code-detailed:executed',
'CodyVSCodeExtension:recipe:find-code-smells:executed',
'CodyVSCodeExtension:recipe:git-history:executed',
'CodyVSCodeExtension:recipe:rate-code:executed'
]) AS key
timestamp >= %s::timestamp - '1 month'::interval
AND iscodyactiveevent(name)
)
SELECT
key,
name,
current_month,
current_week,
current_day,
Expand All @@ -1634,25 +1593,23 @@ SELECT
COUNT(DISTINCT user_id) FILTER (WHERE month = current_month) AS uniques_month,
COUNT(DISTINCT user_id) FILTER (WHERE week = current_week) AS uniques_week,
COUNT(DISTINCT user_id) FILTER (WHERE day = current_day) AS uniques_day,
SUM(case when month = current_month and key in
(SELECT * FROM code_generation_keys)
SUM(case when month = current_month and iscodygenerationevent(name)
then 1 else 0 end) as code_generation_month,
SUM(case when week = current_week and key in
(SELECT * FROM explanation_keys)
SUM(case when week = current_week and iscodygenerationevent(name)
then 1 else 0 end) as code_generation_week,
SUM(case when day = current_day and key in (SELECT * FROM code_generation_keys)
SUM(case when day = current_day and iscodygenerationevent(name)
then 1 else 0 end) as code_generation_day,
SUM(case when month = current_month and key in (SELECT * FROM explanation_keys)
SUM(case when month = current_month and iscodyexplanationevent(name)
then 1 else 0 end) as explanation_month,
SUM(case when week = current_week and key in (SELECT * FROM explanation_keys)
SUM(case when week = current_week and iscodyexplanationevent(name)
then 1 else 0 end) as explanation_week,
SUM(case when day = current_day and key in (SELECT * FROM explanation_keys)
SUM(case when day = current_day and iscodyexplanationevent(name)
then 1 else 0 end) as explanation_day,
0 as invalid_month,
0 as invalid_week,
0 as invalid_day
FROM events
GROUP BY key, current_month, current_week, current_day
GROUP BY name, current_month, current_week, current_day
`

var searchLatencyEventNames = []string{
Expand Down
61 changes: 36 additions & 25 deletions internal/database/event_logs_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1317,7 +1317,7 @@ func TestEventLogs_AggregatedCodyEvents(t *testing.T) {

codyEventNames := []string{"CodyVSCodeExtension:recipe:rewrite-to-functional:executed",
"CodyVSCodeExtension:recipe:explain-code-high-level:executed"}
users := []uint32{1, 2}
users := []uint32{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20}

days := []time.Time{
now, // Today
Expand All @@ -1332,7 +1332,7 @@ func TestEventLogs_AggregatedCodyEvents(t *testing.T) {
for _, user := range users {
for _, name := range codyEventNames {
for _, day := range days {
for i := 0; i < 25; i++ {
for i := 0; i < 250; i++ {
e := &Event{
UserID: user,
Name: name,
Expand Down Expand Up @@ -1361,35 +1361,46 @@ func TestEventLogs_AggregatedCodyEvents(t *testing.T) {

expectedEvents := []types.CodyAggregatedEvent{
{
Name: "CodyVSCodeExtension:recipe:explain-code-high-level:executed",
Month: time.Date(now.Year(), now.Month(), 1, 0, 0, 0, 0, time.UTC),
Week: now.Truncate(time.Hour * 24).Add(-time.Hour * 24 * 5),
Day: now.Truncate(time.Hour * 24),
TotalMonth: 200,
TotalWeek: 150,
TotalDay: 50,
UniquesMonth: 2,
UniquesWeek: 2,
UniquesDay: 2,
CodeGenerationWeek: 150,
CodeGenerationDay: 0,
ExplanationMonth: 200,
ExplanationWeek: 150,
ExplanationDay: 50,
Name: "CodyVSCodeExtension:recipe:explain-code-high-level:executed",
Month: time.Date(now.Year(), now.Month(), 1, 0, 0, 0, 0, time.UTC),
Week: now.Truncate(time.Hour * 24).Add(-time.Hour * 24 * 5),
Day: now.Truncate(time.Hour * 24),
TotalMonth: 20000,
TotalWeek: 15000,
TotalDay: 5000,
UniquesMonth: 20,
UniquesWeek: 20,
UniquesDay: 20,
CodeGenerationMonth: 0,
CodeGenerationWeek: 0,
CodeGenerationDay: 0,
ExplanationMonth: 20000,
ExplanationWeek: 15000,
ExplanationDay: 5000,
InvalidMonth: 0,
InvalidWeek: 0,
InvalidDay: 0,
},
{
Name: "CodyVSCodeExtension:recipe:rewrite-to-functional:executed",
Month: time.Date(now.Year(), now.Month(), 1, 0, 0, 0, 0, time.UTC),
Week: now.Truncate(time.Hour * 24).Add(-time.Hour * 24 * 5),
Day: now.Truncate(time.Hour * 24),
TotalMonth: 200,
TotalWeek: 150,
TotalDay: 50,
UniquesMonth: 2,
UniquesWeek: 2,
UniquesDay: 2,
CodeGenerationMonth: 200,
CodeGenerationDay: 50,
TotalMonth: 20000,
TotalWeek: 15000,
TotalDay: 5000,
UniquesMonth: 20,
UniquesWeek: 20,
UniquesDay: 20,
CodeGenerationMonth: 20000,
CodeGenerationWeek: 15000,
CodeGenerationDay: 5000,
ExplanationMonth: 0,
ExplanationWeek: 0,
ExplanationDay: 0,
InvalidMonth: 0,
InvalidWeek: 0,
InvalidDay: 0,
},
}

Expand Down
42 changes: 42 additions & 0 deletions internal/database/schema.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,18 @@
"Name": "invalidate_session_for_userid_on_password_change",
"Definition": "CREATE OR REPLACE FUNCTION public.invalidate_session_for_userid_on_password_change()\n RETURNS trigger\n LANGUAGE plpgsql\nAS $function$\n BEGIN\n IF OLD.passwd != NEW.passwd THEN\n NEW.invalidated_sessions_at = now() + (1 * interval '1 second');\n RETURN NEW;\n END IF;\n RETURN NEW;\n END;\n$function$\n"
},
{
"Name": "iscodyactiveevent",
"Definition": "CREATE OR REPLACE FUNCTION public.iscodyactiveevent(name text)\n RETURNS boolean\n LANGUAGE plpgsql\n IMMUTABLE\nAS $function$\nBEGIN\n RETURN\n (name LIKE '%%cody%%' OR name LIKE '%%Cody%%')\n AND name NOT IN (\n '%completion:started%',\n '%completion:suggested%',\n '%cta%',\n '%Cta%',\n 'CodyVSCodeExtension:CodySavedLogin:executed',\n 'web:codyChat:tryOnPublicCode',\n 'web:codyEditorWidget:viewed',\n 'web:codyChat:pageViewed',\n 'CodyConfigurationPageViewed',\n 'ClickedOnTryCodySearchCTA',\n 'TryCodyWebOnboardingDisplayed',\n 'AboutGetCodyPopover',\n 'TryCodyWeb',\n 'CodySurveyToastViewed',\n 'SiteAdminCodyPageViewed',\n 'CodyUninstalled',\n 'SpeakToACodyEngineerCTA'\n );\nEND;\n$function$\n"
},
{
"Name": "iscodyexplanationevent",
"Definition": "CREATE OR REPLACE FUNCTION public.iscodyexplanationevent(name text)\n RETURNS boolean\n LANGUAGE plpgsql\n IMMUTABLE\nAS $function$\nBEGIN\n RETURN name = ANY(ARRAY[\n 'CodyVSCodeExtension:recipe:explain-code-high-level:executed',\n 'CodyVSCodeExtension:recipe:explain-code-detailed:executed',\n 'CodyVSCodeExtension:recipe:find-code-smells:executed',\n 'CodyVSCodeExtension:recipe:git-history:executed',\n 'CodyVSCodeExtension:recipe:rate-code:executed'\n ]);\nEND;\n$function$\n"
},
{
"Name": "iscodygenerationevent",
"Definition": "CREATE OR REPLACE FUNCTION public.iscodygenerationevent(name text)\n RETURNS boolean\n LANGUAGE plpgsql\n IMMUTABLE\nAS $function$\nBEGIN\n RETURN name = ANY(ARRAY[\n 'CodyVSCodeExtension:recipe:rewrite-to-functional:executed',\n 'CodyVSCodeExtension:recipe:improve-variable-names:executed',\n 'CodyVSCodeExtension:recipe:replace:executed',\n 'CodyVSCodeExtension:recipe:generate-docstring:executed',\n 'CodyVSCodeExtension:recipe:generate-unit-test:executed',\n 'CodyVSCodeExtension:recipe:rewrite-functional:executed',\n 'CodyVSCodeExtension:recipe:code-refactor:executed',\n 'CodyVSCodeExtension:recipe:fixup:executed',\n\t'CodyVSCodeExtension:recipe:translate-to-language:executed'\n ]);\nEND;\n$function$\n"
},
{
"Name": "merge_audit_log_transitions",
"Definition": "CREATE OR REPLACE FUNCTION public.merge_audit_log_transitions(internal hstore, arrayhstore hstore[])\n RETURNS hstore\n LANGUAGE plpgsql\n IMMUTABLE\nAS $function$\n DECLARE\n trans hstore;\n BEGIN\n FOREACH trans IN ARRAY arrayhstore\n LOOP\n internal := internal || hstore(trans-\u003e'column', trans-\u003e'new');\n END LOOP;\n\n RETURN internal;\n END;\n$function$\n"
Expand Down Expand Up @@ -10400,6 +10412,36 @@
"ConstraintType": "",
"ConstraintDefinition": ""
},
{
"Name": "event_logs_name_is_cody_active_event",
"IsPrimaryKey": false,
"IsUnique": false,
"IsExclusion": false,
"IsDeferrable": false,
"IndexDefinition": "CREATE INDEX event_logs_name_is_cody_active_event ON event_logs USING btree (iscodyactiveevent(name))",
"ConstraintType": "",
"ConstraintDefinition": ""
},
{
"Name": "event_logs_name_is_cody_explanation_event",
"IsPrimaryKey": false,
"IsUnique": false,
"IsExclusion": false,
"IsDeferrable": false,
"IndexDefinition": "CREATE INDEX event_logs_name_is_cody_explanation_event ON event_logs USING btree (iscodyexplanationevent(name))",
"ConstraintType": "",
"ConstraintDefinition": ""
},
{
"Name": "event_logs_name_is_cody_generation_event",
"IsPrimaryKey": false,
"IsUnique": false,
"IsExclusion": false,
"IsDeferrable": false,
"IndexDefinition": "CREATE INDEX event_logs_name_is_cody_generation_event ON event_logs USING btree (iscodygenerationevent(name))",
"ConstraintType": "",
"ConstraintDefinition": ""
},
{
"Name": "event_logs_name_timestamp",
"IsPrimaryKey": false,
Expand Down
3 changes: 3 additions & 0 deletions internal/database/schema.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1355,6 +1355,9 @@ Referenced by:
Indexes:
"event_logs_pkey" PRIMARY KEY, btree (id)
"event_logs_anonymous_user_id" btree (anonymous_user_id)
"event_logs_name_is_cody_active_event" btree (iscodyactiveevent(name))
"event_logs_name_is_cody_explanation_event" btree (iscodyexplanationevent(name))
"event_logs_name_is_cody_generation_event" btree (iscodygenerationevent(name))
"event_logs_name_timestamp" btree (name, "timestamp" DESC)
"event_logs_source" btree (source)
"event_logs_timestamp" btree ("timestamp")
Expand Down
3 changes: 3 additions & 0 deletions migrations/BUILD.bazel
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 11 additions & 0 deletions migrations/frontend/1691043630_event_logs_indexing/down.sql
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
DROP INDEX IF EXISTS event_logs_name_is_cody_explanation_event;

DROP INDEX IF EXISTS event_logs_name_is_cody_generation_event;

DROP INDEX IF EXISTS event_logs_name_is_cody_active_event;

DROP FUNCTION IF EXISTS isCodyGenerationEvent(name text);

DROP FUNCTION IF EXISTS isCodyExplanationEvent(name text);

DROP FUNCTION IF EXISTS isCodyActiveEvent(name text);
2 changes: 2 additions & 0 deletions migrations/frontend/1691043630_event_logs_indexing/metadata.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
name: event_logs_indexing
parents: [1690323910]
65 changes: 65 additions & 0 deletions migrations/frontend/1691043630_event_logs_indexing/up.sql
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
CREATE OR REPLACE FUNCTION iscodyactiveevent(name text) RETURNS boolean
LANGUAGE plpgsql IMMUTABLE
AS $$
BEGIN
RETURN
(name LIKE '%%cody%%' OR name LIKE '%%Cody%%')
AND name NOT IN (
'%completion:started%',
'%completion:suggested%',
'%cta%',
'%Cta%',
'CodyVSCodeExtension:CodySavedLogin:executed',
'web:codyChat:tryOnPublicCode',
'web:codyEditorWidget:viewed',
'web:codyChat:pageViewed',
'CodyConfigurationPageViewed',
'ClickedOnTryCodySearchCTA',
'TryCodyWebOnboardingDisplayed',
'AboutGetCodyPopover',
'TryCodyWeb',
'CodySurveyToastViewed',
'SiteAdminCodyPageViewed',
'CodyUninstalled',
'SpeakToACodyEngineerCTA'
);
END;
$$;

CREATE OR REPLACE FUNCTION iscodyexplanationevent(name text) RETURNS boolean
LANGUAGE plpgsql IMMUTABLE
AS $$
BEGIN
RETURN name = ANY(ARRAY[
'CodyVSCodeExtension:recipe:explain-code-high-level:executed',
'CodyVSCodeExtension:recipe:explain-code-detailed:executed',
'CodyVSCodeExtension:recipe:find-code-smells:executed',
'CodyVSCodeExtension:recipe:git-history:executed',
'CodyVSCodeExtension:recipe:rate-code:executed'
]);
END;
$$;

CREATE OR REPLACE FUNCTION iscodygenerationevent(name text) RETURNS boolean
LANGUAGE plpgsql IMMUTABLE
AS $$
BEGIN
RETURN name = ANY(ARRAY[
'CodyVSCodeExtension:recipe:rewrite-to-functional:executed',
'CodyVSCodeExtension:recipe:improve-variable-names:executed',
'CodyVSCodeExtension:recipe:replace:executed',
'CodyVSCodeExtension:recipe:generate-docstring:executed',
'CodyVSCodeExtension:recipe:generate-unit-test:executed',
'CodyVSCodeExtension:recipe:rewrite-functional:executed',
'CodyVSCodeExtension:recipe:code-refactor:executed',
'CodyVSCodeExtension:recipe:fixup:executed',
'CodyVSCodeExtension:recipe:translate-to-language:executed'
]);
END;
$$;

CREATE INDEX IF NOT EXISTS event_logs_name_is_cody_explanation_event ON event_logs USING btree (iscodyexplanationevent(name));

CREATE INDEX IF NOT EXISTS event_logs_name_is_cody_generation_event ON event_logs USING btree (iscodygenerationevent(name));

CREATE INDEX IF NOT EXISTS event_logs_name_is_cody_active_event ON event_logs USING btree (iscodyactiveevent(name));
Loading

0 comments on commit 72d8422

Please sign in to comment.