[Backport 5.0] Offline drift check improvements (#50374)

closes #49813

This PR started as an exploration of embedding schema descriptions in
the migrator image via a new factory generating function. It turns out
this functionality always existed in
[cmd/migrator/build.sh](https://sourcegraph.com/github.com/sourcegraph/sourcegraph@38fcc28/-/blob/cmd/migrator/build.sh?L92-96)

We need to maintain the list of tags found in that file during releases
-- this PR brings the list up to the current build
```sh
git_versions=(
  v3.42.0 v3.42.1 v3.42.2
  v3.43.0 v3.43.1 v3.43.2
  v4.0.0 v4.0.1 v4.1.0
  v4.1.1 v4.1.2 v4.1.3
  v4.2.0 v4.2.1
  v4.3.0 v4.3.1
  v4.4.0 v4.4.1 v4.4.2
  v4.5.0 v4.5.1
  v5.0.0
)
```

This PR also changes the order in which schema description are sought
such that locally built schema descriptions in
[schema-descriptions](https://sourcegraph.com/github.com/sourcegraph/sourcegraph@38fcc28/-/blob/cmd/migrator/Dockerfile?L18)
are always checked first.

## Test plan

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->

This was tested by building a test image of the migrator with the
addition of these version tags and then running a few drift checks
against the latest versions with the `version` column in the database
set to the desired schema check.

### v5.0.0 Positive Case _(note drift is expected since the db schema is
actually latest/dev)_
```
λ ~/sourcegraph/ wg/drift-offline-auto-infer* docker run -e PGHOST=host.docker.internal -e PGUSER=sourcegraph -e PGPASSWORD=sg --network=host dev-migrator drift --db frontend
✱ Sourcegraph migrator
ℹ️ Checking drift against version "v5.0.0"
ℹ️ Locating schema description
✅ Schema found in Local file (/schema-descriptions/v5.0.0-internal_database_schema.json).
❌ Drift detected!

❌ Unexpected definition of function "soft_deleted_repository_name"

strings.Join({
  	"CREATE OR REPLACE FUNCTION
public.soft_deleted_repository_name(name text)",
  	" RETURNS text",
  	" LANGUAGE plpgsql",
- 	" STRICT",
  	"AS $function$",
  	"BEGIN",
- 	"    RETURN 'DELETED-' || extract(epoch from
transaction_timestamp()) || '-' || name;",
+ 	"    IF name LIKE 'DELETED-%' THEN",
+ 	"        RETURN name;",
+ 	"    ELSE",
+ 	"        RETURN 'DELETED-' || extract(epoch from
transaction_timestamp()) || '-' || name;",
+ 	"    END IF;",
  	"END;",
  	"$function$",
  	"",
  }, "\n")

💡 Suggested action: replace the function definition.

CREATE OR REPLACE FUNCTION
public.soft_deleted_repository_name(name text)
 RETURNS text
 LANGUAGE plpgsql
 STRICT
AS $function$
BEGIN
    RETURN 'DELETED-' || extract(epoch from
transaction_timestamp()) || '-' || name;
END;
$function$;

❌ Unexpected properties of sequence "codeintel_ranking_references_processed_id_seq"

schemas.SequenceDescription{
  	Name:
"codeintel_ranking_references_processed_id_seq",
- 	TypeName:     "integer",
+ 	TypeName:     "bigint",
  	StartValue:   1,
  	MinimumValue: 1,
- 	MaximumValue: 2147483647,
+ 	MaximumValue: 9223372036854775807,
  	Increment:    1,
  	CycleOption:  "NO",
  }

💡 Hint: redefine the sequence using the definition at the following URL:

👉 https://sourcegraph.com/search?patternType=regexp&q=repo%3A%5Egithub%5C.com%2Fsourcegraph%2Fsourcegraph%24%40v5.0.0+file%3A%5Emigrations%2Ffrontend%2Fsquashed%5C.sql%24+%28%5E%7C%5Cb%29CREATE%5CsSEQUENCE%5Cscodeintel_ranking_references_processed_id_seq%28%24%7C%5Cb%29+OR+%28%5E%7C%5Cb%29nextval%5C%28%27codeintel_ranking_references_processed_id_seq%27%3A%3Aregclass%5C%29%3B%28%24%7C%5Cb%29

❌ Unexpected index "cm_action_jobs"."cm_action_jobs_trigger_event"
💡 Suggested action: drop the index.

DROP INDEX cm_action_jobs_trigger_event;

❌ Unexpected properties of column "codeintel_initial_path_ranks"."document_path"

schemas.ColumnDescription{
  	... // 2 identical fields
  	TypeName:               "text",
  	IsNullable:             false,
- 	Default:                "",
+ 	Default:                "''::text",
  	CharacterMaximumLength: 0,
  	IsIdentity:             false,
  	... // 4 identical fields
  }

💡 Suggested action: change the column default.

ALTER TABLE codeintel_initial_path_ranks ALTER COLUMN
document_path SET DEFAULT ;

❌ Unexpected column "codeintel_initial_path_ranks"."document_paths"
💡 Suggested action: drop the column.

ALTER TABLE codeintel_initial_path_ranks DROP COLUMN
document_paths;

❌ Unexpected index "codeintel_ranking_path_counts_inputs"."codeintel_ranking_path_counts_inputs_graph_key_id"
💡 Suggested action: drop the index.

DROP INDEX codeintel_ranking_path_counts_inputs_graph_key_id;

❌ Unexpected properties of column "codeintel_ranking_references_processed"."id"

schemas.ColumnDescription{
  	Name:       "id",
  	Index:      -1,
- 	TypeName:   "integer",
+ 	TypeName:   "bigint",
  	IsNullable: false,
  	Default:
"nextval('codeintel_ranking_references_processed_id_seq'::regclass)",
  	... // 6 identical fields
  }

💡 Hint: redefine the column using the definition at the following URL:

👉 https://sourcegraph.com/search?patternType=regexp&q=repo%3A%5Egithub%5C.com%2Fsourcegraph%2Fsourcegraph%24%40v5.0.0+file%3A%5Emigrations%2Ffrontend%2Fsquashed%5C.sql%24+%28%5E%7C%5Cb%29CREATE%5CsTABLE%5Cscodeintel_ranking_references_processed%28%24%7C%5Cb%29+OR+%28%5E%7C%5Cb%29ALTER%5CsTABLE%5CsONLY%5Cscodeintel_ranking_references_processed%28%24%7C%5Cb%29

❌ Unexpected index "codeintel_ranking_references_processed"."codeintel_ranking_references_processed_reference_id"
💡 Suggested action: drop the index.

DROP INDEX codeintel_ranking_references_processed_reference_id;

❌ Unexpected column "gitserver_repos"."cloning_progress"
💡 Suggested action: drop the column.

ALTER TABLE gitserver_repos DROP COLUMN cloning_progress;

❌ Unexpected column "permission_sync_jobs"."is_partial_success"
💡 Suggested action: drop the column.

ALTER TABLE permission_sync_jobs DROP COLUMN is_partial_success;

{"SeverityText":"FATAL","Timestamp":1680647088348251300,"InstrumentationScope":"migrator","Caller":"migrator/main.go:21","Function":"main.main","Body":"database schema is out of sync","Resource":{"service.name":"migrator","service.instance.id":"79d6aa1d-a298-4869-b084-90d4c54ba534"},"Attributes":{}}
```

### `v0.0.0` Failed Case

```
λ ~/sourcegraph/ wg/drift-offline-auto-infer* docker run -e PGHOST=host.docker.internal -e PGUSER=sourcegraph -e PGPASSWORD=sg --network=host dev-migrator drift --db frontend
✱ Sourcegraph migrator
ℹ️ Checking drift against version "v0.0.0"
ℹ️ Locating schema description
ℹ️ Reading schema definition in Local file (/schema-descriptions/v0.0.0-internal_database_schema.json)... Schema not found (open /schema-descriptions/v0.0.0-internal_database_schema.json: no such file or directory). Will attempt a fallback source.
ℹ️ Reading schema definition in GitHub (https://raw.githubusercontent.com/sourcegraph/sourcegraph/v0.0.0/internal/database/schema.json)... Schema not found (HTTP 404: https://raw.githubusercontent.com/sourcegraph/sourcegraph/v0.0.0/internal/database/schema.json). Will attempt a fallback source.
ℹ️ Reading schema definition in GCS (https://storage.googleapis.com/sourcegraph-assets/migrations/drift/v0.0.0-internal_database_schema.json)... Schema not found (HTTP 404: https://storage.googleapis.com/sourcegraph-assets/migrations/drift/v0.0.0-internal_database_schema.json).
💡 Schema not found. Check if the following resources exist. If they do, then the context in which this migrator is being run may not be permitted to reach the public internet.
  - https://raw.githubusercontent.com/sourcegraph/sourcegraph/v0.0.0/internal/database/schema.json
  - https://storage.googleapis.com/sourcegraph-assets/migrations/drift/v0.0.0-internal_database_schema.json
Alternative, provide a different version that matches one of the following patterns:
  - 4.1 (version branch)
  - 57b1f56 (12-character sha)
  - 57b1f56787619464dc62f469127d64721b428b76 (40-character sha)
```

### Build command

```
frontend	5.0.0	2023-04-03 15:02:27.100241-07	0.0.0	true
``` <br> Backport 5fbbcda from #50207

Co-authored-by: Warren Gifford <[email protected]>

cmd/migrator/build.sh

@@ -75,7 +75,12 @@ git_versions=(
   v3.42.0 v3.42.1 v3.42.2
   v3.43.0 v3.43.1 v3.43.2
   v4.0.0 v4.0.1
-  v4.1.0 v4.1.1 v4.1.2
+  v4.1.0 v4.1.1 v4.1.2 v4.1.3
+  v4.2.0 v4.2.1
+  v4.3.0 v4.3.1
+  v4.4.0 v4.4.1 v4.4.2
+  v4.5.0 v4.5.1
+  v5.0.0
 )
 for version in "${git_versions[@]}"; do
   echo "Persisting schemas for ${version} from Git..."

cmd/migrator/shared/main.go

@@ -48,9 +48,9 @@ func NewRunnerWithSchemas(observationCtx *observation.Context, logger log.Logger
 // DefaultSchemaFactories is a list of schema factories to be used in
 // non-exceptional cases.
 var DefaultSchemaFactories = []cliutil.ExpectedSchemaFactory{
+	cliutil.LocalExpectedSchemaFactory,
 	cliutil.GitHubExpectedSchemaFactory,
 	cliutil.GCSExpectedSchemaFactory,
-	cliutil.LocalExpectedSchemaFactory,
 }
 
 func Start(logger log.Logger, registerEnterpriseMigrators registerMigratorsUsingConfAndStoreFactoryFunc) error {