feat(bulk-import): add import source

[dzemanov]

Hey, I just made a Pull Request!

Adds field source to GET /imports endpoint indicating from which Catalog source the import came from (config, location, integration). In case of duplicates, the first source is taken (in the order of config, location, integration), since there is no way of knowing if entity returned from locations endpoint and from entities endpoint is a duplicate or only a location entity.

This is needed so UI is able to show correct dialog when deleting bulk-import entities.

(Backstage in unregister shows dialog for location entity, even if it is a duplicate for e.g. config. Then after unregistration of this entity, it will show again 'This entity does not seem to originate from a registered location.')

Fixes

RHIDP-3931

How to test

• You can add in app-config.yaml locations accessible from GH integrations e.g.:

catalog:
  locations:
    - type: url
      target: https://github.com/.../catalog-info.yaml

• Add GH Discovery plugin to backend workspace and configurate it, for example:

catalog:
  providers:
    github:
      myorg:
        organization: ${GH_ORG}'
        schedule:
          frequency: { minutes: 30 }
          timeout: { minutes: 3 }

• Add some location manually via Create... -> Register Existing Component

• Example request/response:
  Request:

http -A bearer -a "${ADMIN_CURL_TOKEN}" \
      GET http://localhost:7007/api/bulk-import/imports \
      page==1 size==3

Response:

[
    {
        "approvalTool": "GIT",
        "id": "https://github.com/dzemanov/example",
        "lastUpdate": "2024-11-19T07:37:01Z",
        "repository": {
            "defaultBranch": "main",
            "id": "dzemanov/example",
            "name": "example",
            "organization": "dzemanov",
            "url": "https://github.com/dzemanov/example"
        },
        "source": "location",
        "status": "ADDED"
    },
    {
        "approvalTool": "GIT",
        "id": "https://github.com/dzemanov/hello-world",
        "lastUpdate": "2024-11-19T06:54:29Z",
        "repository": {
            "defaultBranch": "master",
            "id": "dzemanov/hello-world",
            "name": "hello-world",
            "organization": "dzemanov",
            "url": "https://github.com/dzemanov/hello-world"
        },
        "source": "config",
        "status": "ADDED"
    },
    {
        "approvalTool": "GIT",
        "id": "https://github.com/dzemanov-org/test",
        "lastUpdate": "2024-10-31T18:40:10Z",
        "repository": {
            "defaultBranch": "main",
            "id": "dzemanov-org/test",
            "name": "test",
            "organization": "dzemanov-org",
            "url": "https://github.com/dzemanov-org/test"
        },
        "source": "integration",
        "status": "ADDED"
    }
]

✔️ Checklist

• A changeset describing the change and affected packages. (more info)
• Added or Updated documentation
• Tests for new functionality and regression tests for bug fixes
• Screenshots attached (for UI changes)

[dzemanov]

/cc @ciiay

[ciiay]

Thank you @dzemanov , your solution looks good 👍 I was able to test the cases where the request returns source of location and config. How did you add a repository where the source value returns other?

[dzemanov]

@ciiay I have added external provider - GH Discovery plugin:
yarn workspace backend add @backstage/plugin-catalog-backend-module-github.
And configured it via:

catalog:
  providers:
    github:
      myorg:
        organization: ${GH_ORG}'
        schedule:
          frequency: { minutes: 30 }
          timeout: { minutes: 3 }

[ciiay]

@dzemanov thanks for the reply. Is this GH Discovery plugin the only way considered source value of other, or there are other ways to add a repository other than via GH Discovery plugin that is other case? I'm asking because @ShiranHi is considering to use "This repository added to the app-config file. To remove it modify the file directly" as a informative message to disable the deletion to repositories that are added via sources other than "location", and we want to make sure this wording covers both "config" and "other". Or we can mention like "This repository is added via app-config file", does this cover both config and other cases? Any suggestions?

[dzemanov]

@ciiay thanks for mentioning this. I changed 'other' to 'integration'. Entities can be ingested using GH integrations, so that could be multiple configured integrations under:

integrations:
    github:

Or I guess other GH entity provider plugins besides GH Discovery plugin.

These are all configured in 'app-config', so it makes sense to point the user there. To remove entities ingested by provider, one must remove the provider (which can remove a whole more bunch of entities) / update the repo in a way that provider removes it automatically in the next run, e.g removing catalog-info.yaml from repo.