feat: create new rest dlt factory

[Jabolol]

This PR closes #2396, closes #619, closes #2665, closes #2666 by creating a new factory: create_rest_factory_asset. It uses dlt's REST API source under the hood, abstracting all our infrastructure-specific details, so that new contributors can add new sources effortlessly. Usage is as follows:

Tip

For the full reference, check dlt docs.

config: RESTAPIConfig = {
    "client": {
        "base_url": "https://api.llama.fi/",
    },
    "resources": list(
        map(
            lambda protocol: {
                "name": f"{protocol.replace('-', '_')}",
                "endpoint": {
                    "path": f"protocol/{protocol}",
                    "data_selector": "$",
                },
            },
            DEFI_LLAMA_PROTOCOLS,
        )
    ),
}

dlt_assets = create_rest_factory_asset(
    config=config,
)

defillama_tvl_assets = dlt_assets(
    key_prefix=["defillama", "tvl"],
)

This will create new assets, such as defillama/tvl/aave-v3, which will fetch the endpoint and automatically upload the new source data to BigQuery. Since it uses dlt_factory under the hood, all the concurrency, tags, etc. implementations are available out of the box.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
kariba-network	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 21, 2024 9:47pm
oso-www	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 21, 2024 9:47pm

[oso-prs]

Auto-approved! please merge responsibly 😄

[oso-prs]

Test deployment unnecessary, no dbt files have been changed.