From 00ec0666573dc9acef4b7fc6d6f32bcf950c3e30 Mon Sep 17 00:00:00 2001 From: Raymond Cheng Date: Sat, 14 Dec 2024 15:31:07 -0800 Subject: [PATCH] docs: more reorganization of docs (#2643) --- README.md | 2 +- .../index.md | 4 +- .../index.md | 2 +- .../blog/2024-07-16-oso-data-portal/index.md | 16 ++-- .../2024-08-12-war-for-public-goods/index.md | 2 +- .../contribute-data/bigquery/_category_.json | 2 +- .../docs/contribute-data/bigquery/index.md | 4 +- apps/docs/docs/contribute-data/quickstart.md | 2 +- .../2024-04-05-data-challenge-01.md | 2 +- .../challenges/2024-07-30-openrank.md | 2 +- apps/docs/docs/contribute-models/index.mdx | 36 -------- .../docs/contribute-models/share-insights.md | 2 +- apps/docs/docs/get-started/api.mdx | 2 +- apps/docs/docs/get-started/bigquery.mdx | 6 +- apps/docs/docs/guides/ops/_category_.json | 2 +- apps/docs/docs/guides/style-guide.md | 2 +- apps/docs/docs/integrate/3rd-party.mdx | 2 +- apps/docs/docs/integrate/api.md | 77 +++++++++--------- .../{overview => datasets}/arbitrum.png | Bin .../bigquery_starred_datasets.png | Bin .../bigquery_subscribe.png | Bin .../integrate/{overview => datasets}/eas.png | Bin .../{overview => datasets}/ethereum.png | Bin .../{overview => datasets}/farcaster.jpg | Bin .../{overview => datasets}/filecoin.png | Bin .../{overview => datasets}/gitcoin.png | Bin .../{overview => datasets}/github.png | Bin .../{overview => datasets}/index.mdx | 10 +-- .../{overview => datasets}/lens-protocol.png | Bin .../integrate/{overview => datasets}/op.png | Bin .../open-collective.png | Bin .../open-source-insights.png | Bin .../{overview => datasets}/openrank.png | Bin .../{overview => datasets}/oso-primary.png | Bin apps/docs/docs/integrate/index.md | 5 +- apps/docs/docs/integrate/python-notebooks.md | 4 +- apps/docs/docs/integrate/query-data.mdx | 4 +- .../{guides => references}/architecture.md | 4 +- .../ecosystems/_category_.json | 2 +- .../ecosystems/arbitrum.md | 0 .../{guides => references}/ecosystems/pln.md | 0 .../ecosystems/superchain.md | 0 .../docs/docs/{guides => references}/event.md | 2 +- apps/docs/docs/references/faq.md | 4 +- .../impact-metrics/_category_.json | 2 +- .../impact-metrics/code.md | 0 .../impact-metrics/dependencies.md | 0 .../impact-metrics/index.mdx | 0 .../impact-metrics/onchain.md | 2 +- .../impact-metrics/package.md | 0 .../join-team.mdx | 0 .../docs/{guides => references}/mission.md | 0 apps/docs/docs/references/talks.md | 2 +- apps/docs/docs/tutorials/collection-view.md | 2 +- apps/docs/docs/tutorials/network-graph.md | 2 +- apps/docs/docs/tutorials/project-deepdive.md | 2 +- warehouse/dbt/models/homepage.md | 2 +- 57 files changed, 92 insertions(+), 124 deletions(-) rename apps/docs/docs/integrate/{overview => datasets}/arbitrum.png (100%) rename apps/docs/docs/integrate/{overview => datasets}/bigquery_starred_datasets.png (100%) rename apps/docs/docs/integrate/{overview => datasets}/bigquery_subscribe.png (100%) rename apps/docs/docs/integrate/{overview => datasets}/eas.png (100%) rename apps/docs/docs/integrate/{overview => datasets}/ethereum.png (100%) rename apps/docs/docs/integrate/{overview => datasets}/farcaster.jpg (100%) rename apps/docs/docs/integrate/{overview => datasets}/filecoin.png (100%) rename apps/docs/docs/integrate/{overview => datasets}/gitcoin.png (100%) rename apps/docs/docs/integrate/{overview => datasets}/github.png (100%) rename apps/docs/docs/integrate/{overview => datasets}/index.mdx (98%) rename apps/docs/docs/integrate/{overview => datasets}/lens-protocol.png (100%) rename apps/docs/docs/integrate/{overview => datasets}/op.png (100%) rename apps/docs/docs/integrate/{overview => datasets}/open-collective.png (100%) rename apps/docs/docs/integrate/{overview => datasets}/open-source-insights.png (100%) rename apps/docs/docs/integrate/{overview => datasets}/openrank.png (100%) rename apps/docs/docs/integrate/{overview => datasets}/oso-primary.png (100%) rename apps/docs/docs/{guides => references}/architecture.md (98%) rename apps/docs/docs/{guides => references}/ecosystems/_category_.json (81%) rename apps/docs/docs/{guides => references}/ecosystems/arbitrum.md (100%) rename apps/docs/docs/{guides => references}/ecosystems/pln.md (100%) rename apps/docs/docs/{guides => references}/ecosystems/superchain.md (100%) rename apps/docs/docs/{guides => references}/event.md (99%) rename apps/docs/docs/{guides => references}/impact-metrics/_category_.json (83%) rename apps/docs/docs/{guides => references}/impact-metrics/code.md (100%) rename apps/docs/docs/{guides => references}/impact-metrics/dependencies.md (100%) rename apps/docs/docs/{guides => references}/impact-metrics/index.mdx (100%) rename apps/docs/docs/{guides => references}/impact-metrics/onchain.md (97%) rename apps/docs/docs/{guides => references}/impact-metrics/package.md (100%) rename apps/docs/docs/{contribute-models => references}/join-team.mdx (100%) rename apps/docs/docs/{guides => references}/mission.md (100%) diff --git a/README.md b/README.md index 21fcc0003..b3bb7ed39 100644 --- a/README.md +++ b/README.md @@ -212,4 +212,4 @@ is released under Apache 2.0 This repository does not contain data. Datasets may include material that may be subject to third party rights. For details on each dataset, see -the [Data Overview](https://docs.opensource.observer/docs/integrate/overview/). +the [Data Overview](https://docs.opensource.observer/docs/integrate/datasets/). diff --git a/apps/docs/blog/2024-03-20-request-for-impact-metrics/index.md b/apps/docs/blog/2024-03-20-request-for-impact-metrics/index.md index e836c2ef3..f1a522456 100644 --- a/apps/docs/blog/2024-03-20-request-for-impact-metrics/index.md +++ b/apps/docs/blog/2024-03-20-request-for-impact-metrics/index.md @@ -17,7 +17,7 @@ If you'd like to get involved, here's what to do: 1. Apply to [join the Data Collective](https://www.kariba.network/). It only takes a few minutes. We'll review, and reach out to schedule an onboarding call. 2. Join our [Discord server](https://www.opensource.observer/discord) and say hello. 3. Get inspiration from our Colab directory of [starter notebooks for impact metrics](https://drive.google.com/drive/folders/1I4exiLfZYgPwIGBEzuAMeGtJUhtS_DE7?usp=drive_link). We also have of them in our [Insights repo](https://github.com/opensource-observer/insights/tree/main/community/notebooks) if you prefer to run them locally. -4. Consult [our docs](../../docs/) and especially our [impact metric spec](../../docs/guides/impact-metrics/) as you prepare your analysis. +4. Consult [our docs](../../docs/) and especially our [impact metric spec](../../docs/references/impact-metrics/) as you prepare your analysis. The rest of this post includes more details on the types of impact metrics we're interested in. @@ -62,7 +62,7 @@ _Metrics that attempt to differentiate onchain users based on behavior and trust This is a hot topic right now, with a number of projects attempting to create reputation systems for onchain users. We're integrating with many of the leading projects and bringing the data they generate into the OSO data warehouse. From there, there are all sorts of directions you can take the analysis! -The **Onchain Activity** metric is a good starting point for new analysts. Have a look at our working definitions for [onchain users](../../docs/guides/impact-metrics/onchain) and help us create some better ones. The **Network Loyalty** is a spicy metric that will get even spicier as we add more networks to the OSO dataset! +The **Onchain Activity** metric is a good starting point for new analysts. Have a look at our working definitions for [onchain users](../../docs/references/impact-metrics/onchain) and help us create some better ones. The **Network Loyalty** is a spicy metric that will get even spicier as we add more networks to the OSO dataset! | Impact Metric | Description | Comments | | :--------------- | :---------------------------------------------------- | :----------------------------------------- | diff --git a/apps/docs/blog/2024-06-29-impact-metrics-rf4-deep-dive/index.md b/apps/docs/blog/2024-06-29-impact-metrics-rf4-deep-dive/index.md index 6413b390a..35899bef8 100644 --- a/apps/docs/blog/2024-06-29-impact-metrics-rf4-deep-dive/index.md +++ b/apps/docs/blog/2024-06-29-impact-metrics-rf4-deep-dive/index.md @@ -116,7 +116,7 @@ We're excited to see how this type of model evolves and is extended for future u ## Developing the impact metrics -The following design principles guide the development and evolution of [impact metrics](../../docs/guides/impact-metrics/#principles): +The following design principles guide the development and evolution of [impact metrics](../../docs/references/impact-metrics/#principles): - Verifiability: Metrics should be based on public data that can be independently verified. They should not rely on proprietary data sources or private APIs. - Reproducibility: Metrics should be easy to reproduce, simulate, and audit to ensure they are achieving the intended results. They should not have a "black box" element that makes them difficult to understand or replicate. diff --git a/apps/docs/blog/2024-07-16-oso-data-portal/index.md b/apps/docs/blog/2024-07-16-oso-data-portal/index.md index 79249ac7c..a7b973631 100644 --- a/apps/docs/blog/2024-07-16-oso-data-portal/index.md +++ b/apps/docs/blog/2024-07-16-oso-data-portal/index.md @@ -14,17 +14,17 @@ OSO Data Exchange on Google BigQuery. Here, we will publish every data set we have as live, up-to-date, and free to use datasets. In addition to sharing every model in the -[OSO production data pipeline](https://docs.opensource.observer/docs/integrate/overview/#oso-production-data-pipeline), +[OSO production data pipeline](../../docs/integrate/datasets#oso-production-data-pipeline), we are sharing source data for blocks/transactions/traces across -[7 chains in the OP Superchain](https://docs.opensource.observer/docs/integrate/overview/#superchain-data) +[7 chains in the OP Superchain](../../docs/integrate/datasets#optimism-data) (including Optimism, Base, Frax, Metal, Mode, PGN, Zora), -[Gitcoin Data](https://docs.opensource.observer/docs/integrate/overview/#gitcoin-and-passport-data), -and [OpenRank](https://docs.opensource.observer/docs/integrate/overview/#openrank-data). +[Gitcoin Data](../../docs/integrate/datasets#gitcoin-and-passport-data), +and [OpenRank](../../docs/integrate/datasets#openrank-data). This builds on the existing BigQuery public data ecosystem that includes -[GitHub](https://docs.opensource.observer/docs/integrate/overview/#github-data), -[Ethereum](https://docs.opensource.observer/docs/integrate/overview/#ethereum-data), -[Farcaster](https://docs.opensource.observer/docs/integrate/overview/#farcaster-data), -and [Lens](https://docs.opensource.observer/docs/integrate/overview/#lens-data) data. +[GitHub](../../docs/integrate/datasets#github-data), +[Ethereum](../../docs/integrate/datasets#ethereum-data), +[Farcaster](../../docs/integrate/datasets#farcaster-data), +and [Lens](../../docs/integrate/datasets#lens-data) data. To learn more, check out the data portal here:

diff --git a/apps/docs/blog/2024-08-12-war-for-public-goods/index.md b/apps/docs/blog/2024-08-12-war-for-public-goods/index.md index f4db09d59..1329e1d12 100644 --- a/apps/docs/blog/2024-08-12-war-for-public-goods/index.md +++ b/apps/docs/blog/2024-08-12-war-for-public-goods/index.md @@ -97,4 +97,4 @@ Once again, there are valuable lessons from baseball that we can apply: - **Embrace Competing Implementations**: As mentioned earlier, there is no single standard calculation for WAR. Even though the data and methodologies are public, there’s vibrant debate about which method is best. In an industry as dynamic as crypto, this diversity of approaches is a strength, encouraging innovation and continuous improvement. - **Engage the Research Community**: WAR models are like catnip for data enthusiasts. They offer endless opportunities for simulation, experimentation, and visualization. By providing well-documented, easy-to-use datasets and stoking interest in the results from protocol leaders and governance teams, we can create a garden of catnip for researchers and data nerds. -At Open Source Observer, we want to build the equivalent of a sabermetrics for open source impact. If you have ideas for a WAR model, check out our existing library of [data](../../docs/integrate/overview) and [metrics](../../docs/guides/impact-metrics/), and come say hi on our [Discord](https://www.opensource.observer/discord). +At Open Source Observer, we want to build the equivalent of a sabermetrics for open source impact. If you have ideas for a WAR model, check out our existing library of [data](../../docs/integrate/datasets) and [metrics](../../docs/references/impact-metrics/), and come say hi on our [Discord](https://www.opensource.observer/discord). diff --git a/apps/docs/docs/contribute-data/bigquery/_category_.json b/apps/docs/docs/contribute-data/bigquery/_category_.json index db77c4917..4fbc04a20 100644 --- a/apps/docs/docs/contribute-data/bigquery/_category_.json +++ b/apps/docs/docs/contribute-data/bigquery/_category_.json @@ -1,6 +1,6 @@ { "label": "Connect via BigQuery", - "position": 1, + "position": 2, "link": { "type": "doc", "id": "index" diff --git a/apps/docs/docs/contribute-data/bigquery/index.md b/apps/docs/docs/contribute-data/bigquery/index.md index 73ae685d2..566245300 100644 --- a/apps/docs/docs/contribute-data/bigquery/index.md +++ b/apps/docs/docs/contribute-data/bigquery/index.md @@ -8,7 +8,9 @@ trivially easy to integrate any public dataset into the OSO data pipeline, provided the dataset exists in the US multi-region. -If the dataset needs to be replicated, see our guide on +If you want OSO to host a copy of +the dataset in the US multi-region, +see our guide on [BigQuery Data Transfer Service](./replication.md). ## Make the data available in the US region diff --git a/apps/docs/docs/contribute-data/quickstart.md b/apps/docs/docs/contribute-data/quickstart.md index 0fb130a1a..df4990fbc 100644 --- a/apps/docs/docs/contribute-data/quickstart.md +++ b/apps/docs/docs/contribute-data/quickstart.md @@ -1,6 +1,6 @@ --- title: Dagster Quickstart -sidebar_position: 2 +sidebar_position: 1 --- # Dagster quickstart diff --git a/apps/docs/docs/contribute-models/challenges/2024-04-05-data-challenge-01.md b/apps/docs/docs/contribute-models/challenges/2024-04-05-data-challenge-01.md index 9fa232377..5717d88a9 100644 --- a/apps/docs/docs/contribute-models/challenges/2024-04-05-data-challenge-01.md +++ b/apps/docs/docs/contribute-models/challenges/2024-04-05-data-challenge-01.md @@ -72,7 +72,7 @@ Here’s what you need to do to participate in this (and future) data challenges 1. Join the [Kariba Data Collective](https://www.kariba.network/). Participation is open to anyone in the data collective (free to join, but we review applications). 2. Join our Discord and receive a **_data-collective_** role; ask questions and share code snippets in the **_#data-challenges_** channel. -3. Bookmark [our docs and tutorials](../../guides/impact-metrics/) for creating impact metrics. Also make sure to browse our [Colab Notebooks](https://drive.google.com/drive/folders/1mzqrSToxPaWhsoGOR-UVldIsaX1gqP0F) for examples and inspiration. +3. Bookmark [our docs and tutorials](../../references/impact-metrics/) for creating impact metrics. Also make sure to browse our [Colab Notebooks](https://drive.google.com/drive/folders/1mzqrSToxPaWhsoGOR-UVldIsaX1gqP0F) for examples and inspiration. 4. Submit your impact metric(s) by opening an issue on our Insight repo [here](https://github.com/opensource-observer/insights/issues/new/choose). Every impact metric submission should include: diff --git a/apps/docs/docs/contribute-models/challenges/2024-07-30-openrank.md b/apps/docs/docs/contribute-models/challenges/2024-07-30-openrank.md index b24e7821e..1c73000cd 100644 --- a/apps/docs/docs/contribute-models/challenges/2024-07-30-openrank.md +++ b/apps/docs/docs/contribute-models/challenges/2024-07-30-openrank.md @@ -34,7 +34,7 @@ We've queued up a few ideas to get you started: - **Protocol Rank**. This [notebook](https://github.com/opensource-observer/insights/blob/main/community/data_challenges/openrank/OpenRank_ProtocolRank.ipynb) uses the same logic as the Repo Rank notebook, but narrows the curation space to only include projects that have deployed contracts on an Ethereum Layer 2. -- **Gitcoin Project Rank**. This [notebook](https://github.com/opensource-observer/insights/blob/main/community/data_challenges/openrank/OpenRank_GitcoinProjectRank.ipynb) builds on top of our recently-added [Gitcoin Grants dataset](https://docs.opensource.observer/docs/integrate/overview/#gitcoin-and-passport-data), which includes a history of every peer-to-peer donation on the Gitcoin platform. It also taps the Farcaster dataset to identify usernames associated with donors. +- **Gitcoin Project Rank**. This [notebook](https://github.com/opensource-observer/insights/blob/main/community/data_challenges/openrank/OpenRank_GitcoinProjectRank.ipynb) builds on top of our recently-added [Gitcoin Grants dataset](../../integrate/datasets/index.mdx#gitcoin-and-passport-data), which includes a history of every peer-to-peer donation on the Gitcoin platform. It also taps the Farcaster dataset to identify usernames associated with donors. ### Requirements diff --git a/apps/docs/docs/contribute-models/index.mdx b/apps/docs/docs/contribute-models/index.mdx index f27be11e5..cc387785a 100644 --- a/apps/docs/docs/contribute-models/index.mdx +++ b/apps/docs/docs/contribute-models/index.mdx @@ -17,42 +17,6 @@ There are a variety of ways you can contribute to OSO. This doc features some of - - - Update Project Data - - - - oss-directory - - - Add a new project or update info for an existing project. - OSS Projects, Analysts, General Public - - - - Add Funding Data - - - - oss-funding - - - Add to our database of OSS funding via CSV upload. - OSS Funders, Analysts - - - - Connect Your Data - - - oso - - - Write a plugin or help us replicate your data in the OSO data warehouse. - - Data Engineers, Developers - Propose an Impact Data Model diff --git a/apps/docs/docs/contribute-models/share-insights.md b/apps/docs/docs/contribute-models/share-insights.md index 35e99221a..04c0de043 100644 --- a/apps/docs/docs/contribute-models/share-insights.md +++ b/apps/docs/docs/contribute-models/share-insights.md @@ -31,7 +31,7 @@ Contribute your own data visualization templates by adding them to the `./visual ### Impact Metrcs -We're building scripts and analysis frameworks for community-generated impact metrics. Submissions must be replicable with data sourced entirely from the OSO data warehouse and should follow the follow the [Impact Metrics Specification](../guides/impact-metrics/). +We're building scripts and analysis frameworks for community-generated impact metrics. Submissions must be replicable with data sourced entirely from the OSO data warehouse and should follow the follow the [Impact Metrics Specification](../references/impact-metrics/). In most cases, impact vectors will be reviewed and recognized via one of our Data Challenges, with separate instructions for showcasing work and submitting pull requests. diff --git a/apps/docs/docs/get-started/api.mdx b/apps/docs/docs/get-started/api.mdx index ada4568be..976153085 100644 --- a/apps/docs/docs/get-started/api.mdx +++ b/apps/docs/docs/get-started/api.mdx @@ -9,7 +9,7 @@ import TabItem from '@theme/TabItem'; The [OSO](https://www.opensource.observer) API serves queries on metrics and metadata about open source projects. It is a [GraphQL](https://graphql.org/) API backed by the -[OSO data pipeline](../guides/architecture.md). +[OSO data pipeline](../references/architecture.md). Let's make your first query in under five minutes. diff --git a/apps/docs/docs/get-started/bigquery.mdx b/apps/docs/docs/get-started/bigquery.mdx index 5e99ac2f9..cf03306ec 100644 --- a/apps/docs/docs/get-started/bigquery.mdx +++ b/apps/docs/docs/get-started/bigquery.mdx @@ -70,7 +70,7 @@ Click on the following link to subscribe to the OSO production dataset: Create a linked dataset in your own GCP project. -![link dataset](../integrate/overview/bigquery_subscribe.png) +![link dataset](../integrate/datasets/bigquery_subscribe.png) ## Make your first query @@ -96,8 +96,8 @@ The results will appear in a table at the bottom of the console. The console will help you complete your query as you type, and will also provide you with a preview of the results and computation time. You can save your queries, download the results, and even make simple visualizations directly from the console. :::tip -To explore all the OSO datasets available, see the -[Data Overview](../integrate/overview/index.mdx). +To explore all the OSO datasets available, see the +[Data Overview](../integrate/datasets/index.mdx). - **oso_production** contains all production data. This can be quite large depending on the dataset. - **oso_playground** contains only the last 2 weeks for every dataset. We recommend using this for development and testing. diff --git a/apps/docs/docs/guides/ops/_category_.json b/apps/docs/docs/guides/ops/_category_.json index 418b93856..74464319e 100644 --- a/apps/docs/docs/guides/ops/_category_.json +++ b/apps/docs/docs/guides/ops/_category_.json @@ -1,5 +1,5 @@ { - "label": "🏗️ Operations Guides", + "label": "🏗️ Operations", "position": 8, "link": { "type": "doc", diff --git a/apps/docs/docs/guides/style-guide.md b/apps/docs/docs/guides/style-guide.md index 98bf24a39..04a3ad618 100644 --- a/apps/docs/docs/guides/style-guide.md +++ b/apps/docs/docs/guides/style-guide.md @@ -1,6 +1,6 @@ --- title: Code Style Guide -sidebar_position: 6 +sidebar_position: 1 --- ## dbt Models diff --git a/apps/docs/docs/integrate/3rd-party.mdx b/apps/docs/docs/integrate/3rd-party.mdx index 4884320dc..eeee617c8 100644 --- a/apps/docs/docs/integrate/3rd-party.mdx +++ b/apps/docs/docs/integrate/3rd-party.mdx @@ -12,7 +12,7 @@ OSO datasets and models are public and can be accessed on BigQuery. This allows First, we need to subscribe to an OSO dataset in your own Google Cloud account. You can see all of our available datasets in the -[Data Overview](./overview/index.mdx). +[Data Overview](./datasets/index.mdx). We recommend starting with the OSO production data pipeline here: diff --git a/apps/docs/docs/integrate/api.md b/apps/docs/docs/integrate/api.md index 710ee9724..40ef0a5a8 100644 --- a/apps/docs/docs/integrate/api.md +++ b/apps/docs/docs/integrate/api.md @@ -10,22 +10,6 @@ If you want access to the full dataset for data exploration, check out the guide [performing queries](./query-data.mdx) and [Python notebooks](./python-notebooks.md). -## How to Generate an API key - -First, go to [www.opensource.observer](https://www.opensource.observer) and create a new account. - -If you already have an account, log in. Then create a new personal API key: - -1. Go to [Account settings](https://www.opensource.observer/app/settings) -2. In the "API Keys" section, click "+ New" -3. Give your key a label - this is just for you, usually to describe a key's purpose. -4. You should see your brand new key. **Immediately** save this value, as you'll **never** see it again after refreshing the page. -5. Click "Create" to save the key. - -**You can create as many keys as you like.** - -![generate API key](./generate-api-key.png) - ## GraphQL Endpoint All API requests are sent to the following URL: @@ -34,25 +18,26 @@ All API requests are sent to the following URL: https://www.opensource.observer/api/v1/graphql ``` +### GraphQL Explorer + You can navigate to our [public GraphQL explorer](https://www.opensource.observer/graphql) to explore the schema and execute test queries. -## How to Authenticate +![GraphQL explorer](./api-explorer.gif) -In order to authenticate with the API service, you have to use the `Authorization` HTTP header and `Bearer` authentication on all HTTP requests, like so: +:::tip +As shown in the video, you must "Inline Variables" in order for queries to run in the explorer. +::: -```js -const headers = { - Authorization: `Bearer ${DEVELOPER_API_KEY}`, -}; -``` +The GraphQL schema is automatically generated from [`oso/dbt/models/marts`](https://github.com/opensource-observer/oso/tree/main/dbt/models/marts). Any dbt model defined there will automatically be exported to our GraphQL API. See the guide on [adding DBT models](../contribute-models/data-models.md) for more information on contributing to our marts models. -:::tip -Our API opens a small rate limit for anonymous queries without authentication. Feel free to use for low volume queries. +:::warning +Our data pipeline is under heavy development and all table schemas are subject to change until we introduce versioning to marts models. +Please join us on [Discord](https://www.opensource.observer/discord) to stay up to date on updates. ::: -## Example +### Example queries This query will fetch the first 10 projects in [oss-directory](https://github.com/opensource-observer/oss-directory). @@ -85,22 +70,40 @@ query GetCodeMetrics { } ``` -## GraphQL Explorer +## Authentication -You can navigate to our [public GraphQL explorer](https://www.opensource.observer/graphql) to explore the schema and execute test queries. +All requests to the API must be authenticated. +OSO is currently supported by generous grants that +allow us to provide this as a free service for the +community. +We will monitor usage of models, both to ensure quality of service +and to deprecate models that are no longer used. -![GraphQL explorer](./api-explorer.gif) +### Generate an API key -:::tip -As shown in the video, you must "Inline Variables" in order for queries to run in the explorer. -::: +First, go to [www.opensource.observer](https://www.opensource.observer) and create a new account. -The GraphQL schema is automatically generated from [`oso/dbt/models/marts`](https://github.com/opensource-observer/oso/tree/main/dbt/models/marts). Any dbt model defined there will automatically be exported to our GraphQL API. See the guide on [adding DBT models](../contribute-models/data-models.md) for more information on contributing to our marts models. +If you already have an account, log in. Then create a new personal API key: -:::warning -Our data pipeline is under heavy development and all table schemas are subject to change until we introduce versioning to marts models. -Please join us on [Discord](https://www.opensource.observer/discord) to stay up to date on updates. -::: +1. Go to [Account settings](https://www.opensource.observer/app/settings) +2. In the "API Keys" section, click "+ New" +3. Give your key a label - this is just for you, usually to describe a key's purpose. +4. You should see your brand new key. **Immediately** save this value, as you'll **never** see it again after refreshing the page. +5. Click "Create" to save the key. + +**You can create as many keys as you like.** + +![generate API key](./generate-api-key.png) + +### How to Authenticate + +In order to authenticate with the API service, you have to use the `Authorization` HTTP header and `Bearer` authentication on all HTTP requests, like so: + +```js +const headers = { + Authorization: `Bearer ${DEVELOPER_API_KEY}`, +}; +``` ## Rate Limits diff --git a/apps/docs/docs/integrate/overview/arbitrum.png b/apps/docs/docs/integrate/datasets/arbitrum.png similarity index 100% rename from apps/docs/docs/integrate/overview/arbitrum.png rename to apps/docs/docs/integrate/datasets/arbitrum.png diff --git a/apps/docs/docs/integrate/overview/bigquery_starred_datasets.png b/apps/docs/docs/integrate/datasets/bigquery_starred_datasets.png similarity index 100% rename from apps/docs/docs/integrate/overview/bigquery_starred_datasets.png rename to apps/docs/docs/integrate/datasets/bigquery_starred_datasets.png diff --git a/apps/docs/docs/integrate/overview/bigquery_subscribe.png b/apps/docs/docs/integrate/datasets/bigquery_subscribe.png similarity index 100% rename from apps/docs/docs/integrate/overview/bigquery_subscribe.png rename to apps/docs/docs/integrate/datasets/bigquery_subscribe.png diff --git a/apps/docs/docs/integrate/overview/eas.png b/apps/docs/docs/integrate/datasets/eas.png similarity index 100% rename from apps/docs/docs/integrate/overview/eas.png rename to apps/docs/docs/integrate/datasets/eas.png diff --git a/apps/docs/docs/integrate/overview/ethereum.png b/apps/docs/docs/integrate/datasets/ethereum.png similarity index 100% rename from apps/docs/docs/integrate/overview/ethereum.png rename to apps/docs/docs/integrate/datasets/ethereum.png diff --git a/apps/docs/docs/integrate/overview/farcaster.jpg b/apps/docs/docs/integrate/datasets/farcaster.jpg similarity index 100% rename from apps/docs/docs/integrate/overview/farcaster.jpg rename to apps/docs/docs/integrate/datasets/farcaster.jpg diff --git a/apps/docs/docs/integrate/overview/filecoin.png b/apps/docs/docs/integrate/datasets/filecoin.png similarity index 100% rename from apps/docs/docs/integrate/overview/filecoin.png rename to apps/docs/docs/integrate/datasets/filecoin.png diff --git a/apps/docs/docs/integrate/overview/gitcoin.png b/apps/docs/docs/integrate/datasets/gitcoin.png similarity index 100% rename from apps/docs/docs/integrate/overview/gitcoin.png rename to apps/docs/docs/integrate/datasets/gitcoin.png diff --git a/apps/docs/docs/integrate/overview/github.png b/apps/docs/docs/integrate/datasets/github.png similarity index 100% rename from apps/docs/docs/integrate/overview/github.png rename to apps/docs/docs/integrate/datasets/github.png diff --git a/apps/docs/docs/integrate/overview/index.mdx b/apps/docs/docs/integrate/datasets/index.mdx similarity index 98% rename from apps/docs/docs/integrate/overview/index.mdx rename to apps/docs/docs/integrate/datasets/index.mdx index 9cb2368bb..673e8c240 100644 --- a/apps/docs/docs/integrate/overview/index.mdx +++ b/apps/docs/docs/integrate/datasets/index.mdx @@ -1,5 +1,5 @@ --- -title: Data Overview +title: Access Public Datasets sidebar_position: 1 --- @@ -18,9 +18,9 @@ import ArbitrumLogo from "./arbitrum.png"; import EasLogo from "./eas.png"; import OcLogo from "./open-collective.png"; -First, you need to set up your BigQuery account. You can do this by going to the -[Get Started](../../get-started/index.md) -page. +OSO shares every dataset on BigQuery Analytics Hub. +To get started, +[set up your BigQuery account](../../get-started/bigquery.mdx). ## OSO Data Exchange on Analytics Hub @@ -87,7 +87,7 @@ in a live production application._ From source data, we produce a "universal event table", currently stored at [`int_events`](https://models.opensource.observer/#!/model/model.opensource_observer.int_events). -Each event consists of an [event_type](../../guides/event.md) +Each event consists of an [event_type](../../references/event.md) (e.g. a git commit or contract invocation), [to/from artifacts](../../guides/oss-directory/artifact.md), a timestamp, and an amount. diff --git a/apps/docs/docs/integrate/overview/lens-protocol.png b/apps/docs/docs/integrate/datasets/lens-protocol.png similarity index 100% rename from apps/docs/docs/integrate/overview/lens-protocol.png rename to apps/docs/docs/integrate/datasets/lens-protocol.png diff --git a/apps/docs/docs/integrate/overview/op.png b/apps/docs/docs/integrate/datasets/op.png similarity index 100% rename from apps/docs/docs/integrate/overview/op.png rename to apps/docs/docs/integrate/datasets/op.png diff --git a/apps/docs/docs/integrate/overview/open-collective.png b/apps/docs/docs/integrate/datasets/open-collective.png similarity index 100% rename from apps/docs/docs/integrate/overview/open-collective.png rename to apps/docs/docs/integrate/datasets/open-collective.png diff --git a/apps/docs/docs/integrate/overview/open-source-insights.png b/apps/docs/docs/integrate/datasets/open-source-insights.png similarity index 100% rename from apps/docs/docs/integrate/overview/open-source-insights.png rename to apps/docs/docs/integrate/datasets/open-source-insights.png diff --git a/apps/docs/docs/integrate/overview/openrank.png b/apps/docs/docs/integrate/datasets/openrank.png similarity index 100% rename from apps/docs/docs/integrate/overview/openrank.png rename to apps/docs/docs/integrate/datasets/openrank.png diff --git a/apps/docs/docs/integrate/overview/oso-primary.png b/apps/docs/docs/integrate/datasets/oso-primary.png similarity index 100% rename from apps/docs/docs/integrate/overview/oso-primary.png rename to apps/docs/docs/integrate/datasets/oso-primary.png diff --git a/apps/docs/docs/integrate/index.md b/apps/docs/docs/integrate/index.md index be765fb14..6122973b6 100644 --- a/apps/docs/docs/integrate/index.md +++ b/apps/docs/docs/integrate/index.md @@ -6,9 +6,8 @@ sidebar_position: 0 Open Source Observer is a fully open data pipeline for measuring the impact of open source efforts. That means all source code, data, and infrastructure is publicly available for use. -- [**Get Started**](../get-started/index.md): to setup your Google account for data access and run your first query -- [**Data Overview**](./overview/index.mdx): for an overview of all data available -- [**SQL Query Guide**](./query-data.mdx): to quickly query and download any data +- [**Access Public Datasets**](./datasets/index.mdx): for an overview of all data available +- [**BigQuery Studio Guide**](./query-data.mdx): to quickly query and download any data - [**Python notebooks**](./python-notebooks.md): to do more in-depth data science and processing - [**Connect OSO to 3rd Party tools**](./3rd-party.mdx): like Hex.tech, Tableau, and Metabase - [**Fork the data pipeline**](./fork-pipeline.md): to setup your own data pipeline off any OSO model diff --git a/apps/docs/docs/integrate/python-notebooks.md b/apps/docs/docs/integrate/python-notebooks.md index 95eb93d0e..290a36e6a 100644 --- a/apps/docs/docs/integrate/python-notebooks.md +++ b/apps/docs/docs/integrate/python-notebooks.md @@ -496,7 +496,7 @@ An **impact metric** is essentially a SQL query made against the OSO dataset tha There are a variety of statistical techniques for analyzing data about impact metrics and identifying trends. This section provides a basic example of how to create an impact metric and run a distribution analysis. :::tip -The complete specification for an impact metric is available [here](../guides/impact-metrics/index.mdx). +The complete specification for an impact metric is available [here](../references/impact-metrics/index.mdx). ::: ### General guide for creating an impact metric @@ -508,7 +508,7 @@ The complete specification for an impact metric is available [here](../guides/im #### 2. Define the Metric and Selection Criteria -- **Metric**: Get inspiration from some of our [impact metrics](../guides/impact-metrics/index.mdx) or [propose a new metric](../contribute-models/data-models.md). Examples: "Number of Full-Time Developer Months", "Number of Dependent Onchain Apps", "Layer 2 Gas Fees", "Number of New Contributors". +- **Metric**: Get inspiration from some of our [impact metrics](../references/impact-metrics/index.mdx) or [propose a new metric](../contribute-models/data-models.md). Examples: "Number of Full-Time Developer Months", "Number of Dependent Onchain Apps", "Layer 2 Gas Fees", "Number of New Contributors". - **Time Period**: Specify a time interval for applying the metric. Examples: "Last 6 months", "Since the project's inception". - **Selection Filter**: Make explicit the criteria to identify which projects are eligible (or ineligible) to be included in the analysis. Examples: "Projects with developer activity in the last 90 days", "Projects with NPM packages used by at least 5 onchain projects", "Projects with a permissive open source license (e.g., MIT, Apache 2.0) and a codebase that is at least 6 months old". diff --git a/apps/docs/docs/integrate/query-data.mdx b/apps/docs/docs/integrate/query-data.mdx index cbd67f520..d2de4603e 100644 --- a/apps/docs/docs/integrate/query-data.mdx +++ b/apps/docs/docs/integrate/query-data.mdx @@ -1,5 +1,5 @@ --- -title: Run SQL Queries +title: Query BigQuery sidebar_position: 3 --- @@ -19,7 +19,7 @@ guide to first get your BigQuery account setup. First, we need to subscribe to an OSO dataset in your own Google Cloud account. You can see all of our available datasets in the -[Data Overview](./overview/index.mdx). +[Data Overview](./datasets/index.mdx). We recommend starting with the OSO production data pipeline here: diff --git a/apps/docs/docs/guides/architecture.md b/apps/docs/docs/references/architecture.md similarity index 98% rename from apps/docs/docs/guides/architecture.md rename to apps/docs/docs/references/architecture.md index 23b0f9f69..6d460eabc 100644 --- a/apps/docs/docs/guides/architecture.md +++ b/apps/docs/docs/references/architecture.md @@ -1,6 +1,6 @@ --- title: Technical Architecture -sidebar_position: 5 +sidebar_position: 2 --- :::info @@ -41,7 +41,7 @@ Anyone with can view, query, or build off of any stage in the pipeline. In the future we plan to explore a decentralized lakehouse. To see all datasets that you can subscribe to, check out our -[Data Overview](../integrate/overview/index.mdx). +[Data Overview](../integrate/datasets/index.mdx). ### dbt pipeline diff --git a/apps/docs/docs/guides/ecosystems/_category_.json b/apps/docs/docs/references/ecosystems/_category_.json similarity index 81% rename from apps/docs/docs/guides/ecosystems/_category_.json rename to apps/docs/docs/references/ecosystems/_category_.json index 22333561f..2af565977 100644 --- a/apps/docs/docs/guides/ecosystems/_category_.json +++ b/apps/docs/docs/references/ecosystems/_category_.json @@ -1,6 +1,6 @@ { "label": "Ecosystems", - "position": 7, + "position": 5, "link": { "type": "generated-index" } diff --git a/apps/docs/docs/guides/ecosystems/arbitrum.md b/apps/docs/docs/references/ecosystems/arbitrum.md similarity index 100% rename from apps/docs/docs/guides/ecosystems/arbitrum.md rename to apps/docs/docs/references/ecosystems/arbitrum.md diff --git a/apps/docs/docs/guides/ecosystems/pln.md b/apps/docs/docs/references/ecosystems/pln.md similarity index 100% rename from apps/docs/docs/guides/ecosystems/pln.md rename to apps/docs/docs/references/ecosystems/pln.md diff --git a/apps/docs/docs/guides/ecosystems/superchain.md b/apps/docs/docs/references/ecosystems/superchain.md similarity index 100% rename from apps/docs/docs/guides/ecosystems/superchain.md rename to apps/docs/docs/references/ecosystems/superchain.md diff --git a/apps/docs/docs/guides/event.md b/apps/docs/docs/references/event.md similarity index 99% rename from apps/docs/docs/guides/event.md rename to apps/docs/docs/references/event.md index 8617d316e..2dd5fec13 100644 --- a/apps/docs/docs/guides/event.md +++ b/apps/docs/docs/references/event.md @@ -1,5 +1,5 @@ --- -title: Event Data Indexers +title: Event Types sidebar_position: 3 --- diff --git a/apps/docs/docs/references/faq.md b/apps/docs/docs/references/faq.md index c68de058b..25f23d3b2 100644 --- a/apps/docs/docs/references/faq.md +++ b/apps/docs/docs/references/faq.md @@ -1,6 +1,6 @@ --- title: FAQ -sidebar_position: 5 +sidebar_position: 7 --- ## Community @@ -9,7 +9,7 @@ sidebar_position: 5 Open Source Observer is a public good. All of our code is [open source](https://github.com/opensource-observer/oso/). -All of our data sets and models are [open data](../integrate/overview/index.mdx). +All of our data sets and models are [open data](../integrate/datasets/index.mdx). All of our infrastructure is [open](https://dagster.opensource.observer/locations) too. If we are going to transform funding for public goods, we need to live by an open source ethos. diff --git a/apps/docs/docs/guides/impact-metrics/_category_.json b/apps/docs/docs/references/impact-metrics/_category_.json similarity index 83% rename from apps/docs/docs/guides/impact-metrics/_category_.json rename to apps/docs/docs/references/impact-metrics/_category_.json index 7a884e50f..3f6264be0 100644 --- a/apps/docs/docs/guides/impact-metrics/_category_.json +++ b/apps/docs/docs/references/impact-metrics/_category_.json @@ -1,6 +1,6 @@ { "label": "Impact Metrics", - "position": 3, + "position": 4, "link": { "type": "doc", "id": "index" diff --git a/apps/docs/docs/guides/impact-metrics/code.md b/apps/docs/docs/references/impact-metrics/code.md similarity index 100% rename from apps/docs/docs/guides/impact-metrics/code.md rename to apps/docs/docs/references/impact-metrics/code.md diff --git a/apps/docs/docs/guides/impact-metrics/dependencies.md b/apps/docs/docs/references/impact-metrics/dependencies.md similarity index 100% rename from apps/docs/docs/guides/impact-metrics/dependencies.md rename to apps/docs/docs/references/impact-metrics/dependencies.md diff --git a/apps/docs/docs/guides/impact-metrics/index.mdx b/apps/docs/docs/references/impact-metrics/index.mdx similarity index 100% rename from apps/docs/docs/guides/impact-metrics/index.mdx rename to apps/docs/docs/references/impact-metrics/index.mdx diff --git a/apps/docs/docs/guides/impact-metrics/onchain.md b/apps/docs/docs/references/impact-metrics/onchain.md similarity index 97% rename from apps/docs/docs/guides/impact-metrics/onchain.md rename to apps/docs/docs/references/impact-metrics/onchain.md index 9ac2da657..08157590c 100644 --- a/apps/docs/docs/guides/impact-metrics/onchain.md +++ b/apps/docs/docs/references/impact-metrics/onchain.md @@ -12,7 +12,7 @@ Onchain metrics include projects' smart contracts, transaction volumes, and sequ --- -The latest version of our code metrics model can be viewed [here](https://models.opensource.observer/#!/model/model.opensource_observer.onchain_metrics_by_project_v1#description). For information about how addresses are tagged, please see: [Address Tagging](../oss-directory/artifact.md#tagging-addresses). +The latest version of our code metrics model can be viewed [here](https://models.opensource.observer/#!/model/model.opensource_observer.onchain_metrics_by_project_v1#description). For information about how addresses are tagged, please see: [Address Tagging](../../guides/oss-directory/artifact.md#tagging-addresses). ### days_since_first_transaction diff --git a/apps/docs/docs/guides/impact-metrics/package.md b/apps/docs/docs/references/impact-metrics/package.md similarity index 100% rename from apps/docs/docs/guides/impact-metrics/package.md rename to apps/docs/docs/references/impact-metrics/package.md diff --git a/apps/docs/docs/contribute-models/join-team.mdx b/apps/docs/docs/references/join-team.mdx similarity index 100% rename from apps/docs/docs/contribute-models/join-team.mdx rename to apps/docs/docs/references/join-team.mdx diff --git a/apps/docs/docs/guides/mission.md b/apps/docs/docs/references/mission.md similarity index 100% rename from apps/docs/docs/guides/mission.md rename to apps/docs/docs/references/mission.md diff --git a/apps/docs/docs/references/talks.md b/apps/docs/docs/references/talks.md index c93156c2f..deb3f26a1 100644 --- a/apps/docs/docs/references/talks.md +++ b/apps/docs/docs/references/talks.md @@ -1,6 +1,6 @@ --- title: Talks -sidebar_position: 4 +sidebar_position: 6 --- diff --git a/apps/docs/docs/tutorials/collection-view.md b/apps/docs/docs/tutorials/collection-view.md index 763ce0d7b..4959ceec2 100644 --- a/apps/docs/docs/tutorials/collection-view.md +++ b/apps/docs/docs/tutorials/collection-view.md @@ -11,7 +11,7 @@ All **collections** are defined as YAML files in [OSS Directory](https://github. ## BigQuery -If you haven't already, then the first step is to subscribe to OSO public datasets in BigQuery. You can do this by clicking the "Subscribe" button on our [Datasets page](../integrate/overview/#oso-production-data-pipeline). +If you haven't already, then the first step is to subscribe to OSO public datasets in BigQuery. You can do this by clicking the "Subscribe" button on our [Datasets page](../integrate/datasets/#oso-production-data-pipeline). The following queries should work if you copy-paste them into your [BigQuery console](https://console.cloud.google.com/bigquery). diff --git a/apps/docs/docs/tutorials/network-graph.md b/apps/docs/docs/tutorials/network-graph.md index e9b0dfe92..fe48413f4 100644 --- a/apps/docs/docs/tutorials/network-graph.md +++ b/apps/docs/docs/tutorials/network-graph.md @@ -13,7 +13,7 @@ Here's a visualization of the final result: ## BigQuery -If you haven't already, then the first step is to subscribe to OSO public datasets in BigQuery. You can do this by clicking the "Subscribe" button on our [Datasets page](../integrate/overview/#oso-production-data-pipeline). +If you haven't already, then the first step is to subscribe to OSO public datasets in BigQuery. You can do this by clicking the "Subscribe" button on our [Datasets page](../integrate/datasets/#oso-production-data-pipeline). The following queries should work if you copy-paste them into your [BigQuery console](https://console.cloud.google.com/bigquery). diff --git a/apps/docs/docs/tutorials/project-deepdive.md b/apps/docs/docs/tutorials/project-deepdive.md index b4d95bcae..43abba87f 100644 --- a/apps/docs/docs/tutorials/project-deepdive.md +++ b/apps/docs/docs/tutorials/project-deepdive.md @@ -11,7 +11,7 @@ All **projects** are defined as YAML files in [OSS Directory](https://github.com ## BigQuery -If you haven't already, then the first step is to subscribe to OSO public datasets in BigQuery. You can do this by clicking the "Subscribe" button on our [Datasets page](../integrate/overview/#oso-production-data-pipeline). +If you haven't already, then the first step is to subscribe to OSO public datasets in BigQuery. You can do this by clicking the "Subscribe" button on our [Datasets page](../integrate/datasets#oso-production-data-pipeline). The following queries should work if you copy-paste them into your [BigQuery console](https://console.cloud.google.com/bigquery). diff --git a/warehouse/dbt/models/homepage.md b/warehouse/dbt/models/homepage.md index 254e8fb84..c4dbfb7dc 100644 --- a/warehouse/dbt/models/homepage.md +++ b/warehouse/dbt/models/homepage.md @@ -23,7 +23,7 @@ Note that you can also right-click on models to interactively filter and explore - [Website](https://www.opensource.observer/) - [Documentation](https://docs.opensource.observer/) -- [Technical architecture](https://docs.opensource.observer/docs/guides/architecture) +- [Technical architecture](https://docs.opensource.observer/docs/references/architecture) - [GitHub repo](https://github.com/opensource-observer/oso) - [Dagster dashboard](https://dagster.opensource.observer/)