Initial commit

.env.test

@@ -0,0 +1,4 @@
+#copy and paste this file into a .env file to run scripts locally. You will also need to add the DUNE_API_KEY into the repo settings under "Secrets and Variables"
+
+#add a dune API key - you can create one under team settings (https://dune.com/settings/teams/manage/{team_name}/api). You must be on the premium plan.
+DUNE_API_KEY=

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,17 @@
+---
+name: Bug Report
+about: Tell us about the data bug or broken query you are seeing
+title: "[BUG] Query data quality issue"
+labels: 'bug'
+assignees: ''
+
+---
+
+**What is the query id, and describe the bug**
+A clear and concise description of what the bug is, with query id linked.
+
+**Examples of the bug**
+Please provide links to etherscan or your own queries showing the errors/discrepencies.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.

.github/ISSUE_TEMPLATE/chart-improvment.md

@@ -0,0 +1,17 @@
+---
+name: Chart Improvment
+about: Suggest an idea for improving a chart
+title: "[CHART] Suggestion for improving a chart"
+labels: 'enhancements'
+assignees: ''
+
+---
+
+**Which query would you like to improve, and why?"**
+A clear and concise description of what the problem is, with query id (if this is an improvement to an existing chart)
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Any work or examples you've seen that are similar to the solution**
+Links and screenshots to any guiding charts/tables/queries you've seen.

.github/ISSUE_TEMPLATE/generic-issue.md

@@ -0,0 +1,11 @@
+---
+name: Generic Question
+about: Tell us about your question
+title: "[GENERIC] Question"
+labels: 'question'
+assignees: ''
+
+---
+
+**What is your question?**
+I don't quite understand how/why this query does XYZ...

.github/ISSUE_TEMPLATE/query-improvement.md

@@ -0,0 +1,17 @@
+---
+name: Query Improvement
+about: Suggest an idea for improving a query
+title: "[QUERY] Suggestion for improving a chart"
+labels: 'enhancement'
+assignees: ''
+
+---
+
+**Which query would you like to improve, and why?"**
+A clear and concise description of what the problem is, with query id (if this is an improvement to an existing chart)
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen. Please suggest data sources/tables that this enhancement could come from.
+
+**Any work or examples you've seen that are similar to the solution**
+Links and screenshots to any guiding charts/tables/queries you've seen.

.github/workflows/push_to_dune.yml

@@ -0,0 +1,32 @@
+name: Push to Dune on Commit
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - 'queries/**'
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v2
+
+      - uses: actions/setup-python@v4
+        with:
+          python-version: '3.9'
+
+      - name: Log directory structure
+        run: |
+          pwd
+          ls -R
+    
+      - name: pip requirements
+        run: pip install -r requirements.txt
+
+      - name: Update all queries from Dune, by overwriting queries with repo query text
+        env:
+          DUNE_API_KEY: ${{ secrets.DUNE_API_KEY }}
+        run: python -u scripts/push_to_dune.py

.github/workflows/upload_to_dune.yml

@@ -0,0 +1,32 @@
+name: Upload CSVs to Dune on Commit
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - 'uploads/**'
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v2
+
+      - uses: actions/setup-python@v4
+        with:
+          python-version: '3.9'
+
+      - name: Log directory structure
+        run: |
+          pwd
+          ls -R
+    
+      - name: pip requirements
+        run: pip install -r requirements.txt
+
+      - name: Update all queries from Dune, by overwriting queries with repo query text
+        env:
+          DUNE_API_KEY: ${{ secrets.DUNE_API_KEY }}
+        run: python -u scripts/upload_to_dune.py

.gitignore

@@ -0,0 +1,2 @@
+
+.env

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 Dune
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,19 @@
+**Is this linked to an existing issue**
+If so, link that issue(s) here.
+
+**Fill out the following table describing your edits:**
+
+| Original | Updated | Change | Reasoning |
+|---|---|---|---|
+| [3237745](https://dune.com/queries/3237745) | [3237938](https://dune.com/queries/3238935) | Remove sandwich traders using dex.sandwiches | We should only care about traders who are not doing MEV |
+| `evm_grants.csv` | n/a | added grants for Optimism | captures approved grants only |
+
+**Provide any other context or screenshots that explain or justify the changes above:**
+
+---
+
+**Note to Contributor:**
+
+Make sure your PR edits the original `query_id.sql` file with the new query text. If you are proposing adding a new query completely, make sure to add it to `queries.yml` and as a file in `/queries` as well.
+
+Thanks for contributing! 🙏

README.md

@@ -0,0 +1,60 @@
+# Dune Query Repo
+
+A template for creating repos to [manage your Dune queries](https://dune.mintlify.app/api-reference/crud/endpoint/create) and any [CSVs as Dune tables](https://dune.mintlify.app/api-reference/upload/endpoint/upload).
+
+### Setup Your Repo
+
+1. Generate an API key from your Dune account and put that in both your `.env` file and [github action secrets](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions#creating-secrets-for-a-repository) (name it `DUNE_API_KEY`). You can create a key under your Dune team settings. *The api key must be from a plus plan for this repo to work.*
+
+2. Type your intended query ids into the `queries.yml` file. The id can be found from the link `https://dune.com/queries/<query_id>/...`. If you're creating this for a dashboard, go to the dashboard you want to create a repo and click on the "github" button in the top right of your dashboard to see the query ids.
+
+3. Then, run `pull_from_dune.py` to bring in all queries into `/query_{id}.sql` files within the `/queries` folder. Directions to setup and run this python script are below.
+
+### Updating Queries or CSV Tables
+
+1. Make any changes you need to directly in the repo. Any time you push a commit to MAIN branch, `push_to_dune.py` will save your changes into Dune directly. You can run this manually too if you want.
+
+2. For CSVs, update the files in the `/uploads` folder. `upload_to_dune.py` will run on commit, or can be run manually. The table name in Dune will be `dune.team_name.dataset_<filename>`.
+
+---
+
+### Query Management Scripts
+
+You'll need python and pip installed to run the script commands. If you don't have a package manager set up, then use either [conda](https://www.anaconda.com/download) or [poetry](https://python-poetry.org/) . Then install the required packages:
+
+```
+pip install -r requirements.txt
+```
+
+| Script | Action                                                                                                                                                    | Command |
+|---|-----------------------------------------------------------------------------------------------------------------------------------------------------------|---|
+| `pull_from_dune.py` | updates/adds queries to your repo based on ids in `queries.yml`                                                                                           | `python scripts/pull_from_dune.py` |
+| `push_to_dune.py` | updates queries to Dune based on files in your `/queries` folder                                                                                          | `python scripts/push_to_dune.py` |
+| `preview_query.py` | gives you the first 20 rows of results by running a query from your `/queries` folder. Specify the id. This uses Dune API credits | `python scripts/preview_query.py 2615782` |
+| `upload_to_dune.py` | uploads/updates any tables from your `/uploads` folder. Must be in CSV format, and under 200MB. | `python scripts/upload_to_dune.py` |
+
+---
+
+### Things to be aware of
+
+💡: Names of queries are pulled into the file name the first time `pull_from_dune.py` is run. Changing the file name in app or in folder will not affect each other (they aren't synced). **Make sure you leave the `___id.sql` at the end of the file, otherwise the scripts will break!**
+
+🟧: Make sure to leave in the comment `-- already part of a query repo` at the top of your file. This will hopefully help prevent others from using it in more than one repo.
+
+🔒: Queries must be owned by the team the API key was created under - otherwise you won't be able to update them from the repo.
+
+➕: If you want to add a query, add it in Dune app first then pull the query id (from URL `dune.com/queries/{id}/other_stuff`) into `queries.yml`
+
+🛑: If you accidently merge a PR or push a commit that messes up your query in Dune, you can roll back any changes using [query version history](https://dune.com/docs/app/query-editor/version-history).
+
+---
+
+### For Contributors
+
+I've set up four types of issues right now:
+- `bugs`: This is for data quality issues like miscalculations or broken queries.
+- `chart improvements`: This is for suggesting improvements to the visualizations.
+- `query improvements`: This is for suggesting improvements to the query itself, such as adding an extra column or table that enhances the results.
+- `generic questions`: This is a catch all for other questions or suggestions you may have about the dashboard.
+
+If you want to contribute, either start an issue or go directly into making a PR (using the same labels as above). Once the PR is merged, the queries will get updated in the frontend.

queries.yml

@@ -0,0 +1,2 @@
+query_ids:
+  - 123456

queries/filler.txt

@@ -0,0 +1 @@
+this file is just here so the folder can exist. Feel free to delete this after you've pulled your queries in.

requirements.txt

@@ -0,0 +1,4 @@
+dune-client
+pyyaml
+python-dotenv
+pandas

scripts/preview_query.py

@@ -0,0 +1,39 @@
+import os
+from dune_client.client import DuneClient
+from dotenv import load_dotenv
+import sys
+import pandas as pd
+
+dotenv_path = os.path.join(os.path.dirname(__file__), '..', '.env')
+load_dotenv(dotenv_path)
+
+dune = DuneClient.from_env()
+
+#get id passed in python script invoke
+id = sys.argv[1]
+
+queries_path = os.path.join(os.path.dirname(__file__), '..', 'queries')
+files = os.listdir(queries_path)
+found_files = [file for file in files if str(id) == file.split('___')[-1].split('.')[0]]
+
+if len(found_files) != 0:
+    query_file = os.path.join(os.path.dirname(__file__), '..', 'queries', found_files[0])
+
+    print('getting 20 line preview for query {}...'.format(id))
+
+    with open(query_file, 'r', encoding='utf-8') as file:
+        query_text = file.read()
+
+    print('select * from (\n' + query_text + '\n) limit 20')
+
+    results = dune.run_sql('select * from (\n' + query_text + '\n) limit 20')
+    # print(results.result.rows)
+    results = pd.DataFrame(data=results.result.rows)
+    print('\n')
+    print(results)
+    print('\n')
+    print(results.describe())
+    print('\n')
+    print(results.info())
+else:
+    print('query id file not found, try again')

scripts/pull_from_dune.py

@@ -0,0 +1,60 @@
+import os
+import yaml
+from dune_client.client import DuneClient
+from dotenv import load_dotenv
+import sys
+import codecs
+
+# Set the default encoding to UTF-8
+sys.stdout = codecs.getwriter("utf-8")(sys.stdout.detach())
+
+dotenv_path = os.path.join(os.path.dirname(__file__), '..', '.env')
+load_dotenv(dotenv_path)
+
+dune = DuneClient.from_env()
+
+# Read the queries.yml file
+queries_yml = os.path.join(os.path.dirname(__file__), '..', 'queries.yml')
+with open(queries_yml, 'r', encoding='utf-8') as file:
+    data = yaml.safe_load(file)
+
+# Extract the query_ids from the data
+query_ids = [id for id in data['query_ids']]
+
+for id in query_ids:
+    query = dune.get_query(id)
+    print('PROCESSING: query {}, {}'.format(query.base.query_id, query.base.name))
+
+    # Check if query file exists in /queries folder
+    queries_path = os.path.join(os.path.dirname(__file__), '..', 'queries')
+    files = os.listdir(queries_path)
+    found_files = [file for file in files if str(id) == file.split('___')[-1].split('.')[0]]
+
+    if len(found_files) != 0:
+        # Update existing file
+        file_path = os.path.join(os.path.dirname(__file__), '..', 'queries', found_files[0])
+
+        print('UPDATE: existing query file: {}'.format(found_files[0]))
+        with open(file_path, 'r+', encoding='utf-8') as file:
+            #if "query repo:" is in the file, then don't add the text header again
+            if '-- part of a query repo' in query.sql:
+                file.write(query.sql)
+            else:
+                file.write(f'-- part of a query repo\n-- query name: {query.base.name}\n-- query link: https://dune.com/queries/{query.base.query_id}\n\n\n{query.sql}')
+    else:
+        # Create new file and directories if they don't exist
+        new_file = f'{query.base.name.replace(" ", "_").lower()[:30]}___{query.base.query_id}.sql'
+        file_path = os.path.join(os.path.dirname(__file__), '..', 'queries', new_file)
+        os.makedirs(os.path.dirname(file_path), exist_ok=True)
+
+        if '-- part of a query repo' in query.sql:
+            print('WARNING!!! This query is part of a query repo')
+            with open(file_path, 'w', encoding='utf-8') as file:
+                file.write(f'-- WARNING: this query may be part of multiple repos\n{query.sql}')
+        else:
+            with open(file_path, 'w', encoding='utf-8') as file:
+                file.write(f'-- part of a query repo\n-- query name: {query.base.name}\n-- query link: https://dune.com/queries/{query.base.query_id}\n\n\n{query.sql}')
+        print('CREATE: new query file: {}'.format(new_file))
+
+
+