Skip to content

inigohidalgo/prefect-polygon-etl

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

71 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ETL from polygon.io using prefect

Setup

Start prefect server

https://github.com/inigohidalgo/prefect-server-docker

Deploy flows

python src/polygon_etl/flows/deploy_all.py

Set up credentials

Polygon

Sign up for an account at polygon.io and set up an API key.

Run the flow and save a secret with name polygon-api-key and the value of your API key from polygon.io

prefect deployment run save-prefect-secret/save-prefect-secret --param secret_name=polygon-api-key --param secret_value=$POLYGON_API_KEY

Minio

Get your minio credentials and run the flow to save them as secrets.

prefect deployment run save-minio-credentials/save-minio-credentials --param minio_access_key=$MINIO_ACCESS_KEY --param minio_secret_key=$MINIO_SECRET_KEY --param host=http://localhost:9000

Set up deployments

python src/flows/deploy_all.py

Save the following block:

JSON / polygon-daily-aggregates-tickers - ["AAPL", "MSFT"...]

Run the flow:

prefect deployment run get-aggregates-from-preconfigured-list-of-tickers-raw-to-bronze/daily-aggregates-raw-to-bronze-multiple-tickers

This will load the data from the list of tickers into the bronze bucket.

About

ELT for Polygon.io financial data using Prefect

Topics

Resources

Stars

Watchers

Forks