Create ETL that joins crashes and Moped components #1596
+242
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Charlie-Henry
requested review from
johnclary,
chiaberry,
roseeichelmann,
frankhereford and
mddilley
johnclary
changed the title
johnclary
approved these changes
Nov 7, 2024
4 tasks
chiaberry
approved these changes
Nov 13, 2024
mddilley
approved these changes
Nov 13, 2024
|
|
||
| It is recommended to run this script using the docker container. You can build it using: | ||
|
|
||
| Note, if you are on Apple Silicon you may need to add `--platform linux/amd64` to get GDAL to install correctly. |
There was a problem hiding this comment.
🙏 thanks for this detail because this was the case for me
| query_moped = """ | ||
| { | ||
| component_arcgis_online_view(where: { geometry: { _is_null: false } }) { | ||
| project_component_id, |
There was a problem hiding this comment.
Commas not creating a syntax error in GraphQL is a TIL for me. Cool to know! https://spec.graphql.org/June2018/#sec-Insignificant-Commas
| components.set_index("project_component_id", inplace=True) | ||
| crashes.set_index("id", inplace=True) | ||
| logger.info(f"Joining crashes spatially to project geometry") | ||
| crashes_near_projects = gpd.sjoin(crashes, components, how="inner") |
There was a problem hiding this comment.
nice! this is a pretty powerful line, and a TIL for me. 😎
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Associated issues
cityofaustin/atd-data-tech#19453
Testing
Steps to test:
First fill out the env_template for Hasura, from 1pass. Make sure you use Moped production and VZ staging credentials
Note, if you are on Apple Silicon you may need to add
--platform linux/amd64to get GDAL to install correctly.The logging should be pretty self explanatory to see if things succeeded or not.
You can also then check if
moped_component_crasheswas populated with the lookup table. (maybe delete everything in that table yourself to double check if the script did anything).Ship list
mainbranch