Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add workflow to gen data #362

Open
wants to merge 299 commits into
base: feat/pull-v2-api
Choose a base branch
from
Open

Add workflow to gen data #362

wants to merge 299 commits into from

Conversation

ckingbailey
Copy link
Collaborator

replaying PR #329

ChenglimEar and others added 30 commits November 1, 2023 08:50
@ChenglimEar
separate out step to check postgres so that we can easily check the
ac64ba0 
output
@ChenglimEar
remove unused script
f08c7c1
@ChenglimEar
remove new version of sqlalchemy since it breaks csvsql in make import
8b55cd9
@ChenglimEar
add some conditions for job execution in main workflow
d239c1f
@ChenglimEar
add check for version of sqlalchemy
9a64554
@ChenglimEar
get container to rebuild when requirements change
1ee8282
@ChenglimEar
adjust logic for when jobs run in main workflow
6ca18db
@ChenglimEar
try summarizing tables created in main workflow
aacdec8
@ChenglimEar
make sure we clean up downloads before we start
3e11a12
@ChenglimEar
Merge branch 'master' into add-workflow-to-gen-data
b883c21
@ChenglimEar
try testing csvsql early
565aede
@ChenglimEar
some logging to test csvsql
a80b6e7
@ChenglimEar
turn on verbose output while testing csvsql
c9988aa
@ChenglimEar
try sending csv file through stdin for csvsql
0dd3c03
@ChenglimEar
upgrade csvkit to 1.3.0 and upgraded its dependencies where needed
9678551
@ChenglimEar
remove files no longer needed in download dir
a1812ff
@ChenglimEar
set postgres version in dev container and workflow to 9.6 to match
062d4b1 
travic CI
@ChenglimEar
Merge branches 'feat/pull-v2-api' and 'feat/pull-v2-api' of github.co…
82408d9 
…m:caciviclab/disclosure-backend-static into feat/pull-v2-api
@ChenglimEar
Merge branch 'feat/pull-v2-api' into add-workflow-to-gen-data
cf785f2
@ChenglimEar
update workflow names
db8c504
@ChenglimEar
make use of sql files to create tables
c176028
@ChenglimEar
Update import-file to display schema of created table
48f6259
@ChenglimEar
Update import-file to log more info about postgresql tables
ad869ef
@ChenglimEar
Update import-file to point psql to DATABASE_NAME
dde12b4
@ChenglimEar
Update import-file to use the right quote around table name in psql
c33d743
@ChenglimEar
Update import-file to remove debug logging
f9f2654
@ChenglimEar
Update Makefile to use saved sql for creating tables from spreadsheet…
23ff55b 
… data
@ChenglimEar
fix Makefile by moving bash into file and saved generated sql for tab…
84c3a26 
…les that hold spreadsheet data
@ChenglimEar
some fixes to get csvkit 1.3.0 working - not fully working yet...
a51ee6c
@ChenglimEar
Merge branch 'upgrade-csvkit' of github.com:caciviclab/disclosure-bac…
bb4354b 
…kend-static into upgrade-csvkit
OpenDisclosure Deploybot and others added 29 commits July 2, 2024 22:18
Run make clean download import process
7986727 
This is an automated update by travis-ci at
Tue Jul  2 22:18:22 PDT 2024

[skip ci]
@ChenglimEar
provide means to switch to ruby 2.7.1 if needed
5d7c25a
@ChenglimEar
Merge branch 'add-workflow-to-gen-data' of github.com:caciviclab/disc…
407b274 
…losure-backend-static into add-workflow-to-gen-data
Run make clean download import process
2760215 
This is an automated update by travis-ci at
Sun Jul 14 13:22:34 PDT 2024

[skip ci]
@ChenglimEar
remove whitespace from data_warning column
e83eaa5
@ChenglimEar
Merge branch 'add-workflow-to-gen-data' of github.com:caciviclab/disc…
25c580c 
…losure-backend-static into add-workflow-to-gen-data
Run make clean download import process
f10a7a6 
This is an automated update by travis-ci at
Mon Jul 15 20:20:04 PDT 2024

[skip ci]
@ChenglimEar
make data_warning empty instead of null
a0a322f
@ChenglimEar
make make-null-empty executable
ca3d2a0
Run make clean download import process
fd68dd5 
This is an automated update by travis-ci at
Mon Jul 15 21:07:07 PDT 2024

[skip ci]
@ChenglimEar
maintain a consistent order for the candidates report
9d0cb41
Run make clean download import process
bcaafa8 
This is an automated update by travis-ci at
Mon Jul 15 22:28:11 PDT 2024

[skip ci]
@ChenglimEar
make null empty for data_warning in committees
9759958
Run make clean download import process
efa9973 
This is an automated update by travis-ci at
Mon Jul 15 22:48:24 PDT 2024

[skip ci]
@ChenglimEar
output consistent order to ensure that output doesn't change when
a6e89bd 
postgres version changes
Run make clean download import process
b4e4113 
This is an automated update by travis-ci at
Mon Jul 15 23:29:20 PDT 2024

[skip ci]
@ChenglimEar
set null committee name to empty string so that we can get consistent
710218c 
outputs when the postgres version changes
Run make clean download import process
37eb448 
This is an automated update by travis-ci at
Tue Jul 16 00:04:38 PDT 2024

[skip ci]
@ChenglimEar
use floats everywhere when calculating totals in create-digests
af67d7e
@ChenglimEar
increase column size for instagram column in candidates table
b46cccc
Run make clean download import process
a07acd9 
This is an automated update by travis-ci at
Thu Jul 18 21:48:37 PDT 2024

[skip ci]
@ChenglimEar
add some additional totals for oakland-2024 election in digests.json to
2a538a3 
help with debugging
Run make clean download import process
f2358af 
This is an automated update by travis-ci at
Thu Jul 18 22:14:25 PDT 2024

[skip ci]
@ChenglimEar
change election name in digests.json to include full date to correctly
0d6d8a2 
capture multiple elections in the same year
Run make clean download import process
8da4ef6 
This is an automated update by travis-ci at
Thu Jul 18 22:42:01 PDT 2024

[skip ci]
@ChenglimEar
remove commented postgres 9.6 in workflow
7ff8420
Run make clean download import process
e0362c9 
This is an automated update by travis-ci at
Sat Jul 20 20:51:29 PDT 2024

[skip ci]
@ChenglimEar
only run netfile v2 download when credentials are set up
f7d5769
@ChenglimEar
Merge branch 'add-workflow-to-gen-data' of github.com:caciviclab/disc…
d9c741b 
…losure-backend-static into add-workflow-to-gen-data
@ckingbailey ckingbailey mentioned this pull request Dec 14, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ckingbailey @ChenglimEar