Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split ingestion pipeline #61

Merged
merged 20 commits into from
Aug 6, 2024
Merged

Split ingestion pipeline #61

merged 20 commits into from
Aug 6, 2024

Conversation

pipliggins
Copy link
Collaborator

@pipliggins pipliggins commented Jul 30, 2024

Splits the ingestion pipeline into two parts; fhirflat file creation and validation.

The initial file creation doesn't pack/unpack any pydantic FHIR resource classes, while validation tries to create resources but doesn't then convert them back into fhirflat files. Together, this provides a significant speedup even when running both file creation and validation together to mimic the previous ingest_to_flat behaviour.

Previous speeds on a sample dataset:

Patient took 1.19 seconds to convert 67 rows.
Encounter took 1.31 seconds to convert 67 rows.
Observation took 6.85 seconds to convert 2194 rows.
Condition took 5.99 seconds to convert 461 rows.

New speeds doing ingestion + validation together (no parallel processing implemented):

Patient took 0.78 seconds to convert 67 rows. 
Encounter took 0.68 seconds to convert 67 rows. 
2 resources not created due to validation errors. Errors saved to encounter_errors.csv
Observation took 1.61 seconds to convert 2194 rows. 
Condition took 0.78 seconds to convert 461 rows. 

Fixes #54

@codecov-commenter
Copy link

codecov-commenter commented Jul 30, 2024

Codecov Report

Attention: Patch coverage is 98.77049% with 3 lines in your changes missing coverage. Please review.

Project coverage is 97.70%. Comparing base (3f3fa96) to head (6ba5617).
Report is 2 commits behind head on main.

Files Patch % Lines
fhirflat/util.py 96.29% 2 Missing ⚠️
fhirflat/resources/base.py 97.14% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #61      +/-   ##
==========================================
+ Coverage   97.58%   97.70%   +0.12%     
==========================================
  Files          39       40       +1     
  Lines        1779     1962     +183     
==========================================
+ Hits         1736     1917     +181     
- Misses         43       45       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@pipliggins pipliggins marked this pull request as ready for review August 5, 2024 11:49
@pipliggins pipliggins requested a review from abhidg August 5, 2024 11:49
Copy link
Contributor

@abhidg abhidg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, few minor comments!

fhirflat/ingest.py Show resolved Hide resolved
fhirflat/ingest.py Show resolved Hide resolved
docs/howto/conversion-data.md Outdated Show resolved Hide resolved
fhirflat/resources/base.py Outdated Show resolved Hide resolved
fhirflat/resources/base.py Outdated Show resolved Hide resolved
fhirflat/resources/base.py Outdated Show resolved Hide resolved
fhirflat/util.py Show resolved Hide resolved
fhirflat/util.py Outdated Show resolved Hide resolved
@pipliggins pipliggins requested a review from abhidg August 6, 2024 13:33
@pipliggins pipliggins merged commit 627c07e into main Aug 6, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Split data ingestion into process and validate
3 participants