Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

locations.geojson POC phase2: End-to-end partial support of json data #1810

Open
wants to merge 18 commits into
base: master
Choose a base branch
from

Conversation

jcpitre
Copy link
Contributor

@jcpitre jcpitre commented Sep 4, 2024

Summary:

Builds on the poc written by @davidgamez (#1805), related to locations.geojson

What it does:

  • Add the necessary classes to process locations.geojson (Classes with Geojson in the name)
  • Read parts of the file
  • Add a sample validator (GtfsGeojsonFeatureUniqueLocationIdValidator) and notice (UniqueLocationIdViolationNotice)

Not covered:

  • Reading and processing geometry data.
  • Processing the other files from Flex
  • Adding other validators or notices apart from the sample one.

To be done:

  • Add tests
  • Comments

This is NOT ready to merge yet.

I modified an existing dataset that has flex data to create an error where the id of a feature in locations.geojson is the same as an id in stops.txt.
browncounty-mn-us--flex-v2-broken.zip

Here is a capture of the sample notice in the report:
image

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with gradle test to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@jcpitre jcpitre added the do not merge This PR needs more work/discussion or is not meant to be merged label Sep 4, 2024
Copy link
Contributor

github-actions bot commented Sep 4, 2024

This contribution does not follow the conventions set by the Google Java style guide. Please run the following command line at the root of the project to fix formatting errors: ./gradlew goJF.

Copy link
Contributor

github-actions bot commented Sep 4, 2024

📝 Acceptance Test Report

📋 Summary

✅ The rule acceptance has passed for commit 15f1fa2
Download the full acceptance test report here (report will disappear after 90 days).

📊 Notices Comparison

New Errors (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Errors (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

New Warnings (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Warnings (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

🛡️ Corruption Check

0 out of 1575 sources (~0 %) are corrupted.

⏱️ Performance Assessment

📈 Validation Time

Assess the performance in terms of seconds taken for the validation process.

Time Metric Dataset ID Reference (s) Latest (s) Difference (s)
Average -- 3.90 3.97 ⬆️+0.07
Median -- 1.37 1.44 ⬆️+0.07
Standard Deviation -- 11.14 11.09 ⬇️-0.05
Minimum in References Reports us-california-catalina-express-gtfs-299 0.50 0.57 ⬆️+0.07
Maximum in Reference Reports gb-unknown-uk-aggregate-feed-gtfs-2014 320.06 317.04 ⬇️-3.02
Minimum in Latest Reports us-california-santa-clarita-transit-gtfs-812 0.56 0.55 ⬇️-0.01
Maximum in Latest Reports gb-unknown-uk-aggregate-feed-gtfs-2014 320.06 317.04 ⬇️-3.02

@jcpitre jcpitre changed the title Poc/json files phase2: Modifications from #1805 locations.geojson phase2: Modifications from #1805 Sep 4, 2024
@jcpitre jcpitre requested a review from qcdyx September 4, 2024 13:42
Copy link
Contributor

github-actions bot commented Sep 4, 2024

📝 Acceptance Test Report

📋 Summary

✅ The rule acceptance has passed for commit e989e06
Download the full acceptance test report here (report will disappear after 90 days).

📊 Notices Comparison

New Errors (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Errors (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

New Warnings (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Warnings (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

🛡️ Corruption Check

0 out of 1575 sources (~0 %) are corrupted.

⏱️ Performance Assessment

📈 Validation Time

Assess the performance in terms of seconds taken for the validation process.

Time Metric Dataset ID Reference (s) Latest (s) Difference (s)
Average -- 3.92 4.02 ⬆️+0.10
Median -- 1.38 1.44 ⬆️+0.05
Standard Deviation -- 11.40 11.37 ⬇️-0.03
Minimum in References Reports us-california-flex-v2-developer-test-feed-3-gtfs-1819 0.50 0.59 ⬆️+0.10
Maximum in Reference Reports gb-unknown-uk-aggregate-feed-gtfs-2014 331.99 327.84 ⬇️-4.15
Minimum in Latest Reports us-massachusetts-massachusetts-area-express-max-gtfs-431 0.60 0.57 ⬇️-0.03
Maximum in Latest Reports gb-unknown-uk-aggregate-feed-gtfs-2014 331.99 327.84 ⬇️-4.15

@jcpitre jcpitre changed the title locations.geojson phase2: Modifications from #1805 locations.geojson POC phase2: End-to-end partial support of json data Sep 4, 2024
Copy link
Contributor

github-actions bot commented Sep 4, 2024

📝 Acceptance Test Report

📋 Summary

✅ The rule acceptance has passed for commit a5d8762
Download the full acceptance test report here (report will disappear after 90 days).

📊 Notices Comparison

New Errors (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Errors (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

New Warnings (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Warnings (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

🛡️ Corruption Check

0 out of 1575 sources (~0 %) are corrupted.

⏱️ Performance Assessment

📈 Validation Time

Assess the performance in terms of seconds taken for the validation process.

Time Metric Dataset ID Reference (s) Latest (s) Difference (s)
Average -- 3.91 4.00 ⬆️+0.09
Median -- 1.37 1.42 ⬆️+0.05
Standard Deviation -- 10.99 11.10 ⬆️+0.11
Minimum in References Reports us-california-flex-v2-developer-test-feed-3-gtfs-1819 0.49 0.57 ⬆️+0.07
Maximum in Reference Reports gb-unknown-uk-aggregate-feed-gtfs-2014 312.22 316.36 ⬆️+4.14
Minimum in Latest Reports us-california-flex-v2-developer-test-feed-2-gtfs-1818 0.51 0.53 ⬆️+0.02
Maximum in Latest Reports gb-unknown-uk-aggregate-feed-gtfs-2014 312.22 316.36 ⬆️+4.14

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add class comments to help understanding the purpose of this abstract class?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added class comments.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, class comment could be useful.

Copy link
Contributor Author

@jcpitre jcpitre Sep 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added class comments.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the benefit of pulling TableStatus out from GtfsTableContainer? Just curious, coz nothing is changed in this enum class.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say it's not essential but it's not bad to have a file dedicated to one enum.

Copy link
Contributor

github-actions bot commented Sep 4, 2024

📝 Acceptance Test Report

📋 Summary

✅ The rule acceptance has passed for commit ffecd98
Download the full acceptance test report here (report will disappear after 90 days).

📊 Notices Comparison

New Errors (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Errors (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

New Warnings (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Warnings (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

🛡️ Corruption Check

0 out of 1575 sources (~0 %) are corrupted.

⏱️ Performance Assessment

📈 Validation Time

Assess the performance in terms of seconds taken for the validation process.

Time Metric Dataset ID Reference (s) Latest (s) Difference (s)
Average -- 3.95 4.01 ⬆️+0.06
Median -- 1.38 1.44 ⬆️+0.05
Standard Deviation -- 11.42 11.26 ⬇️-0.16
Minimum in References Reports us-california-santa-clarita-transit-gtfs-812 0.53 0.69 ⬆️+0.16
Maximum in Reference Reports gb-unknown-uk-aggregate-feed-gtfs-2014 328.59 320.34 ⬇️-8.26
Minimum in Latest Reports us-massachusetts-massachusetts-area-express-max-gtfs-431 0.57 0.54 ⬇️-0.03
Maximum in Latest Reports gb-unknown-uk-aggregate-feed-gtfs-2014 328.59 320.34 ⬇️-8.26

Copy link
Contributor

github-actions bot commented Sep 4, 2024

📝 Acceptance Test Report

📋 Summary

✅ The rule acceptance has passed for commit 0d8ddb7
Download the full acceptance test report here (report will disappear after 90 days).

📊 Notices Comparison

New Errors (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Errors (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

New Warnings (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Warnings (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

🛡️ Corruption Check

0 out of 1575 sources (~0 %) are corrupted.

⏱️ Performance Assessment

📈 Validation Time

Assess the performance in terms of seconds taken for the validation process.

Time Metric Dataset ID Reference (s) Latest (s) Difference (s)
Average -- 3.91 3.98 ⬆️+0.07
Median -- 1.38 1.42 ⬆️+0.05
Standard Deviation -- 11.20 11.39 ⬆️+0.19
Minimum in References Reports us-florida-citrus-county-transit-gtfs-630 0.50 0.53 ⬆️+0.03
Maximum in Reference Reports gb-unknown-uk-aggregate-feed-gtfs-2014 321.01 329.04 ⬆️+8.02
Minimum in Latest Reports us-florida-citrus-county-transit-gtfs-630 0.50 0.53 ⬆️+0.03
Maximum in Latest Reports gb-unknown-uk-aggregate-feed-gtfs-2014 321.01 329.04 ⬆️+8.02

Copy link
Contributor

github-actions bot commented Sep 5, 2024

📝 Acceptance Test Report

📋 Summary

✅ The rule acceptance has passed for commit 4f9e3db
Download the full acceptance test report here (report will disappear after 90 days).

📊 Notices Comparison

New Errors (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Errors (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

New Warnings (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Warnings (0 out of 1575 datasets, ~0%) ✅

No changes were detected due to the code change.

🛡️ Corruption Check

0 out of 1575 sources (~0 %) are corrupted.

⏱️ Performance Assessment

📈 Validation Time

Assess the performance in terms of seconds taken for the validation process.

Time Metric Dataset ID Reference (s) Latest (s) Difference (s)
Average -- 3.91 3.98 ⬆️+0.08
Median -- 1.38 1.46 ⬆️+0.07
Standard Deviation -- 11.28 11.21 ⬇️-0.08
Minimum in References Reports us-florida-citrus-county-transit-gtfs-630 0.51 0.59 ⬆️+0.09
Maximum in Reference Reports gb-unknown-uk-aggregate-feed-gtfs-2014 326.83 321.61 ⬇️-5.22
Minimum in Latest Reports us-california-catalina-express-gtfs-299 0.52 0.56 ⬆️+0.03
Maximum in Latest Reports gb-unknown-uk-aggregate-feed-gtfs-2014 326.83 321.61 ⬇️-5.22

@emmambd emmambd linked an issue Sep 5, 2024 that may be closed by this pull request
@emmambd
Copy link
Contributor

emmambd commented Sep 5, 2024

@jcpitre One note re: the test notice of unique_location_id_violation. We won't actually be able to merge this one until the schema for location_groups.txt in #1749 is added, since the rule requires checking location id across all 3 files.

So there are two options:

  1. Leave the notice is purely for testing purposes and then take it out of this PR. (Moving it to the other issue/PR for Flex: Add location group schema and update stop time schema #1749)
  2. Add the location_groups.txt schema within this PR, which seems to make the PR extra big.

@jcpitre
Copy link
Contributor Author

jcpitre commented Sep 5, 2024

@jcpitre One note re: the test notice of unique_location_id_violation. We won't actually be able to merge this one until the schema for location_groups.txt in #1749 is added, since the rule requires checking location id across all 3 files.

So there are two options:

  1. Leave the notice is purely for testing purposes and then take it out of this PR. (Moving it to the other issue/PR for Flex: Add location group schema and update stop time schema #1749)
  2. Add the location_groups.txt schema within this PR, which seems to make the PR extra big.

Yes, I put anything related to location_groups.txt in comments in the validator class so we hopefully don't forget.
I thought though that there is some value in the validation that is done. After all, having the duplication of location_id in stops.txt and locations.geojson is an error in itself.
And until all validators are implemented for Flex we will have false negatives, be it this one or other rules not yet implemented.

@emmambd
Copy link
Contributor

emmambd commented Sep 5, 2024

@jcpitre I wouldn't want to release with it in the present state (meaning #1749 would block release, which it already does), but I'm okay with that. False positives are the main focus of this release, not negatives.

@emmambd
Copy link
Contributor

emmambd commented Sep 9, 2024

@jcpitre I tested duplicate_key with this PR using
flex.zip to try to generate a duplicate_key notice.

It worked! But I realized that we don't have the csvRowNumber lines...because it's not a csv.
Screenshot 2024-09-09 at 3 30 34 PM

Do we have a plan or need to discuss how to modify these rowNumber columns for generic notices that can affect either a geojson file or a csv file? cc @davidgamez

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
do not merge This PR needs more work/discussion or is not meant to be merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Flex: Add locations.geojson schema
4 participants