GeoParquet 1.0.0 beta.1 and the path to GeoParquet 1.0.0

[cholmes]

Earlier today a group of GeoParquet contributors met to talk about the path to 1.0.0 and made a few decisions, though we welcome further discussion here.

The main decision was to focus on 'interoperability' for version 1.0.0, to push ahead with a simple, stable specification that helps any existing user of geospatial data in parquet to have a clear, stable target to rely upon. We still plan to explore more advanced topics like spatial indexing, alternate geometry representations and 'streaming' use cases, but those will be either a version 1.1 or a 2.0 (we'll use SemVer, so we hope to do it as a 1.1, but if there are breaking changes it'll be a 2.0).

In line with that we'll keep the encoding field, but for 1.0 it will only have 1 available option - WKB. This allows us to introduce an alternate encoding in the future without having it be a breaking change, but we're not trying to get in a new format for version 1.0. We'll aim to closely track GeoArrow Spec, and when it matures and releases a 1.0 we'll likely make a new GeoParquet release. We talked about also keeping open in the GeoParquet repo a pull request that describes a Arrow Native / columnar geometry format, linking to the geoarrow description for continued experimentation.

Practically this is an update to our articulated roadmap, as we won't do a 'spatial index'-focused release, and we'd see work on that coming after 1.0.

The plan is to work to a 1.0.0-beta.1 release within a month. This release will mean 'we don't think we'll be making any changes before 1.0.0, but we want to get a wider set of feedback and will consider any changes that come during that time - including breaking changes'. But we should be able to get a wider group of people implementing, since the changes between 1.0.0-beta.1 and 1.0.0 will hopefully be minimal.

To get to 1.0.0 we'll aim to satisfy a set of criteria (listed below) that represents broad community acceptance and will mean that we've gotten sufficient feedback to feel comfortable with a 1.0.0 release that will be stable. Once the criteria are hit we'll issue a 1.0.0-rc.1 - a true 'release candidate' where we'll leave it out for a few weeks. If no one has any changes then that version will become 1.0.0. If there are any fixes (even minor ones) to the spec itself we'll issue a next release candidate and repeat the process.

The group discussed the 'criteria' for a bit, and came up with the following:

• At least 10 'implementations' of various software packages. Right now there are 4 listed on the main site - GeoPandas (Python), sfarrow (R), GDAL/OGR (C++, bindings in several languages), GeoParquet.jl (Julia). So we'll need at least 6 more implementations, ideally spanning different languages / environments.
• At least 5 different 'data providers' who have adopted this. They should be providing at least 1 'production' dataset, and ideally have much of their data available in GeoParquet. 'Production' just means that it shouldn't be just little demos / sample data - it should be a full dataset that is useful. Today we'd count Microsoft with their Planetary Computer, as it provides their Building Footprints dataset as GeoParquet, as well as GeoParquet for their STAC Collections
• At least 20 different 'production' datasets. We want new users to the GeoParquet world to be able to instantly access a decent set of data, not just a few little example datasets. This obviously works in conjunction with number of data providers, but we want to be sure at least one or two of those providers is making significant data available, not just one or two experiments.
• Accessible validation engine - it should be easy for anyone to test if a parquet file they produced or obtained is fully valid and compliant 'GeoParquet'. This will help implementors easily test to ensure they're doing things right, and will help users make sure the data they got or produced is fully spec compliant. There is a validator in the geoparquet repo today, but right now it just checks the schema, not the data itself. We want a validator that checks the schema as well, and it should also be accessible to non-developers - ideally a webpage (at least for smaller datasets) and an easily installable desktop or command-line tool (-> Validator improvements #124).
• Test data - we didn't put a clear number on how much of this we want, but we want to be sure that new implementors can easily work with a range of data that shows off the 'edges' of the spec where they can make sure they're doing everything right. We'd have geoparquet that does the defaults, but also shows off different projections, multiple data columns, nested data structures, etc. (-> Example data to test implementations #123)

That should be the core set of requirements for 1.0.0. And then there are a few other activities that people in the group will likely work on, but that aren't hard requirements to get to 1.0.0:

• A GeoParquet website, that gives a nice introduction to the specification, points people at implementations and datasets, etc. Ideally we get a nice logo as well, but that's definitely 'nice to have'.
• Online geoparquet creation tool, a 'geoparquet.io' modeled after http://geojson.io that makes it easy for people to convert some of their existing data into geoparquet and try it out. This may be where we also present the full validation tools.
• Registering as an OGC 'Community Standard'. Their process has this happen after 1.0 version of the spec, but we'll aim to have this done as quickly as possible after 1.0. We may also pursue formation of an OGC 'Standards Working Group' (SWG) at some point.
• Media Type / MIME Type - The general consensus is that we'd like to just use the general Parquet Media type, instead of having a special 'geo parquet' type, but Parquet doesn't have one. But we'd like to at least help push that, as the geospatial api's lean on media types extensively. (-> Media type #115)

The very next steps are to organize the issue tracker to represent the milestones, and then to work through the issues for 1.0.0-beta.1. I hope to take on the organization in the next day or two, and everyone is welcome and encouraged to help triage and make PR's for the remaining issues. We'll likely have another synchronous meeting on 10/7 to hopefully finish off any needed discussion from the remaining issues, and hope to cut the release soon after. It'll be at 9am pacific time.

Comments/feedback welcome, especially on the overall plan. If you're wanting to go deep on a particular issue mentioned here then it'd likely be better to open a discussion or issue and discuss there.

[kylebarron]

Should the title be geoparquet? Or are you meaning to explain how STAC went from beta to 1.0?

[cholmes]

D'oh! Yes, yes it should. Guess that's what happens in my brain when I switch between the two projects.

[jorisvandenbossche]

One question related to actual versioning of files: if we "release" a 1.0.0.beta1 next month, do we expect that implementors actually write this version into the parquet files? Or do we still release for example a 0.5.0 which is equal in specification to 1.0.0beta1, so you don't have "beta" written in a file?

[cholmes]

I had been thinking it'd write out 1.0.0-beta.1. But if that's a pain I'd be open to having it be 0.5.0 in the file and 'branding' it as our first beta. What's the downside of having 'beta' written in the file?

[jatorre]

Thanks chris for the effort. Right now nothing else to add.