[WIP] i.sentinel3.import: New addon to import Sentinel-3 data #612
+992
−0
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| MODULE_TOPDIR = ../.. | ||
|
|
||
| PGM = i.sentinel3.import | ||
|
|
||
| include $(MODULE_TOPDIR)/include/Make/Script.make | ||
|
|
||
| default: script |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,131 @@ | ||
| <h2>DESCRIPTION</h2> | ||
|
|
||
| The <em>i.sentinel3.import</em> module allows importing Copernicus Sentinel-3 products | ||
| downloaded by the <em><a href="i.sentinel.download.html">i.sentinel.download</a></em> | ||
| module. | ||
|
|
||
| <p> | ||
| By default <em>i.sentinel3.import</em> imports all Sentinel-3 scene files found | ||
| in the <b>input</b> directory. The number of scene files can be optionally | ||
| reduced by the <b>pattern</b> option or filtered by modification time using the | ||
| <b>modified_after</b> and/or <b>modified_before</b> option. In the <b>pattern</b> | ||
| option, a regular expression for filtering the file names should be given, e.g. | ||
| "0179_076_100_0900_LN2" for importing only specific tiles. | ||
| <p> | ||
| The <em>Sentinel-3</em> format is currently not supported by GDAL and consists | ||
| of several netCDF4 files. Therefore, a GRASS GIS specific import routine is implemented. | ||
| It imports the data pixel-wise after transforming them from WGS84 into the projection | ||
| of the current LOCATION. <em>r.in.xyz<em> is then used to read the coordinates into | ||
| a GRASS GIS raster map. Thus, the the user has to set the computational region extent and | ||
| resolution appropriately, to import the data of interest in the correct resolution. | ||
|
|
||
| <p> | ||
| By default, ancillary data and quality flag data is imported as well - as artificial | ||
ninsbl marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| bands. Import of those additional data layers can be deactivated using the <b>ancillary_bands</b> | ||
| and <b>quality_bands</b> option. | ||
|
|
||
| <p> | ||
| For each imported band both scene and band specific metadata are written into the map history | ||
ninsbl marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| (<em><a href="https://grass.osgeo.org/grass-stable/manuals/r.support.html">r.support</a></em>). | ||
| In addition, the scene name is stored as <em>source1</em> and the imported or linked file name as | ||
| <em>source2</em>. Also, sensing time is written into the timestamp of the | ||
| map. After import, the metadata can be retrieved with <em>r.info -e</em> | ||
| as shown below. | ||
|
|
||
|
|
||
| <h2>NOTES</h2> | ||
|
|
||
| <p> | ||
| Currently, only import of SL_2_LST__ products is supported. | ||
| <p> | ||
| By <b>register_file</b> option <em>i.sentinel3.import</em> allows | ||
ninsbl marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| creating a file which can be used to register imported imagery data | ||
| into space-time raster dateset (STRDS) with | ||
ninsbl marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| <em><a href="https://grass.osgeo.org/grass-stable/manuals/t.register.html">t.register</a></em>. | ||
| See example below. | ||
|
|
||
| <p> | ||
| <h3>Metadata storage</h3> | ||
|
|
||
| By using the <b>-j</b> flag the band metadata are additionally stored | ||
| in JSON format (in the current mapset under <tt>cell_misc</tt>). | ||
|
|
||
|
|
||
| <h2>EXAMPLES</h2> | ||
|
|
||
| <h3>List Sentinel bands</h3> | ||
|
|
||
| At first, print list of raster files to be imported by <b>-p</b>. For | ||
ninsbl marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| each file also product content is printed: | ||
|
|
||
| <div class="code"><pre> | ||
| i.sentinel3.import -p input="data" nprocs=4 modified_before="2021-09-09" product=LST basename=S3_LST | ||
|
|
||
| ... | ||
|
There was a problem hiding this comment. Why "data"? Are the quotes necessary there? Also, maybe put some of the resulting printed info as example and not only the |
||
| </pre></div> | ||
|
|
||
| <h3>Import Sentinel-3 data</h3> | ||
| Import all Sentinel-3 data found in <i>data</i> directory and store metadata | ||
| as JSON files within the GRASS GIS database directory: | ||
| <div class="code"><pre> | ||
| i.sentinel3.import -a -c -j input="data" nprocs=4 modified_before="2021-09-09" product=LST basename=S3_LST | ||
| </pre></div> | ||
|
|
||
|
|
||
| <h3>Register imported Sentinel-3 data into STRDS</h3> | ||
|
|
||
| <div class="code"><pre> | ||
| i.sentinel3.import -a -c -j input="data" register_output=data/reg.txt nprocs=4 modified_before="2021-09-09" product=LST basename=S3_LST | ||
|
|
||
| # register imported data into existing STRDS | ||
| t.register input=Sentinel_3 file=data/reg.txt | ||
| </pre></div> | ||
|
|
||
| A register file typically contains two columns: imported raster map | ||
| name and timestamp separated by <tt>|</tt>. In the case of current | ||
| development version of GRASS GIS which supports band references concept | ||
| (see <em><a href="https://grass.osgeo.org/grass-devel/manuals/g.bands.html">g.bands</a></em> module for details) a | ||
| register file is extended by a third column containg band reference | ||
| information, see the examples below. | ||
|
|
||
| <div class="code"><pre> | ||
| # register file produced by stable GRASS GIS 7.8 version | ||
| S3_imp_surface_temperature_20210907T205026|2021-09-07T20:50:26.285902+00:00|2021-09-07T20:53:25.977846+00:00 | ||
| S3_imp_surface_temperature_standard_error_20210907T205026|2021-09-07T20:50:26.285902+00:00|2021-09-07T20:53:25.977846+00:00 | ||
| S3_imp_confidence_20210907T205026|2021-09-07T20:50:26.285902+00:00|2021-09-07T20:53:25.977846+00:00 | ||
| ... | ||
| # register file produced by development GRASS GIS 7.9 version | ||
| S3_imp_surface_temperature_20210907T205026|2021-09-07T20:50:26.285902+00:00|2021-09-07T20:53:25.977846+00:00|S3_surface_temperature | ||
| S3_imp_surface_temperature_standard_error_20210907T205026|2021-09-07T20:50:26.285902+00:00|2021-09-07T20:53:25.977846+00:00|S3_surface_temperature_standard_error | ||
| S3_imp_confidence_20210907T205026|2021-09-07T20:50:26.285902+00:00|2021-09-07T20:53:25.977846+00:00|S3_confidence | ||
| </pre></div> | ||
|
|
||
| <h2>REQUIREMENTS</h2> | ||
| The following Python libraries are required | ||
| <ul> | ||
| <li>GDAL (install through system software management)</li> | ||
| <li>numpy</li> | ||
| <li>netcdf4</li> | ||
| <li>cf-units (for unit conversion)</li> | ||
| </ul> | ||
|
|
||
| <h2>SEE ALSO</h2> | ||
|
|
||
| <em> | ||
| <a href="i.sentinel.html">Overview of i.sentinel toolset</a> | ||
| </em> | ||
| <p> | ||
| <em> | ||
| <a href="i.sentinel.download.html">i.sentinel.download</a>, | ||
| <a href="https://grass.osgeo.org/grass-stable/manuals/r.in.xyz.html">r.in.xyz</a>, | ||
| <a href="https://grass.osgeo.org/grass-stable/manuals/t.register.html">t.register</a> | ||
| </em> | ||
|
|
||
| <h2>AUTHOR</h2> | ||
|
|
||
| Stefan Blumentrath, <a href="https://www.nina.no/">NINA</a>, Norway | ||
|
|
||
| <!-- | ||
| <p> | ||
| <i>Last changed: $Date$</i> | ||
| --> | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What does QGIS uses then to open the nc files within the S3 zip files? I tested with a S3SY2VG1 product. A double click just opens the ndvi layer in QGIS and
r.importworks just fine for that layer as well. Maybe, this statement is a bit too strong or only valid for S3 LST products...?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you mean the semi automated classification plugin? That one has a similar (but not the same) workaround for the S3-format...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, plain QGIS opens the
.ncmentioned file just fine, I do not use the semi automated classification pluginThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, thanks for clarifying.
Yes, QGIS does open the
.ncbut - at least the QGIS 3.16 version I am running does not resolve the geometrical information correctly. The image is flipped and distorted....The sentinel-3 format is pretty peculiar and one needs to combine the geometry and data nc files in order to get a reasonble geographical representation of the data....
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NDVI products look just fine
LST is flipped
This is what I meant originally with my comment, some of the S3 products are read just fine (NDVI from the S3A_SY_2_VG1 above). Hence, it's not completely true that none of the S3 formats is supported by GDAL as stated.
I use GDAL 3.2.2 and QGIS 3.20
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for clarifying and bearing with me. My focus has been on SLSTR. So, the different products seem to come in different formats / packaging also across processing levels...
So a "final"
i.sentinel3.importmodule should support the formats for the different instrumentsfor level 1 and 2.
Not sure if I find the time to investigate and implement all of them (can be as simple as
r.import/r.in.gdal/r.external).In any case this should be made clear in the documentation, as you correctly point out...