Landsat8 download problem from USGS-EROS for L1C product not available on AWS S3 or Google Storage

[gybely]

I am not having success trying to load a set of specific Landsat-8 scenes from USGS EROS site -- for scenes not available on AWS or Google-- trying this from an OSX machine.

I get the scene names using '> landsat search' just fine... With '> landsat download', I get "product not available on AWS S3 or Google Storage"... one of these scenes, as an example, is LC81290582016007LGN02.

When I supply my USGS EROS credentials --username --password (I do have m2m / special inventory access enabled on this account), I get the error message below. Any help greatly appreciated...

#----
Time spent : 0.75 seconds
Traceback (most recent call last):
File "/usr/local/bin/landsat", line 11, in
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/landsat/landsat.py", line 491, in main
exit(*main(args))
File "/usr/local/lib/python2.7/site-packages/landsat/landsat.py", line 408, in main
files = d.download(args.scenes, bands)
File "/usr/local/lib/python2.7/site-packages/landsat/downloader.py", line 80, in download
files.append(self.usgs_eros(scene, self.download_dir))
File "/usr/local/lib/python2.7/site-packages/landsat/downloader.py", line 95, in usgs_eros
error_tree = ElementTree.fromstring(str(e.message))
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1311, in XML
parser.feed(text)
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1653, in feed
self._raiseerror(v)
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1517, in _raiseerror
raise err
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 171, column 22

[j08lue]

I had this issue, too, and solved it by updating usgs to v0.2.0.

pip install -U usgs

But there might be more issues because USGS is not using SOAP anymore?

[qianliuGit]

I have the problem, is there a solution??

[j08lue]

yes

[gybely]

After pip -U install usgs I still have the problem with some scenes from USGS...
My guess is this is still related to the scene reprocessing for Collection-1 L1.

Error below:
usgs.USGSError: DATASET_INVALID: Dataset is not configured for Inventory Service

[j08lue]

@gybely @qianliuGit

You also need this change to landsat-util: #235

You can install it from my fork, if you like:

pip install https://github.com/DHI-GRAS/landsat-util/archive/usgs_c1.zip

[qianliuGit]

@j08lue
I uninstalled my landsat-util, then i have ran the "pip install -U usgs" , and then ran the "pip install https://github.com/DHI-GRAS/landsat-util/archive/usgs_c1.zip".
When i ran "landsat search --start "january 1 2017" --end "january 30 2017" -p 122,036", i got this:
===> 2 items were found
{
"limit": 10,
"results": [
{
"cloud": 4.01,
"date": "2017-01-24",
"path": "122",
"row": "036",
"sat_type": "L8",
"sceneID": "LC81220362017024LGN01",
"thumbnail": "http://earthexplorer.usgs.gov/browse/landsat_8/2017/122/036/LC08_L1TP_122036_20170124_20170311_01_T1.jpg"
},
{
"cloud": 75.02,
"date": "2017-01-08",
"path": "122",
"row": "036",
"sat_type": "L8",
"sceneID": "LC81220362017008LGN01",
"thumbnail": "http://earthexplorer.usgs.gov/browse/landsat_8/2017/122/036/LC08_L1GT_122036_20170108_20170311_01_T2.jpg"
}
],
"status": "SUCCESS",
"total": 2,
"total_returned": 2
}
['Search completed!']
,

then "landsat download LC81220362017024LGN01"
i have got this "LC81220362017024LGN01 is not available on AWS S3 or Google Storage",

Any help is greatly appreciated, thanks a lot!

[j08lue]

You did not specify your USGS account

                --username          USGS Eros account Username (only works if the account has special
                                    inventory access). Username and password as a fallback if the image
                                    is not found on AWS S3 or Google Storage
                --password          USGS Eros account Password

which means that USGS is not available as an option when Google and Amazon fail.

I take it you did not look that much into this ;), so you probably do not have applied for machine-to-machine access for your USGS EROS account. For that you need to write to their support and wait a couple of weeks.

[qianliuGit]

@j08lue Thanks, i will have a try !!

[jmellian]

I think the problem is that as of 1 may 2017, there has been a change in dataformat. Because of that the AWS en Google changed the location of the landsatfiles. I have fix that in pull-request #241, but there is a problem with the unittest for Geocoder (no changes there).

See: Download Collection 1 datastructure with AWS en Google-storage #241
Added download Collection 1 landsat 8 data from AWS after 1 may 2017:

Try the change from:
https://github.com/jmellian/landsat-util :only geocoder is (probably) not working, because of a api-key problem

[minorsecond]

@jmellian I've tried both the version here and yours and still get the "not available" error. I've applied for machine-to-machine access and am hoping it works.

[j08lue]

I've applied for machine-to-machine access and am hoping it works.

USGS machine-to-machine access, I can confirm, works.

[matthewhanson]

I've updated the usgs library in requirements to 0.2.1 on the develop branch.
If any of the issues raised in here persist please open as a new issue.