Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PODAAC-5173: Add OPeNDAP Tutorials on Cloud Set Up and Subsetting #97

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

PODAAC-5173: Add OPeNDAP Tutorials on Cloud Set Up and Subsetting #97

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
92e5f36
Add opendap tutorials for cloud credentials and subsetting
phoeberepo Mar 3, 2023
441be09
Merge remote-tracking branch 'origin/master' into feature/PODAAC-5173
phoeberepo Mar 4, 2023
bf1f699
Remove test statement
phoeberepo Mar 4, 2023
7b98585
Update subsetting-tutorial with outputs
phoeberepo Mar 4, 2023
948fb8b
Add tutorial template summaries to notebook and provide additional in…
phoeberepo Mar 17, 2023
f676db2
Add tutorial template to opendap subsetting tutorial
phoeberepo Mar 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
364 changes: 364 additions & 0 deletions notebooks/opendap/opendap-cloud-credentials-tutorial.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,364 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# OPeNDAP Cloud Set Up"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Step 1. Register a username and password at: https://urs.earthdata.nasa.gov/"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Step 2: Set Up ~/.netrc\n",
"With your usernamd and password from Step 1, input your username and password by creating a .netrc on your local machine by running the command below.\n",
"\n",
"`\n",
"echo \"machine urs.earthdata.nasa.gov login your_username password your_password\" > ~/.netrc\n",
"`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Confirm your netrc file generated successfully by running `cat ~/.netrc`\n",
"\n",
"It should look like this:\n",
"\n",
"```machine urs.earthdata.nasa.gov login your_username password your_password```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Step 3: Set Up ~/.urs_cookies\n",
"This saves http cookies in a local file named .urs_cookies which allows you to access any URS-protected data file.\n",
"Adapted from: https://urs.earthdata.nasa.gov/documentation/for_users/data_access/curl_and_wget"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```wget --quiet --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies --auth-no-challenge \"https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/VIIRS_NPP-STAR-L3U-v2.80/20120201000000-STAR-L3U_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.80-v02.0-fv01.0.nc\"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Confirm your netrc file generated successfully by running `cat ~/.urs_cookies`\n",
"\n",
"It should look like this:\n",
"\n",
"```\n",
"# HTTP Cookie File\n",
"# Generated by Wget on date-today.\n",
"# Edit at your own risk.\n",
"\n",
"archive.podaac.earthdata.nasa.gov\tFALSE\t/\tFALSE\tyour-number\tyour-alpha-char your-long-string-of-char\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Step 4: Set Up .dodsrc\n",
"Create a .dodsrc file in the same directory as your python or notebook. If it is not in the same directory, your credentials will fail."
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Overwriting .dodsrc\n"
]
}
],
"source": [
"%%file .dodsrc\n",
"HTTP.NETRC=~/.netrc\n",
"HTTP.COOKIEJAR=~/.urs_cookies\n",
"# OPeNDAP client configuration file. See the OPeNDAP users guide for information.\n",
"USE_CACHE=0\n",
"# Cache and object size are given in megabytes (20 ==> 20Mb).\n",
"CACHE_ROOT=~/.dods_cache/\n",
"ALWAYS_VALIDATE=0\n",
"MAX_CACHE_SIZE=20\n",
"MAX_CACHED_OBJ=5\n",
"IGNORE_EXPIRES=0\n",
"DEFAULT_EXPIRES=86400\n",
"# Request servers compress responses if possible? 1 (yes) or 0 (false)\n",
"DEFLATE=0\n",
"# Should SSL certificates and hosts be validated? SSL will only work with signed certificates.\n",
"VALIDATE_SSL=1\n",
"# Proxy configuration (optional parts in []s).\n",
"# You may also use the 'http_proxy' environment variable\n",
"# but a value in this file will override that env variable.\n",
"#PROXY_SERVER=[http://][username:password@]host[:port]\n",
"#NO_PROXY_FOR=\n",
"#AIS_DATABASE=\n",
"#COOKIE_JAR=.dods_cookies\n",
"# The cookie jar is a file that holds cookies sent from\n",
"# servers such as single signon systems. Uncomment this\n",
"# option and provide a file name to activate this feature.\n",
"# If the value is a filename, it will be created in this\n",
"# directory; a full pathname can be used to force a specific\n",
"# location."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Confirm your .dodsrc file was generated successfully by running `cat .dodsrc`\n",
"\n",
"It should look like this: \n",
"\n",
"```\n",
"HTTP.NETRC=~/.netrc\n",
"HTTP.COOKIEJAR=~/.urs_cookies\n",
"# OPeNDAP client configuration file. See the OPeNDAP users guide for information.\n",
"USE_CACHE=0\n",
"# Cache and object size are given in megabytes (20 ==> 20Mb).\n",
"CACHE_ROOT=~/.dods_cache/\n",
"ALWAYS_VALIDATE=0\n",
"MAX_CACHE_SIZE=20\n",
"MAX_CACHED_OBJ=5\n",
"IGNORE_EXPIRES=0\n",
"DEFAULT_EXPIRES=86400\n",
"# Request servers compress responses if possible? 1 (yes) or 0 (false)\n",
"DEFLATE=0\n",
"# Should SSL certificates and hosts be validated? SSL will only work with signed certificates.\n",
"VALIDATE_SSL=1\n",
"# Proxy configuration (optional parts in []s).\n",
"# You may also use the 'http_proxy' environment variable\n",
"# but a value in this file will override that env variable.\n",
"#PROXY_SERVER=[http://][username:password@]host[:port]\n",
"#NO_PROXY_FOR=\n",
"#AIS_DATABASE=\n",
"#COOKIE_JAR=.dods_cookies\n",
"# The cookie jar is a file that holds cookies sent from\n",
"# servers such as single signon systems. Uncomment this\n",
"# option and provide a file name to activate this feature.\n",
"# If the value is a filename, it will be created in this\n",
"# directory; a full pathname can be used to force a specific\n",
"# location.\\\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Step 5: Test Your .netrc Credentials\n",
"Test your .netrc credentials are cached correctly by running a wget command on an opendap URL. If this wget test fails, then OPeNDAP access in step 6 will also fail. \n",
"\n",
"```wget 'https://opendap.earthdata.nasa.gov/collections/C1968979597-POCLOUD/granules/S6A_P4_2__LR_STD__NR_085_057_20230302T191818_20230302T210949_F07'```"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"--2023-03-02 16:33:29-- https://opendap.earthdata.nasa.gov/collections/C1968979597-POCLOUD/granules/S6A_P4_2__LR_STD__NR_085_057_20230302T191818_20230302T210949_F07\n",
"Resolving opendap.earthdata.nasa.gov (opendap.earthdata.nasa.gov)... 13.226.228.88, 13.226.228.111, 13.226.228.33, ...\n",
"Connecting to opendap.earthdata.nasa.gov (opendap.earthdata.nasa.gov)|13.226.228.88|:443... connected.\n",
"HTTP request sent, awaiting response... 302 \n",
"Location: /login/urs [following]\n",
"--2023-03-02 16:33:29-- https://opendap.earthdata.nasa.gov/login/urs\n",
"Reusing existing connection to opendap.earthdata.nasa.gov:443.\n",
"HTTP request sent, awaiting response... 302 \n",
"Location: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=SSCATdwCtNkEWliH3_WzDw&response_type=code&redirect_uri=https://opendap.earthdata.nasa.gov/login/urs [following]\n",
"--2023-03-02 16:33:29-- https://urs.earthdata.nasa.gov/oauth/authorize?client_id=SSCATdwCtNkEWliH3_WzDw&response_type=code&redirect_uri=https://opendap.earthdata.nasa.gov/login/urs\n",
"Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33\n",
"Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.\n",
"HTTP request sent, awaiting response... 401 Unauthorized\n",
"Authentication selected: Basic realm=\"Please enter your Earthdata Login credentials. If you do not have a Earthdata Login, create one at https://urs.earthdata.nasa.gov//users/new\"\n",
"Reusing existing connection to urs.earthdata.nasa.gov:443.\n",
"HTTP request sent, awaiting response... 302 Found\n",
"Location: https://opendap.earthdata.nasa.gov/login/urs?code=b534ed56515ee90770ba16a1c12fd06e7eb4557abac4a1f3f7389e1e0ebc9f2a [following]\n",
"--2023-03-02 16:33:29-- https://opendap.earthdata.nasa.gov/login/urs?code=b534ed56515ee90770ba16a1c12fd06e7eb4557abac4a1f3f7389e1e0ebc9f2a\n",
"Connecting to opendap.earthdata.nasa.gov (opendap.earthdata.nasa.gov)|13.226.228.88|:443... connected.\n",
"HTTP request sent, awaiting response... 302 \n",
"Location: https://opendap.earthdata.nasa.gov/collections/C1968979597-POCLOUD/granules/S6A_P4_2__LR_STD__NR_085_057_20230302T191818_20230302T210949_F07 [following]\n",
"--2023-03-02 16:33:30-- https://opendap.earthdata.nasa.gov/collections/C1968979597-POCLOUD/granules/S6A_P4_2__LR_STD__NR_085_057_20230302T191818_20230302T210949_F07\n",
"Reusing existing connection to opendap.earthdata.nasa.gov:443.\n",
"HTTP request sent, awaiting response... 302 \n",
"Location: https://opendap.earthdata.nasa.gov/collections/C1968979597-POCLOUD/granules/S6A_P4_2__LR_STD__NR_085_057_20230302T191818_20230302T210949_F07.dmr.html [following]\n",
"--2023-03-02 16:33:30-- https://opendap.earthdata.nasa.gov/collections/C1968979597-POCLOUD/granules/S6A_P4_2__LR_STD__NR_085_057_20230302T191818_20230302T210949_F07.dmr.html\n",
"Reusing existing connection to opendap.earthdata.nasa.gov:443.\n",
"HTTP request sent, awaiting response... 200 \n",
"Length: unspecified [text/html]\n",
"Saving to: ‘S6A_P4_2__LR_STD__NR_085_057_20230302T191818_20230302T210949_F07’\n",
"\n",
"S6A_P4_2__LR_STD__N [ <=> ] 2.31M 7.72MB/s in 0.3s \n",
"\n",
"2023-03-02 16:33:34 (7.72 MB/s) - ‘S6A_P4_2__LR_STD__NR_085_057_20230302T191818_20230302T210949_F07’ saved [2425870]\n",
"\n"
]
}
],
"source": [
"!wget 'https://opendap.earthdata.nasa.gov/collections/C1968979597-POCLOUD/granules/S6A_P4_2__LR_STD__NR_085_057_20230302T191818_20230302T210949_F07'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"An .nc file named S6A_P4_2__LR_STD__NR_085_057_20230302T191818_20230302T210949_F0 should download into your directory"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Step 6. Test Cloud OPeNDAP Access Works\n",
"Using netCDF4 library (https://unidata.github.io/netcdf4-python/), test that your OPeNDAP cloud set up is sucessful by running the script below. If it is successful, the OPeNDAP variables will be outputted. \n",
"\n",
"It is essential that the OPeNDAP URL used in testing is compatible with nc4 formats. If the OPeNDAP is not compatible with netCDF4, this script will yield an error message. "
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'netCDF4._netCDF4.Dataset'>\n",
"root group (NETCDF3_CLASSIC data model, file format DAP2):\n",
" acknowledgement: This research was carried out by the Jet Propulsion Laboratory, managed by the California Institute of Technology under a contract with the National Aeronautics and Space Administration.\n",
" author: Ian Fenty and Ou Wang\n",
" cdm_data_type: Grid\n",
" comment: Fields provided on a regular lat-lon grid. They have been mapped to the regular lat-lon grid from the original ECCO lat-lon-cap 90 (llc90) native model grid.\n",
" Conventions: CF-1.8, ACDD-1.3\n",
" coordinates: time_bnds latitude_bnds longitude_bnds\n",
" coordinates_comment: Note: the global 'coordinates' attribute describes auxillary coordinates.\n",
" creator_email: [email protected]\n",
" creator_institution: NASA Jet Propulsion Laboratory (JPL)\n",
" creator_name: ECCO Consortium\n",
" creator_type: group\n",
" creator_url: https://ecco-group.org\n",
" date_created: 2020-12-17T02:23:56\n",
" date_issued: 2020-12-17T02:23:56\n",
" date_metadata_modified: 2021-03-15T23:56:30\n",
" date_modified: 2021-03-15T23:56:30\n",
" geospatial_bounds_crs: EPSG:4326\n",
" geospatial_lat_max: 90.0\n",
" geospatial_lat_min: -90.0\n",
" geospatial_lat_resolution: 0.5\n",
" geospatial_lat_units: degrees_north\n",
" geospatial_lon_max: 180.0\n",
" geospatial_lon_min: -180.0\n",
" geospatial_lon_resolution: 0.5\n",
" geospatial_lon_units: degrees_east\n",
" history: Inaugural release of an ECCO Central Estimate solution to PO.DAAC\n",
" id: 10.5067/ECG5D-ATM44\n",
" institution: NASA Jet Propulsion Laboratory (JPL)\n",
" instrument_vocabulary: GCMD instrument keywords\n",
" keywords: EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC WINDS > SURFACE WINDS > WIND SPEED, EARTH SCIENCE > OCEANS > OCEAN WINDS > SURFACE WINDS, EARTH SCIENCE > OCEANS > OCEAN PRESSURE > SEA LEVEL PRESSURE, EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC WATER VAPOR > WATER VAPOR INDICATORS > HUMIDITY > SPECIFIC HUMIDITY, EARTH SCIENCE SERVICES > MODELS > EARTH SCIENCE REANALYSES/ASSIMILATION MODELS, EARTH SCIENCE > ATMOSPHERE > ATMOSPHERIC TEMPERATURE > SURFACE TEMPERATURE > AIR TEMPERATURE\n",
" keywords_vocabulary: NASA Global Change Master Directory (GCMD) Science Keywords\n",
" license: Public Domain\n",
" metadata_link: https://cmr.earthdata.nasa.gov/search/collections.umm_json?ShortName=ECCO_L4_ATM_STATE_05DEG_DAILY_V4R4\n",
" naming_authority: gov.nasa.jpl\n",
" platform: ERS-1/2, TOPEX/Poseidon, Geosat Follow-On (GFO), ENVISAT, Jason-1, Jason-2, CryoSat-2, SARAL/AltiKa, Jason-3, AVHRR, Aquarius, SSM/I, SSMIS, GRACE, DTU17MDT, Argo, WOCE, GO-SHIP, MEOP, Ice Tethered Profilers (ITP)\n",
" platform_vocabulary: GCMD platform keywords\n",
" processing_level: L4\n",
" product_name: ATM_SURFACE_TEMP_HUM_WIND_PRES_day_mean_2017-12-31_ECCO_V4r4_latlon_0p50deg.nc\n",
" product_time_coverage_end: 2018-01-01T00:00:00\n",
" product_time_coverage_start: 1992-01-01T12:00:00\n",
" product_version: Version 4, Release 4\n",
" program: NASA Physical Oceanography, Cryosphere, Modeling, Analysis, and Prediction (MAP)\n",
" project: Estimating the Circulation and Climate of the Ocean (ECCO)\n",
" publisher_email: [email protected]\n",
" publisher_institution: PO.DAAC\n",
" publisher_name: Physical Oceanography Distributed Active Archive Center (PO.DAAC)\n",
" publisher_type: institution\n",
" publisher_url: https://podaac.jpl.nasa.gov\n",
" references: ECCO Consortium, Fukumori, I., Wang, O., Fenty, I., Forget, G., Heimbach, P., & Ponte, R. M. 2020. Synopsis of the ECCO Central Production Global Ocean and Sea-Ice State Estimate (Version 4 Release 4). doi:10.5281/zenodo.3765928\n",
" source: The ECCO V4r4 state estimate was produced by fitting a free-running solution of the MITgcm (checkpoint 66g) to satellite and in situ observational data in a least squares sense using the adjoint method\n",
" standard_name_vocabulary: NetCDF Climate and Forecast (CF) Metadata Convention\n",
" summary: This dataset provides daily-averaged atmosphere surface temperature, humidity, wind, and pressure interpolated to a regular 0.5-degree grid from the ECCO Version 4 Release 4 (V4r4) ocean and sea-ice state estimate. Estimating the Circulation and Climate of the Ocean (ECCO) state estimates are dynamically and kinematically-consistent reconstructions of the three-dimensional, time-evolving ocean, sea-ice, and surface atmospheric states. ECCO V4r4 is a free-running solution of a global, nominally 1-degree configuration of the MIT general circulation model (MITgcm) that has been fit to observations in a least-squares sense. Observational data constraints used in V4r4 include sea surface height (SSH) from satellite altimeters [ERS-1/2, TOPEX/Poseidon, GFO, ENVISAT, Jason-1,2,3, CryoSat-2, and SARAL/AltiKa]; sea surface temperature (SST) from satellite radiometers [AVHRR], sea surface salinity (SSS) from the Aquarius satellite radiometer/scatterometer, ocean bottom pressure (OBP) from the GRACE satellite gravimeter; sea-ice concentration from satellite radiometers [SSM/I and SSMIS], and in-situ ocean temperature and salinity measured with conductivity-temperature-depth (CTD) sensors and expendable bathythermographs (XBTs) from several programs [e.g., WOCE, GO-SHIP, Argo, and others] and platforms [e.g., research vessels, gliders, moorings, ice-tethered profilers, and instrumented pinnipeds]. V4r4 covers the period 1992-01-01T12:00:00 to 2018-01-01T00:00:00.\n",
" time_coverage_duration: P1D\n",
" time_coverage_end: 2017-12-31T12:00:00\n",
" time_coverage_resolution: P1D\n",
" time_coverage_start: 2017-12-31T00:00:00\n",
" title: ECCO Atmosphere Surface Temperature, Humidity, Wind, and Pressure - Daily Mean 0.5 Degree (Version 4 Release 4)\n",
" uuid: f8578d30-4051-11eb-8ae0-0cc47a3f8727\n",
" dimensions(sizes): latitude(360), longitude(720), nv(2), time(1)\n",
" variables(dimensions): float32 EXFatemp(time, latitude, longitude), float32 EXFewind(time, latitude, longitude), float32 EXFwspee(time, latitude, longitude), float32 EXFnwind(time, latitude, longitude), float32 latitude_bnds(latitude, nv), float32 EXFpress(time, latitude, longitude), int32 time_bnds(time, nv), float32 EXFaqh(time, latitude, longitude), float32 longitude_bnds(longitude, nv), float32 latitude(latitude), float32 longitude(longitude), float32 nv(nv), int32 time(time)\n",
" groups: \n"
]
}
],
"source": [
"import netCDF4 as nc\n",
"FILE = \"https://opendap.earthdata.nasa.gov/providers/POCLOUD/collections/ECCO%20Atmosphere%20Surface%20Temperature%2C%20Humidity%2C%20Wind%2C%20and%20Pressure%20-%20Daily%20Mean%200.5%20Degree%20(Version%204%20Release%204)/granules/ATM_SURFACE_TEMP_HUM_WIND_PRES_day_mean_2017-12-31_ECCO_V4r4_latlon_0p50deg\"\n",
"ds = nc.Dataset(FILE, mode=\"r\")\n",
"\n",
"print(ds)"
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"ds.close()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.6"
}
},
"nbformat": 4,
"nbformat_minor": 1
}
Loading