Skip to content
This repository has been archived by the owner on Aug 29, 2023. It is now read-only.

UC9 CLI Demo

Jump to bottom
Krzysztof (Chris) Bernat edited this page Feb 23, 2017 · 26 revisions

Use Case 9

Problem Definition:

A climate scientist wishes to analyse potential correlations between Ozone and Cloud ECVs.

Required Toolbox Features:

  • Access to and ingestion of ESA CCI Ozone and Cloud data (Atmosphere Mole Content of Ozone and Cloud Cover)
  • Geometric adjustments (coregistration)
  • Spatial (point, polygon) and temporal subsetting
  • Visualisation of time series
  • Correlation analysis, scatter-plot of correlation statistics, saving of image and correlation statistics

1. Data product ingestion

Use ds list to list available products. You can filter them according to some name.

cate ds list -n ozone
3 data sources found
   1: esacci.OZONE.day.L3S.TC.GOME-2.Metop-A.MERGED.fv0100.r1
   2: esacci.OZONE.day.L3S.TC.GOME.ERS-2.MERGED.fv0100.r1
   3: esacci.OZONE.mon.L3.NP.multi-sensor.multi-platform.MERGED.fv0002.r1

cate ds list -n cloud
14 data sources found
   1: esacci.CLOUD.day.L3U.CLD_PRODUCTS.AVHRR.NOAA-15.AVHRR_NOAA.1-0.r1
   2: esacci.CLOUD.day.L3U.CLD_PRODUCTS.AVHRR.NOAA-16.AVHRR_NOAA.1-0.r1
   3: esacci.CLOUD.day.L3U.CLD_PRODUCTS.AVHRR.NOAA-17.AVHRR_NOAA.1-0.r1
   4: esacci.CLOUD.day.L3U.CLD_PRODUCTS.AVHRR.NOAA-18.AVHRR_NOAA.1-0.r1
   5: esacci.CLOUD.day.L3U.CLD_PRODUCTS.MODIS.Aqua.MODIS_AQUA.1-0.r1
   6: esacci.CLOUD.day.L3U.CLD_PRODUCTS.MODIS.Terra.MODIS_TERRA.1-0.r1
   7: esacci.CLOUD.mon.L3C.CLD_PRODUCTS.AVHRR.NOAA-15.AVHRR_NOAA.1-0.r1
   8: esacci.CLOUD.mon.L3C.CLD_PRODUCTS.AVHRR.NOAA-16.AVHRR_NOAA.1-0.r1
   9: esacci.CLOUD.mon.L3C.CLD_PRODUCTS.AVHRR.NOAA-17.AVHRR_NOAA.1-0.r1
  10: esacci.CLOUD.mon.L3C.CLD_PRODUCTS.AVHRR.NOAA-18.AVHRR_NOAA.1-0.r1
  11: esacci.CLOUD.mon.L3C.CLD_PRODUCTS.MODIS.Aqua.MODIS_AQUA.1-0.r1
  12: esacci.CLOUD.mon.L3C.CLD_PRODUCTS.MODIS.Terra.MODIS_TERRA.1-0.r1
  13: esacci.CLOUD.mon.L3S.CLD_PRODUCTS.AVHRR.multi-platform.AVHRR_MERGED.1-0.r1
  14: esacci.CLOUD.mon.L3S.CLD_PRODUCTS.MODIS.multi-platform.MODIS_MERGED.1-0.r1

Create a new workspace.

cate ws new

INFO:tornado.access:200 GET / (127.0.0.1) 0.50ms
INFO:tornado.access:200 GET /ws/new?description=&base_dir=%2Fhome%2Fccitbx%2FDesktop&save=False (127.0.0.1) 0.66ms
Workspace created.

Open the desired datasets, by providing their name and desired time-span.

cate res open cl07 esacci.CLOUD.mon.L3C.CLD_PRODUCTS.AVHRR.NOAA-17.AVHRR_NOAA.1-0.r1 2007-01-01 2007-12-30

INFO:tornado.access:200 GET / (127.0.0.1) 0.48ms
Resource "cl07" set.
INFO:tornado.access:200 POST /ws/res/set/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core/cl07 (127.0.0.1) 1386.67ms
cate res open oz07 esacci.OZONE.mon.L3.NP.multi-sensor.multi-platform.MERGED.fv0002.r1 2007-01-01 2007-12-30

INFO:tornado.access:200 GET / (127.0.0.1) 0.49ms
Resource "oz07" set.
INFO:tornado.access:200 POST /ws/res/set/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core/oz07 (127.0.0.1) 349.86ms

2. Dataset variable selection

To select particular geophysical quantities to work with, use the 'select_var' operation together with cate res set command:

cate res set cc_tot select_var ds=cl07 var=cc_total

INFO:tornado.access:200 GET / (127.0.0.1) 0.60ms
Executing 2 workflow step(s): done   
Resource "cc_tot" set.
INFO:tornado.access:200 POST /ws/res/set/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core/cc_tot (127.0.0.1) 7.61ms
cate res set oz_tot select_var ds=oz07 var=O3_du_tot

INFO:tornado.access:200 GET / (127.0.0.1) 0.54ms
Executing 2 workflow step(s): done   
INFO:tornado.access:200 POST /ws/res/set/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core/oz_tot (127.0.0.1) 1.84ms
Resource "oz_tot" set.

We can plot the datasets and save the plots using the plot_map operation:

cate ws run plot_map ds=cc_tot var=cc_total file=fig1.png

INFO:tornado.access:200 GET / (127.0.0.1) 0.70ms
Running operation 'plot_map': [------------------------------]   0% Executing 4 workflow 
Running operation 'plot_map': Executing 4 workflow step(s)                                      
Operation 'plot_map' executed.
INFO:tornado.access:200 POST /ws/run_op/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core (127.0.0.1) 2065.44ms

cate ws run plot_map ds=oz_tot var=O3_du_tot file=fig2.png

INFO:tornado.access:200 GET / (127.0.0.1) 0.68ms
Running operation 'plot_map': Executing 4 workflow step(s)                                      
Operation 'plot_map' executed.
INFO:tornado.access:200 POST /ws/run_op/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core (127.0.0.1) 492.28ms

3. Co-Register the datasets

The datasets now have different lat/lon definitions. This can be verified by using cate res print

cate res print cc_tot

INFO:tornado.access:200 GET / (127.0.0.1) 0.59ms
<xarray.Dataset>
Dimensions:       (hist_cot: 7, hist_cot_bin: 6, hist_ctp: 8, hist_ctp_bin: 7, hist_phase: 2, lat: 360, lon: 720, time: 12)
Coordinates:
  * lat           (lat) float32 -89.75 -89.25 -88.75 -88.25 -87.75 -87.25 ...
  * lon           (lon) float32 -179.75 -179.25 -178.75 -178.25 -177.75 ...
  * hist_cot      (hist_cot) float32 0.3 1.3 3.6 9.4 23.0 60.0 100.0
  * hist_cot_bin  (hist_cot_bin) float32 1.0 2.0 3.0 4.0 5.0 6.0
  * hist_ctp      (hist_ctp) float32 1100.0 800.0 680.0 560.0 440.0 310.0 ...
  * hist_ctp_bin  (hist_ctp_bin) float32 1.0 2.0 3.0 4.0 5.0 6.0 7.0
  * hist_phase    (hist_phase) int32 0 1
  * time          (time) float64 2.454e+06 2.454e+06 2.454e+06 2.454e+06 ...
Data variables:
    cc_total      (time, lat, lon) float64 0.1076 0.3423 0.2857 0.2318 ...
INFO:tornado.access:200 GET /ws/res/print/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core?res_name_or_expr=cc_tot (127.0.0.1) 16.48ms
cate res print oz_tot

INFO:tornado.access:200 GET / (127.0.0.1) 0.48ms
<xarray.Dataset>
Dimensions:       (air_pressure: 17, lat: 180, layers: 16, lon: 360, time: 12)
Coordinates:
  * lon           (lon) float32 -179.5 -178.5 -177.5 -176.5 -175.5 -174.5 ...
  * lat           (lat) float32 -89.5 -88.5 -87.5 -86.5 -85.5 -84.5 -83.5 ...
  * layers        (layers) int32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
  * air_pressure  (air_pressure) float32 1013.0 446.05 196.35 113.63 65.75 ...
  * time          (time) datetime64[ns] 2007-01-04 2007-02-01 2007-03-01 ...
Data variables:
    O3_du_tot     (time, lat, lon) float32 260.176 264.998 267.394 265.048 ...
INFO:tornado.access:200 GET /ws/res/print/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core?res_name_or_expr=oz_tot (127.0.0.1) 9.27ms
cate op list --tag geom

2 operations found
   1: coregister
   2: subset_spatial

will list all commands that have a tag that matches 'geom'. To find out more about a particular operation, use cate op info

cate op info coregister

Operation cate.ops.coregistration.coregister
===========================================

Perform coregistration of two datasets by resampling the slave dataset unto the
grid of the master. If upsampling has to be performed, this is achieved using
interpolation, if downsampling has to be performed, the pixels of the slave dataset
are aggregated to form a coarser grid.

This operation works on datasets whose spatial dimensions are defined on global,
pixel-registered and equidistant in lat/lon coordinates grids. E.g., data points
define the middle of a pixel and pixels have the same size across the dataset.

This operation will resample all variables in a dataset, as the lat/lon grid is
defined per dataset. It works only if all variables in the dataset have (time/lat/lon)
dimensions.

For an overview of downsampling/upsampling methods used in this operation, please
see https://github.com/CAB-LAB/gridtools

Whether upsampling or downsampling has to be performed is determined automatically
based on the relationship of the grids of the provided datasets.

Inputs:
  ds_master (Dataset)
      The dataset whose grid is used for resampling
  ds_slave (Dataset)
      The dataset that will be resampled
  method_us (str)
      Interpolation method to use for upsampling.
      default value: linear
      value set: ['nearest', 'linear']
  method_ds (str)
      Interpolation method to use for downsampling.
      default value: mean
      value set: ['first', 'last', 'mean', 'mode', 'var', 'std']

Output:
  return (Dataset)
      The slave dataset resampled on the grid of the master

To carry out coregistration, use cate res set again with appropriate operation parameters

cate res set cc_tot_res coregister ds_master=oz_tot ds_slave=cc_tot

INFO:tornado.access:200 GET / (127.0.0.1) 0.50ms
Executing 5 workflow step(s): done   
Resource "cc_tot_res" set.
INFO:tornado.access:200 POST /ws/res/set/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core/cc_tot_res (127.0.0.1) 2964.39ms
cate ws run plot_map ds=cc_tot_res var=cc_total file=fig3.png

INFO:tornado.access:200 GET / (127.0.0.1) 0.50ms
Running operation 'plot_map': [------------------------------]   0% Executing 5 workflow step(s)/home/ccitbx/miniconda3/envs/ect_env/lib/python3.5/site-packages/matplotlib/artist.py:221: MatplotlibDeprecationWarning: This has been deprecated in mpl 1.5, please use the
axes property.  A removal date has not been set.
  warnings.warn(_get_axes_msg, mplDeprecation, stacklevel=1)
Running operation 'plot_map': Executing 5 workflow step(s)                                      
Operation 'plot_map' executed.
INFO:tornado.access:200 POST /ws/run_op/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core (127.0.0.1) 661.99ms

4. Spatial Filtering

To filter the datasets to contain only a particular region use the subset_spatial operation.

cate res set oz_africa subset_spatial ds=oz_tot lat_min=-40 lat_max=40 lon_min=-20 lon_max=60

INFO:tornado.access:200 GET / (127.0.0.1) 0.49ms
Executing 3 workflow step(s): done   
Resource "oz_africa" set.
INFO:tornado.access:200 POST /ws/res/set/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core/oz_africa (127.0.0.1) 5.55ms
cate res set cc_africa subset_spatial ds=cc_tot_res lat_min=-40 lat_max=40 lon_min=-20 lon_max=60

INFO:tornado.access:200 GET / (127.0.0.1) 0.61ms
Executing 6 workflow step(s): done   
Resource "cc_africa" set.
INFO:tornado.access:200 POST /ws/res/set/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core/cc_africa (127.0.0.1) 6.69ms
cate ws run plot_map ds=cc_africa var=cc_total file=fig4.png

INFO:tornado.access:200 GET / (127.0.0.1) 0.55ms
Running operation 'plot_map': Executing 7 workflow step(s)                                      
Operation 'plot_map' executed.
INFO:tornado.access:200 POST /ws/run_op/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core (127.0.0.1) 512.15ms

cate ws run plot_map ds=cc_africa var=cc_total lat_min=-40 lat_max=40 lon_min=-20 lon_max=60 file=fig5.png

INFO:tornado.access:200 GET / (127.0.0.1) 0.63ms
Running operation 'plot_map': Executing 7 workflow step(s)                                      
Operation 'plot_map' executed.
INFO:tornado.access:200 POST /ws/run_op/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core (127.0.0.1) 451.17ms

5. Temporal Filtering

To further filter the datasets to contain only a particular time-span, use subset_temporal operation

cate res set oz_africa_janoct subset_temporal ds=oz_africa time_min='2007-01-01' time_max='2007-10-30'
cate res set cc_africa_janoct subset_temporal ds=cc_africa time_min='2007-01-01' time_max='2007-10-30'

If on Linux, quotes enclosing datetime strings should be additionally escaped:

cate res set oz_africa_janoct subset_temporal ds=oz_africa time_min=\'2007-01-01\' time_max=\'2007-10-30\'

INFO:tornado.access:200 GET / (127.0.0.1) 0.52ms
Executing 4 workflow step(s): done   
Resource "oz_africa_janoct" set.
INFO:tornado.access:200 POST /ws/res/set/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core/oz_africa_janoct (127.0.0.1) 8.37ms
cate res set cc_africa_janoct subset_temporal ds=cc_africa time_min=\'2007-01-01\' time_max=\'2007-10-30\'

INFO:tornado.access:200 GET / (127.0.0.1) 0.60ms
Executing 7 workflow step(s): done   
Resource "cc_africa_janoct" set.
INFO:tornado.access:200 POST /ws/res/set/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core/cc_africa_janoct (127.0.0.1) 5.78ms

6. Extract time series

We'll extract spatial mean timeseries from both datasets using tseries_mean operation.

cate res set cc_africa_ts tseries_mean ds=cc_africa_janoct var=cc_total

INFO:tornado.access:200 GET / (127.0.0.1) 0.80ms
Executing 8 workflow step(s): done   
Resource "cc_africa_ts" set.
INFO:tornado.access:200 POST /ws/res/set/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core/cc_africa_ts (127.0.0.1) 13.74ms
cate res set oz_africa_ts tseries_mean ds=oz_africa_janoct var=O3_du_tot

INFO:tornado.access:200 GET / (127.0.0.1) 0.52ms
Executing 5 workflow step(s): done   
Resource "oz_africa_ts" set.
INFO:tornado.access:200 POST /ws/res/set/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core/oz_africa_ts (127.0.0.1) 9.68ms

This creates datasets that contain mean and std variables for both time-series.

7. Time Series Plot

To plot the time-series and save the plot plot_1D operation can be used together with cate ws run operation:

cate ws run plot_1D ds=cc_africa_ts var=cc_total file=fig6.png

INFO:tornado.access:200 GET / (127.0.0.1) 0.59ms
Running operation 'plot_1D': Executing 11 workflow step(s)                                      
Operation 'plot_1D' executed.
INFO:tornado.access:200 POST /ws/run_op/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core (127.0.0.1) 543.42ms

cate ws run plot_1D ds=oz_africa_ts var=O3_du_tot file=fig7.png

INFO:tornado.access:200 GET / (127.0.0.1) 0.47ms
Running operation 'plot_1D': Executing 11 workflow step(s)                                      
Operation 'plot_1D' executed.
INFO:tornado.access:200 POST /ws/run_op/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core (127.0.0.1) 395.39ms
(ect_env)ccitbx@ccitbx-VirtualBox:~/Development/cate-core$

8. Product-Moment Correlation (Pearson)

To carry out Pearson correlation on the mean time-series, pearson_correlation operation can be used.

cate op list --tag correlation

One operation found
   1: pearson_correlation
cate res set pearson pearson_correlation ds_y=cc_africa_ts ds_x=oz_africa_ts var_y=cc_total var_x=O3_du_tot file=pearson.txt

INFO:tornado.access:200 GET / (127.0.0.1) 0.49ms
Executing 12 workflow step(s): done   
Resource "pearson" set.
INFO:tornado.access:200 POST /ws/res/set/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core/pearson (127.0.0.1) 32.45ms

This will calculate the correlation coefficient along with the associated p_value for both mean time-series, as well as save the information in the given file. We can view the result using cate res print:

cate res print pearson

INFO:tornado.access:200 GET / (127.0.0.1) 0.54ms
<xarray.Dataset>
Dimensions:    ()
Coordinates:
    *empty*
Data variables:
    corr_coef  float64 -0.2924
    p_value    float64 0.4123
Attributes:
    ECT_Description: Correlation between cc_total O3_du_tot

If both variables provided to the pearson_correlation operation have time/lat/lon dimensions and the lat/lon definition is the same, a pixel by pixel correlation will be carried out and result in the creation of two variables of the same lat/lon dimension - corr_coeff and p_value that can then be plotted on a map.

cate res set pearson_map pearson_correlation ds_y=cc_africa_janoct ds_x=oz_africa_janoct var_y=cc_total var_x=O3_du_tot

INFO:tornado.access:200 GET / (127.0.0.1) 0.56ms
Executing 10 workflow step(s): done   
Resource "pearson_map" set.
INFO:tornado.access:200 POST /ws/res/set/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core/pearson_map (127.0.0.1) 6491.09ms
cate ws run plot_map ds=pearson_map var=corr_coef lat_min=-40 lat_max=40 lon_min=-20 lon_max=60 file=fig8.png

INFO:tornado.access:200 GET / (127.0.0.1) 0.62ms
Running operation 'plot_map': [------------------------------]   0% Executing 13 workflow step(s)/home/ccitbx/miniconda3/envs/ect_env/lib/python3.5/site-packages/matplotlib/artist.py:221: MatplotlibDeprecationWarning: This has been deprecated in mpl 1.5, please use the
axes property.  A removal date has not been set.
  warnings.warn(_get_axes_msg, mplDeprecation, stacklevel=1)
Running operation 'plot_map': Executing 13 workflow step(s)                                      
Operation 'plot_map' executed.
INFO:tornado.access:200 POST /ws/run_op/%2Fhome%2Fccitbx%2FDevelopment%2Fect-core (127.0.0.1) 458.77ms

Clone this wiki locally