Getting Started with the NetCDF Diag Files

[esheehan-gsl]

@ian-noaa since I’m out next week, I thought it’d be good to leave you with the steps I used to start looking into the NetCDF files, even though I didn’t get to look at them very closely. If you have a different approach you prefer, feel free to ignore this.

I downloaded and extracted the archive that Guoqing shared in Drive into tmp/data (tmp/ is in .gitignore so you don’t have to worry about committing the NetCDF files). Then I fired up JupyterLab in a Docker container.

docker run -it --name ug_jupyter \
    -v "$(pwd)/tmp/:/home/jovyan/work" \
    -p 8888:8888 \
    jupyter/scipy-notebook \
    start.sh jupyter lab --LabApp.token=''

This will make JupyterLab available to you on http://localhost:8888/ (the token thing just ensures you don’t have to copy any security tokens, since you’re running in a secured environment).

Inside of JupyterLab, you’ll need to open a new terminal and install xarray and netcdf4.

pip install xarray netcdf4

Then you can open either a Python console or a new Python notebook and begin exploring the NetCDF files.

import xarray as xr
diag = xr.open_dataset("data/ncdiag_conv_ps_anl.nc4.2022050514")
diag

That should load the file so that you can start exploring it. If you save your notebooks, they’ll end up in tmp/ (so they won’t be committed).

[esheehan-gsl]

@guoqing-noaa @terraladwig @christinaholtNOAA now that I’m back from vacation and sick leave, I’m making some headway on these NetCDF diag files, I think. I’ve got a list of variables from the NetCDF file; I have a guess as to what some of them mean, but not others, so I’m going to list them here and you can hopefully fill in the gaps for us and correct anything I’ve misunderstood.

Here are my guesses:

• Analysis_Use_Flag: 1 if the observation was assimilated, 0 if it was rejected (or is this an enumerated type with three values, one for assimilated, rejected, or monitored?)
• Forecast_adjusted: forecast value from the background, interpolated to the observation site from the forecast grid
• Forecast_unadjusted: original forecast value from the background on the forecast grid
• Height: height of the observation (in what units?)
• Latitude: latitude of the observation site
• Longitude: longitude of the observation site
• Obs_Minus_Forecast_adjusted: difference between the observed value and the adjusted background value
• Obs_Minus_Forecast_unadjusted: difference between the observed value and the original background value (on the forecast grid)
• Observation: observed value
• Pressure: Pressure at the observation site
• Time: time of the observation

And here are the terms for which I don’t have a good guess:

• Bias_Correction_Terms:
• Data_Pof:
• Data_Vertical_Velocity:
• Dominant_Sfc_Type:
• Errinv_Adjust:
• Errinv_Final:
• Errinv_Input:
• Model_Terrain:
• Nonlinear_QC_Rel_Wgt:
• Observation_Class:
• Observation_Subtype:
• Observation_Type:
• Prep_QC_Mark:
• Prep_Use_Flag:
• Provider_Name:
• Setup_QC_Mark:
• Station_Elevation:
• Station_ID:
• Subprovider_Name:
• atmosphere_pressure_coordinate:

Assuming that my guesses are correct, and that one of the first things we want to do is display a histogram of obs - bg like we see in this grid of distributions that @guoqing-noaa shared with us over in #15:

Then the fields that we need are Analysis_Use_Flag and Obs_Minus_Forecast_adjusted. We use Analysis_Use_Flag to isolate assimilated, rejected (and perhaps monitored) observations to create each of the distributions of Obs_Minus_Forecast_adjusted.

As a sanity check, I created this histogram from ncdiag_conv_t_anl.nc4.2022050514 for Obs_Minus_Forecast_adjusted (I did not use the Analysis_Use_Flag, so this is for all observations). I limited the range of values in the histogram to [-8, 8] because it looks like it has long tails in both directions (I have a min value of -93.7 and a max value of 72.2 for Obs_Minus_Forecast_adjusted), and that seemed to be the domain used in the sample image above.

Does this seem reasonable?

Once we iron out our understanding of what’s in these files, we should be able to start progressing rapidly on making these histograms available in the application.

[guoqing-noaa]

See below for my interpretation of variables presented in NetCDF diag files:

Analysis_Use_Flag: in NetCDF diag files, it takes only 1 or -1, where 1 means assimilated while -1 means not-assimilated.
Forecast_unadjusted: forecast value interpolated to the observation site from the forecast grid
Obs_Minus_Forecast_unadjusted: OMF, observation minus forecast_unadjusted, (or OMB, observtion minus background_adjusted)
Obs_Minus_Forecast_adjusted: similar as the above OMF, but the difference is adjusted through different procedures, such as bias correction, error tuning, obs perturbing, etc.
Forecast_adjusted: it is observation - OMF_adjusted (Obs_Minus_Forecast_adjusted)
Station_Elevation: station elevation (meters)
Station_ID: station ID (or name)
Height: height of the observation (meters)
Pressure: Pressure at the observation site (mb)
Time: observation time (hours relative to analysis time, e.g. - 0.25 indicates 15 minutes before the nominal analysis time. Be careful that for subhourly files, the nominal analysis time is NOT hourly, it might be 20220615_0815z)
Errinv_Input: inversed observation error, the original one read from the prepbufr files.
Errinv_Adjust: inversed observation error, adjusted (such as inflation, etc)
Errinv_Final: inversed observation error, final, used by GSI
Observation_Class: the observation class, a string, such as " t"
observation type: observation type, e.g. 187 means surface METAR observations. Need to reference a PrepBufr table to understand each type
Observation_Subtype: observation subtype, some observation types need subtypes to be further categoried.
Prep_QC_Mark: Quality Control (QC) mark, directly read from PrepBufr files
Prep_Use_Flag: data use flag determined during PrepBufr file reading
Provider_Name: data provider name, a string
Subprovider_Name: sub provider name, a string
Setup_QC_Mark: QC marks, only meaningful for Temperature (indicating virtual temperature or not), wind and GPS RO observations.




Data_Pof: Used by aircraft observations, 5 (ascending); 6 (descending); 3 (cruise level)
Data_Vertical_Velocity: data vertical velocity, used by aircraft observations
Bias_Correction_Terms: bias correction value, used by aircraft observations


Dominant_Sfc_Type: dominant land surface type determined from surrounding land surface types (used by 2D-RTMA), 0 sea, 1 land 2 sea ice 3 snow 4 mixed
Model_Terrain: Model terrain height interpolated to the obs location (Used by 2D-RTMA)


Nonlinear_QC_Rel_Wgt: non-linear qc relative weight
atmosphere_pressure_coordinate: this is used by JEDI only.

[esheehan-gsl]

Quick question about the values of longitude: I’m seeing values ranging from 225–299, are these degrees East? If the extent of the longitude is 225–299°E, I would subtract those numbers from 360° to get degrees west, so that I have a range of 61–135°W, is that correct?

[terraladwig]

Yes, range of 61 to 135 degrees west makes sense for data over CONUS.