get_hycom_3hr.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Python script to extract 3-hourly hycom model hindcast output\n",
        "\n",
        "This script downloads 3-hourly hycom hindcast data of sea surface elevation, and water temperature, salinity, u-velocity, and v-velocity at all model depths for any selected ocean sub-area and for any selected time period between 1994 to present. HYCOM data are downloaded from www.hycom.org and saved in netcdf files on your google drive.\n",
        "___\n",
        "\n",
        "Adapted from a LiveOcean script by Parker MacCready (https://github.com/parkermac/LiveOcean)\n",
        "\n",
        "Modified by Greg Pelletier (gjpelletier@gmail.com) for standalone use to download 3-hourly hycom data (https://github.com/gjpelletier/get_hycom)\n",
        "\n",
        "___\n",
        "\n",
        "INSTRUCTIONS\n",
        "\n",
        "Specify the following in the code sections below:\n",
        "  - list of variables to be extracted from any combination of var_list = 'surf_el,water_temp,salinity,water_u,water_v'\n",
        "  - west, east, south, and north extent of the ocean sub-area where data will be extracted\n",
        "  - name of the resultDirectory where the hycom data will be saved as output\n",
        "  - the date_start and number_of_days of the time period to be extracted, and corresponding hycom codes for the model glb and expt\n",
        "\n",
        "During execution you should see the progress of each 3-hourly file that is extracted during the period of interest from beginning to end. Each nc file name has the format yyyyMMdd_HH.nc to indicate the datetime stamp in UTC\n",
        "\n",
        "\n",
        "___\n",
        "\n",
        "Import the required python packages:\n"
      ],
      "metadata": {
        "id": "oNVN8kLuLBWI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import sys\n",
        "from datetime import *\n",
        "import time\n",
        "from urllib.request import urlretrieve\n",
        "from urllib.error import URLError\n",
        "from socket import timeout"
      ],
      "metadata": {
        "id": "JbMGcYTRLCHc"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Mount your google drive folder to make it possible to store the output nc files of 3-hourly data in your google drive:"
      ],
      "metadata": {
        "id": "17kPphF8MAYQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "id": "cmrh2bIxMA8d",
        "outputId": "a2cbbc71-fad4-42a1-8a67-ec415dcb79e6"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Specify the name of the resultDirectory folder where the output files will be saved in your mounted google drive folder. Edit the name of resultDirectory below to use any name you want, as long as you start the name with 'drive/MyDrive/' if you are using Google Colab. This subfolder will be created by the script later if it does not already exist in your google drive:"
      ],
      "metadata": {
        "id": "mNB9loQQMRV5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "resultDirectory = 'drive/MyDrive/Colab Notebooks/hycom/'   # include the ending '/' "
      ],
      "metadata": {
        "id": "FKx5ZcETMRlY"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Specify the date_start in ISO format for the starting datetime when the data to be extracted. \n",
        "\n",
        "The starting hour must be either 00, 03, 06, 09, 12, 15, 18, or 21, or must be 12 if the date_start is Jan 1 of the year or first day of the expt. The date_start must be within the range of dates for the glb and expt as described at www.hycom.org\n",
        "\n",
        "Also specify the number_of_days of 3-hourly data to be downloaded. There will be separate output nc files downloaded for eight consecutive 3-hourly datetimes for each day in the number_of_days. For example, if number_of_days=1 there will be 8 nc files, if number_of_days=7 there will be 56 nc files, etc.\n",
        "\n",
        "Each output nc file name will be generated by the script and will have the format yyyyMMdd_HH.nc to indicate the datetime stamp in UTC \n",
        "\n",
        "You can download up to one year of 3-hourly data at a time from any given calendar year. Note that it can take between about 10 seconds up to over a minute for each 3-hourly file to download. In other words, if number_of_days=1 then it will take up to a few minutes, and if number_of_days is 365 then it will take up to several hours to download all of the 3-hourly nc files.\n",
        "\n",
        "Also note that some experiments may have some missing 3-hourly times which can cause the script to get stuck in those places. If that happens you can begin the script again with a new date_start to resume again starting with the first non-missing date_start."
      ],
      "metadata": {
        "id": "7LQtrZimRyRf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "date_start = '2020-01-01 12:00:00'      \n",
        "number_of_days  = 1                     "
      ],
      "metadata": {
        "id": "-Z9Yl5QeRzO6"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Specify the HYCOM codes for glb and expt corresponding to the time period that will be downloaded\n",
        "\n",
        "The following list shows the correct correct glb and expt to use for the dates to be downloaded (more info on the glb and expt is available at www.hycom.org if needed):\n",
        "\n",
        "*   Use glb = 'GLBv0.08' and expt = '53.X' for dates between 1994-2015\n",
        "*   Use glb = 'GLBv0.08' and expt = '56.3' for dates between 1/1/2016 or 7/1/2014 to 4/30/2016\n",
        "*   Use glb = 'GLBv0.08' and expt = '57.2' for dates between 5/1/2016 to 1/31/2017\n",
        "*   Use glb = 'GLBv0.08' and expt = '92.8' for dates between 2/1/2017 to 5/31/2017\n",
        "*   Use glb = 'GLBv0.08' and expt = '57.7' for dates between 6/1/2017 to 9/30/2017\n",
        "*   Use glb = 'GLBv0.08' and expt = '92.9' for dates between 10/1/2017 to 12/31/2017Use glb = 'GLBv0.08' and expt = '92.9' for dates between 10/1/2017 to 12/31/2017\n",
        "*   Use glb = 'GLBv0.08' and expt = '93.0' for dates between 1/1/2018 to 12/31/2018 or 2/18/2020\n",
        "*   Use glb = 'GLBy0.08' and expt = '93.0' for dates between 2019-present\n",
        "\n"
      ],
      "metadata": {
        "id": "FG4sYf9gN3xD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "glb = 'GLBy0.08'                     \n",
        "expt = '93.0'                        "
      ],
      "metadata": {
        "id": "-_IabiD2O382"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Specify spatial limits (default below is Parker MacCready's HYCOM bounding box for the boundary of the LiveOcean model):"
      ],
      "metadata": {
        "id": "kVknszGdOigL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "north = 53              # -80 to 80 degN          \n",
        "south = 39              # -80 to 80 degN\n",
        "west = -131 + 360       # 0 to 360 degE\n",
        "east = -121 + 360       # 0 to 360 degE"
      ],
      "metadata": {
        "id": "z1yUAqs1OizP"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Edit the following var_list as needed to download any subset of these available variables:"
      ],
      "metadata": {
        "id": "fqH1zO1KOu6a"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "var_list = 'surf_el,water_temp,salinity,water_u,water_v'"
      ],
      "metadata": {
        "id": "7hSN-e-JOvJw"
      },
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Make a function to create a directory if it does not already exist:"
      ],
      "metadata": {
        "id": "fLHH32O0T5z4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def ensure_dir(file_path):\n",
        "    # create a folder if it does not already exist\n",
        "    directory = os.path.dirname(file_path)\n",
        "    if not os.path.exists(directory):\n",
        "        os.makedirs(directory)"
      ],
      "metadata": {
        "id": "_aJZAAa2T6De"
      },
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Make a function to extract the hycom data during the loop through a list of all datetimes to be extracted:"
      ],
      "metadata": {
        "id": "1u6r3JaLUGt0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def get_extraction(dt, out_fn, var_list):\n",
        "    dstr0 = dt.strftime('%Y-%m-%d-T%H:00:00Z')\n",
        "    print(dstr0)\n",
        "    if expt == '53.X':\n",
        "        url = ('http://ncss.hycom.org/thredds/ncss/' + glb + '/expt_' + expt + '/data/' + dt.strftime('%Y') + \n",
        "            '?var='+var_list +\n",
        "            '&north='+str(north)+'&south='+str(south)+'&west='+str(west)+'&east='+str(east) +\n",
        "            '&disableProjSubset=on&horizStride=1' +\n",
        "            '&time_start='+dstr0+'&time_end='+dstr0+'&timeStride=8' +\n",
        "            '&vertCoord=&addLatLon=true&accept=netcdf4')\n",
        "    else:\n",
        "        url = ('http://ncss.hycom.org/thredds/ncss/' + glb + '/expt_' + expt + \n",
        "            '?var='+var_list +\n",
        "            '&north='+str(north)+'&south='+str(south)+'&west='+str(west)+'&east='+str(east) +\n",
        "            '&disableProjSubset=on&horizStride=1' +\n",
        "            '&time_start='+dstr0+'&time_end='+dstr0+'&timeStride=8' +\n",
        "            '&vertCoord=&addLatLon=true&accept=netcdf4')\n",
        "    # get the data and save as a netcdf file\n",
        "    counter = 1\n",
        "    got_file = False\n",
        "    while (counter <= 10) and (got_file == False):\n",
        "        print('  Attempting to get data, counter = ' + str(counter))\n",
        "        tt0 = time.time()\n",
        "        try:\n",
        "            (a,b) = urlretrieve(url, out_fn)\n",
        "            # a is the output file name\n",
        "            # b is a message you can see with b.as_string()\n",
        "        except URLError as ee:\n",
        "            if hasattr(ee, 'reason'):\n",
        "                print('  *We failed to reach a server.')\n",
        "                print('  -Reason: ', ee.reason)\n",
        "            elif hasattr(ee, 'code'):\n",
        "                print('  *The server could not fulfill the request.')\n",
        "                print('  -Error code: ', ee.code)\n",
        "        except timeout:\n",
        "            print('  *Socket timed out')\n",
        "        else:\n",
        "            got_file = True\n",
        "            print('  Downloaded data')\n",
        "        print('  Time elapsed: %0.1f seconds' % (time.time() - tt0))\n",
        "        counter += 1\n",
        "    if got_file:\n",
        "        result = 'success'\n",
        "    else:\n",
        "        result = 'fail'\n",
        "    return result"
      ],
      "metadata": {
        "id": "al3DlRroUG8f"
      },
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Make a dt_list of all of the 3-hourly datetimes to extract from hycom:"
      ],
      "metadata": {
        "id": "f502We_tUHTQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "base = datetime.fromisoformat(date_start)\n",
        "if base.strftime('%H') == '12' and number_of_days >= 365:\n",
        "    ndt = number_of_days * 8 - 4\n",
        "else:\n",
        "    ndt = number_of_days * 8\n",
        "dt_list = []\n",
        "dt_list = [base + timedelta(hours=3*x) for x in range(ndt)]"
      ],
      "metadata": {
        "id": "lcbGFOPpUHi0"
      },
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Loop through all of the datetimes in dt_list and download all of the nc files for the number_of_days of 3-hourly data:"
      ],
      "metadata": {
        "id": "jFN6xvz9UmKG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "out_dir = resultDirectory                  # specify output directory adding the ending '/'\n",
        "ensure_dir(out_dir)                        # make sure the output directory exists, make one if not\n",
        "f = open(out_dir + 'log.txt', 'w+')        # open log of successful downloads\n",
        "print('\\n** Working on ' + glb + '/expt_' + expt + ' **')\n",
        "f.write('\\n\\n** Working on ' + glb + '/expt_' + expt + ' **')\n",
        "tt1 = time.time()                          # tic for total elapsed time\n",
        "force_overwrite = True                     # overwrite any already existing nc files in the output folder\n",
        "for dt in dt_list:\n",
        "    out_fn = out_dir + datetime.strftime(dt, '%Y%m%d_%H') + '.nc'\n",
        "    print(out_fn)\n",
        "    if os.path.isfile(out_fn):\n",
        "        if force_overwrite:\n",
        "            os.remove(out_fn)\n",
        "    if not os.path.isfile(out_fn):\n",
        "        result = get_extraction(dt, out_fn, var_list)\n",
        "        f.write('\\n ' + datetime.strftime(dt, '%Y%m%d_%H') + ' ' + result)\n",
        "\n",
        "totmin = (time.time() - tt1)/60             # total time elapsed for loop over all datetimes in minutes\n",
        "print('')\n",
        "print('All downloads are completed.')\n",
        "print('Total time elapsed: %0.1f minutes' % totmin)\n",
        "f.close()       # close log of successful downloads"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "id": "WyXnVjSMUmWt",
        "outputId": "f2bd51ea-270d-43d1-b000-1e22e783c5bc"
      },
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "** Working on GLBy0.08/expt_93.0 **\n",
            "drive/MyDrive/Colab Notebooks/hycom/20200101_12.nc\n",
            "2020-01-01-T12:00:00Z\n",
            "  Attempting to get data, counter = 1\n",
            "  Downloaded data\n",
            "  Time elapsed: 46.5 seconds\n",
            "drive/MyDrive/Colab Notebooks/hycom/20200101_15.nc\n",
            "2020-01-01-T15:00:00Z\n",
            "  Attempting to get data, counter = 1\n",
            "  Downloaded data\n",
            "  Time elapsed: 13.6 seconds\n",
            "drive/MyDrive/Colab Notebooks/hycom/20200101_18.nc\n",
            "2020-01-01-T18:00:00Z\n",
            "  Attempting to get data, counter = 1\n",
            "  Downloaded data\n",
            "  Time elapsed: 11.8 seconds\n",
            "drive/MyDrive/Colab Notebooks/hycom/20200101_21.nc\n",
            "2020-01-01-T21:00:00Z\n",
            "  Attempting to get data, counter = 1\n",
            "  Downloaded data\n",
            "  Time elapsed: 10.0 seconds\n",
            "drive/MyDrive/Colab Notebooks/hycom/20200102_00.nc\n",
            "2020-01-02-T00:00:00Z\n",
            "  Attempting to get data, counter = 1\n",
            "  Downloaded data\n",
            "  Time elapsed: 7.6 seconds\n",
            "drive/MyDrive/Colab Notebooks/hycom/20200102_03.nc\n",
            "2020-01-02-T03:00:00Z\n",
            "  Attempting to get data, counter = 1\n",
            "  Downloaded data\n",
            "  Time elapsed: 11.7 seconds\n",
            "drive/MyDrive/Colab Notebooks/hycom/20200102_06.nc\n",
            "2020-01-02-T06:00:00Z\n",
            "  Attempting to get data, counter = 1\n",
            "  Downloaded data\n",
            "  Time elapsed: 11.6 seconds\n",
            "drive/MyDrive/Colab Notebooks/hycom/20200102_09.nc\n",
            "2020-01-02-T09:00:00Z\n",
            "  Attempting to get data, counter = 1\n",
            "  Downloaded data\n",
            "  Time elapsed: 9.3 seconds\n",
            "\n",
            "All downloads are completed.\n",
            "Total time elapsed: 2.0 minutes\n"
          ]
        }
      ]
    }
  ]
}