Colombia_Time_Series_Data.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Colombia_Time_Series_Data.ipynb",
      "provenance": [],
      "mount_file_id": "1d1LzuRUWI-EijmC-vaXdM92EAPWMhSvJ",
      "authorship_tag": "ABX9TyMQs/ZzMjJqGBXQ7hEp2xM/",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Max-FM/SPRINT-Colombia/blob/main/Colombia_Time_Series_Data.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ruk4Ycv3XHvS"
      },
      "source": [
        "#Colombia Time Series Data\n",
        "\n",
        "**Disclaimer:** This notebook is partially adapted from [this](https://colab.research.google.com/github/google/earthengine-community/blob/master/tutorials/time-series-visualization-with-altair/tutorial.ipynb) tutorial on visualising time-series data using Google Earth Engine and Python. This is a good reference to further understand how this notebook works.\n",
        "\n",
        "**Note:** This notebook will not work straight 'out of the box' and will require a bit of editing to suit your purposes. The idea of this notebook is to show the method I used to obtain time-series data from Google Earth Engine and is intended to be used as a template for future work."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "68_R807C3qLZ"
      },
      "source": [
        "##Import required packages"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i9FdeMqVZCYU"
      },
      "source": [
        "import ee\n",
        "import numpy as np\n",
        "import pandas as pd"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tubT3goC3ua3"
      },
      "source": [
        "##Authenticate Google Earth Engine\n",
        "\n",
        "To access the Google Earth Engine API you require an account. To request access, go to [https://signup.earthengine.google.com](https://signup.earthengine.google.com/). You may have to wait up to a day or so to be granted access and it's possible you will not recieve any email communication. To manually check whether you have access, try to log into [https://code.earthengine.google.com](https://code.earthengine.google.com/), or attempt to run the next cell and follow the instructions provided in the output cell."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F-uLIrdLZRoc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b573651f-68c3-4f36-9a10-4148ed8e2b01"
      },
      "source": [
        "# Trigger the authentication flow.\n",
        "ee.Authenticate()\n",
        "\n",
        "# Initialize the library.\n",
        "ee.Initialize()"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "To authorize access needed by Earth Engine, open the following URL in a web browser and follow the instructions. If the web browser does not start automatically, please manually browse the URL below.\n",
            "\n",
            "    https://accounts.google.com/o/oauth2/auth?client_id=517222506229-vsmmajv00ul0bs7p89v5m89qs8eb9359.apps.googleusercontent.com&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fearthengine+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.full_control&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&code_challenge=LXTMpPf0P2K_C_Bg8-P29-WHztUlfq7VlE3KuptWQXA&code_challenge_method=S256\n",
            "\n",
            "The authorization workflow will generate a code, which you should paste in the box below. \n",
            "Enter verification code: 4/1AY0e-g6aGDcnjvJXB8gGijow-bqsDBVuPhTm1tpuheh2Tal0SgEl_DiVjMg\n",
            "\n",
            "Successfully saved authorization token.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WGrgSnJuXmk6"
      },
      "source": [
        "## Define Request Function\n",
        "\n",
        "This function contains a series of image collections on Google Earth Engine that we wish to obtain time series data from."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iDmWaq6BXybF"
      },
      "source": [
        "def calculate_RVI(image):\n",
        "    bandDict = {\n",
        "                'VV': image.select('VV'), \n",
        "                'VH': image.select('VH')\n",
        "                }\n",
        "\n",
        "    rvi = image.expression('(4*VH)/(VH+VV)', bandDict) \\\n",
        "               .rename('RVI')\n",
        "\n",
        "    return image.addBands(rvi)\n",
        "\n",
        "def get_Sentinel_1_RVI(criteria):\n",
        "    SENTINEL_1_SAR_GRD_C_BAND = ee.ImageCollection(\"COPERNICUS/S1_GRD\") \\\n",
        "                                  .filter(criteria) \\\n",
        "                                  .filter(ee.Filter.listContains('transmitterReceiverPolarisation', 'VV')) \\\n",
        "                                  .filter(ee.Filter.listContains('transmitterReceiverPolarisation', 'VH')) \\\n",
        "                                  .filter(ee.Filter.eq('instrumentMode', 'IW')) \\\n",
        "                                  .select(['VV', 'VH'])\n",
        "\n",
        "    SENTINEL_1_RVI = SENTINEL_1_SAR_GRD_C_BAND.map(calculate_RVI).select('RVI')\n",
        "\n",
        "    return SENTINEL_1_RVI\n",
        "\n",
        "def obtain_data(region):\n",
        "    criteria  = ee.Filter.geometry(region) \n",
        "\n",
        "    SENTINEL_1_RVI = get_Sentinel_1_RVI(criteria)\n",
        "\n",
        "    MODIS_16D_TERRA_VEG = ee.ImageCollection(\"MODIS/006/MOD13Q1\") \\\n",
        "                             .filter(criteria) \\\n",
        "                             .select(['NDVI', 'EVI'])\n",
        "\n",
        "    MODIS_16D_AQUA_VEG = ee.ImageCollection(\"MODIS/006/MYD13Q1\") \\\n",
        "                           .filter(criteria) \\\n",
        "                           .select(['NDVI', 'EVI'])\n",
        "\n",
        "    TERRA_CLIMATE_MONTHLY = ee.ImageCollection(\"IDAHO_EPSCOR/TERRACLIMATE\") \\\n",
        "                              .filter(criteria)\n",
        "\n",
        "    PERSIANN_DAILY = ee.ImageCollection(\"NOAA/PERSIANN-CDR\") \\\n",
        "                       .filter(criteria)\n",
        "                       \n",
        "    CHIRPS_DAILY = ee.ImageCollection(\"UCSB-CHG/CHIRPS/DAILY\") \\\n",
        "                     .filter(criteria)\n",
        "\n",
        "    IMERG_MONTHLY = ee.ImageCollection(\"NASA/GPM_L3/IMERG_MONTHLY_V06\") \\\n",
        "                      .filter(criteria)\n",
        "\n",
        "    ERA5_DAILY = ee.ImageCollection(\"ECMWF/ERA5/DAILY\") \\\n",
        "                   .filter(criteria)\n",
        "\n",
        "    ERA5_MONTHLY = ee.ImageCollection(\"ECMWF/ERA5/MONTHLY\") \\\n",
        "                     .filter(criteria)\n",
        "\n",
        "    # Above image collections are placed in a Python dictionary for convenience.\n",
        "    image_collections = {\n",
        "                         'SENTINEL_1_RVI': SENTINEL_1_RVI,\n",
        "                         'MODIS_16D_TERRA_VEG': MODIS_16D_TERRA_VEG,\n",
        "                         'MODIS_16D_AQUA_VEG': MODIS_16D_AQUA_VEG,\n",
        "                         'TERRA_CLIMATE_MONTHLY': TERRA_CLIMATE_MONTHLY,\n",
        "                         'PERSIANN_DAILY': PERSIANN_DAILY,\n",
        "                         'CHIRPS_DAILY': CHIRPS_DAILY,\n",
        "                         'IMERG_MONTHLY': IMERG_MONTHLY,\n",
        "                         'ERA5_DAILY': ERA5_DAILY,\n",
        "                         'ERA5_MONTHLY': ERA5_MONTHLY\n",
        "                         }\n",
        "\n",
        "    return image_collections"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eUufiuUk8Wrk"
      },
      "source": [
        "## Define Region Reduction and Data Collection Functions\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XJgSyFoX8FcZ"
      },
      "source": [
        "def create_reduce_region_function(\n",
        "                                  geometry,\n",
        "                                  reducer=ee.Reducer.median(),\n",
        "                                  scale=250,\n",
        "                                  crs='EPSG:4326',\n",
        "                                  bestEffort=True,\n",
        "                                  maxPixels=1e13,\n",
        "                                  tileScale=4\n",
        "                                  ):\n",
        "  \"\"\"Creates a region reduction function.\n",
        "\n",
        "  Creates a region reduction function intended to be used as the input function\n",
        "  to ee.ImageCollection.map() for reducing pixels intersecting a provided region\n",
        "  to a statistic for each image in a collection. See ee.Image.reduceRegion()\n",
        "  documentation for more details.\n",
        "\n",
        "  Args:\n",
        "    geometry:\n",
        "      An ee.Geometry that defines the region over which to reduce data.\n",
        "    reducer:\n",
        "      Optional; An ee.Reducer that defines the reduction method.\n",
        "    scale:\n",
        "      Optional; A number that defines the nominal scale in meters of the\n",
        "      projection to work in.\n",
        "    crs:\n",
        "      Optional; An ee.Projection or EPSG string ('EPSG:5070') that defines\n",
        "      the projection to work in.\n",
        "    bestEffort:\n",
        "      Optional; A Boolean indicator for whether to use a larger scale if the\n",
        "      geometry contains too many pixels at the given scale for the operation\n",
        "      to succeed.\n",
        "    maxPixels:\n",
        "      Optional; A number specifying the maximum number of pixels to reduce.\n",
        "    tileScale:\n",
        "      Optional; A number representing the scaling factor used to reduce\n",
        "      aggregation tile size; using a larger tileScale (e.g. 2 or 4) may enable\n",
        "      computations that run out of memory with the default.\n",
        "\n",
        "  Returns:\n",
        "    A function that accepts an ee.Image and reduces it by region, according to\n",
        "    the provided arguments.\n",
        "  \"\"\"\n",
        "\n",
        "  def reduce_region_function(img):\n",
        "    \"\"\"Applies the ee.Image.reduceRegion() method.\n",
        "\n",
        "    Args:\n",
        "      img:\n",
        "        An ee.Image to reduce to a statistic by region.\n",
        "\n",
        "    Returns:\n",
        "      An ee.Feature that contains properties representing the image region\n",
        "      reduction results per band and the image timestamp formatted as\n",
        "      milliseconds from Unix epoch (included to enable time series plotting).\n",
        "    \"\"\"\n",
        "\n",
        "    stat = img.reduceRegion(\n",
        "                            reducer=reducer,\n",
        "                            geometry=geometry,\n",
        "                            scale=scale,\n",
        "                            crs=crs,\n",
        "                            bestEffort=bestEffort,\n",
        "                            maxPixels=maxPixels,\n",
        "                            tileScale=tileScale\n",
        "                            )\n",
        "\n",
        "    return ee.Feature(geometry, stat).set({'millis': img.date().millis()})\n",
        "  return reduce_region_function\n",
        "\n",
        "# Define a function to transfer feature properties to a dictionary.\n",
        "def feature_collection_to_dict(fc):\n",
        "  prop_names = fc.first().propertyNames()\n",
        "  prop_lists = fc.reduceColumns(\n",
        "      reducer=ee.Reducer.toList().repeat(prop_names.size()),\n",
        "      selectors=prop_names).get('list')\n",
        "\n",
        "  return ee.Dictionary.fromLists(prop_names, prop_lists)\n",
        "\n",
        "# Function to add date variables to DataFrame.\n",
        "def add_date_info(df):\n",
        "  df['Timestamp'] = pd.to_datetime(df['millis'], unit='ms')\n",
        "  df['Year'] = pd.DatetimeIndex(df['Timestamp']).year\n",
        "  df['Month'] = pd.DatetimeIndex(df['Timestamp']).month\n",
        "  df['Day'] = pd.DatetimeIndex(df['Timestamp']).day\n",
        "  df['DOY'] = pd.DatetimeIndex(df['Timestamp']).dayofyear\n",
        "  return df\n",
        "\n",
        "# Aggregates pixel values over a defined region in a Google Earth Engine image \n",
        "# collection and returns a time-series Pandas dataframe.\n",
        "def obtain_timeseries_dataframe(\n",
        "                                image_collection, \n",
        "                                region, \n",
        "                                aggregation, \n",
        "                                scale=250,\n",
        "                                crs='EPSG:4326',\n",
        "                                bestEffort=True,\n",
        "                                maxPixels=1e13,\n",
        "                                tileScale=4\n",
        "                                ):\n",
        "    \n",
        "    reduce_dict = {'median': ee.Reducer.median(),\n",
        "                   'stdDev': ee.Reducer.stdDev(),\n",
        "                   'count': ee.Reducer.count()}\n",
        "    \n",
        "    # Defines the reduction function to apply to the image collection.\n",
        "    reduce_fn = create_reduce_region_function(geometry=region, \n",
        "                                              reducer=reduce_dict[aggregation], \n",
        "                                              scale=scale,\n",
        "                                              crs=crs,\n",
        "                                              bestEffort=bestEffort,\n",
        "                                              maxPixels=maxPixels,\n",
        "                                              tileScale=tileScale)\n",
        "    \n",
        "    # Creates a time-series feature collection by aggregating over the image \n",
        "    # collection.\n",
        "    feature_collection_reduced = ee.FeatureCollection(image_collection.map(reduce_fn)) \\\n",
        "                                   .filter(ee.Filter.notNull(image_collection.first().bandNames()))\n",
        "\n",
        "    # Converts the feature collection to a Python dictionary.\n",
        "    reduced_dict = feature_collection_to_dict(feature_collection_reduced).getInfo()\n",
        "\n",
        "    # Converts the Python dictionary to a Pandas dataframe.\n",
        "    df = pd.DataFrame(reduced_dict)\n",
        "\n",
        "    # Renames data columms to reflect aggregation type.\n",
        "    df.columns = df.columns.map(lambda colName : colName+f'_{aggregation}' if colName !='system:index' and colName !='millis' else colName)\n",
        "    # Drops the 'millis' column.\n",
        "    df = add_date_info(df).drop(columns=['millis'])\n",
        "\n",
        "    return df\n",
        "\n",
        "# Filters an image collection to within a certain date range.\n",
        "def filter_collection_by_date(collection, start_date, end_date):\n",
        "    start_date = ee.Date(start_date)\n",
        "    end_date = ee.Date (end_date)\n",
        "\n",
        "    date_filter = ee.Filter.date(start_date, end_date)\n",
        "\n",
        "    collection_filtered = collection.filter(date_filter)\n",
        "\n",
        "    return collection_filtered\n",
        "\n",
        "# Sorts the columns alphabetically with the exeption of a pre-detemined list of\n",
        "# columns.\n",
        "def reorder_dataframe(df, preordered_columns):\n",
        "    df = df.reindex(sorted(df.columns), axis=1)\n",
        "    new_order = preordered_columns + [c for c in df.columns if c not in preordered_columns]\n",
        "    df = df.reindex(columns=new_order)\n",
        "    return df\n",
        "\n",
        "def add_info_columns(df, district_name, farm_name, collection_name):\n",
        "    df['District'] = district_name\n",
        "    df['Farm'] = farm_name\n",
        "    df['Image_Collection'] = collection_name\n",
        "\n",
        "    return df"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z53TZcE0uPL1"
      },
      "source": [
        "## Initialising Parameters\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ww90qA1AuPWF"
      },
      "source": [
        "# Splitting date ranges into yearly chunks to reduce timeouts on GEE side.\n",
        "date_ranges = [(f'{year}-01-01', f'{year+1}-01-01') for year in np.arange(1997, 2021)]\n",
        "\n",
        "# Resample pixels to resolution in metres. Setting native scale for Sentinel 1.\n",
        "scaleDict = {\n",
        "            'SENTINEL_1_RVI': 10,\n",
        "            'MODIS_16D_TERRA_VEG': 50,\n",
        "            'MODIS_16D_AQUA_VEG': 50,\n",
        "            'TERRA_CLIMATE_MONTHLY': 50,\n",
        "            'PERSIANN_DAILY': 50,\n",
        "            'CHIRPS_DAILY': 50,\n",
        "            'IMERG_MONTHLY': 50,\n",
        "            'ERA5_DAILY': 50,\n",
        "            'ERA5_MONTHLY': 50\n",
        "            }\n",
        "\n",
        "# List of pre-ordered columns to prefix the dataframe with.\n",
        "preordered_columns = ['District', 'Farm', 'Year', 'Month', 'Day', 'DOY', 'Timestamp', 'Image_Collection', 'system:index']\n",
        "# List of columns to merge the median and standard deviation dataframes on.\n",
        "merge_columns = ['system:index', 'Timestamp', 'Year', 'Month', 'Day', 'DOY'] "
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uHxxFF-Cl_iN"
      },
      "source": [
        "## Getting District Level Time Series Data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XqVoxDval_iP"
      },
      "source": [
        "# Set to 'None' to iterate through all image collections defined above.\n",
        "# Otherwise provide a list of strings \n",
        "# e.g. ['MODIS_16D_TERRA_VEG', 'PERSIANN_DAILY', etc]\n",
        "desired_collections = None\n",
        "\n",
        "# Set to 'True' to overwrite existing files.\n",
        "overwrite = False"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GGfqRRBal_iQ"
      },
      "source": [
        "%%time\n",
        "import os.path\n",
        "\n",
        "# Loops through the shapefiles for each district.\n",
        "for district_name in ['Dosquebradas', 'Versailles']:\n",
        "    # Shape files have been pre-uploaded to Google Earth Engine.\n",
        "    region = ee.FeatureCollection(f'users/maxfoxley-marrable/SPRINT/{district_name}')\n",
        "    aoi = region.geometry()\n",
        "\n",
        "    print(district_name)\n",
        "\n",
        "    # Obtains all image collections defined above for said region.\n",
        "    image_collections = obtain_data(region)\n",
        "    \n",
        "    # Filters out unwanted image collections.\n",
        "    if desired_collections:\n",
        "        image_collections = {collection: image_collections[collection] \\\n",
        "                            for collection in desired_collections}\n",
        "\n",
        "    # Loops for each image collection obtained above.\n",
        "    for collection_name, collection in image_collections.items():\n",
        "        \n",
        "        print(collection_name)\n",
        "\n",
        "        # Edit as required. For now I have it directly saving to the shared \n",
        "        # Google Drive.\n",
        "        filepath = f'/content/drive/Shared drives/Colombia SPRINT/Test Districts/{district_name}/CSV Data/{district_name}_{collection_name}.csv'\n",
        "\n",
        "        # Skips if CSV file for a specific region and image collection already \n",
        "        # exists, provided overwrite = False.\n",
        "        if overwrite == False and os.path.isfile(filepath) == True:\n",
        "            print(f'{district_name}_{collection_name}.csv already exists, skipping.')\n",
        "            continue\n",
        "\n",
        "        # Creates an empty master dataframe which will be iteratively appended \n",
        "        # to create a full ~20 year timeseries. \n",
        "        dataframe = pd.DataFrame()\n",
        "\n",
        "        # Skips the image collection if it contains no images.\n",
        "        if collection.size().getInfo() == 0:\n",
        "            continue\n",
        "        \n",
        "        # Loops though dates in batches of about 8 years to avoid timeouts on\n",
        "        # GEE server side. See above. \n",
        "        for start_date, end_date in date_ranges:\n",
        "            date_range_str = f'{start_date}_{end_date}'\n",
        "\n",
        "            # Filters image collection to within desired date range.\n",
        "            collection_filtered = filter_collection_by_date(collection, start_date, end_date)\n",
        "\n",
        "            if collection_filtered.size().getInfo() == 0:\n",
        "                continue\n",
        "\n",
        "            # Gets the median pixel values for a region across all images in\n",
        "            # the filtered image collection.\n",
        "            median_df = obtain_timeseries_dataframe(collection_filtered,\n",
        "                                                    region=aoi, \n",
        "                                                    aggregation='median', \n",
        "                                                    scale=scaleDict[collection_name])\n",
        "            \n",
        "            # Same as above, but gets the standard deviation.\n",
        "            std_dev_df = obtain_timeseries_dataframe(collection_filtered,\n",
        "                                                     region=aoi, \n",
        "                                                     aggregation='stdDev', \n",
        "                                                     scale=scaleDict[collection_name])\n",
        "            \n",
        "            # Gets the pixel count for the region, but there has to be a better \n",
        "            # way of pixel counting.\n",
        "            count_df = obtain_timeseries_dataframe(collection_filtered,\n",
        "                                                   region=aoi, \n",
        "                                                   aggregation='count', \n",
        "                                                   scale=scaleDict[collection_name])\n",
        "            \n",
        "\n",
        "            # Merges median and standard deviation time-series data frames into\n",
        "            # a single dataframe.\n",
        "            merge_df = pd.merge_ordered(median_df, std_dev_df, on=merge_columns)\n",
        "            merge_df = add_info_columns(merge_df, district_name, None, collection_name)\n",
        "            merge_df = reorder_dataframe(merge_df, preordered_columns)\n",
        "            merge_df['Pixel_Scale'] = scaleDict[collection_name] # Pixel scale (in metres) defined above.\n",
        "            merge_df['Pixel_Count'] = count_df.iloc[:,0] # Pixel count, this might be a bad way of doing this.\n",
        "                \n",
        "            # Appends merged dataframe for a master dataframe containing the \n",
        "            # full ~20 year time-series.\n",
        "            dataframe = dataframe.append(merge_df, ignore_index=True)\n",
        "\n",
        "        dataframe.drop('Farm', axis=1, inplace=True)\n",
        "        # Saves master dataframe to a CSV file.\n",
        "        dataframe.to_csv(filepath)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P5mBKGbdY1NX"
      },
      "source": [
        "## Getting Farm Level Time Series Data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C_NM37-QmqLt"
      },
      "source": [
        "# List of feature collections that contain shapefiles for farms.\n",
        "farms_list = [\n",
        "              'Dosquebradas_Central_N_Farm_C', \n",
        "              'Dosquebradas_Dos_Cosechas_C',\n",
        "              'Dosquebradas_Fincas_campesinas_C',\n",
        "              'Dosquebradas_La_Union_C',\n",
        "              'Dosquebradas_Las_Camillas_C',\n",
        "              'Dosquebradas_Minas_del_Socorro_C',\n",
        "              'Dosquebradas_NE_area_C',\n",
        "              'Dosquebradas_N_Farm_C',\n",
        "              'Dosquebradas_New_coffee_farm_C',\n",
        "              'Dosquebradas_Santa_Ana_Alto_C',\n",
        "              'Dosquebradas_Santa_Ana_Bajo_La_Union_C',\n",
        "              'Versailles_Barcelona',\n",
        "              'Versailles_El_Chalet',\n",
        "              'Versailles_Guyabel',\n",
        "              'Versailles_La_Primavera',\n",
        "              'Versailles_La_Rivera',\n",
        "              'Versailles_Miraflores',\n",
        "              'Versailles_Predio_Valery',\n",
        "              'Versailles_Roberto',\n",
        "              'Versailles_San_Fernando',\n",
        "              'Versailles_Villa_Emanuel'\n",
        "              ]\n",
        "\n",
        "# Set to 'None' to iterate through all image collections defined above.\n",
        "# Otherwise provide a list of strings \n",
        "# e.g. ['MODIS_16D_TERRA_VEG', 'PERSIANN_DAILY', etc]\n",
        "desired_collections = None\n",
        "\n",
        "# Set to 'True' to overwrite existing files.\n",
        "overwrite = False"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RAjjASWZY64r",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 578
        },
        "outputId": "9a8647bc-31a3-499e-c2de-3ea367f4108c"
      },
      "source": [
        "%%time\n",
        "import os.path\n",
        "\n",
        "# Loops through the shapefiles for each farm.\n",
        "for farm in farms_list:\n",
        "    # Shape files have been pre-uploaded to Google Earth Engine.\n",
        "    region = ee.FeatureCollection(f'users/maxfoxley-marrable/SPRINT/Farms/{farm}')\n",
        "    aoi = region.geometry()\n",
        "\n",
        "    # Splits the filename string to obtain the district and farm name.\n",
        "    district_name = farm.partition('_')[0]\n",
        "    farm_name = farm.partition('_')[-1]\n",
        "\n",
        "    print(district_name, farm_name)\n",
        "\n",
        "    # Obtains all image collections defined above for said region.\n",
        "    image_collections = obtain_data(region)\n",
        "    \n",
        "    # Filters out unwanted image collections.\n",
        "    if desired_collections:\n",
        "        image_collections = {collection: image_collections[collection] \\\n",
        "                            for collection in desired_collections}\n",
        "\n",
        "    # Loops for each image collection obtained above.\n",
        "    for collection_name, collection in image_collections.items():\n",
        "        \n",
        "        print(collection_name)\n",
        "\n",
        "        # Edit as required. For now I have it directly saving to the shared \n",
        "        # Google Drive.\n",
        "        # filepath = f'/content/drive/Shared drives/Colombia SPRINT/Test Districts/{district_name}/Farms/CSV Data/{farm_name}/{farm_name}_{collection_name}.csv'\n",
        "        filepath = f'/content/{farm_name}_{collection_name}.csv'\n",
        "\n",
        "        # Skips if CSV file for a specific region and image collection already \n",
        "        # exists, provided overwrite = False.\n",
        "        if overwrite == False and os.path.isfile(filepath) == True:\n",
        "            print(f'{farm_name}_{collection_name}.csv already exists, skipping.')\n",
        "            continue\n",
        "\n",
        "        # Creates an empty master dataframe which will be iteratively appended \n",
        "        # to create a full ~20 year timeseries. \n",
        "        dataframe = pd.DataFrame()\n",
        "\n",
        "        # Skips the image collection if it contains no images.\n",
        "        if collection.size().getInfo() == 0:\n",
        "            continue\n",
        "        \n",
        "        # Loops though dates in batches of about 8 years to avoid timeouts on\n",
        "        # GEE server side. See above. \n",
        "        for start_date, end_date in date_ranges:\n",
        "            date_range_str = f'{start_date}_{end_date}'\n",
        "\n",
        "            # print(date_range_str)\n",
        "\n",
        "            # Filters image collection to within desired date range.\n",
        "            collection_filtered = filter_collection_by_date(collection, start_date, end_date)\n",
        "\n",
        "            if collection_filtered.size().getInfo() == 0:\n",
        "                continue\n",
        "\n",
        "            # Gets the median pixel values for a region across all images in\n",
        "            # the filtered image collection.\n",
        "            median_df = obtain_timeseries_dataframe(collection_filtered,\n",
        "                                                    region=aoi, \n",
        "                                                    aggregation='median', \n",
        "                                                    scale=scaleDict[collection_name])\n",
        "            \n",
        "            # Same as above, but gets the standard deviation.\n",
        "            std_dev_df = obtain_timeseries_dataframe(collection_filtered,\n",
        "                                                     region=aoi, \n",
        "                                                     aggregation='stdDev', \n",
        "                                                     scale=scaleDict[collection_name])\n",
        "            \n",
        "            # Gets the pixel count for the region, but there has to be a better \n",
        "            # way of pixel counting.\n",
        "            count_df = obtain_timeseries_dataframe(collection_filtered,\n",
        "                                                   region=aoi, \n",
        "                                                   aggregation='count', \n",
        "                                                   scale=scaleDict[collection_name])\n",
        "            \n",
        "\n",
        "            # Merges median and standard deviation time-series data frames into\n",
        "            # a single dataframe.\n",
        "            merge_df = pd.merge_ordered(median_df, std_dev_df, on=merge_columns)\n",
        "            merge_df = add_info_columns(merge_df, district_name, farm_name, collection_name)\n",
        "            merge_df = reorder_dataframe(merge_df, preordered_columns)\n",
        "            merge_df['Pixel_Scale'] = scaleDict[collection_name] # Pixel scale (in metres) defined above.\n",
        "            merge_df['Pixel_Count'] = count_df.iloc[:,0] # Pixel count, this might be a bad way of doing this.\n",
        "                \n",
        "            # Appends merged dataframe for a master dataframe containing the \n",
        "            # full ~20 year time-series.\n",
        "            dataframe = dataframe.append(merge_df, ignore_index=True)\n",
        "\n",
        "        # Saves master dataframe to a CSV file.\n",
        "        dataframe.to_csv(filepath)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Dosquebradas Central_N_Farm_C\n",
            "SENTINEL_1_RVI\n",
            "MODIS_16D_TERRA_VEG\n",
            "MODIS_16D_AQUA_VEG\n",
            "TERRA_CLIMATE_MONTHLY\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36m_make_request\u001b[0;34m(self, conn, method, url, timeout, chunked, **httplib_request_kw)\u001b[0m\n\u001b[1;32m    376\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# Python 2.7, use buffering of HTTP responses\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 377\u001b[0;31m                 \u001b[0mhttplib_response\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetresponse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbuffering\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    378\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# Python 3\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mTypeError\u001b[0m: getresponse() got an unexpected keyword argument 'buffering'",
            "\nDuring handling of the above exception, another exception occurred:\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-12-e7948c0b1e8c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_cell_magic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'time'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"import os.path\\n\\n# Loops through the shapefiles for each farm.\\nfor farm in farms_list:\\n    # Shape files have been pre-uploaded to Google Earth Engine.\\n    region = ee.FeatureCollection(f'users/maxfoxley-marrable/SPRINT/Farms/{farm}')\\n    aoi = region.geometry()\\n\\n    # Splits the filename string to obtain the district and farm name.\\n    district_name = farm.partition('_')[0]\\n    farm_name = farm.partition('_')[-1]\\n\\n    print(district_name, farm_name)\\n\\n    # Obtains all image collections defined above for said region.\\n    image_collections = obtain_data(region)\\n    \\n    # Filters out unwanted image collections.\\n    if desired_collections:\\n        image_collections = {collection: image_collections[collection] \\\\\\n                            for collection in desired_collections}\\n\\n    # Loops for each image collection obtained above.\\n    for collection_name, collection in image_collections.items():\\n        \\n        print(collection_name)\\n\\n        # Edit as required. For now I have it directly saving to the shared \\n        # Google Drive.\\n        # filepath = f'/content/drive/Shared drives/Colombia SPRINT/Test Districts/{district_name}/Farms/CSV Data/{farm_name}/{farm_name}_{collection_name}.csv'\\n        filepath = f'/content/{farm_name}_{collection_name}.csv'\\n\\n        # Skip...\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py\u001b[0m in \u001b[0;36mrun_cell_magic\u001b[0;34m(self, magic_name, line, cell)\u001b[0m\n\u001b[1;32m   2115\u001b[0m             \u001b[0mmagic_arg_s\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvar_expand\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstack_depth\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2116\u001b[0m             \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuiltin_trap\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2117\u001b[0;31m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmagic_arg_s\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2118\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2119\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<decorator-gen-60>\u001b[0m in \u001b[0;36mtime\u001b[0;34m(self, line, cell, local_ns)\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/IPython/core/magic.py\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(f, *a, **k)\u001b[0m\n\u001b[1;32m    186\u001b[0m     \u001b[0;31m# but it's overkill for just that one bit of state.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    187\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mmagic_deco\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 188\u001b[0;31m         \u001b[0mcall\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    189\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    190\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mcallable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/IPython/core/magics/execution.py\u001b[0m in \u001b[0;36mtime\u001b[0;34m(self, line, cell, local_ns)\u001b[0m\n\u001b[1;32m   1191\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1192\u001b[0m             \u001b[0mst\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclock2\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1193\u001b[0;31m             \u001b[0mexec\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mglob\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlocal_ns\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1194\u001b[0m             \u001b[0mend\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclock2\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1195\u001b[0m             \u001b[0mout\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<timed exec>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32m<ipython-input-6-ff818ef2b0e1>\u001b[0m in \u001b[0;36mobtain_timeseries_dataframe\u001b[0;34m(image_collection, region, aggregation, scale, crs, bestEffort, maxPixels, tileScale)\u001b[0m\n\u001b[1;32m    117\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    118\u001b[0m     \u001b[0;31m# Converts the feature collection to a Python dictionary.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 119\u001b[0;31m     \u001b[0mreduced_dict\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfeature_collection_to_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfeature_collection_reduced\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetInfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    120\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    121\u001b[0m     \u001b[0;31m# Converts the Python dictionary to a Pandas dataframe.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/ee/computedobject.py\u001b[0m in \u001b[0;36mgetInfo\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     93\u001b[0m       \u001b[0mThe\u001b[0m \u001b[0mobject\u001b[0m \u001b[0mcan\u001b[0m \u001b[0mevaluate\u001b[0m \u001b[0mto\u001b[0m \u001b[0manything\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     94\u001b[0m     \"\"\"\n\u001b[0;32m---> 95\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcomputeValue\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     96\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     97\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0mencode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mencoder\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/ee/data.py\u001b[0m in \u001b[0;36mcomputeValue\u001b[0;34m(obj)\u001b[0m\n\u001b[1;32m    708\u001b[0m             \u001b[0mbody\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0;34m'expression'\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mserializer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mencode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfor_cloud_api\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    709\u001b[0m             \u001b[0mproject\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0m_get_projects_path\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 710\u001b[0;31m             prettyPrint=False))['result']\n\u001b[0m\u001b[1;32m    711\u001b[0m   return send_('/value', {\n\u001b[1;32m    712\u001b[0m       \u001b[0;34m'json'\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mserialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfor_cloud_api\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/ee/data.py\u001b[0m in \u001b[0;36m_execute_cloud_call\u001b[0;34m(call, num_retries)\u001b[0m\n\u001b[1;32m    343\u001b[0m   \"\"\"\n\u001b[1;32m    344\u001b[0m   \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 345\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mcall\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexecute\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnum_retries\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnum_retries\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    346\u001b[0m   \u001b[0;32mexcept\u001b[0m \u001b[0mgoogleapiclient\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mHttpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    347\u001b[0m     \u001b[0;32mraise\u001b[0m \u001b[0m_translate_cloud_exception\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/googleapiclient/_helpers.py\u001b[0m in \u001b[0;36mpositional_wrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    132\u001b[0m                 \u001b[0;32melif\u001b[0m \u001b[0mpositional_parameters_enforcement\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mPOSITIONAL_WARNING\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    133\u001b[0m                     \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwarning\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 134\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mwrapped\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    135\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    136\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mpositional_wrapper\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/googleapiclient/http.py\u001b[0m in \u001b[0;36mexecute\u001b[0;34m(self, http, num_retries)\u001b[0m\n\u001b[1;32m    890\u001b[0m             \u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    891\u001b[0m             \u001b[0mbody\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbody\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 892\u001b[0;31m             \u001b[0mheaders\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mheaders\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    893\u001b[0m         )\n\u001b[1;32m    894\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/googleapiclient/http.py\u001b[0m in \u001b[0;36m_retry_request\u001b[0;34m(http, num_retries, req_type, sleep, rand, uri, method, *args, **kwargs)\u001b[0m\n\u001b[1;32m    170\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    171\u001b[0m             \u001b[0mexception\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 172\u001b[0;31m             \u001b[0mresp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcontent\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhttp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0muri\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    173\u001b[0m         \u001b[0;31m# Retry on SSL errors and socket timeout errors.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    174\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0m_ssl_SSLError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mssl_error\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/google_auth_httplib2.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(self, uri, method, body, headers, **kwargs)\u001b[0m\n\u001b[1;32m    199\u001b[0m         \u001b[0;31m# Make the request.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    200\u001b[0m         response, content = self.http.request(\n\u001b[0;32m--> 201\u001b[0;31m             uri, method, body=body, headers=request_headers, **kwargs)\n\u001b[0m\u001b[1;32m    202\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    203\u001b[0m         \u001b[0;31m# If the response indicated that the credentials needed to be\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/httplib2/__init__.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(self, uri, method, body, headers, redirections, connection_type)\u001b[0m\n\u001b[1;32m   1989\u001b[0m                         \u001b[0mheaders\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1990\u001b[0m                         \u001b[0mredirections\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1991\u001b[0;31m                         \u001b[0mcachekey\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1992\u001b[0m                     )\n\u001b[1;32m   1993\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/httplib2/__init__.py\u001b[0m in \u001b[0;36m_request\u001b[0;34m(self, conn, host, absolute_uri, request_uri, method, body, headers, redirections, cachekey)\u001b[0m\n\u001b[1;32m   1649\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1650\u001b[0m         (response, content) = self._conn_request(\n\u001b[0;32m-> 1651\u001b[0;31m             \u001b[0mconn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrequest_uri\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbody\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mheaders\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1652\u001b[0m         )\n\u001b[1;32m   1653\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/httplib2shim/__init__.py\u001b[0m in \u001b[0;36m_conn_request\u001b[0;34m(self, conn, request_uri, method, body, headers)\u001b[0m\n\u001b[1;32m    146\u001b[0m                 \u001b[0mretries\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0murllib3\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mRetry\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtotal\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mredirect\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    147\u001b[0m                 \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0murllib3\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTimeout\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtotal\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 148\u001b[0;31m                 decode_content=decode)\n\u001b[0m\u001b[1;32m    149\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    150\u001b[0m             \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_map_response\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murllib3_response\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdecode\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdecode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/urllib3/request.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(self, method, url, fields, headers, **urlopen_kw)\u001b[0m\n\u001b[1;32m     70\u001b[0m             return self.request_encode_body(method, url, fields=fields,\n\u001b[1;32m     71\u001b[0m                                             \u001b[0mheaders\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mheaders\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 72\u001b[0;31m                                             **urlopen_kw)\n\u001b[0m\u001b[1;32m     73\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     74\u001b[0m     def request_encode_url(self, method, url, fields=None, headers=None,\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/urllib3/request.py\u001b[0m in \u001b[0;36mrequest_encode_body\u001b[0;34m(self, method, url, fields, headers, encode_multipart, multipart_boundary, **urlopen_kw)\u001b[0m\n\u001b[1;32m    148\u001b[0m         \u001b[0mextra_kw\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murlopen_kw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    149\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 150\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0murlopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mextra_kw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/urllib3/poolmanager.py\u001b[0m in \u001b[0;36murlopen\u001b[0;34m(self, method, url, redirect, **kw)\u001b[0m\n\u001b[1;32m    322\u001b[0m             \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0murlopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    323\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 324\u001b[0;31m             \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0murlopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mu\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequest_uri\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    325\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    326\u001b[0m         \u001b[0mredirect_location\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mredirect\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_redirect_location\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36murlopen\u001b[0;34m(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)\u001b[0m\n\u001b[1;32m    598\u001b[0m                                                   \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtimeout_obj\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    599\u001b[0m                                                   \u001b[0mbody\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbody\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mheaders\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mheaders\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 600\u001b[0;31m                                                   chunked=chunked)\n\u001b[0m\u001b[1;32m    601\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    602\u001b[0m             \u001b[0;31m# If we're going to release the connection in ``finally:``, then\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36m_make_request\u001b[0;34m(self, conn, method, url, timeout, chunked, **httplib_request_kw)\u001b[0m\n\u001b[1;32m    378\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# Python 3\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    379\u001b[0m                 \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 380\u001b[0;31m                     \u001b[0mhttplib_response\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetresponse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    381\u001b[0m                 \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    382\u001b[0m                     \u001b[0;31m# Remove the TypeError from the exception chain in Python 3;\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/http/client.py\u001b[0m in \u001b[0;36mgetresponse\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1371\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1372\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1373\u001b[0;31m                 \u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbegin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1374\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mConnectionError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1375\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/http/client.py\u001b[0m in \u001b[0;36mbegin\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    309\u001b[0m         \u001b[0;31m# read until we get a non-100 response\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    310\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 311\u001b[0;31m             \u001b[0mversion\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstatus\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreason\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_read_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    312\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mstatus\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mCONTINUE\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    313\u001b[0m                 \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/http/client.py\u001b[0m in \u001b[0;36m_read_status\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    270\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    271\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_read_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 272\u001b[0;31m         \u001b[0mline\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreadline\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_MAXLINE\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"iso-8859-1\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    273\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0m_MAXLINE\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    274\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mLineTooLong\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"status line\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/socket.py\u001b[0m in \u001b[0;36mreadinto\u001b[0;34m(self, b)\u001b[0m\n\u001b[1;32m    584\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    585\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 586\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecv_into\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    587\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    588\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_timeout_occurred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/ssl.py\u001b[0m in \u001b[0;36mrecv_into\u001b[0;34m(self, buffer, nbytes, flags)\u001b[0m\n\u001b[1;32m   1010\u001b[0m                   \u001b[0;34m\"non-zero flags not allowed in calls to recv_into() on %s\"\u001b[0m \u001b[0;34m%\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1011\u001b[0m                   self.__class__)\n\u001b[0;32m-> 1012\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnbytes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1013\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1014\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0msocket\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecv_into\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnbytes\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mflags\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/ssl.py\u001b[0m in \u001b[0;36mread\u001b[0;34m(self, len, buffer)\u001b[0m\n\u001b[1;32m    872\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Read on closed or unwrapped SSL socket.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    873\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 874\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sslobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    875\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mSSLError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    876\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mSSL_ERROR_EOF\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msuppress_ragged_eofs\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.6/ssl.py\u001b[0m in \u001b[0;36mread\u001b[0;34m(self, len, buffer)\u001b[0m\n\u001b[1;32m    629\u001b[0m         \"\"\"\n\u001b[1;32m    630\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mbuffer\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 631\u001b[0;31m             \u001b[0mv\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sslobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    632\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    633\u001b[0m             \u001b[0mv\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sslobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    }
  ]
}