Flight_line_processing_vignette.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f4aef327-6a42-4f72-b6c5-6c34f07e887f",
   "metadata": {},
   "source": [
    "# Processing and Correcting NEON Hyperspectral Flight Lines for Scalable Spectral Data Analysis\n",
    "\n",
    "Welcome to this vignette! This guide provides a detailed walkthrough for processing NEON (National Ecological Observatory Network) flight line data, taking you from raw downloads to actionable outputs. The workflow includes converting raw NEON flight lines into ENVI-compatible formats, applying essential data corrections (such as topographic and BRDF adjustments), and extracting hyperspectral data to build comprehensive tables for numerical and statistical analysis.\n",
    "\n",
    "This workflow has been carefully designed to address the challenges of processing large datasets, ensuring efficient memory usage and high data integrity, even within the constraints of a machine with 250 GB of RAM. By the end of this guide, you will have the tools and understanding to transform raw hyperspectral data into corrected, high-quality datasets ready for advanced ecological and environmental research.\n",
    "\n",
    "---\n",
    "\n",
    "## Table of Contents\n",
    "1. [Introduction](#1-introduction)\n",
    "2. [Prerequisites](#2-prerequisites)\n",
    "3. [Environment Setup](#3-environment-setup)\n",
    "4. [Understanding NEON Flight Lines](#4-understanding-neon-flight-lines)\n",
    "5. [Finding NEON Flight Codes](#5-finding-neon-flight-codes)\n",
    "6. [Running the `jefe` Function](#6-running-the-jefe-function)\n",
    "7. [Handling Large Data Processing](#7-handling-large-data-processing)\n",
    "8. [Extracting Data and Building Tables](#8-extracting-data-and-building-tables)\n",
    "9. [Workarounds for RAM Limitations](#9-workarounds-for-ram-limitations)\n",
    "10. [Conclusion](#10-conclusion)\n",
    "11. [References](#11-references)\n",
    "\n",
    "---\n",
    "\n",
    "## 1. Introduction\n",
    "\n",
    "Hyperspectral data collected through NEON (National Ecological Observatory Network) flight lines provides a high-resolution spectral view of the Earth's surface, capturing detailed information about vegetation, soils, water, and other environmental components. However, the raw NEON data comes in specialized formats that require processing and correction before they can be used for meaningful analysis. \n",
    "\n",
    "This vignette provides a detailed, step-by-step guide to download, convert, and process NEON flight line data, ensuring efficient memory usage and high data integrity throughout the workflow. Along the way, we will create specific file types needed for both corrections and analysis, bridging the gap between raw hyperspectral data and actionable insights.\n",
    "\n",
    "---\n",
    "\n",
    "### **Why Extract Hyperspectral Signals?**\n",
    "Hyperspectral data is invaluable for ecological and environmental research as it provides detailed spectral signatures across hundreds of bands. By extracting these signals and applying corrections, researchers can:\n",
    "- **Translate Patterns Across Scales:** Connect fine-scale field measurements to broader regional or global observations.\n",
    "- **Quantify Environmental Changes:** Monitor vegetation health, water quality, or land cover changes over time.\n",
    "- **Improve Decision-Making Tools:** Build robust models for ecological resilience, biodiversity, and conservation planning.\n",
    "\n",
    "---\n",
    "\n",
    "### **Applications of This Workflow**\n",
    "\n",
    "This workflow is particularly suited for:\n",
    "- **Scaling Insights Across Spatial Domains:** Translating fine-scale hyperspectral data to broader landscapes ensures consistency and comparability across scales.\n",
    "- **Monitoring Environmental Changes:** Creating corrected and high-fidelity datasets to track vegetation health, water quality, or land cover over time.\n",
    "- **Enabling Cross-Sensor Calibration:** Harmonizing hyperspectral data across platforms by applying consistent corrections and resampling techniques.\n",
    "\n",
    "---\n",
    "\n",
    "## 2. Prerequisites\n",
    "\n",
    "Before you begin, ensure you have the following:\n",
    "\n",
    "- **Hardware Requirements:**\n",
    "  - A machine with at least **250 GB RAM** to handle large datasets efficiently.\n",
    "\n",
    "- **Software Requirements:**\n",
    "  - Access to a pre-configured Python environment with necessary libraries installed, including:\n",
    "    - `geopandas`, `rasterio`, `pandas`, `numpy`, `hytools`, `scikit-learn`, `matplotlib`, `requests`, `h5py`, `ray`.\n",
    "\n",
    "- **Data Requirements:**\n",
    "  - NEON flight line data.\n",
    "  - Corresponding flight codes to identify and process the relevant flight lines.\n",
    "\n",
    "- **Additional Tools:**\n",
    "  - A Jupyter Notebook interface to follow this vignette step by step.\n",
    "\n",
    "---\n",
    "\n",
    "## 3. Python Setup\n",
    "\n",
    "To follow this vignette, you'll need a Python environment configured with the necessary dependencies. If you haven't set up your environment yet, follow the steps below to install the required tools and libraries. This guide assumes you are working in a Jupyter Notebook.\n",
    "\n",
    "### Required Libraries\n",
    "Ensure the following libraries are available in your environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3ef913d0-cb8a-4b4a-8367-fa6ff501f027",
   "metadata": {},
   "outputs": [],
   "source": [
    "import hytools as ht\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import random\n",
    "import nbconvert\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f3e25445-87f1-4374-ba9c-fc46a40cf35c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: spectral in /opt/conda/envs/macrosystems/lib/python3.10/site-packages (0.23.1)\n",
      "Requirement already satisfied: numpy in /opt/conda/envs/macrosystems/lib/python3.10/site-packages (from spectral) (1.26.3)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install spectral"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "feecdd2a-7c7c-47fe-9b1e-fa11b646ec05",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ENVIProcessor', 'GradientBoostingRegressor', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'apply_topo_and_brdf_corrections', 'boosted_quantile_plot', 'boosted_quantile_plot_by_sensor', 'box', 'clean_data_and_write_to_csv', 'concatenate_sensors', 'control_function', 'download_neon_file', 'download_neon_flight_lines', 'extract_overlapping_layers_to_2d_dataframe', 'find_raster_files', 'fit_models_with_different_alpha', 'flight_lines_to_envi', 'generate_config_json', 'generate_correction_configs', 'generate_correction_configs_for_directory', 'get_spectral_data_and_wavelengths', 'glob', 'go_forth_and_multiply', 'gpd', 'h5py', 'ht', 'jefe', 'json', 'load_and_combine_rasters', 'load_spectra', 'mask', 'np', 'os', 'pd', 'plot_each_sensor_with_highlight', 'plot_spectral_data', 'plot_with_highlighted_sensors', 'plt', 'prepare_spectral_data', 'process_all_subdirectories', 'process_and_flatten_array', 'process_hdf5_with_neon2envi', 'random', 'rasterio', 'rasterize', 'rasterize_polygons_to_match_envi', 'ray', 'requests', 'resample_translation_to_other_sensors', 'reshape_spectra', 'show', 'show_rgb', 'subprocess', 'time', 'translate_to_other_sensors']\n"
     ]
    }
   ],
   "source": [
    "### Loading Earth Lab Spectral Tools\n",
    "\n",
    "# 1. Enable autoreload in your Jupyter Notebook:\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "# 2. Import the custom tools module:\n",
    "\n",
    "import spectral_unmixing_tools_original as el_spectral\n",
    "\n",
    "# 3. Verify that the tools loaded correctly by printing the module's directory:\n",
    "\n",
    "print(dir(el_spectral))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87aede84-ae4b-4f38-8cc1-584a862069f6",
   "metadata": {},
   "source": [
    "\n",
    "## 4. Understanding and Finding NEON Flight Lines\n",
    "\n",
    "NEON flight lines are aerial survey paths designed to collect high-resolution spectral data across various ecological sites. These datasets are vital for studying vegetation, soil, water bodies, and other environmental parameters, forming the foundation for many ecological and environmental analyses.\n",
    "\n",
    "### **How to Find Flight Codes**\n",
    "To process NEON flight lines with the `jefe` function, you’ll need the flight codes corresponding to your desired data. Follow these steps to find them:\n",
    "1. **Access the NEON Data Portal:** Visit the [NEON Data Portal](https://data.neonscience.org/) to browse available datasets.\n",
    "2. **Navigate to Flight Line Data:** Locate the section for flight line spectral data at your site of interest.\n",
    "3. **Identify Relevant Flight Codes:** Each flight line has associated metadata, including its unique flight code. Record the codes for the lines you wish to process.\n",
    "\n",
    "### **Important Considerations**\n",
    "1. **Data Availability:**\n",
    "   - NEON’s Airborne Observation Platform (AOP) data is generally available 60 days after the last collection day at a site.\n",
    "   - Data collection schedules may shift due to weather or logistical factors. For the latest updates, consult the [NEON Flight Schedules and Coverage page](https://www.neonscience.org/data-collection/flight-schedules-coverage).\n",
    "\n",
    "2. **Data Quality Updates:**\n",
    "   - NEON regularly updates its data products to address quality concerns or implement new processing methods.\n",
    "   - Stay informed about updates or changes that could affect your datasets by checking the [AOP Data Availability Notification](https://www.neonscience.org/impact/observatory-blog/aop-data-availability-notification-release-2024).\n",
    "\n",
    "---\n",
    "\n",
    "## 5. Running the `jefe` Function\n",
    "\n",
    "The `jefe` function orchestrates the entire workflow, including converting flight lines into appropriate file formats, applying corrections, and extracting pixel data to build tables.\n",
    "\n",
    "### Parameters for `jefe`\n",
    "\n",
    "To effectively utilize the `jefe` function for processing NEON flight line data, it's crucial to understand and accurately specify its parameters. Below is a detailed guide on each parameter, including how to obtain the necessary information.\n",
    "\n",
    "#### **`base_folder` (str)**\n",
    "- **Description:** The directory where output files will be stored.\n",
    "- **How to Specify:** Choose or create a directory path on your local system where you want the processed data to be saved.\n",
    "\n",
    "#### **`site_code` (str)**\n",
    "- **Description:** The NEON site code representing the specific field site.\n",
    "- **How to Find:**\n",
    "  - NEON assigns unique four-letter codes to each field site (e.g., \"NIWO\" for Niwot Ridge).\n",
    "  - You can find these codes on the [NEON Field Sites page](https://www.neonscience.org/field-sites/explore).\n",
    "\n",
    "#### **`product_code` (str)**\n",
    "- **Description:** The NEON data product code identifying the specific data product.\n",
    "- **How to Find:**\n",
    "  - NEON data products have unique identifiers (e.g., \"DP1.30003.001\" for discrete return LiDAR point cloud data).\n",
    "  - Browse the [NEON Data Products Catalog](https://data.neonscience.org/data-products/explore) to locate the product code relevant to your research.\n",
    "\n",
    "#### **`year_month` (str)**\n",
    "- **Description:** The year and month of data collection in `'YYYY-MM'` format.\n",
    "- **How to Determine:**\n",
    "  - Data collection periods vary by site and product. Consult the [NEON Data Availability page](https://data.neonscience.org/visualizations/data-availability) to check when data was collected for your site and product of interest.\n",
    "  - **Important Note:** Data availability is subject to change due to factors like weather conditions and program planning adjustments.\n",
    "\n",
    "#### **`flight_lines` (list)**\n",
    "- **Description:** A list of flight line codes to process.\n",
    "- **How to Find:**\n",
    "  - Flight line codes correspond to specific aerial survey paths.\n",
    "  - Access the [NEON Data Portal](https://data.neonscience.org/) and navigate to the desired data product and site.\n",
    "  - Flight line codes are typically listed in the metadata associated with each dataset.\n",
    "\n",
    "---\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61b48a48-e40b-4702-8ac5-56253fe5a20f",
   "metadata": {},
   "source": [
    "### Example Usage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a0b8e5c2-6fb5-43d2-9eae-3e30aa63c8c7",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing flight line: D13_NIWO_DP1_20200807_163444\n",
      "Data retrieved successfully for 2020-08!\n",
      "Downloading NEON_D13_NIWO_DP1_20200807_163444_reflectance.h5 from https://storage.googleapis.com/neon-aop-products/2020/FullSite/D13/2020_NIWO_4/L1/Spectrometer/ReflectanceH5/2020080714/NEON_D13_NIWO_DP1_20200807_163444_reflectance.h5\n",
      "Download completed for NEON_D13_NIWO_DP1_20200807_163444_reflectance.h5\n",
      "Download completed.\n",
      "\n",
      "Processing flight line: D13_NIWO_DP1_20200807_170802\n",
      "Data retrieved successfully for 2020-08!\n",
      "Downloading NEON_D13_NIWO_DP1_20200807_170802_reflectance.h5 from https://storage.googleapis.com/neon-aop-products/2020/FullSite/D13/2020_NIWO_4/L1/Spectrometer/ReflectanceH5/2020080714/NEON_D13_NIWO_DP1_20200807_170802_reflectance.h5\n",
      "Download completed for NEON_D13_NIWO_DP1_20200807_170802_reflectance.h5\n",
      "Download completed.\n",
      "\n",
      "Processing: ./NEON_D13_NIWO_DP1_20200807_170802_reflectance.h5\n",
      "Command executed successfully\n",
      "Standard Output: Here we GO!\n",
      "\n",
      "Exporting ancillary data\n",
      "\u001b[36m(HyTools pid=49393)\u001b[0m Exported NEON_D13_NIWO_DP1_20200807_170802_reflectance to ENVI format.\n",
      "\u001b[36m(HyTools pid=49393)\u001b[0m Ancillary_Imagery/Path_Length /NIWO/Reflectance/Metadata/Ancillary_Imagery/Path_Length\n",
      "\u001b[36m(HyTools pid=49393)\u001b[0m to-sensor_Azimuth_Angle /NIWO/Reflectance/Metadata/to-sensor_Azimuth_Angle\n",
      "\u001b[36m(HyTools pid=49393)\u001b[0m to-sensor_Zenith_Angle /NIWO/Reflectance/Metadata/to-sensor_Zenith_Angle\n",
      "\u001b[36m(HyTools pid=49393)\u001b[0m Logs/Solar_Azimuth_Angle /NIWO/Reflectance/Metadata/Logs/Solar_Azimuth_Angle\n",
      "\u001b[36m(HyTools pid=49393)\u001b[0m Logs/Solar_Zenith_Angle /NIWO/Reflectance/Metadata/Logs/Solar_Zenith_Angle\n",
      "\u001b[36m(HyTools pid=49393)\u001b[0m Ancillary_Imagery/Slope /NIWO/Reflectance/Metadata/Ancillary_Imagery/Slope\n",
      "\u001b[36m(HyTools pid=49393)\u001b[0m Ancillary_Imagery/Aspect /NIWO/Reflectance/Metadata/Ancillary_Imagery/Aspect\n",
      "\u001b[36m(HyTools pid=49393)\u001b[0m [10424078, 10424078, 10424078, 1, 1, 10424078, 10424078]\n",
      "Processing complete.\n",
      "\u001b[36m(HyTools pid=49393)\u001b[0m Ancillary data exported to NIWOT_calibration_flight_08_2020/NEON_D13_NIWO_DP1_20200807_170802_reflectance/NEON_D13_NIWO_DP1_20200807_170802_reflectance_ancillary\n",
      "\n",
      "Processing: ./NEON_D13_NIWO_DP1_20200807_163444_reflectance.h5\n",
      "Command executed successfully\n",
      "Standard Output: Here we GO!\n",
      "\n",
      "Exporting ancillary data\n",
      "\u001b[36m(HyTools pid=50294)\u001b[0m Exported NEON_D13_NIWO_DP1_20200807_163444_reflectance to ENVI format.\n",
      "\u001b[36m(HyTools pid=50294)\u001b[0m Ancillary_Imagery/Path_Length /NIWO/Reflectance/Metadata/Ancillary_Imagery/Path_Length\n",
      "\u001b[36m(HyTools pid=50294)\u001b[0m to-sensor_Azimuth_Angle /NIWO/Reflectance/Metadata/to-sensor_Azimuth_Angle\n",
      "\u001b[36m(HyTools pid=50294)\u001b[0m to-sensor_Zenith_Angle /NIWO/Reflectance/Metadata/to-sensor_Zenith_Angle\n",
      "\u001b[36m(HyTools pid=50294)\u001b[0m Logs/Solar_Azimuth_Angle /NIWO/Reflectance/Metadata/Logs/Solar_Azimuth_Angle\n",
      "\u001b[36m(HyTools pid=50294)\u001b[0m Logs/Solar_Zenith_Angle /NIWO/Reflectance/Metadata/Logs/Solar_Zenith_Angle\n",
      "\u001b[36m(HyTools pid=50294)\u001b[0m Ancillary_Imagery/Slope /NIWO/Reflectance/Metadata/Ancillary_Imagery/Slope\n",
      "\u001b[36m(HyTools pid=50294)\u001b[0m Ancillary_Imagery/Aspect /NIWO/Reflectance/Metadata/Ancillary_Imagery/Aspect\n",
      "\u001b[36m(HyTools pid=50294)\u001b[0m [11746020, 11746020, 11746020, 1, 1, 11746020, 11746020]\n",
      "Processing complete.\n",
      "\u001b[36m(HyTools pid=50294)\u001b[0m Ancillary data exported to NIWOT_calibration_flight_08_2020/NEON_D13_NIWO_DP1_20200807_163444_reflectance/NEON_D13_NIWO_DP1_20200807_163444_reflectance_ancillary\n",
      "\n",
      "Generating configuration files for directory: NIWOT_calibration_flight_08_2020/NEON_D13_NIWO_DP1_20200807_170802_reflectance\n",
      "Configuration saved to NIWOT_calibration_flight_08_2020/NEON_D13_NIWO_DP1_20200807_170802_reflectance/NEON_D13_NIWO_DP1_20200807_170802_reflectance_config__anc.json\n",
      "Configuration files generation completed.\n",
      "\n",
      "Generating configuration files for directory: NIWOT_calibration_flight_08_2020/NEON_D13_NIWO_DP1_20200807_163444_reflectance\n",
      "Configuration saved to NIWOT_calibration_flight_08_2020/NEON_D13_NIWO_DP1_20200807_163444_reflectance/NEON_D13_NIWO_DP1_20200807_163444_reflectance_config__anc.json\n",
      "Configuration files generation completed.\n",
      "\n",
      "Starting topo and BRDF correction. This takes a long time.\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[4], line 14\u001b[0m\n\u001b[1;32m      6\u001b[0m flight_lines \u001b[38;5;241m=\u001b[39m [\n\u001b[1;32m      7\u001b[0m     \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mD13_NIWO_DP1_20200807_163444\u001b[39m\u001b[38;5;124m'\u001b[39m,\n\u001b[1;32m      8\u001b[0m     \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mD13_NIWO_DP1_20200807_170802\u001b[39m\u001b[38;5;124m'\u001b[39m\n\u001b[1;32m      9\u001b[0m ]\n\u001b[1;32m     10\u001b[0m \u001b[38;5;66;03m# BRDF correction is failing with only one flight line provided but works when a list is longer than one. \u001b[39;00m\n\u001b[1;32m     11\u001b[0m \n\u001b[1;32m     12\u001b[0m \u001b[38;5;66;03m# Error out after correction when it should be moving to translation. That's the function I've been trying to work and have been trying to isolate. \u001b[39;00m\n\u001b[1;32m     13\u001b[0m \u001b[38;5;66;03m# Run the jefe function with the provided example parameters\u001b[39;00m\n\u001b[0;32m---> 14\u001b[0m \u001b[43mel_spectral\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mjefe\u001b[49m\u001b[43m(\u001b[49m\u001b[43mbase_folder\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43msite_code\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mproduct_code\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43myear_month\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mflight_lines\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/data-store/cross-sensor-cal/spectral_unmixing_tools_original.py:78\u001b[0m, in \u001b[0;36mjefe\u001b[0;34m(base_folder, site_code, product_code, year_month, flight_lines)\u001b[0m\n\u001b[1;32m     65\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m     66\u001b[0m \u001b[38;5;124;03mA control function that orchestrates the processing of spectral data.\u001b[39;00m\n\u001b[1;32m     67\u001b[0m \u001b[38;5;124;03mIt first calls go_forth_and_multiply to generate necessary data and structures,\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     75\u001b[0m \u001b[38;5;124;03m- flight_lines (list): A list of flight lines for go_forth_and_multiply.\u001b[39;00m\n\u001b[1;32m     76\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m     77\u001b[0m \u001b[38;5;66;03m# First, call go_forth_and_multiply with the provided parameters\u001b[39;00m\n\u001b[0;32m---> 78\u001b[0m \u001b[43mgo_forth_and_multiply\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m     79\u001b[0m \u001b[43m    \u001b[49m\u001b[43mbase_folder\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mbase_folder\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     80\u001b[0m \u001b[43m    \u001b[49m\u001b[43msite_code\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43msite_code\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     81\u001b[0m \u001b[43m    \u001b[49m\u001b[43mproduct_code\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mproduct_code\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     82\u001b[0m \u001b[43m    \u001b[49m\u001b[43myear_month\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43myear_month\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m     83\u001b[0m \u001b[43m    \u001b[49m\u001b[43mflight_lines\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mflight_lines\u001b[49m\n\u001b[1;32m     84\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     86\u001b[0m \u001b[38;5;66;03m# Next, process all subdirectories within the base_folder\u001b[39;00m\n\u001b[1;32m     87\u001b[0m process_all_subdirectories(base_folder)\n",
      "File \u001b[0;32m~/data-store/cross-sensor-cal/spectral_unmixing_tools_original.py:107\u001b[0m, in \u001b[0;36mgo_forth_and_multiply\u001b[0;34m(base_folder, **kwargs)\u001b[0m\n\u001b[1;32m    104\u001b[0m generate_config_json(base_folder)\n\u001b[1;32m    106\u001b[0m \u001b[38;5;66;03m# Step 4: Apply topographic and BRDF corrections\u001b[39;00m\n\u001b[0;32m--> 107\u001b[0m \u001b[43mapply_topo_and_brdf_corrections\u001b[49m\u001b[43m(\u001b[49m\u001b[43mbase_folder\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    109\u001b[0m \u001b[38;5;66;03m# Step 5: Resample and translate data to other sensor formats\u001b[39;00m\n\u001b[1;32m    110\u001b[0m resample_translation_to_other_sensors(base_folder)\n",
      "File \u001b[0;32m~/data-store/cross-sensor-cal/spectral_unmixing_tools_original.py:433\u001b[0m, in \u001b[0;36mapply_topo_and_brdf_corrections\u001b[0;34m(base_folder_path, conda_env_path)\u001b[0m\n\u001b[1;32m    430\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m os\u001b[38;5;241m.\u001b[39mpath\u001b[38;5;241m.\u001b[39misfile(json_file_path):\n\u001b[1;32m    431\u001b[0m     \u001b[38;5;66;03m# Call the script with the JSON file path\u001b[39;00m\n\u001b[1;32m    432\u001b[0m     command \u001b[38;5;241m=\u001b[39m [python_executable, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mimage_correct.py\u001b[39m\u001b[38;5;124m\"\u001b[39m, json_file_path]  \u001b[38;5;66;03m# Use list to avoid shell interpretation\u001b[39;00m\n\u001b[0;32m--> 433\u001b[0m     process \u001b[38;5;241m=\u001b[39m \u001b[43msubprocess\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcommand\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mstdout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43msubprocess\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mPIPE\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mstderr\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43msubprocess\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mPIPE\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtext\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m)\u001b[49m\n\u001b[1;32m    435\u001b[0m     \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mProcessed \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mjson_file_path\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m    436\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m process\u001b[38;5;241m.\u001b[39mreturncode \u001b[38;5;241m!=\u001b[39m \u001b[38;5;241m0\u001b[39m:\n",
      "File \u001b[0;32m/opt/conda/envs/macrosystems/lib/python3.10/subprocess.py:505\u001b[0m, in \u001b[0;36mrun\u001b[0;34m(input, capture_output, timeout, check, *popenargs, **kwargs)\u001b[0m\n\u001b[1;32m    503\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m Popen(\u001b[38;5;241m*\u001b[39mpopenargs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs) \u001b[38;5;28;01mas\u001b[39;00m process:\n\u001b[1;32m    504\u001b[0m     \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 505\u001b[0m         stdout, stderr \u001b[38;5;241m=\u001b[39m \u001b[43mprocess\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcommunicate\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43minput\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtimeout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtimeout\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    506\u001b[0m     \u001b[38;5;28;01mexcept\u001b[39;00m TimeoutExpired \u001b[38;5;28;01mas\u001b[39;00m exc:\n\u001b[1;32m    507\u001b[0m         process\u001b[38;5;241m.\u001b[39mkill()\n",
      "File \u001b[0;32m/opt/conda/envs/macrosystems/lib/python3.10/subprocess.py:1154\u001b[0m, in \u001b[0;36mPopen.communicate\u001b[0;34m(self, input, timeout)\u001b[0m\n\u001b[1;32m   1151\u001b[0m     endtime \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m   1153\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m-> 1154\u001b[0m     stdout, stderr \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_communicate\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43minput\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mendtime\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtimeout\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1155\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mKeyboardInterrupt\u001b[39;00m:\n\u001b[1;32m   1156\u001b[0m     \u001b[38;5;66;03m# https://bugs.python.org/issue25942\u001b[39;00m\n\u001b[1;32m   1157\u001b[0m     \u001b[38;5;66;03m# See the detailed comment in .wait().\u001b[39;00m\n\u001b[1;32m   1158\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m timeout \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n",
      "File \u001b[0;32m/opt/conda/envs/macrosystems/lib/python3.10/subprocess.py:2021\u001b[0m, in \u001b[0;36mPopen._communicate\u001b[0;34m(self, input, endtime, orig_timeout)\u001b[0m\n\u001b[1;32m   2014\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_check_timeout(endtime, orig_timeout,\n\u001b[1;32m   2015\u001b[0m                         stdout, stderr,\n\u001b[1;32m   2016\u001b[0m                         skip_check_and_raise\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m)\n\u001b[1;32m   2017\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mRuntimeError\u001b[39;00m(  \u001b[38;5;66;03m# Impossible :)\u001b[39;00m\n\u001b[1;32m   2018\u001b[0m         \u001b[38;5;124m'\u001b[39m\u001b[38;5;124m_check_timeout(..., skip_check_and_raise=True) \u001b[39m\u001b[38;5;124m'\u001b[39m\n\u001b[1;32m   2019\u001b[0m         \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mfailed to raise TimeoutExpired.\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[0;32m-> 2021\u001b[0m ready \u001b[38;5;241m=\u001b[39m \u001b[43mselector\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mselect\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtimeout\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   2022\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_check_timeout(endtime, orig_timeout, stdout, stderr)\n\u001b[1;32m   2024\u001b[0m \u001b[38;5;66;03m# XXX Rewrite these to use non-blocking I/O on the file\u001b[39;00m\n\u001b[1;32m   2025\u001b[0m \u001b[38;5;66;03m# objects; they are no longer using C stdio!\u001b[39;00m\n",
      "File \u001b[0;32m/opt/conda/envs/macrosystems/lib/python3.10/selectors.py:416\u001b[0m, in \u001b[0;36m_PollLikeSelector.select\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m    414\u001b[0m ready \u001b[38;5;241m=\u001b[39m []\n\u001b[1;32m    415\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 416\u001b[0m     fd_event_list \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_selector\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpoll\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtimeout\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    417\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mInterruptedError\u001b[39;00m:\n\u001b[1;32m    418\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m ready\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# el jefe takes 3-5 hours to run and it creates a lot of files. You should have 200+GB of RAM and Storage available.\n",
    "base_folder = \"NIWOT_calibration_flight_08_2020\"\n",
    "site_code = 'NIWO'\n",
    "product_code = 'DP1.30006.001'\n",
    "year_month = '2020-08'\n",
    "flight_lines = [\n",
    "    'D13_NIWO_DP1_20200807_170802'\n",
    "]\n",
    "# BRDF correction is failing with only one flight line provided but works when a list is longer than one. \n",
    "\n",
    "# Error out after correction when it should be moving to translation. That's the function I've been trying to work and have been trying to isolate. \n",
    "# Run the jefe function with the provided example parameters\n",
    "el_spectral.jefe(base_folder, site_code, product_code, year_month, flight_lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef1c9354-0020-4901-8e8b-0c89c755d759",
   "metadata": {},
   "source": [
    "### What Happens When `jefe` Runs\n",
    "\n",
    "When you run the `jefe` function, a sequence of operations is executed, and multiple outputs are generated. Here's a detailed breakdown:\n",
    "\n",
    "1. **Downloading Raw Data:**\n",
    "   - The original NEON flight line folder is downloaded to the specified output directory.\n",
    "   - The raw folder contains the reflectance data and associated metadata files.\n",
    "\n",
    "2. **Conversion to Multiple Formats:**\n",
    "   - The downloaded folder is processed to generate additional formats required for analysis.\n",
    "   - These files are named systematically to represent the processing step or correction applied. For example:\n",
    "     - **`_envi`:** Reflectance data in ENVI format.\n",
    "     - **`_envi_mask`:** Mask files indicating areas to include or exclude during analysis.\n",
    "     - **`.hdr`:** Header files describing the structure of the associated data.\n",
    "     - **`.json`:** Configuration files for corrections and processing steps.\n",
    "\n",
    "3. **Application of Corrections:**\n",
    "   - Topographic corrections (TOPO) and bidirectional reflectance distribution function (BRDF) corrections are applied to ensure data accuracy.\n",
    "   - Outputs include:\n",
    "     - **`_brdf_coeffs__envi.json`:** Coefficients for BRDF corrections.\n",
    "     - **`_topo_coeffs__envi.json`:** Coefficients for topographic corrections.\n",
    "\n",
    "4. **Data Extraction and Processing:**\n",
    "   - Spectral data is extracted pixel by pixel and saved in tabular formats for further analysis.\n",
    "   - These extractions are saved incrementally to avoid memory overuse.\n",
    "\n",
    "---\n",
    "\n",
    "### Example Outputs from a Single Flight Line\n",
    "\n",
    "After running the `jefe` function, the output directory contains processed files at the top level and a folder for the original raw data. Here’s what you can expect for a single trial run:\n",
    "\n",
    "---\n",
    "\n",
    "#### **Main Output Directory:**\n",
    "- **Processed Files:** Includes ENVI-format files, masks, headers, and configuration files. These represent the final processed outputs ready for analysis.\n",
    "- **Raw Folder:** A subdirectory containing the original reflectance data downloaded from NEON.\n",
    "\n",
    "| File Name                                             | Description                                         |\n",
    "|-------------------------------------------------------|-----------------------------------------------------|\n",
    "| `NEON_D13_NIWO_DP1_20200807_170802_reflectance__envi` | Reflectance data converted to ENVI format.         |\n",
    "| `NEON_D13_NIWO_DP1_20200807_170802_reflectance__mask` | Mask file for the reflectance data.                |\n",
    "| `NEON_D13_NIWO_DP1_20200807_170802_reflectance.hdr`   | Header file describing the ENVI data structure.    |\n",
    "| `NEON_D13_NIWO_DP1_20200807_170802_reflectance__brdf_coeffs__envi.json` | BRDF correction coefficients. |\n",
    "| `NEON_D13_NIWO_DP1_20200807_170802_reflectance__topo_coeffs__envi.json` | TOPO correction coefficients. |\n",
    "\n",
    "---\n",
    "\n",
    "#### **Raw Folder (Inside the Output Directory):**\n",
    "- **Original Files:** Contains the raw reflectance data downloaded directly from NEON before any processing steps.\n",
    "\n",
    "| File Name                                             | Description                                         |\n",
    "|-------------------------------------------------------|-----------------------------------------------------|\n",
    "| `NEON_D13_NIWO_DP1_20200807_170802_reflectance`       | Original reflectance data from NEON.               |\n",
    "| `NEON_D13_NIWO_DP1_20200807_170802_reflectance_ancillary` | Ancillary metadata for corrections.               |\n",
    "| `NEON_D13_NIWO_DP1_20200807_170802_reflectance_config__envi.json` | Configuration for ENVI data processing. |\n",
    "| `NEON_D13_NIWO_DP1_20200807_170802_reflectance_config__anc.json`  | Configuration for ancillary corrections. |\n",
    "\n",
    "---\n",
    "\n",
    "This structure ensures that:\n",
    "1. The **processed files** are readily available in the main directory for analysis.\n",
    "2. The **raw data** is preserved in its original form for reference or reprocessing if needed.\n",
    "\n",
    "By organizing outputs this way, you can easily navigate between raw and processed data while maintaining a clear workflow history.\n",
    "\n",
    "\n",
    "### Process Overview\n",
    "\n",
    "1. **Data Conversion:**\n",
    "   - Converts NEON reflectance data to formats compatible with ENVI tools and downstream analyses.\n",
    "\n",
    "2. **Data Corrections:**\n",
    "   - Applies topographic and BRDF corrections to improve data quality.\n",
    "\n",
    "3. **Outputs Generated:**\n",
    "   - Reflectance data in corrected formats.\n",
    "   - Mask files for regions of interest.\n",
    "   - Configuration files describing the processing steps.\n",
    "   - Coefficients for TOPO and BRDF corrections.\n",
    "\n",
    "By the end of this process, you will have a comprehensive set of files ready for analysis, including corrected reflectance data, metadata, and configurations.\n",
    "\n",
    "---\n",
    "\n",
    "## 6. Handling Large Data Processing<a name=\"handling-large-data-processing\"></a>\n",
    "\n",
    "Processing NEON flight lines involves managing large amounts of spectral data. This workflow incorporates strategies to optimize memory usage and prevent bottlenecks.\n",
    "\n",
    "### Key Strategies\n",
    "\n",
    "1. **Chunk Processing:** Processes data in smaller chunks to avoid memory overload.\n",
    "2. **Direct Disk Writing:** Saves intermediate and final results directly to storage.\n",
    "3. **Optimized Data Structures:** Uses efficient formats like NumPy arrays and Pandas DataFrames.\n",
    "4. **Parallel Processing:** Utilizes libraries like `ray` for distributed processing.\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "## 9. Conclusion<a name=\"conclusion\"></a>\n",
    "\n",
    "This vignette provided a comprehensive, step-by-step guide to processing NEON flight line data, highlighting key techniques and strategies for handling large, complex datasets. The workflow included downloading NEON flight lines, converting them into suitable file formats, applying critical corrections, and extracting hyperspectral data from pixels before writing the results to CSV files for further numerical analysis.\n",
    "\n",
    "By completing this process, you gain the ability to transform raw NEON airborne data into actionable datasets, enabling robust ecological and environmental research. This workflow is designed to balance efficiency, accuracy, and scalability, ensuring that even massive datasets can be processed on machines with limited resources.\n",
    "\n",
    "### **Key Takeaways**\n",
    "1. **Efficient Data Handling:** \n",
    "   - From downloading raw flight line data to saving corrected and processed outputs, this workflow demonstrates how to manage large-scale operations effectively.\n",
    "   - Chunk processing and direct-to-disk writing ensure that memory constraints are respected while maintaining high data fidelity.\n",
    "\n",
    "2. **Robust Data Corrections:** \n",
    "   - The inclusion of topographic and BRDF corrections ensures that the processed data is accurate and reliable for downstream analysis, accounting for variability in reflectance and terrain.\n",
    "\n",
    "3. **Hyperspectral Data for Analysis:** \n",
    "   - The extraction of hyperspectral data from individual pixels provides a valuable resource for detailed numerical and statistical studies, enabling deeper insights into ecological and environmental processes.\n",
    "\n",
    "4. **Scalability and Reproducibility:** \n",
    "   - This workflow is scalable to handle additional flight lines, datasets, and sites, making it a versatile tool for researchers working across diverse geographies and ecological systems.\n",
    "   - By following standardized steps and leveraging robust tools, you can ensure that your processing is reproducible and aligned with scientific best practices.\n",
    "\n",
    "---\n",
    "\n",
    "## 10. References<a name=\"references\"></a>\n",
    "\n",
    "- **NEON Data Portal:** [https://data.neonscience.org/](https://data.neonscience.org/)\n",
    "- **GeoPandas Documentation:** [https://geopandas.org/](https://geopandas.org/)\n",
    "- **Rasterio Documentation:** [https://rasterio.readthedocs.io/](https://rasterio.readthedocs.io/)\n",
    "- **NumPy Documentation:** [https://numpy.org/doc/](https://numpy.org/doc/)\n",
    "- **HyTools Documentation:** [https://hytools.readthedocs.io/](https://hytools.readthedocs.io/)\n",
    "- **Ray Documentation:** [https://docs.ray.io/en/latest/](https://docs.ray.io/en/latest/)\n",
    "- **NEON Field Sites Page:** [https://www.neonscience.org/field-sites/explore](https://www.neonscience.org/field-sites/explore)\n",
    "- **NEON Data Products Catalog:** [https://data.neonscience.org/data-products/explore](https://data.neonscience.org/data-products/explore)\n",
    "- **NEON Data Availability Page:** [https://data.neonscience.org/visualizations/data-availability](https://data.neonscience.org/visualizations/data-availability)\n",
    "- **NEON Flight Schedules and Coverage:** [https://www.neonscience.org/data-collection/flight-schedules-coverage](https://www.neonscience.org/data-collection/flight-schedules-coverage)\n",
    "- **AOP Data Availability Notification:** [https://www.neonscience.org/impact/observatory-blog/aop-data-availability-notification-release-2024](https://www.neonscience.org/impact/observatory-blog/aop-data-availability-notification-release-2024)\n",
    "\n",
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "macrosystems",
   "language": "python",
   "name": "macrosystems"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}