ML Hierarchical Forecasting Missing Fitted Values

[Avidar123]

Im attempting on using the Hierarchical Forecasting feature in nixtal when i ran across this problem:

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Cell In[96], line 13
      5 reconcilers = [
      6     TopDown('forecast_proportions'),
      7     BottomUp(),
      8     MinTrace('ols'),
      9 ]
     11 hrec = HierarchicalReconciliation(reconcilers=reconcilers)
---> 13 Y_rec_mf = hrec.reconcile(Y_hat_df=yhat_test, Y_df=insample_mf, S=S_df, tags=tags)

File /opt/conda/lib/python3.10/site-packages/hierarchicalforecast/core.py:226, in HierarchicalReconciliation.reconcile(self, Y_hat_df, S, tags, Y_df, level, intervals_method, num_samples, seed, sort_df, is_balanced)
    193 """Hierarchical Reconciliation Method.
    194 
    195 The `reconcile` method is analogous to SKLearn `fit_predict` method, it 
   (...)
    222 `Y_tilde_df`: pd.DataFrame, with reconciled predictions.
    223 """
    224 # Check input's validity and sort dataframes
    225 Y_hat_df, S_df, Y_df, self.model_names = \
--> 226             self._prepare_fit(Y_hat_df=Y_hat_df,
    227                               S_df=S,
    228                               Y_df=Y_df,
    229                               tags=tags,
    230                               level=level,
    231                               intervals_method=intervals_method,
    232                               sort_df=sort_df)
    234 # Initialize reconciler arguments
    235 reconciler_args = dict(
    236     idx_bottom=S_df.index.get_indexer(S.columns),
    237     tags={key: S_df.index.get_indexer(val) for key, val in tags.items()}
    238 )

File /opt/conda/lib/python3.10/site-packages/hierarchicalforecast/core.py:174, in HierarchicalReconciliation._prepare_fit(self, Y_hat_df, S_df, Y_df, tags, level, intervals_method, sort_df)
    172     Y_hat_diff = len(Y_hat_df.index.difference(Y_df.index.unique()))
    173     if Y_diff > 0 or Y_hat_diff > 0:
--> 174         raise Exception(f'Check `Y_hat_df`, `Y_df` series difference, Y_hat\Y={Y_hat_diff}, Y\Y_hat={Y_diff}')
    176 # Same Y_hat_df/S_df/Y_df's unique_id order to prevent errors
    177 S_df = S_df.loc[uids]

Exception: Check `Y_hat_df`, `Y_df` series difference, Y_hat\Y=3508, Y\Y_hat=0

Then i attempted to check the number of unique_ids for both the forecast df and insample prediction df and found that both are indeed different. The code i used to check is below:

mf.fit(df = df,
       static_features=[],
       fitted = True,
       )


X_df = mf.make_future_dataframe(h=horizon)
X_df = _get_future_exog_hiech(X_df, df[["ds", "unique_id"]])

yhat_test = mf.predict(h=horizon, X_df=X_df)
yhat_test_unique = set(yhat_test['unique_id'].unique())
insample_mf = mf.forecast_fitted_values()
insample_mf_unique = set(insample_mf['unique_id'].unique())
print(len(insample_mf_unique))
print(len(yhat_test_unique))

the result for 'print(len(insample_mf_unique))' is 11864
and the result for 'print(len(yhat_test_unique))' is 15372
and df has 15372 unique_id's so for some reason the insample fitted values are dropping unique_id's for some reason.
My entire code is below:

import lightgbm as lgb
import xgboost as xgb
import pandas as pd
import numpy as np
import warnings
warnings.filterwarnings("ignore")
from mlforecast import MLForecast
from mlforecast.feature_engineering import transform_exog
from mlforecast.lag_transforms import RollingMean
from utilsforecast.evaluation import evaluate
from typing import List
from mlforecast.auto import (
    AutoLightGBM,
    AutoXGBoost,
    AutoMLForecast
)
from utilsforecast.losses import mae, bias, rmse
import optuna
from mlforecast.lag_transforms import RollingMean, RollingStd, RollingMin, RollingMax, RollingQuantile, SeasonalRollingMean, SeasonalRollingStd, SeasonalRollingMin, SeasonalRollingMax, SeasonalRollingQuantile, ExpandingMin, ExpandingMax, ExpandingMean, ExpandingQuantile, ExpandingStd
import numpy as np
import pandas as pd
import scipy.stats as st
import statsmodels as sm
import warnings
# Matplotlib e Seaborn
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
sns.set(style="white", palette="muted", color_codes=True)
# Plotly
from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot
import plotly.graph_objs as go
from hierarchicalforecast.core import HierarchicalReconciliation
from hierarchicalforecast.methods import BottomUp, TopDown, MinTrace
from hierarchicalforecast.utils import aggregate
from hierarchicalforecast.evaluation import HierarchicalEvaluation

# Read price data
price = pd.read_csv("/kaggle/input/vn1-data-set/Phase 0 - Price.csv", na_values=np.nan)
df_price = price.copy()
price["Value"] = "Price"
price = price.set_index(["Client", "Warehouse","Product", "Value"]).stack()

# Read sales data
sales = pd.read_csv("/kaggle/input/vn1-data-set/Phase 0 - Sales.csv", na_values=np.nan)
df_sales = sales.copy()
sales["Value"] = "Sales"
sales = sales.set_index(["Client", "Warehouse","Product", "Value"]).stack()

# Create single dataframe
df = pd.concat([price, sales]).unstack("Value").reset_index()
df.columns = ["Client", "Warehouse", "Product", "ds", "Price", "y"]
df["ds"] = pd.to_datetime(df["ds"])
df = df.astype({"Price": np.float32,
                "y": np.float32,
                "Client": "category",
                "Warehouse": "category",
                "Product": "category",
                })

# Add unique_id and sort
df["unique_id"] = df["Client"].astype(str) + "/" + df["Warehouse"].astype(str) + "/" + df["Product"].astype(str)
df = df.sort_values(by=["unique_id", "ds"]).reset_index(drop=True)

# There is implied target leakage in the Price feature - where price is unavailable, there is zero sales, and vice versa. So, we first lag the Price feature by one.
# Lag price feature by 1
df_price_lagged = transform_exog(
    df[["unique_id", "ds", "Price"]],
    lags=[1],
)
df = df.merge(df_price_lagged[["unique_id", "ds", "Price_lag1"]], left_on=["unique_id", "ds"], right_on=["unique_id", "ds"])
df = df.drop(columns=["Price"])
df = df.rename(columns={"Price_lag1": "Price"})

#  We subsequently deal with it very basic: we fill the unknown prices with the mean price of the product from the same client at the same week, and the remaining Price NaNs are forward-filled.
mean_price_per_product = df.groupby(["Client", "Product", "ds"], observed=True)["Price"].mean().reset_index()
df = df.merge(mean_price_per_product, left_on=["Client", "Product", "ds"], right_on=["Client", "Product", "ds"], suffixes=("", "_mean"))
df["Price_mean"] =  df.groupby(["Client", "Product"], observed=True)["Price_mean"].ffill()
df["Price"] = df["Price"].fillna(df["Price_mean"])
df = df.drop(columns="Price_mean")

df['Top Level'] = 'Top'
df.dropna(inplace = True)
df['Client'] = df['Client'].astype(str)
df['Warehouse'] = df['Warehouse'].astype(str)
df['Product'] = df['Product'].astype(str)
hierarchy_levels = [['Top Level'],
                    ['Top Level','Client'],
                    ['Top Level','Client', 'Warehouse'],
                    ['Top Level','Client', 'Warehouse', 'Product']]
df_sales = df[['Top Level', 'Client', 'Warehouse', 'Product', 'ds', 'y']].copy()
df_price = df[['Top Level', 'Client', 'Warehouse', 'Product', 'ds', 'Price']].copy()
df_price = df_price.rename(columns = {'Price': 'y'})
Y_hier_df, S_df, tags = aggregate(df=df_sales, spec=hierarchy_levels)
Price_hier_df, S_df_price, tag_price = aggregate(df=df_price, spec=hierarchy_levels)
Price_hier_df = Price_hier_df.rename(columns = {'y': 'Price'})
df = Y_hier_df.merge(Price_hier_df, left_on=["unique_id", "ds"], right_on=["unique_id", "ds"])
df = Y_hier_df.copy()
df = df[df['ds'] != df['ds'].min()].copy()
df.reset_index(inplace=True) 

# Target lag transforms
def _get_lag_transforms(lag_transform_weeks = [2, 4, 6, 8]):
    lag_transforms = []
    for lag in lag_transform_weeks:
        lag_transforms.append(RollingMean(window_size=lag))
        lag_transforms.append(RollingStd(window_size=lag))
        lag_transforms.append(RollingMax(window_size=lag))

    return lag_transforms

# Target lags
def _get_lags(lags = list(range(1, 14)) + [50, 51, 52, 53, 54]):
    return lags

# Price transforms
def _get_price_transforms(df):
# Transform price feature
    transformed = transform_exog(
        df[["unique_id", "ds", "Price"]],
        lags=[1,2,3,4],
    )
# Add deltas
    transformed["Delta_Price_lag1"] = transformed["Price"] - transformed["Price_lag1"]
    transformed["RelDelta_Price_lag1"] = transformed["Price"] / transformed["Price_lag1"] - 1
# Merge
    df_price = transformed.drop(columns=["Price", "Price_lag1"])
    df = df.merge(df_price, left_on = ["unique_id", "ds"], right_on=["unique_id", "ds"])
    return df

# Function required to generate the future exogenous (mainly Price) for the test set
def _get_future_exog(X_df, df):
    X_df_idx = X_df.set_index(["unique_id", "ds"]).index
    df_last_price = df.groupby(["unique_id"])["Price"].last()
    X_df = X_df.merge(df_last_price, left_on=["unique_id"], right_on=["unique_id"])
    X_df[["Client", "Warehouse", "Product"]] = X_df["unique_id"].str.split("/", expand=True)
    df_test = df.copy().merge(X_df, left_on=["unique_id", "ds", "Client", "Warehouse", "Product", "Price"], right_on=["unique_id", "ds", "Client", "Warehouse", "Product", "Price"], how="outer")
    df_test = _get_price_transforms(df_test)
    df_test = _get_date_features(df_test)
    df_test = df_test.drop(columns=["Client", "Warehouse", "Product"])
    df_test = df_test.set_index(["unique_id", "ds"])
    X_df = df_test.loc[X_df_idx]
    price_lag_cols = [col for col in X_df.columns if "Price_lag" in col]
    X_df[price_lag_cols] = X_df[price_lag_cols].fillna(0)
    X_df = X_df.reset_index()
    return X_df

def _get_future_exog_hiech(X_df, df):
    X_df_idx = X_df.set_index(["unique_id", "ds"]).index
    df_last_price = df.groupby(["unique_id"])["Price"].last()
    X_df = X_df.merge(df_last_price, left_on=["unique_id"], right_on=["unique_id"])
    X_df[["Top Level", "Client", "Warehouse", "Product"]] = X_df["unique_id"].str.split("/", expand=True)
    df[["Top Level", "Client", "Warehouse", "Product"]] = df["unique_id"].str.split("/", expand=True)
    df_test = df.copy().merge(X_df, left_on=["unique_id", "ds", "Client", "Warehouse", "Product", "Price", "Top Level"], right_on=["unique_id", "ds", "Client", "Warehouse", "Product", "Price", "Top Level"], how="outer")
    df_test = _get_price_transforms(df_test)
    df_test = _get_date_features(df_test)
    df_test = df_test.drop(columns=["Client", "Warehouse", "Product", "Top Level"])
    df_test = df_test.set_index(["unique_id", "ds"])
    X_df = df_test.loc[X_df_idx]
    price_lag_cols = [col for col in X_df.columns if "Price_lag" in col]
    X_df[price_lag_cols] = X_df[price_lag_cols].fillna(0)
    X_df = X_df.reset_index()
    return X_df
    
# We use the competition metric to evaluate our cross-validation predictions.
def CompetitionMetric(
    df: pd.DataFrame,
    models: List[str],
    id_col: str = "unique_id",
    target_col: str = "y",
) -> pd.DataFrame:
    """Computes the competition metric"""

    error = df[models].sub(df[target_col], axis=0)
    sum_abs_error = error.abs().sum()
    abs_sum_error = error.sum().abs()
    score = sum_abs_error + abs_sum_error
    score /= df[target_col].sum()

    score.index.name = id_col
    score = score.reset_index()

    return score

def loss(df, train_df):
  x = df.shape[0]
  mae_value = mae(df, models=["model"])["model"].mean()
  mae_value = mae_value * x
  bias_value = bias(df, models=["model"])["model"].mean()
  bias_value = bias_value * x
  sum_of_actual = df['y'].astype(np.float64).sum()
  score = (mae_value + abs(bias_value)) / sum_of_actual
  return score

def final_output(df):
  # We use the name of the model
  name = "Submission Phase 1 - LGBM.csv"
  # Apply some data wrangling to ensure everything is in the expected format of the competition
  df[["Client", "Warehouse", "Product"]] = df["unique_id"].str.split("/", expand=True)
  df[["Client", "Warehouse", "Product"]] = df[["Client", "Warehouse", "Product"]].astype(np.int64)
  df = df.sort_values(by=["Client", "Warehouse", "Product"]).reset_index(drop=True)
  df = df.drop(columns = "unique_id")
  df = df.set_index(["Client", "Warehouse", "Product", "ds"])
  df = df.unstack(3)
  df.columns = df.columns.get_level_values(1)
  df.to_csv(name)
  from google.colab import files
  files.download(name)

df_price = df[["unique_id", "ds", "Price"]].copy()
df_price = _get_price_transforms(df_price)
df_price = df_price.drop(columns = "Price")
df = df.merge(df_price, left_on = ["unique_id", "ds"], right_on=["unique_id", "ds"])

# Add date features
df = _get_date_features(df)

df.replace([np.inf, -np.inf], 0, inplace=True)

#%% Model inputs
horizon = 13
freq = "W-MON"

# We use a LightGBMRegressor with Tweedie loss, a loss function that is known to perform well on zero-inflated problems (i.e. demand forecasting)
models = [xgb.XGBRegressor(objective="reg:tweedie", enable_categorical = True, device = "cuda")]


# Instantiate MLForecast
mf = MLForecast(
    models = models,
    freq=freq,
    lags=_get_lags(),
    lag_transforms={
        1: _get_lag_transforms(),
        13: _get_lag_transforms(),
    },
    date_features = ['week', 'month', 'year'],
)

# Fit the model
mf.fit(df = df,
#        static_features=["Client", "Warehouse", "Product"],
       static_features=[],
       fitted = True,
       )

# Create dataframe with future exogenous
X_df = mf.make_future_dataframe(h=horizon)
X_df = _get_future_exog_hiech(X_df, df[["ds", "unique_id"]])


# Generate test predictions
yhat_test = mf.predict(h=horizon, X_df=X_df)
yhat_test_unique = set(yhat_test['unique_id'].unique())
insample_mf = mf.forecast_fitted_values()
insample_mf_unique = set(insample_mf['unique_id'].unique())
print(len(insample_mf_unique))
print(len(yhat_test_unique))

yhat_test.set_index('unique_id', inplace = True)

insample_mf = mf.forecast_fitted_values()

reconcilers = [
    TopDown('forecast_proportions'),
    BottomUp(),
    MinTrace('ols'),
]

hrec = HierarchicalReconciliation(reconcilers=reconcilers)

Y_rec_mf = hrec.reconcile(Y_hat_df=yhat_test, Y_df=insample_mf, S=S_df, tags=tags)

The file i used for this code and uploaded below:

Uploading Phase 0 - Sales.csv…

Uploading Phase 0 - Price.csv…

[elephaint]

Hi, thanks for using MF! I created the starter notebooks for that competition, so let me check. My first hunch would be that the dropped ids have no sales at all during the insample period. I'll have a run with your notebook locally to verify.

[elephaint]

Hi, the code errors on this line:

df_price = df[["unique_id", "ds", "Price"]].copy()

Could you make the example so that it runs until it gives the reconciliation error? I couldn't easily fix your code, and I am sure it's an easy fix for you.

[Avidar123]

let me try and ill send you the code if its not fixed

[Avidar123]

as you said @elephaint I ran this whole code so it should work but i got these print out that i didnt get before:

/usr/local/lib/python3.10/dist-packages/utilsforecast/processing.py:762: UserWarning:

The following series are too short for the window and will be dropped: ['Top/13/267', 'Top/18/72', 'Top/19/29', 'Top/19/293', 'Top/19/32', 'Top/32/205', ...]

/usr/local/lib/python3.10/dist-packages/mlforecast/core.py:417: UserWarning:

The following series were dropped completely due to the transformations and features: ['Top/0/1/11000', 'Top/0/1/14311', 'Top/0/234/10967', 'Top/0/234/14147', 'Top/0/242/11756', 'Top/0/242/1226', ...].
These series won't show up if you use `MLForecast.forecast_fitted_values()`.
You can set `dropna=False` or use transformations that require less samples to mitigate this

/usr/local/lib/python3.10/dist-packages/mlforecast/core.py:594: UserWarning:

Found null values in lag3, lag4, lag5, lag6, lag7, lag8, lag9, lag10, lag11, lag12, lag13, lag50, lag51, lag52, lag53, lag54, rolling_mean_lag1_window_size2, rolling_std_lag1_window_size2, rolling_max_lag1_window_size2, rolling_mean_lag1_window_size4, rolling_std_lag1_window_size4, rolling_max_lag1_window_size4, rolling_mean_lag1_window_size6, rolling_std_lag1_window_size6, rolling_max_lag1_window_size6, rolling_mean_lag1_window_size8, rolling_std_lag1_window_size8, rolling_max_lag1_window_size8, rolling_mean_lag13_window_size2, rolling_std_lag13_window_size2, rolling_max_lag13_window_size2, rolling_mean_lag13_window_size4, rolling_std_lag13_window_size4, rolling_max_lag13_window_size4, rolling_mean_lag13_window_size6, rolling_std_lag13_window_size6, rolling_max_lag13_window_size6, rolling_mean_lag13_window_size8, rolling_std_lag13_window_size8, rolling_max_lag13_window_size8, Price_lag2, Price_lag3, Price_lag4, RelDelta_Price_lag1.

the code i used is below:

# Read price data
price = pd.read_csv("Phase 0 - Price.csv", na_values=np.nan)
df_price = price.copy()
price["Value"] = "Price"
price = price.set_index(["Client", "Warehouse","Product", "Value"]).stack()

# Read sales data
sales = pd.read_csv("Phase 0 - Sales.csv", na_values=np.nan)
df_sales = sales.copy()
sales["Value"] = "Sales"
sales = sales.set_index(["Client", "Warehouse","Product", "Value"]).stack()

# Create single dataframe
df = pd.concat([price, sales]).unstack("Value").reset_index()
df.columns = ["Client", "Warehouse", "Product", "ds", "Price", "y"]
df["ds"] = pd.to_datetime(df["ds"])
df = df.astype({"Price": np.float32,
                "y": np.float32,
                "Client": "category",
                "Warehouse": "category",
                "Product": "category",
                })

# Add unique_id and sort
df["unique_id"] = df["Client"].astype(str) + "/" + df["Warehouse"].astype(str) + "/" + df["Product"].astype(str)
df = df.sort_values(by=["unique_id", "ds"]).reset_index(drop=True)

# There is implied target leakage in the Price feature - where price is unavailable, there is zero sales, and vice versa. So, we first lag the Price feature by one.
# Lag price feature by 1
df_price_lagged = transform_exog(
    df[["unique_id", "ds", "Price"]],
    lags=[1],
)
df = df.merge(df_price_lagged[["unique_id", "ds", "Price_lag1"]], left_on=["unique_id", "ds"], right_on=["unique_id", "ds"])
df = df.drop(columns=["Price"])
df = df.rename(columns={"Price_lag1": "Price"})

#  We subsequently deal with it very basic: we fill the unknown prices with the mean price of the product from the same client at the same week, and the remaining Price NaNs are forward-filled.
mean_price_per_product = df.groupby(["Client", "Product", "ds"], observed=True)["Price"].mean().reset_index()
df = df.merge(mean_price_per_product, left_on=["Client", "Product", "ds"], right_on=["Client", "Product", "ds"], suffixes=("", "_mean"))
df["Price_mean"] =  df.groupby(["Client", "Product"], observed=True)["Price_mean"].ffill()
df["Price"] = df["Price"].fillna(df["Price_mean"])
df = df.drop(columns="Price_mean")

df['Top Level'] = 'Top'
df.dropna(inplace = True)
df['Client'] = df['Client'].astype(str)
df['Warehouse'] = df['Warehouse'].astype(str)
df['Product'] = df['Product'].astype(str)
hierarchy_levels = [['Top Level'],
                    ['Top Level','Client'],
                    ['Top Level','Client', 'Warehouse'],
                    ['Top Level','Client', 'Warehouse', 'Product']]
df_sales = df[['Top Level', 'Client', 'Warehouse', 'Product', 'ds', 'y']].copy()
df_price = df[['Top Level', 'Client', 'Warehouse', 'Product', 'ds', 'Price']].copy()
df_price = df_price.rename(columns = {'Price': 'y'})
Y_hier_df, S_df, tags = aggregate(df=df_sales, spec=hierarchy_levels)
Price_hier_df, S_df_price, tag_price = aggregate(df=df_price, spec=hierarchy_levels)
Price_hier_df = Price_hier_df.rename(columns = {'y': 'Price'})
df = Y_hier_df.merge(Price_hier_df, left_on=["unique_id", "ds"], right_on=["unique_id", "ds"])
df = df[df['ds'] != df['ds'].min()].copy()
df.reset_index(inplace=True) 

# Date features
def _get_date_features(df):
    df["Week"] = df["ds"].dt.isocalendar().week
    df["Month"] = df["ds"].dt.month
    df["Year"] = df["ds"].dt.year

    return df

# Target lag transforms
def _get_lag_transforms(lag_transform_weeks = [2, 4, 6, 8]):
    lag_transforms = []
    for lag in lag_transform_weeks:
        lag_transforms.append(RollingMean(window_size=lag))
        lag_transforms.append(RollingStd(window_size=lag))
        lag_transforms.append(RollingMax(window_size=lag))

    return lag_transforms

# Price transforms
def _get_price_transforms(df):
# Transform price feature
    transformed = transform_exog(
        df[["unique_id", "ds", "Price"]],
        lags=[1,2,3,4],
    )
# Add deltas
    transformed["Delta_Price_lag1"] = transformed["Price"] - transformed["Price_lag1"]
    transformed["RelDelta_Price_lag1"] = transformed["Price"] / transformed["Price_lag1"] - 1
# Merge
    df_price = transformed.drop(columns=["Price", "Price_lag1"])
    df = df.merge(df_price, left_on = ["unique_id", "ds"], right_on=["unique_id", "ds"])
    return df

# Function required to generate the future exogenous (mainly Price) for the test set
def _get_future_exog(X_df, df):
    X_df_idx = X_df.set_index(["unique_id", "ds"]).index
    df_last_price = df.groupby(["unique_id"])["Price"].last()
    X_df = X_df.merge(df_last_price, left_on=["unique_id"], right_on=["unique_id"])
    X_df[["Client", "Warehouse", "Product"]] = X_df["unique_id"].str.split("/", expand=True)
    df_test = df.copy().merge(X_df, left_on=["unique_id", "ds", "Client", "Warehouse", "Product", "Price"], right_on=["unique_id", "ds", "Client", "Warehouse", "Product", "Price"], how="outer")
    df_test = _get_price_transforms(df_test)
    df_test = _get_date_features(df_test)
    df_test = df_test.drop(columns=["Client", "Warehouse", "Product"])
    df_test = df_test.set_index(["unique_id", "ds"])
    X_df = df_test.loc[X_df_idx]
    price_lag_cols = [col for col in X_df.columns if "Price_lag" in col]
    X_df[price_lag_cols] = X_df[price_lag_cols].fillna(0)
    X_df = X_df.reset_index()
    return X_df

def _get_future_exog_hiech(X_df, df):
    X_df_idx = X_df.set_index(["unique_id", "ds"]).index
    df_last_price = df.groupby(["unique_id"])["Price"].last()
    X_df = X_df.merge(df_last_price, left_on=["unique_id"], right_on=["unique_id"])
    X_df[["Top Level", "Client", "Warehouse", "Product"]] = X_df["unique_id"].str.split("/", expand=True)
    df[["Top Level", "Client", "Warehouse", "Product"]] = df["unique_id"].str.split("/", expand=True)
    df_test = df.copy().merge(X_df, left_on=["unique_id", "ds", "Client", "Warehouse", "Product", "Price", "Top Level"], right_on=["unique_id", "ds", "Client", "Warehouse", "Product", "Price", "Top Level"], how="outer")
    df_test = _get_price_transforms(df_test)
    df_test = _get_date_features(df_test)
    df_test = df_test.drop(columns=["Client", "Warehouse", "Product", "Top Level"])
    df_test = df_test.set_index(["unique_id", "ds"])
    X_df = df_test.loc[X_df_idx]
    price_lag_cols = [col for col in X_df.columns if "Price_lag" in col]
    X_df[price_lag_cols] = X_df[price_lag_cols].fillna(0)
    X_df = X_df.reset_index()
    return X_df
    
# We use the competition metric to evaluate our cross-validation predictions.
def CompetitionMetric(
    df: pd.DataFrame,
    models: List[str],
    id_col: str = "unique_id",
    target_col: str = "y",
) -> pd.DataFrame:
    """Computes the competition metric"""

    error = df[models].sub(df[target_col], axis=0)
    sum_abs_error = error.abs().sum()
    abs_sum_error = error.sum().abs()
    score = sum_abs_error + abs_sum_error
    score /= df[target_col].sum()

    score.index.name = id_col
    score = score.reset_index()

    return score

def loss(df, train_df):
  x = df.shape[0]
  mae_value = mae(df, models=["model"])["model"].mean()
  mae_value = mae_value * x
  bias_value = bias(df, models=["model"])["model"].mean()
  bias_value = bias_value * x
  sum_of_actual = df['y'].astype(np.float64).sum()
  score = (mae_value + abs(bias_value)) / sum_of_actual
  return score

def final_output(df):
  # We use the name of the model
  name = "Submission Phase 1 - LGBM.csv"
  # Apply some data wrangling to ensure everything is in the expected format of the competition
  df[["Client", "Warehouse", "Product"]] = df["unique_id"].str.split("/", expand=True)
  df[["Client", "Warehouse", "Product"]] = df[["Client", "Warehouse", "Product"]].astype(np.int64)
  df = df.sort_values(by=["Client", "Warehouse", "Product"]).reset_index(drop=True)
  df = df.drop(columns = "unique_id")
  df = df.set_index(["Client", "Warehouse", "Product", "ds"])
  df = df.unstack(3)
  df.columns = df.columns.get_level_values(1)
  df.to_csv(name)

df_price = df[["unique_id", "ds", "Price"]].copy()
df_price = _get_price_transforms(df_price)
df_price = df_price.drop(columns = "Price")
df = df.merge(df_price, left_on = ["unique_id", "ds"], right_on=["unique_id", "ds"])

# Add date features
df = _get_date_features(df)

df.replace([np.inf, -np.inf], 0, inplace=True)

#%% Model inputs
horizon = 13
freq = "W-MON"

# We use a LightGBMRegressor with Tweedie loss, a loss function that is known to perform well on zero-inflated problems (i.e. demand forecasting)
models = [xgb.XGBRegressor(objective="reg:tweedie", enable_categorical = True)]


# Instantiate MLForecast
mf = MLForecast(
    models = models,
    freq = freq,
    lags = list(range(1, 14)) + [50, 51, 52, 53, 54],
    lag_transforms = {
        1: _get_lag_transforms(),
        13: _get_lag_transforms(),
    },
)

# Cross-validation: create a set of forecasts using cross-validation. We use two test windows, last year and the year before from the same weeks.
yhat_cv = mf.cross_validation(df=df,
                              h=horizon,
                              n_windows=2,
#                               static_features=["Client", "Warehouse", "Product"],
                              static_features=[],
                              step_size=52,
                              refit=True,
                               )

# Evaluate the cross-validation set on the competition metric
cv_metric = evaluate(
    yhat_cv.drop(columns=['cutoff']),
    metrics=[CompetitionMetric],
    agg_fn='mean',
)

print(f"[METRIC CV  ]: {cv_metric[0].item():.4f}")


# Fit the model
mf.fit(df = df,
#        static_features=["Client", "Warehouse", "Product"],
       static_features=[],
       fitted = True,
       )

# Create dataframe with future exogenous
X_df = mf.make_future_dataframe(h=horizon)
X_df = _get_future_exog_hiech(X_df, df[["ds", "unique_id", "Price"]])


# Generate test predictions
yhat_test = mf.predict(h=horizon, X_df=X_df)
yhat_test_unique = set(yhat_test['unique_id'].unique())
insample_mf = mf.forecast_fitted_values()
insample_mf_unique = set(insample_mf['unique_id'].unique())
print(len(insample_mf_unique))
print(len(yhat_test_unique))


yhat_test.set_index('unique_id', inplace = True)

insample_mf = mf.forecast_fitted_values()

reconcilers = [
    TopDown('forecast_proportions'),
    BottomUp(),
    MinTrace('ols'),
]

hrec = HierarchicalReconciliation(reconcilers=reconcilers)

Y_rec_mf = hrec.reconcile(Y_hat_df=yhat_test, Y_df=insample_mf, S=S_df, tags=tags)

I tried setting dropna to False and changed nothing.

[elephaint]

Hi,

The missing id's are those id's for which the model can't do the cross-validation as there's too little data to compute all the features.

You can check it by running:

set_diff = set(sorted(yhat_test_unique)) - set(sorted(insample_mf_unique))
df_not_in_mf = df.query("unique_id in @set_diff")
random_id = df_not_in_mf.sample()["unique_id"]
df.query("unique_id == @random_id.item()")

Using less lags will increase the set of unique id's in insample_mf_unique. To get to parity with yhat_test_unique, I'd suggest to just add the mising id - date combinations with zero values for the predictions.

[Avidar123]

This works @elephaint, but shouldn't setting dropna = False fix this as well ? and also wouldn't setting the prediction to zero foe the dropped ID's affect the hierarchical calculation greatly ?