resample_apply - skewed result?

[mmarihart]

Started using resample_apply to resample a 1 minute dataframe to a 5 minute dataframe. I think the data from the resampled timeframe is added a row too late on the base timeframe, I am not sure though. Here's a minimal example code that just resamples the close:

import pandas as pd
from pandas import Timestamp
from backtesting.lib import resample_apply,

df = pd.DataFrame(data={'Close': [18120.0, 18120.25, 18118.75, 18119.5, 18119.25, 18119.25, 18119.0, 18117.25, 18117.25, 18117.0, 18117.0, 18118.5, 18118.5, 18120.5, 18121.75]}, index=[Timestamp('2024-03-01 00:00:00'), Timestamp('2024-03-01 00:01:00'), Timestamp('2024-03-01 00:02:00'), Timestamp('2024-03-01 00:03:00'), Timestamp('2024-03-01 00:04:00'), Timestamp('2024-03-01 00:05:00'), Timestamp('2024-03-01 00:06:00'), Timestamp('2024-03-01 00:07:00'), Timestamp('2024-03-01 00:08:00'), Timestamp('2024-03-01 00:09:00'), Timestamp('2024-03-01 00:10:00'), Timestamp('2024-03-01 00:11:00'), Timestamp('2024-03-01 00:12:00'), Timestamp('2024-03-01 00:13:00'), Timestamp('2024-03-01 00:14:00')])

resample_apply('5T', lambda x: x, df['Close'])

The result is:

2024-03-01 00:00:00         NaN
2024-03-01 00:01:00         NaN
2024-03-01 00:02:00         NaN
2024-03-01 00:03:00         NaN
2024-03-01 00:04:00         NaN
2024-03-01 00:05:00    18119.25
2024-03-01 00:06:00    18119.25
2024-03-01 00:07:00    18119.25
2024-03-01 00:08:00    18119.25
2024-03-01 00:09:00    18119.25
2024-03-01 00:10:00    18117.00
2024-03-01 00:11:00    18117.00
2024-03-01 00:12:00    18117.00
2024-03-01 00:13:00    18117.00
2024-03-01 00:14:00    18117.00
Name: C[5T], dtype: float64

From my understanding we always operate on the close of the 1m candle. So at 2024-03-01 00:04:00 we are at the end of the 1m candle as well as the end of the 5m candle.

Therefore, shouldn't the value 18119.25 (which is the close of the 5m candle that started at 2024-03-01 00:00:00) start one row above at 2024-03-01 00:04:00?

Thank you for any clarifications you can provide.

[tani3010]

I'm also feeling strange for the resampled timeseries, not sure it's the same thing of the topic though. The output of the resample_apply doesn't hold original timestamp, which is also mentioned #970 . Below is the code to replicate the situation.

import pandas as pd
from pandas import Timestamp
from backtesting.lib import resample_apply, OHLCV_AGG

df = pd.DataFrame(data={'Close': [18120.0, 18120.25, 18118.75, 18119.5, 18119.25, 18119.25, 18119.0, 18117.25, 18117.25, 18117.0, 18117.0, 18118.5, 18118.5, 18120.5, 18121.75]}, index=[Timestamp('2024-03-01 00:00:00'), Timestamp('2024-03-01 00:01:00'), Timestamp('2024-03-01 00:02:00'), Timestamp('2024-03-01 00:03:00'), Timestamp('2024-03-01 00:04:00'), Timestamp('2024-03-01 00:05:00'), Timestamp('2024-03-01 00:06:00'), Timestamp('2024-03-01 00:07:00'), Timestamp('2024-03-01 00:08:00'), Timestamp('2024-03-01 00:09:00'), Timestamp('2024-03-01 00:10:00'), Timestamp('2024-03-01 00:11:00'), Timestamp('2024-03-01 00:12:00'), Timestamp('2024-03-01 00:13:00'), Timestamp('2024-03-01 00:14:00')])
df.Close = range(len(df))  # use more simple data

# replication of current resample: resample_apply('5T', lambda x: x, df['Close'])
resampled_1 = df.resample('5T', label='right').agg(OHLCV_AGG['Close']).reindex(index=df.index, method='ffill')

# passing closed='right'. if this is not passed, closed='left' will be used
resampled_2 = df.resample('5T', label='right', closed='right').agg(OHLCV_AGG['Close']).reindex(index=df.index, method='ffill')

pd.concat([df, resampled_1, resampled_2], axis=1, keys=['PreResample', 'CurrentResample', 'ExpectedResample'])

output

                    PreResample CurrentResample ExpectedResample
                          Close           Close            Close
2024-03-01 00:00:00           0             NaN                0
2024-03-01 00:01:00           1             NaN                0
2024-03-01 00:02:00           2             NaN                0
2024-03-01 00:03:00           3             NaN                0
2024-03-01 00:04:00           4             NaN                0
2024-03-01 00:05:00           5             4.0                5
2024-03-01 00:06:00           6             4.0                5
2024-03-01 00:07:00           7             4.0                5
2024-03-01 00:08:00           8             4.0                5
2024-03-01 00:09:00           9             4.0                5
2024-03-01 00:10:00          10             9.0               10
2024-03-01 00:11:00          11             9.0               10
2024-03-01 00:12:00          12             9.0               10
2024-03-01 00:13:00          13             9.0               10
2024-03-01 00:14:00          14             9.0               10

I thought that data would be updated every 5min and others are filled with the previous value like ExpectedResample, by keeping the same timestamp with the original timeseries. I mean that if PreResample has 5 at 2024-03-01 00:05:00, the resampled should have the same value on the same timestamp in every 5min but current implementation chose 4 instead.

Any indicator calculated from resample_apply would be shifted and aggregate the values using previous bar.
Is the current implementation actually expected one? or backtesting assumes the timestamp on Open-time, instead on Close-time?