Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

can't resample data #17

Open
mr-m0nst3r opened this issue Jun 21, 2019 · 8 comments
Open

can't resample data #17

mr-m0nst3r opened this issue Jun 21, 2019 · 8 comments
Labels
bug Something isn't working

Comments

@mr-m0nst3r
Copy link

Hi there,

I'm trying to add 1 minute data to CCXTFeed, and then resample it to 5 minutes, so I can get MACD of 1m and 5m. But failed.

Here's the code:

# !/usr/bin/env python
# -*- coding: utf-8; py-indent-offset:4 -*-
from __future__ import (absolute_import, division, print_function,
                        unicode_literals)
import sys
import time
from datetime import datetime, timedelta

import backtrader as bt
from ccxtbt import CCXTFeed

class TestStrategy(bt.Strategy):
    params = (
        ('printlog', True),
    )

    def log(self, txt, dt=None, doprint=False):
        ''' Logging function fot this strategy'''
        if self.params.printlog or doprint:
            dt = dt or bt.num2date(self.data.datetime[0])
            print('%s, %s' % (dt, txt))

    def start(self):
        self.counter = 0
        print('START')

    def prenext(self):
        self.counter += 1
        print('prenext len %d - counter %d' % (len(self), self.counter))

    def __init__(self):
        self.macd = bt.indicators.MACDHisto(self.datas[0])
        self.macd2 = bt.indicators.MACDHisto(self.datas[1])


    def next(self):
        self.counter += 1
        price_txt = "Counter: " + str(self.counter) + " Open/Close/High/Low/Volume: " + str(self.data0.open[0]) + " - "+ str(self.data0.close[0]) + " - " + str(self.data0.high[0]) + " - " + str(self.data0.low[0])  + " - " + str(self.data0.volume[0]) + " Len: "+ str(len(self.data0))# + " Time Frame:" + bt.TimeFrame.getname(self.data0._timeframe) + " Len: "+ str(len(self.data0))
        self.log(price_txt)
        macd_txt = "MACD: {:.2f}, Histo: {:.2f}".format(self.macd.macd[0],self.macd.histo[0])
        self.log("MACD#1: " + macd_txt)
        macd2_txt = "MACD: {:.2f}, Histo: {:.2f}".format(self.macd2.macd[0],self.macd2.histo[0])
        self.log("MACD#2: " + macd2_txt)
    
    def notify_data(self, data, status, *args, **kwargs):
        dn = data._name
        dt = datetime.now()
        msg= 'Data Status: {}'.format(data._getstatusname(status))
        print(dt,dn,msg)
        if data._getstatusname(status) == 'LIVE':
            self.live_data = True
        else:
            self.live_data = False

if __name__ == '__main__':
    #cerebro = bt.Cerebro(quicknotify=True)
    cerebro = bt.Cerebro()

    

    #exchange = sys.argv[1] if len(sys.argv) > 1 else 'gdax'
    exchange = sys.argv[1] if len(sys.argv) > 1 else 'binance'
    symbol = sys.argv[2] if len(sys.argv) > 2 else 'ETH/USDT'

    #store = CCXTStore(exchange=exchange, currency='LTC', config=config, retries=5, debug=False)

    hist_start_date = datetime.utcnow() - timedelta(minutes=10)
    print('UTC NOW: ', datetime.utcnow())
    print('hist_start_data: ', hist_start_date)
    print('Using symbol: ', symbol)

    data = CCXTFeed(exchange=exchange,
                             dataname=symbol,
                             timeframe=bt.TimeFrame.Minutes,
                             fromdate=hist_start_date,
                             #todate=datetime(2019, 1, 1, 0, 2),
                             compression=1,
                             ohlcv_limit=2,
                             currency='USDT',
                             retries=5,

                             # 'apiKey' and 'secret' are skipped
                             config={'enableRateLimit': True, 'nonce': lambda: str(int(time.time() * 1000))})

    cerebro.adddata(data)
    cerebro.resampledata(data, timeframe=bt.TimeFrame.Minutes, compression=2)

    cerebro.addstrategy(TestStrategy)
    # Run the strategy
    cerebro.run()

And here's the output:

python ccxtbttest.py             
UTC NOW:  2019-06-21 07:19:54.687534
hist_start_data:  2019-06-21 07:09:54.687358
Using symbol:  ETH/USDT
START
2019-06-21 15:20:01.656669 ETH/USDT Data Status: DELAYED
prenext len 1 - counter 1
prenext len 2 - counter 2
2019-06-21 15:20:03.363956 ETH/USDT Data Status: LIVE
prenext len 3 - counter 3
prenext len 4 - counter 4
prenext len 5 - counter 5
prenext len 6 - counter 6
prenext len 7 - counter 7
prenext len 8 - counter 8
prenext len 9 - counter 9
prenext len 10 - counter 10
prenext len 11 - counter 11
prenext len 12 - counter 12
Traceback (most recent call last):
  File "ccxtbttest.py", line 90, in <module>
    cerebro.run()
  File "/Users/michael/.venv/bt-ccxt-store/lib/python3.6/site-packages/backtrader-1.9.74.123-py3.6.egg/backtrader/cerebro.py", line 1127, in run
    runstrat = self.runstrategies(iterstrat)
  File "/Users/michael/.venv/bt-ccxt-store/lib/python3.6/site-packages/backtrader-1.9.74.123-py3.6.egg/backtrader/cerebro.py", line 1298, in runstrategies
    self._runnext(runstrats)
  File "/Users/michael/.venv/bt-ccxt-store/lib/python3.6/site-packages/backtrader-1.9.74.123-py3.6.egg/backtrader/cerebro.py", line 1557, in _runnext
    dt0 = min((d for i, d in enumerate(dts)
ValueError: min() arg is an empty sequence

Any idea?

Thank you.
@Dave-Vallance Dave-Vallance added the bug Something isn't working label Jun 21, 2019
@Dave-Vallance
Copy link
Owner

Hi @mr-m0nst3r

Thanks for the submission and report. Yes, resampling currently does not work from my testing. It is an item to tackle in the future. I have marked this as a bug and will keep it open.

@mr-m0nst3r
Copy link
Author

Okay, thank you.

Btw, I guess I have to calculate the macd without using the bt.Indicators.

Thanks and hope someone could make this bug fixed.

@mr-m0nst3r
Copy link
Author

@Dave-Vallance

I find that if we remove fromdate parameter, the script runs well, but started at about 1 day before, without the error described above.

Any help with this finding?

Here's the question I posted on the backtrader community:
https://community.backtrader.com/topic/1914/resample-live-feed-from-ccxt-problem/4

@LatchRazu
Copy link

LatchRazu commented Mar 16, 2020 •
edited
Loading

After testing it seems that this bug only triggers when a bar which needs to be resampled is only a few seconds old. I haven't been able to trigger it while walking through the code.

I came up with this ugly fix:
Inside of ccxtfeed.py -> _load

                    start = datetime.utcnow()
                    if start.second < 15:
                        time.sleep(15 - start.second)

                    self._fetch_ohlcv()

Log snippets from some testing I did with 1 and 2 minute bars:

2020-03-16 16:22:22.377627 - fetch_ohlcv - Attempt 0
03-16 16:22 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:24:22.207071 - fetch_ohlcv - Attempt 0
03-16 16:24 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:26:21.639287 - fetch_ohlcv - Attempt 0
03-16 16:26 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:32:11.773225 - fetch_ohlcv - Attempt 0
03-16 16:32 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:42:17.367415 - fetch_ohlcv - Attempt 0
03-16 16:42 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:46:17.382720 - fetch_ohlcv - Attempt 0
03-16 16:46 - custom_classes.f_cerebro: WARNING  works as intended



2020-03-16 16:34:08.293936 - fetch_ohlcv - Attempt 0
03-16 16:34 - custom_classes.f_cerebro: WARNING  resampled bug triggered
2020-03-16 16:40:03.369245 - fetch_ohlcv - Attempt 0
03-16 16:40 - custom_classes.f_cerebro: WARNING  resampled bug triggered
2020-03-16 16:44:05.686620 - fetch_ohlcv - Attempt 0
03-16 16:44 - custom_classes.f_cerebro: WARNING  resampled bug triggered
2020-03-16 16:48:07.024158 - fetch_ohlcv - Attempt 0
03-16 16:48 - custom_classes.f_cerebro: WARNING  resampled bug triggered

@geraybos
Copy link

No, at least it is not working for me

@Dave-Vallance

I find that if we remove fromdate parameter, the script runs well, but started at about 1 day before, without the error described above.

Any help with this finding?

Here's the question I posted on the backtrader community:
https://community.backtrader.com/topic/1914/resample-live-feed-from-ccxt-problem/4

@dogenkigen
Copy link

For me it worked when I checked while escape condition to:
if len(self.data) != 0: break

@JiazhengChai
Copy link

After testing it seems that this bug only triggers when a bar which needs to be resampled is only a few seconds old. I haven't been able to trigger it while walking through the code.

I came up with this ugly fix:
Inside of ccxtfeed.py -> _load

                    start = datetime.utcnow()
                    if start.second < 15:
                        time.sleep(15 - start.second)

                    self._fetch_ohlcv()

Log snippets from some testing I did with 1 and 2 minute bars:

2020-03-16 16:22:22.377627 - fetch_ohlcv - Attempt 0
03-16 16:22 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:24:22.207071 - fetch_ohlcv - Attempt 0
03-16 16:24 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:26:21.639287 - fetch_ohlcv - Attempt 0
03-16 16:26 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:32:11.773225 - fetch_ohlcv - Attempt 0
03-16 16:32 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:42:17.367415 - fetch_ohlcv - Attempt 0
03-16 16:42 - custom_classes.f_cerebro: WARNING  works as intended
2020-03-16 16:46:17.382720 - fetch_ohlcv - Attempt 0
03-16 16:46 - custom_classes.f_cerebro: WARNING  works as intended



2020-03-16 16:34:08.293936 - fetch_ohlcv - Attempt 0
03-16 16:34 - custom_classes.f_cerebro: WARNING  resampled bug triggered
2020-03-16 16:40:03.369245 - fetch_ohlcv - Attempt 0
03-16 16:40 - custom_classes.f_cerebro: WARNING  resampled bug triggered
2020-03-16 16:44:05.686620 - fetch_ohlcv - Attempt 0
03-16 16:44 - custom_classes.f_cerebro: WARNING  resampled bug triggered
2020-03-16 16:48:07.024158 - fetch_ohlcv - Attempt 0
03-16 16:48 - custom_classes.f_cerebro: WARNING  resampled bug triggered

This works for me. Thanks.
But are there any specific reasons to use 15 seconds? I used 5 seconds and it seems to work as well.

@LatchRazu
Copy link

This works for me. Thanks.
But are there any specific reasons to use 15 seconds? I used 5 seconds and it seems to work as well.

No specific reason. Excessive caution.

@JiazhengChai JiazhengChai mentioned this issue Dec 8, 2020
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@dogenkigen @geraybos @Dave-Vallance @mr-m0nst3r @JiazhengChai @LatchRazu