Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add caching and rate limiting support? #376

Closed
SnowCheetos opened this issue Jul 18, 2024 · 2 comments
Closed

Add caching and rate limiting support? #376

SnowCheetos opened this issue Jul 18, 2024 · 2 comments

Comments

@SnowCheetos
Copy link

When working in a notebook or when fetching from static endpoints, there often come times where the same endpoint is called, which would benefit from having a cached URL session. Also sometimes when I need to make a large volume of requests in a short amount of time, it risks hitting the AV rate limit (which I believe is 25/day for free tier, 75/min for the next tier, not sure after that). It'd be nice if I can pass in a custom requests session that handles these things. I know that for yfinance, you can construct a custom session and use it with the Ticker class like this

from requests import Session
from requests_cache import CacheMixin, SQLiteCache
from requests_ratelimiter import LimiterMixin, MemoryQueueBucket
from pyrate_limiter import Duration, RequestRate, Limiter
class CachedLimiterSession(CacheMixin, LimiterMixin, Session):
    pass

session = CachedLimiterSession(
    limiter=Limiter(RequestRate(2, Duration.SECOND*5)),  # max 2 requests per 5 seconds
    bucket_class=MemoryQueueBucket,
    backend=SQLiteCache("yfinance.cache"),
)

ticker = yf.Ticker('msft', session=session)
# The scraped response will be stored in the cache
ticker.actions

I think the AV client could also benefit from something like this. Maybe add an optional attribute session to class AlphaVantage(object): so that def _handle_api_call(self, url): can default to use the custom session if available?

@SnowCheetos
Copy link
Author

Created a pull request #377 for the implementation

@AlphaVantageSupport
Copy link
Collaborator

Thanks! We will keep the PR open and merge it upon community review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants