Feature proposal: improved caching support #1665
Replies: 9 comments 21 replies
-
|
Hey @kschmid this seems like a great idea. What kind of caching were you thinking about? I can see how we could implement caching for |
Beta Was this translation helpful? Give feedback.
-
|
Dear @rickturner2001 first off, I am thinking about this: Another (minor) issue would be to forward things like the responses, as this contains information on whether a request was served from a cache or directly. All of this is basically web-caching. When going for smarter ways of caching on the application level than probably something like yfinance-cache comes to mind. Not sure, whether the project wants to go as far as incorporating this. |
Beta Was this translation helpful? Give feedback.
-
|
We might implement this as a feature. Can discussion be referenced by id @ValueRaider ? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, although the ID has changed. |
Beta Was this translation helpful? Give feedback.
-
|
@kschmid why not just use the smart_scraping proposed in the documentation as you pointed out? What problem are you having with that? |
Beta Was this translation helpful? Give feedback.
-
|
Well, I can only make proposals. ;-) Regarding the second half: there is little information on what yfinance-cache does. From this I cannot see that it provides for example "on-request cache bypassing" based on application input or response inspection. But of course, there may be much more which does not become clear from the overview. |
Beta Was this translation helpful? Give feedback.
-
Please explain, because way I see it dumb caching is smarter than caching nothing and getting rate-limited/blocked.
The README can be edited like any other file, you can add a link, like is done for
It's called yfinance-cache, don't try to bypass it. Best to forget all your caching efforts with yfinance and read this tiny section with a fresh mind. |
Beta Was this translation helpful? Give feedback.
-
It does serve some stuff from the cache, which may be invalid (and unexpected), leading unexpected behavior (especially intraday). Needed my some time, may need others as well. But I got over it, others will as well.
You are right, I was wrong. The arguments would probably allow for appropriate caching from an application level. (Though no way to determine whether results come from cache or are fresh as far as I see, but this would then probably better go to yfinance-cache..) |
Beta Was this translation helpful? Give feedback.
-
|
You can simply have a logger at level DEBUG. Cached requests don't log anything |
Beta Was this translation helpful? Give feedback.
-
|
Thanks Rick, will try this. |
Beta Was this translation helpful? Give feedback.
-
|
@kschmid I am not too satisfied with the way yahoo_cache is operating, and as far as I can tell, it's not really the solution you were looking for. As far as knowing whether a request is being pulled from cache or not, I have a simple solution: class CachedLimiterSession(CacheMixin, LimiterMixin, Session):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.is_cached = False
def get(self, url: str, params=None, **kwargs):
expiration = get_expiration_from_interval(params['interval'])
res = super().get(url, params=params,
expire_after=expiration, **kwargs)
self.is_cached = res.from_cache # res.from_cache returns true if the request is being pulled from cache
return res
session = CachedLimiterSession(
limiter=Limiter(RequestRate(2, Duration.SECOND*5)),
bucket_class=MemoryQueueBucket,
backend=SQLiteCache("yfinance.cache"),
methods_allowed=['GET']
)
def get_apple(session: CachedLimiterSession):
yf.download("AAPL", interval="1d", period="1y", session=session)
print("Is cached: ", session.is_cached)
get_apple(session)
get_apple(session)Output However, the expiration for the request IS NOT being set properly, in fact it is |
Beta Was this translation helpful? Give feedback.
-
|
Dear Rick, thanks for this help. I was looking into this mostly for debugging reasons and for this it should be fine. |
Beta Was this translation helpful? Give feedback.
-
|
Ok, here is what I have done: tried it also on a MacBook AIR (M1) as well as the MacBook Pro, which is where all previous attempts were. On the Air, I had first to download the relevant packages. If comment out the expiration part in get, then it works, if not then not (on both). When I open yfinance.cache, it looks like this (MacBook Pro) Hence, even the keys are identical to yours, but I only get binary and not the nice column-headers. |
Beta Was this translation helpful? Give feedback.
-
|
Good, as far as the SQL output it seems that the responses are being cached correctly (I had to install dbeaver to actually see the binary values and get the nice table view) I was unable to reproduce the errors you are getting, would you mind running this on a docker container? assuming you are using a virtual environment main.py import yfinance as yf
from requests_cache import CacheMixin, SQLiteCache
from requests_ratelimiter import LimiterMixin, MemoryQueueBucket
from pyrate_limiter import Duration, RequestRate, Limiter
from requests import Session
def get_expiration_from_interval(interval: str) -> int:
# ...
return 10
class CachedLimiterSession(CacheMixin, LimiterMixin, Session):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.is_cached = False
def get(self, url: str, params=None, **kwargs):
expiration = get_expiration_from_interval(params['interval'])
res = super().get(url, params=params,
expire_after=expiration, **kwargs)
self.is_cached = res.from_cache
return res
session = CachedLimiterSession(
limiter=Limiter(RequestRate(2, Duration.SECOND*5)),
bucket_class=MemoryQueueBucket,
backend=SQLiteCache("yfinance.cache"),
methods_allowed=['GET']
)
def get_ticker_data(session: CachedLimiterSession):
yf.download("AAPL", interval="1d", period="1y", session=session)
print("Is cached: ", session.is_cached)
def get_ticker_history(session: CachedLimiterSession):
ticker = yf.Ticker("AAPL", session=session)
history = ticker.history()
print(history)
print("Is Cached: ", session.is_cached)
get_ticker_history(session)
get_ticker_history(session)
get_ticker_data(session)
get_ticker_data(session)inside .dockerignore dockerfile FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . /app
COPY .dockerignore /app/.dockerignore
CMD ["python", "main.py"] # assuming your script is called main.py
Building the image at this point you should get the expected output |
Beta Was this translation helpful? Give feedback.
-
|
That's interesting: I tried dbeaver-community. (Again Mac) (from the GUI = is there a way to use it in the shell on the mac? According to docu, it should, but I always just get the GUI to start.) What machine are you using? Mac / Linux (which OS?) / Windows. - I am afraid, it may also make a difference. Working on the docker version, now. |
Beta Was this translation helpful? Give feedback.
-
|
Hm, ok, it seems you are assuming that the underlying image is python:3.9-slim (probably the edgelogistics from docker-hub)? |
Beta Was this translation helpful? Give feedback.
-
I am running Ubuntu on my laptop, but as far as the dbeaver output, I made it pretty for the sake of this discussion, I am also getting the _content gibberish. I am not sure how to get dbeaver working on a CLI :( |
Beta Was this translation helpful? Give feedback.
-
|
I tried this, because I wanted to analyze the caching behavior in more detail. (Note that I did it today, i.e., not on a weekday, which may influence the access behavior.) I could identify the expiration handling in the get method as the culprit. If I modify this: This works, but I am not sure regarding other ramifications. Another strange thing, I observed: if instead of downloading the data directly, I create a ticker and use history, then the first ticker gives false, the second gives true. (data is always true). |
Beta Was this translation helpful? Give feedback.
-
|
I should add: the expires is probably not shown, because it is NULL. This is the code modification I had to do in order to make it run. If I insert it, it throws errors and does not modify / update the cache. |
Beta Was this translation helpful? Give feedback.
-
|
Ok, got it. or an alternative code. (The 10 probably does not make too much sense in general). it is shame it is not giving more useful error messages (here is the docker-output), but this is the part obviously that was missing. |
Beta Was this translation helpful? Give feedback.
-
|
Ok. further exploration: the initialization of the ticker (where all base data is effectively downloaded) does not touch modified get method at all. (Thus, after the first Ticker call the is_cached is still "false" from the initialization.) Afterwards, checking this, depends not on the Ticker call, but on whatever download was happening earlier. |
Beta Was this translation helpful? Give feedback.
-
|
Yes I apologize, I meant to omit
What makes you say this? The way I understand this is the following: In TickerBase this is how we get data for a ticker self.session = session
self._data: TickerData = TickerData(self.ticker, session=session)
#...
get_fn = self._data.get
#...
data = get_fn(
url=url,
params=params,
proxy=proxy,
timeout=timeout
)So then we go into def get(self, url, user_agent_headers=None, params=None, proxy=None, timeout=30):
proxy = self._get_proxy(proxy)
response = self._session.get(
url=url,
params=params,
proxies=proxy,
timeout=timeout,
headers=user_agent_headers or self.user_agent_headers)
return responseif you were to print out session here, you would see that session is an instance of either self._session = session or requests
The first call to is_cached must be falsy, given that the request is not coming from cache. if you were to call main twice in the docker container in the same session, you will get 4 truthy values the second time around. But of course, everytime you end your session, the caching gets wiped out, thus resulting in two To prove this, you can run The first time you run
At this point
|
Beta Was this translation helpful? Give feedback.
-
|
Sorry, for not answering sooner.
It was because of my experiments. So, in order to clarify, I went back to the docker. modfied the ticker-routine and the calling as follows: I would expect, as you wrote: that
However what I get (similar to other experiments): When I rerun (fast enough), I get: Thus, the initial ticker request is always false. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @kschmid I hope you're doing great. I have been making some adjustments to Class ChangesWe have now created this function that returns a regex with a dynamic url (note that tthe url is the same used by the Yfiannce API for prices data) def get_dynamic_range_interval(range: str, interval: str):
return re.compile(
rf"https://query2\.finance\.yahoo\.com/v8/finance/chart/([a-zA-Z]+)\?range={range}&interval={interval}&includePrePost=False&events=div%2Csplits%2CcapitalGains"
)We can then use this function to generate url_patterns and give different caching options based on the range-interval combination url_patterns = {
get_dynamic_range_interval("1y", "1d"): calculate_market_open(), # you may define calculate_market_open as you please
}
Also, the class is now a Singletonclass SingletonMeta(type):
_instances = {}
def __call__(cls, *args, **kwargs):
if cls not in cls._instances:
instance = super().__call__(*args, **kwargs)
cls._instances[cls] = instance
return cls._instances[cls]
class CachedLimiterSession(
CacheMixin, LimiterMixin, Session, metaclass=SingletonMeta
):
def __init__(self, is_test: bool = False, *args, **kwargs):
super().__init__(
*args,
**kwargs,
limiter=Limiter(RequestRate(2, Duration.SECOND * 5)),
bucket_class=MemoryQueueBucket,
backend=SQLiteCache(
"yfinance.cache" if not is_test else "yfinance_test.cache"
),
methods_allowed=["GET"],
# Do not cache anything that is not included in url_patterns
# This is effectively the same as {..., *: DO_NOT_CACHE}
expire_after=DO_NOT_CACHE,
urls_expire_after=url_patterns,
)
self.is_caching = True
self.last_request = None
def get(self, url: str, params=None, **kwargs):
if self.is_caching:
res = super().get(url, params=params, **kwargs)
else:
with self.cache_disabled():
res = super().get(url, params=params, **kwargs)
self.last_request = res
return res
def toggle_caching(self):
self.is_caching = not self.is_cachingTestingIn case you wish to match my config.pyimport pathlib
BASE_DIR = pathlib.Path().resolve()
DATABASE_PATH = BASE_DIR / "yfinance.cache"
TEST_DATABASE_PATH = BASE_DIR / "yfinance_test.cache"import os
import unittest
import yfinance as yf
from config import BASE_DIR
from session import CachedLimiterSession
# AAPL default for testing purposes
def download_ticker_data(
session: CachedLimiterSession | None = None,
ticker: str = "AAPL",
):
yf.download(
ticker,
period="1y",
interval="1d",
session=session,
)
def delete_test_cache():
if os.path.isfile(BASE_DIR / "yfinance_test.cache"):
os.remove(BASE_DIR / "yfinance_test.cache")
class TestCache(unittest.TestCase):
def test_global_instance(self):
session = CachedLimiterSession(is_test=True)
session2 = CachedLimiterSession(is_test=True)
self.assertEqual(id(session), id(session2))
delete_test_cache()
def test_caching(self):
session = CachedLimiterSession(is_test=True)
download_ticker_data(session)
self.assertEqual(session.last_request.from_cache, False)
download_ticker_data(session)
self.assertEqual(session.last_request.from_cache, True)
delete_test_cache()
def test_toggle_cache(self):
session = CachedLimiterSession(is_test=True)
session.toggle_caching()
self.assertEqual(session.is_caching, False)
download_ticker_data(session)
self.assertEqual(session.last_request.from_cache, False)
download_ticker_data(session)
self.assertEqual(session.last_request.from_cache, False)
delete_test_cache()
def test_url_patterns(self):
session = CachedLimiterSession(is_test=True)
session.get("https://www.google.com")
self.assertEqual(session.last_request.from_cache, False)
session.get("https://www.google.com")
self.assertEqual(session.last_request.from_cache, False)
delete_test_cache() |
Beta Was this translation helpful? Give feedback.
-
|
This looks pretty well thought out. |
Beta Was this translation helpful? Give feedback.
-
|
@rickturner2001 @kschmid |
Beta Was this translation helpful? Give feedback.
-
Due to rate limiting of the yahoo api, there is certainly a need for caching, which is also exemplified with the proposal of smart scraping, based on caching given.
However, it would be very helpful to control to some extend the way the cache is handled.
a) was a specific request served from cache or from network. This is part of the response, however, as this is inside yfinance, it can't be checked by the application program. It would be great, if this could be made available to the application, e.g., via a call to a get_last_caching_state (or similar), which would provide this part of the response.
b) it would be great, if a ticker could change the assigned session in order to bypass the cache altogether. Right now, I can only do something like this, by creating a second ticker with a different associated session, but that means that also initialization of this ticker will bypass the cache, unnecessarily increasing the number of requests.
Beta Was this translation helpful? Give feedback.
All reactions