Gracefully Handle Download Issues for Automated Calls #2005

jaythedeveloperer · 2024-07-31T19:15:27Z

jaythedeveloperer
Jul 31, 2024

BACKGROUND:

I want to pull data from yfinance 10 times per minute (let’s call it cycles per minute)
Some cycles may encounter errors, for example, a NetworkError, if my internet cuts out.
I am hoping if there is an error before the cycle is done, I want to simple return a string that says “error” without crashing the program, and resume the download on the next cycle.
I’m guessing because all of the threading work, none of my errors are being handled.

EXAMPLE CODE:
In my example, I’m downloading 3 symbols, and running a test by turning my wifi off to reproduce network errors. The goal is to have the exception handling return a list of [“error”,”error”,”error”] right away

“””
import yfinance as yf
import requests
from concurrent.futures import ThreadPoolExecutor, as_completed
import time

List of tickers to download data for

tickers = ['AAPL', 'MSFT', 'GOOGL'] # Example tickers

def download_data(ticker):
try:
print(f"Starting download for {ticker}")
data = yf.download(ticker, period="1y") # Adjust the period as needed
if data.empty:
print(f"No data returned for {ticker}")
else:
print(f"Data downloaded for {ticker}")
# Save or process the data here
except requests.exceptions.ConnectionError:
print(f"ERROR: Connection error for {ticker}")
except Exception as e:
print(f"ERROR: An unexpected error occurred for {ticker}: {e}")

Function to manage threading

def fetch_data(tickers):
with ThreadPoolExecutor(max_workers=5) as executor: # Adjust max_workers as needed
future_to_ticker = {executor.submit(download_data, ticker): ticker for ticker in tickers}
for future in as_completed(future_to_ticker):
ticker = future_to_ticker[future]
try:
future.result() # This will re-raise any exception caught in the thread
except Exception as e:
print(f"ERROR: An unexpected error occurred for {ticker}: {e}")

Start fetching data

fetch_data(tickers)

Allow time for tasks to complete

time.sleep(10)

“””

ERROR MESSAGE:

Exception in thread Thread-3:
Traceback (most recent call last):
File "/Users/macuser/.pyenv/versions/3.8.2/lib/python3.8/site-packages/urllib3/connection.py", line 159, in _new_conn
Exception in thread Thread-1:
Traceback (most recent call last):
File "/Users/macuser/.pyenv/versions/3.8.2/lib/python3.8/site-packages/urllib3/connection.py", line 159, in _new_conn
Exception in thread Thread-2:
Traceback (most recent call last):
File "/Users/macuser/.pyenv/versions/3.8.2/lib/python3.8/site-packages/urllib3/connection.py", line 159, in _new_conn
conn = connection.create_connection(
File "/Users/macuser/.pyenv/versions/3.8.2/lib/python3.8/site-packages/urllib3/util/connection.py", line 61, in create_connection
conn = connection.create_connection(
File "/Users/macuser/.pyenv/versions/3.8.2/lib/python3.8/site-packages/urllib3/util/connection.py", line 61, in create_connection
conn = connection.create_connection(
File "/Users/macuser/.pyenv/versions/3.8.2/lib/python3.8/site-packages/urllib3/util/connection.py", line 61, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/Users/macuser/.pyenv/versions/3.8.2/lib/python3.8/socket.py", line 918, in getaddrinfo
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/Users/macuser/.pyenv/versions/3.8.2/lib/python3.8/socket.py", line 918, in getaddrinfo
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/Users/macuser/.pyenv/versions/3.8.2/lib/python3.8/socket.py", line 918, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known