Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Selenium Chrome Driver #266

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions backend/.devcontainer/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
FROM --platform=linux/amd64 python:3.11-slim-buster

ARG IMAGE_NAME=pennlabs/mobile-backend-devcontainer

# Install build dependencies
RUN apt-get update && apt-get install -y gcc libpq-dev libc-dev git-all wget \
&& rm -rf /var/lib/apt/lists/*


RUN wget -q https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb

RUN apt install -y ./google-chrome-stable_current_amd64.deb

RUN pip install pipenv

ENV DJANGO_SETTINGS_MODULE pennmobile.settings.development
ENV SECRET_KEY 'temporary key just to build the docker image'
7 changes: 7 additions & 0 deletions backend/.devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"name": "Penn Mobile Backend",
"build": {
"dockerfile": "Dockerfile"
},
"runArgs": ["--platform=linux/amd64" ]
}
3 changes: 3 additions & 0 deletions backend/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
FROM pennlabs/django-base:b269ea1613686b1ac6370154debbb741b012de1a-3.11

RUN wget -q https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
RUN apt-get install ./google-chrome-stable_current_amd64.deb -y

LABEL maintainer="Penn Labs"

# Copy project dependencies
Expand Down
2 changes: 1 addition & 1 deletion backend/Pipfile
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ django-redis = "*"
redis = "*"
python-dateutil = "*"
selenium = "*"
webdriver-manager = "*"
chromedriver-binary = "*"

[requires]
python_version = "3.11"
92 changes: 33 additions & 59 deletions backend/Pipfile.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

17 changes: 9 additions & 8 deletions backend/penndata/management/commands/get_penn_today_events.py
Original file line number Diff line number Diff line change
@@ -1,16 +1,14 @@
import datetime
from urllib.parse import urljoin

import chromedriver_binary
from bs4 import BeautifulSoup
from django.core.management.base import BaseCommand
from django.utils import timezone
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.service import Service as FirefoxService
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from webdriver_manager.firefox import GeckoDriverManager

from penndata.models import Event

Expand Down Expand Up @@ -73,7 +71,7 @@ def handle(self, *args, **kwargs):
start_time = datetime.time(0, 0)
else:
start_time = datetime.datetime.strptime(start_time_str, "%I:%M%p").time()
start_date = datetime.datetime.combine(start_date, start_time)
start_date = timezone.make_aware(datetime.datetime.combine(start_date, start_time))

if start_date > now + datetime.timedelta(days=31):
continue
Expand Down Expand Up @@ -106,7 +104,7 @@ def handle(self, *args, **kwargs):
defaults={
"event_type": "Penn Today",
"image_url": "",
"start": timezone.make_aware(start_date),
"start": start_date,
"end": timezone.make_aware(end_date),
"location": location,
"website": event_url,
Expand All @@ -119,11 +117,14 @@ def handle(self, *args, **kwargs):

def connect_and_parse_html(self, event_url, condition):
try:

print(chromedriver_binary.chromedriver_filename)
from selenium.webdriver.chrome.options import Options

options = Options()

options.add_argument("--headless")
driver = webdriver.Firefox(
service=FirefoxService(GeckoDriverManager().install()), options=options
)
driver = webdriver.Chrome(options=options)

driver.get(event_url)
print("WAITING FOR ELEMENT")
Expand Down
13 changes: 13 additions & 0 deletions backend/tests/penndata/test_events.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
from django.core.management import call_command
from django.test import TestCase

from penndata.models import Event


class TestPennTodayEvents(TestCase):
def setUp(self):
call_command("get_penn_today_events")

def test_event(self):
events = Event.objects.all()
self.assertLess(0, events.count())
Loading