diff --git a/.github/workflows/README.md b/.github/workflows/README.md new file mode 100644 index 0000000..585193d --- /dev/null +++ b/.github/workflows/README.md @@ -0,0 +1,85 @@ +GitHub Actions Workflow +============================== +In this section, we detail the GitHub Actions workflows employed in our project. GitHub Actions are a powerful tool for automating software workflows, enabling continuous integration and continuous deployment (CI/CD) practices. Our project leverages these workflows to automate various tasks such as code quality checks, automated testing, and deployment processes. Each YAML file described below represents a specific workflow designed for a particular aspect of our software development process. These workflows ensure that our codebase is rigorously tested, adheres to quality standards, and is consistently deployed to our production environment with minimal manual intervention. + +## `pylint.yml` +This file configures a GitHub Actions workflow to perform static code analysis using Pylint on Python code. It triggers on pushes to the __stage branch__ and runs on an __ubuntu-latest virtual machine__. The workflow supports Python versions 3.8, 3.9, and 3.10. + +The workflow includes the following steps: + +- __Code Checkout__: Uses `actions/checkout@v3` to fetch the source code. +- __Python Setup__: Sets up the Python environment using `actions/setup-python@v3`. +- __Dependencies Installation__: Installs Pylint. +- __Code Analysis with Pylint__: Executes Pylint on Python files in the `src` directory, excluding specific errors and limitations, and generates a textual report. +- __Report Upload__: Uploads the Pylint report as a GitHub Actions artifact, available for review and download. + +This file is crucial for ensuring that the Python code in the repository adheres to the quality and style standards encoded in Pylint's rules. + + +## `pytest.yml` +This file sets up a GitHub Actions workflow for running automated tests using Pytest on Python code. It is activated by pushes to the __stage branch__, specifically targeting changes in the `src`, tests directories, but excluding the `tests/model_testing` directory. + +Key features of this workflow: + +- __Trigger Conditions__: It runs when there are changes in the `src` and `tests` directories (excluding `tests/model_testing`) upon push to __stage__. +- __Environment Setup__: The workflow runs on an __ubuntu-latest virtual machine__. +- __Workflow Steps__: + - __Repository Checkout__: Uses `actions/checkout@v4` for fetching the latest code. + - __Python Environment Setup__: Utilizes `actions/setup-python@v4` with Python version 3.11. + - __DVC (Data Version Control) Setup__: Implements `iterative/setup-dvc@v1` for data and model version control. + - __Installation of Requirements__: Upgrades pip and installs dependencies from requirements.txt. + - __Data and Model Preparation__: Uses DVC to pull training data and model files, and runs preprocessing steps. + - __Testing Phases__: Executes multiple Pytest commands to test dataset, preprocessing functions, model behavior, and APIs. + - __Commented-Out Test for Model Training__: There's a section for model training tests, currently commented out. + +This workflow ensures that the codebase remains stable and functional with each new push, covering a wide range of tests from dataset integrity to API functionality. + +## `azure-static-web-apps-nice-island-02cd56d03.yml` +This file outlines a CI/CD workflow for Azure Static Web Apps, designed to automate the deployment of a frontend application hosted on Azure. It triggers on push events to the main branch and on specific actions (__opened__, __synchronize__, __reopened__, __closed__) of pull requests, again targeting the __main__ branch. + +Key aspects of this workflow: + +- __Triggering Conditions__: It's set to run on push events to the __main branch__, specifically for changes in the `frontend/**` directory, and on pull request activities concerning the main branch. +- __Environment and Jobs__: + - __Build and Deploy Job__: + - __Condition__: Executes if it's a push event or an open/synchronize/reopen action in a pull request (not on pull request closure). + - __Platform__: Runs on __ubuntu-latest virtual machine__. + - __Steps__: + - __Code Checkout__: Uses `actions/checkout@v3` with submodule and Large File Storage (LFS) settings. + - __Build and Deploy Action__: Utilizes `Azure/static-web-apps-deploy@v1` for deployment, configured with secrets for Azure and GitHub tokens, and specifies the locations for app source (`/frontend/`), optional API source, and output directory (`dist`). + - __Close Pull Request Job__: + - __Condition__: Only runs if the event is a pull request closure. + - __Platform__: Also runs on __ubuntu-latest virtual machine__. + - __Steps__: + - __Close Pull Request Action__: Carries out the closure of the pull request using the same Azure deployment action. + +This workflow is crucial for maintaining a streamlined and automated deployment pipeline for the frontend application, ensuring that each update is efficiently built and deployed to Azure Static Web Apps. + +## `main_ITdisambiguation.yml` +This YAML file outlines a GitHub Actions workflow for building and deploying a Docker container app to an Azure Web App named ITdisambiguation. The workflow is triggered on pushes to the __main branch__, specifically focusing on changes within the `src/**` directory, and can also be manually triggered via __workflow_dispatch__. + +Key elements of this workflow: + +- __Trigger Conditions__: Activates on pushes to the main branch (for `src/**` changes) and allows manual triggers. +- __Environment__: Both build and deploy jobs run on __ubuntu-latest virtual machine__. +- __Jobs__: + - __Build Job__: + - __Steps__: + - __Code Checkout__: Uses `actions/checkout@v2`. + - __Docker Buildx Setup__: Prepares Docker Buildx environment using `docker/setup-buildx-action@v2`. + - __Docker Registry Login__: Logs in to Docker registry with credentials stored in GitHub secrets. + - __Python Environment Setup__: Configures Python 3.11 environment. + - __DVC Setup__: Sets up Data Version Control (DVC) for data and model management. + - __Data and Model Preparation__: Pulls the model weights using DVC. + - __Docker Image Build__: Builds the Docker image with the tag based on the commit SHA and pushes it to Docker Hub. + - __Deploy Job__: + - __Dependency__: Depends on the successful completion of the build job. + - __Environment Info__: Specifies production environment and retrieves the web app URL. + - __Deployment Steps__: + - __Azure Web App Deployment__: Deploys the Docker image to the Azure Web App (ITdisambiguation) using `azure/webapps-deploy@v2`, with the necessary configuration details and publish profile provided via GitHub secrets. + +This workflow plays a critical role in automating the continuous integration and deployment process, ensuring a streamlined deployment of the latest version of the app to Azure Web App. + +## Training action +Initially we wanted to make a GitHub Action also to retrain the model whenever code or data changed, but in the end we decided to avoid implementing it. +This choice was due to the fact that GitHub's virtual machines running the actions are not GPU-accelerated, this would make our training so slow that would exceed the maximum job execution time (6 hours). diff --git a/.github/workflows/pylint.yml b/.github/workflows/pylint.yml index 976a1fe..8800835 100644 --- a/.github/workflows/pylint.yml +++ b/.github/workflows/pylint.yml @@ -3,7 +3,7 @@ name: Pylint on: push: branches: - - main + - stage jobs: build: diff --git a/.github/workflows/pytest.yaml b/.github/workflows/pytest.yaml index 6aca507..c0c7c4c 100644 --- a/.github/workflows/pytest.yaml +++ b/.github/workflows/pytest.yaml @@ -3,7 +3,7 @@ name: Pytest on: push: branches: - - main + - stage paths: - 'src/**' - 'tests/**' diff --git a/README.md b/README.md index 3b69e10..e45fc49 100644 --- a/README.md +++ b/README.md @@ -28,55 +28,262 @@ The model was specifically built to address the [SemEval-2023 Visual Word Sense } ``` +Project's documentation +------------ + +Following there are all the READMEs of the project: +- [Dataset card](./data/README.md) +- [Model card](./models/README.md) +- [APIs and Prometheus documentation](./src/api/README.md) +- [Pytest documentation](./tests/README.md) +- [GitHub Actions documentation](./.github/workflows/README.md) + +Lastly, in the `reports/` folder you can find the pylint, flake8, locust and codecarbon reports. + Project Organization ------------ + ├── .dvc <- dvc hidden folder + │   └── config + ├── .github <- github hidden folder + │   └── workflows + │ ├── README.md + │ ├── azure-static-web-apps-nice-island-02cd56d03.yml + │ ├── main_ITdisambiguation.yml + │ ├── pylint.yml + │ └── pytest.yaml + ├── data <- data folder + │   ├── external + │   ├── interim + │   ├── processed + │   ├── raw + │   ├── README.md + │   ├── Test.dvc + │   └── Train.dvc + ├── docker-compose.yml + ├── docs + ├── dvc.lock + ├── dvc.yaml <- DVC metadata + ├── frontend <- frontend application + │   ├── Dockerfile + │   ├── index.html + │   ├── package.json + │   ├── package-lock.json + │   ├── public + │   ├── README.md + │   ├── src + │   └── vite.config.js ├── LICENSE - ├── Makefile <- Makefile with commands like `make data` or `make train` - ├── README.md <- The top-level README for developers using this project. - ├── data - │   ├── external <- Data from third party sources. - │   ├── interim <- Intermediate data that has been transformed. - │   ├── processed <- The final, canonical data sets for modeling. - │   └── raw <- The original, immutable data dump. - │ - ├── docs <- A default Sphinx project; see sphinx-doc.org for details - │ - ├── models <- Trained and serialized models, model predictions, or model summaries - │ - ├── notebooks <- Jupyter notebooks. Naming convention is a number (for ordering), - │ the creator's initials, and a short `-` delimited description, e.g. - │ `1.0-jqp-initial-data-exploration`. - │ - ├── references <- Data dictionaries, manuals, and all other explanatory materials. - │ - ├── reports <- Generated analysis as HTML, PDF, LaTeX, etc. - │   └── figures <- Generated graphics and figures to be used in reporting - │ - ├── requirements.txt <- The requirements file for reproducing the analysis environment, e.g. - │ generated with `pip freeze > requirements.txt` - │ - ├── setup.py <- makes project pip installable (pip install -e .) so src can be imported - ├── src <- Source code for use in this project. - │   ├── __init__.py <- Makes src a Python module - │ │ - │   ├── data <- Scripts to download or generate data - │   │   └── make_dataset.py - │ │ - │   ├── features <- Scripts to turn raw data into features for modeling - │   │   └── build_features.py - │ │ - │   ├── models <- Scripts to train models and then use trained models to make - │ │ │ predictions - │   │   ├── predict_model.py - │   │   └── train_model.py - │ │ - │   └── visualization <- Scripts to create exploratory and results oriented visualizations - │   └── visualize.py - │ - └── tox.ini <- tox file with settings for running tox; see tox.readthedocs.io + ├── locustfile.py <- locustfile for application loads testing + ├── Makefile + ├── metrics <- prediction metrics of the current model + │   ├── hits1.metric + │   ├── hits3.metric + │   └── mrr.metric + ├── models <- trained and serialized models + ├── notebooks + ├── prometheus-deploy.yml + ├── prometheus.yml + ├── README.md + ├── references + ├── reports <- pylint reports + ├── requirements_docker.txt <- requirements to be used by Docker + ├── requirements.txt <- python requirements + ├── setup.py + ├── src + │   ├── api <- code for the APIs + │   ├── conf.py <- training configuration file + │   ├── data <- preprocessing code + │   ├── Dockerfile + │   ├── __init__.py + │   ├── models <- training and evaluation of the model + │   └── utils.py + ├── test_environment.py + ├── tests <- pytest folder + │   ├── api_testing <- tests for the APIs + │   ├── behavioral_testing <- tests of model's behavior + │   ├── dataset_testing <- tests on the dataset + │   ├── model_testing <- tests on the model's training + │   ├── preprocessing_testing <- tests on the preprocessing code + │   └── README.md + └── tox.ini -------- +# Model summary + +First of all, the sense disambiguator receives context and the target word, producing an output that will be given as input, merged with the target word, to the CLIP text encoder. The text encoder generates embeddings representing the word and its sense based on the provided inputs. + +![Model pipeline](./docs/images/pipeline.png) + +Simultaneously, the images in the training set are fed into the image encoder. The encoder processes each image, generating the embedding of the image. Simultaneously, the sense disambiguator receives context and the target word, producing an output that will be given as input, merged with the target word, to the CLIP text encoder. + +We end up with N 1024-dimensional vectors for each textual input and 10 · N 1024-dimensional vectors for each image. + +We then compute the cosine similarity between the textual embeddings and the image embedding of the ten candidate images, ending up with a similarity matrix of size N × 10. Finally we apply the softmax function on the rows with a temperature parameter τ = 0.01 and take the highest scores as the predictions. As for the metrics used to evaluate the models, we will use Mean Reciprocal Rank, Hits@1 and Hits@3. We obtain the following scores for each metric: + +| MRR | Hits@1 | Hits@3 | +|-----|--------|--------| +|0.833|0.737 |0.920 | + +More informations can be found in the [model card](.models/README.md) + +# Pylint +We utilized the Pylint tool to conduct a comprehensive code cleanup process aimed at achieving a more refined, robust, and readable Python codebase. This effort was directed at minimizing errors and streamlining long-term maintenance. + +Through the analyses performed using Pylint, the predominant issues identified were related to indentation errors, extraneous whitespace, lack of module and function-level documentation, overlapping variable names used both locally and globally and instances of excessively long lines. + +Additionally, in order to streamline the analysis process and focus on the most critical aspects, we made a deliberate choice to disable specific pylint error flags. Notably, we disabled 'import-error' flags associated with module imports and errors related to an excessive number of local variables flagged under 'too-many-locals'. + +This approach allowed us to concentrate on addressing the most crucial code quality aspects while improving readability, maintainability, and adherence to coding standards within the project. + +# Flake8 +We also used the Flake8 tool to write cleaner, more readable, and more easily maintainable code, reducing common errors and promoting the adoption of best practices in writing Python code. Even in this case most of the errors were related to indentation errors, extraneous whitespace and instances of excessively long lines. Then, we solve all these errors and we obtain good results from the analyses. +![Flake8 Report](./docs/images/Flake8/Flake8Report.png) + +# Grafana +We used the Grafana tool to be able to graphically display some values ​​obtained from Prometheus metrics by performing queries. +The dashboard is divided into four rows, each displaying data related to the past month: + +The first row contains four counters: +- The first counter represents the total number of HTTP requests made. +- The second counter displays the number of responses that returned error status codes (in the 400 range) for GET requests. +- The third counter shows the number of responses returning error status codes (in the 500 range) for GET requests. +- The fourth counter indicates the number of responses returning error status codes (in the 400 range) for POST requests. +- The fifth counter represents the number of responses returning error status codes (in the 500 range) for POST requests. +![Counters Row](./docs/images/Grafana/grafana1.png) + +The second row features two time series: +- One related to request sizes measured in bytes. +- The other concerning response sizes, also measured in bytes. +![Size Row](./docs/images/Grafana/grafana2.png) + +The third row contains two graphs: +- One displaying the average latency in responses. +- The other highlighting the average latency per handler. +![Latency Row](./docs/images/Grafana/grafana3.png) + +In the fourth row, there are two graphs depicting resource usage: +- The first graph illustrates the usage of machine RAM. +- The second graph focuses on the average CPU usage. +![Usage Row](./docs/images/Grafana/grafana4.png) + + +# Application Deployment Architecture + +This document provides an overview of the deployment architecture for our web application, which consists of monitoring tools, a backend service, and a frontend interface. + +## Architecture Diagram + +![Deployment Architecture](./docs/images/deploy/itdis_architecture_deploy.png) + +## Azure Deployment + +Our web application is hosted on Microsoft Azure, utilizing three separate Azure virtual machines (VMs) to ensure high availability, scalability, and separation of concerns. It was necessary to split the Azure instances because in particular the backend appears to be quite heavy due to the complexity of the taks. + +### Virtual Machine Distribution + +- **VM1 - Monitoring**: Hosts our monitoring stack which consists of Prometheus and Grafana. This VM is dedicated to collecting and visualizing metrics from our backend service. + +- **VM2 - Backend**: Runs our FastAPI backend service. This backend is responsible for handling API requests, processing data, and performing core logic. + +- **VM3 - Frontend**: Serves the React frontend application. Designed to deliver a responsive and user-friendly interface, this VM hosts the static files and assets required for the frontend. + +### VM1 - Monitoring + + - To configure the virtual machine we used docker compose of the default version of grafana and a customized version of prometheus in which, starting from the original version, we modify the prometheus.yml to allow scraping from the VM2 - Backend. + Below is the docker-compose code used + ``` + version: '3.8' + + services: + grafana: + image: grafana/grafana + ports: + - "3000:3000" + + prometheus: + image: giovtemp/it-disambiguation-prometheus:1.1 + ports: + - "9090:9090" + + ``` +[Link to VM1](https://it-disambiguation-monitoring.azurewebsites.net/login) + + + +### VM2 - Backend + +Our FastAPI backend is encapsulated within a Docker container, originating from a `python:3.8-slim-buster` image to ensure a lean and secure deployment environment. The containerization process leverages Docker to facilitate consistent deployment and operational scalability. Dependencies are meticulously managed through `requirements_docker.txt`, optimizing the build process. + +The backend service is hosted on an Azure Basic B3 instance characterized by 4 vCPUs and 7 GB of memory, which is adept for our development and testing workloads. The instance offers an ACU (Azure Compute Unit) of 100, signifying robust computing capabilities to support our application's backend processes. With 10 GB of remote storage and the capacity to scale up to 3 instances, the setup guarantees high availability with an SLA of 99.95%, ensuring the backend's resilience and consistent performance. + +Leveraging CI/CD pipelines, the Docker images built during the GitHub Actions workflows are made available at our Docker Hub repository. Each image is tagged with the SHA of the commit that triggered the action, allowing for precise version control and traceability. You can find the Docker images and their respective tags [here](https://hub.docker.com/r/franchinifelice/itdisambiguation/tags). + +[Link to VM2 redoc](https://itdisambiguation.azurewebsites.net/redoc) + +### VM3 - Frontend + +While we have a Dockerfile set up to build a containerized version of our React-based frontend, strategic decisions for hosting on Azure have led us to a different path to optimize for cost and performance. Considering the lightweight nature of our frontend, we opted to deploy it as a Static Web App on Azure, which offers a free hosting plan perfectly suited for applications like ours. + +The Azure Static Web App service automates the build and deployment process, enhancing developer productivity and operational efficiency. This service simplifies our deployment workflow and reduces overhead by eliminating the need for container orchestration. Moreover, it aligns with our cost-efficiency goals while still providing the scalability and reliability Azure is known for. + +To ensure quality and stability, our CI/CD pipeline is configured to automatically build the frontend and conduct checks before any changes are merged into the `main` branch. This process guarantees that updates are seamlessly and securely deployed to the Azure-hosted application, maintaining a consistent and reliable user experience. + +[Link to VM3](https://nice-island-02cd56d03.4.azurestaticapps.net) + + +Each VM is configured to work in tandem, offering a cohesive and seamless workflow from the user interface down to the data layer. + +## Monitoring with Better Uptime + +In addition to our internal monitoring with Prometheus and Grafana, we utilize Better Uptime to externally monitor the availability of both the backend and frontend services. +By checking [this page](https://itdisambiguation.betteruptime.com/) it is possible to have an overview of the current status of tha listed websites useful for the system, and any future maintenance and previous incidents if occurred. By joining the application, each member of the team receive an alert whenever an error occurs. + +Here's the view to the monitors: + +![First Monitor](./docs/images/BU/monitor1.png) + +![Second Monitor](./docs/images/BU/monitor2.png) + +## Load testing using Locust + +Lastly we tested the performances of the application using **Locust**, all the possible requests are defined in *locustfile.py*, in particular we test all the four different endpoints of our API: +- Get models list (GET request) +- Get model info (GET request) +- Get image prediction (POST request, **the main task of our project**) +- Get context prediction (POST request) + +POST requests are done more frequently as they are the main functionality of our service, in particular the **predict_images** task. + +Our service is quite heavy as the model is around 100M parameters and the user has to send to the server one or more images (the predict_image endpoint is the one that has the highest response time), so our server suffers from high response times even with only a few users connected at the time. +This issue could be solved by simply upgrading the hosting service, as the one we are using is a very basic one. + +[Here](./reports/locust_report.html) you can find a short report generated by Locust. + + +# Data drift detection +As explained in the last milestone presentation during lesson, we ultimately decided to not implement any data drift detection mechanism, below are the reasons behind this choice. + +**Task definition.** The task that we are trying to solve is a **ranking** problem, given a list of images, one ambiguous target word and one context word, we first disambiguate the target word using the context and then select the correct image among the candidates. + +**Problem: data sources.** The first problem we encountered while trying to implement a data drift detector is relative to the sources from where our data is extracted. +Our idea was to extract some features from the input images (like brightness, color histogram, ... ) and then check the drift. +The problem here is that the images in the dataset are scraped from different web sources, this implies that image distribution has a **very high variance**: +Images in the dataset are not "normalized" in any way, have very different subjects, color scales, brightness values and so on, so it is very difficult to detect drifting in this kind of distribution. +In fact, those images do not have any costraint to be used, so the model could technically take an image of any kind as input. + +![Image distribution](./docs/images/dist.png) +Here we plot the pixel distribution of four random images taken from the dataset, we can see that each of them has a very different pixel distribution (mean and variance). +Despite this, we tried without success to implement a drift detector, so we decided to discard it. + +**Drift robustness.** The main benefit of our model is that it is a fine tuned version of **CLIP**, which is very robust to data drifting, as it's pre-trained on a huge amount of data scraped from the web coming from very different data sources (ex. [LAION-2B](https://laion.ai/blog/laion-5b/)). + + + +--- + + +

Project based on the cookiecutter data science project template. #cookiecutterdatascience

diff --git a/docs/images/BU/monitor1.png b/docs/images/BU/monitor1.png new file mode 100644 index 0000000..6918bcf Binary files /dev/null and b/docs/images/BU/monitor1.png differ diff --git a/docs/images/BU/monitor2.png b/docs/images/BU/monitor2.png new file mode 100644 index 0000000..c09db78 Binary files /dev/null and b/docs/images/BU/monitor2.png differ diff --git a/docs/images/Flake8/Flake8Report.png b/docs/images/Flake8/Flake8Report.png new file mode 100644 index 0000000..5bdba6f Binary files /dev/null and b/docs/images/Flake8/Flake8Report.png differ diff --git a/docs/images/Grafana/grafana1.png b/docs/images/Grafana/grafana1.png new file mode 100644 index 0000000..d94614d Binary files /dev/null and b/docs/images/Grafana/grafana1.png differ diff --git a/docs/images/Grafana/grafana2.png b/docs/images/Grafana/grafana2.png new file mode 100644 index 0000000..371a5e8 Binary files /dev/null and b/docs/images/Grafana/grafana2.png differ diff --git a/docs/images/Grafana/grafana3.png b/docs/images/Grafana/grafana3.png new file mode 100644 index 0000000..7f4e3f1 Binary files /dev/null and b/docs/images/Grafana/grafana3.png differ diff --git a/docs/images/Grafana/grafana4.png b/docs/images/Grafana/grafana4.png new file mode 100644 index 0000000..276d53b Binary files /dev/null and b/docs/images/Grafana/grafana4.png differ diff --git a/docs/images/deploy/itdis_architecture_deploy.png b/docs/images/deploy/itdis_architecture_deploy.png new file mode 100644 index 0000000..a4e8150 Binary files /dev/null and b/docs/images/deploy/itdis_architecture_deploy.png differ diff --git a/docs/images/dist.png b/docs/images/dist.png new file mode 100644 index 0000000..32afd8a Binary files /dev/null and b/docs/images/dist.png differ diff --git a/docs/images/pipeline.png b/docs/images/pipeline.png new file mode 100644 index 0000000..666ecab Binary files /dev/null and b/docs/images/pipeline.png differ diff --git a/locustfile.py b/locustfile.py new file mode 100644 index 0000000..4cd7b98 --- /dev/null +++ b/locustfile.py @@ -0,0 +1,49 @@ +from locust import HttpUser, task, between +from urllib.request import urlopen +from random import choice, sample +from io import BytesIO + +class MyUser(HttpUser): + host = "https://itdisambiguation.azurewebsites.net" + wait_time = between(1, 5) + + def on_start(self): + self.rand_images = [BytesIO(urlopen("https://picsum.photos/224/224").read()) for _ in range(15)] + + def get_random_images(self, n: int = 1): + images = sample(self.rand_images, n) + if n == 1: + return images[0] + return images + + @task + def model_list_task(self): + self.client.get("/models") + + @task + def model_detail_task(self): + model = choice(["RN50", "ViT-B-16"]) + self.client.get(f"/models/{model}") + + @task(3) + def predict_context_task(self): + model = choice(["RN50", "ViT-B-16"]) + image = self.get_random_images(1) + self.client.post( + f"/models/{model}/predict_context", + files={"image": image}, + data={"target_word": "test_target", "contexts": "test_context1, test_context2"} + ) + image.seek(0) + + @task(5) + def predict_images_task(self): + model = choice(["RN50", "ViT-B-16"]) + images = self.get_random_images(4) + self.client.post( + f"/models/{model}/predict_images", + files=[("images", x) for x in images], + data={"target_word": "test_target", "context": "test_context"} + ) + for i in images: + i.seek(0) diff --git a/prometheus-deploy.yml b/prometheus-deploy.yml new file mode 100644 index 0000000..04e4d04 --- /dev/null +++ b/prometheus-deploy.yml @@ -0,0 +1,16 @@ +global: + scrape_interval: 10s + external_labels: + monitor: 'it-disambiguation-monitor' + +scrape_configs: + - job_name: "fastapi-it" + scheme: https + tls_config: + insecure_skip_verify: true + metrics_path: '/metrics' + # Scrape targets from this job every 5 seconds + scrape_interval: 5s + + static_configs: + - targets: ["itdisambiguation.azurewebsites.net"] \ No newline at end of file diff --git a/reports/htmldir/back.svg b/reports/flake-report/back.svg similarity index 100% rename from reports/htmldir/back.svg rename to reports/flake-report/back.svg diff --git a/reports/htmldir/file.svg b/reports/flake-report/file.svg similarity index 100% rename from reports/htmldir/file.svg rename to reports/flake-report/file.svg diff --git a/reports/htmldir/flake8_report.html b/reports/flake-report/index.html similarity index 86% rename from reports/htmldir/flake8_report.html rename to reports/flake-report/index.html index 2eba9ca..a2ba7f6 100644 --- a/reports/htmldir/flake8_report.html +++ b/reports/flake-report/index.html @@ -9,7 +9,7 @@

flake8 violations

-

Generated on 2023-11-16 10:06 +

Generated on 2024-01-05 17:31 with Installed plugins: flake8-html: 0.4.3, mccabe: 0.7.0, pycodestyle: 2.11.1, pyflakes: 3.1.0

diff --git a/reports/htmldir/styles.css b/reports/flake-report/styles.css similarity index 100% rename from reports/htmldir/styles.css rename to reports/flake-report/styles.css diff --git a/reports/locust_report.html b/reports/locust_report.html new file mode 100644 index 0000000..c554efd --- /dev/null +++ b/reports/locust_report.html @@ -0,0 +1,722 @@ + + + + Test Report for locustfile.py + + + + +
+

Locust Test Report

+ +
+ +

During: 2023-12-21 09:27:36 - 2023-12-21 09:29:21

+

Target Host: https://itdisambiguation.azurewebsites.net

+

Script: locustfile.py

+
+ +
+

Request Statistics

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MethodName# Requests# FailsAverage (ms)Min (ms)Max (ms)Average size (bytes)RPSFailures/s
GET/models20361174548770.00.0
POST/models/RN50/predict_context11010470430465582690.10.0
POST/models/RN50/predict_images605039958160251910.10.0
GET/models/ViT-B-16103131316870.00.0
POST/models/ViT-B-16/predict_context502645317809421132740.00.0
POST/models/ViT-B-16/predict_images502392215214329071950.00.0
Aggregated3001326731465582430.30.0
+
+ +
+

Response Time Statistics

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MethodName50%ile (ms)60%ile (ms)70%ile (ms)80%ile (ms)90%ile (ms)95%ile (ms)99%ile (ms)100%ile (ms)
GET/models550550550550550550550550
POST/models/RN50/predict_context75014000150001700019000470004700047000
POST/models/RN50/predict_images170017009600960016000160001600016000
GET/models/ViT-B-163131313131313131
POST/models/ViT-B-16/predict_context1900034000340004200042000420004200042000
POST/models/ViT-B-16/predict_images2600027000270003300033000330003300033000
Aggregated1500017000190002600034000420004700047000
+
+ + + + + + +
+

Charts

+
+ + +
+

Final ratio

+
+
+ +
+ + + + + + + + + + + + + \ No newline at end of file diff --git a/reports/conf_after.txt b/reports/pylint/conf_after.txt similarity index 100% rename from reports/conf_after.txt rename to reports/pylint/conf_after.txt diff --git a/reports/conf_before.txt b/reports/pylint/conf_before.txt similarity index 100% rename from reports/conf_before.txt rename to reports/pylint/conf_before.txt diff --git a/reports/evaluate_after.txt b/reports/pylint/evaluate_after.txt similarity index 100% rename from reports/evaluate_after.txt rename to reports/pylint/evaluate_after.txt diff --git a/reports/evaluate_before.txt b/reports/pylint/evaluate_before.txt similarity index 100% rename from reports/evaluate_before.txt rename to reports/pylint/evaluate_before.txt diff --git a/reports/preprocess_after.txt b/reports/pylint/preprocess_after.txt similarity index 100% rename from reports/preprocess_after.txt rename to reports/pylint/preprocess_after.txt diff --git a/reports/preprocess_before.txt b/reports/pylint/preprocess_before.txt similarity index 100% rename from reports/preprocess_before.txt rename to reports/pylint/preprocess_before.txt diff --git a/reports/train_after.txt b/reports/pylint/train_after.txt similarity index 100% rename from reports/train_after.txt rename to reports/pylint/train_after.txt diff --git a/reports/train_before.txt b/reports/pylint/train_before.txt similarity index 100% rename from reports/train_before.txt rename to reports/pylint/train_before.txt diff --git a/reports/utils_after.txt b/reports/pylint/utils_after.txt similarity index 100% rename from reports/utils_after.txt rename to reports/pylint/utils_after.txt diff --git a/reports/utils_before.txt b/reports/pylint/utils_before.txt similarity index 100% rename from reports/utils_before.txt rename to reports/pylint/utils_before.txt diff --git a/requirements.txt b/requirements.txt index 7c5b4bd..e8315bf 100644 --- a/requirements.txt +++ b/requirements.txt @@ -17,6 +17,7 @@ nltk~=3.8.1 Pillow~=10.1.0 tqdm~=4.66.1 setuptools~=65.5.0 +locust pytest # apis diff --git a/src/api/README.md b/src/api/README.md new file mode 100644 index 0000000..437aa45 --- /dev/null +++ b/src/api/README.md @@ -0,0 +1,75 @@ +APIs +================ +This folder contains the source code for the APIs of our model. +APIs are hosted using uvicorn and the server can be launched simply by executing the following command while in the project's root folder +``` +uvicorn src.api.api:app +``` + +Otherwise, if you want to use the APIs hosted by our server, you can connect to the following URL +``` +https://itdisambiguation.azurewebsites.net/ +``` + +Endpoints +---------------- +The following endpoints are exposed by the server: +- **/models**: Gets a list of all the available models +- **/models/{model_name}**: Gets more informations about the specified model +- **/models/{model_name}/predict_context**: Predict the most relevant context given a list of contexts, an image and a target word +- **/models/{model_name}/predict_images**: Predict the most relevant image given a list of images, a context and a target word + +API Documentation +The documentation is available at these links: + +- ReDoc : https://itdisambiguation.azurewebsites.net/redoc +- Swagger : https://itdisambiguation.azurewebsites.net/docs + +# Prometheus Monitoring for FastAPI +In addition to the API, we also implemented Prometheus for resource and performance monitoring. +The monitoring is facilitated by the `prometheus_fastapi_instrumentator` package, which provides a convenient way to instrument a FastAPI application and collect various metrics. + +## Instrumentator Configuration + +The `Instrumentator` object is configured to collect metrics on request size, response size, request latency, and the total number of requests. It is set up to group status codes, ignore endpoints that do not have templated routes, and track in-progress requests. Additionally, the `/metrics` endpoint is excluded from instrumentation to prevent monitoring of the metrics path itself. + +Here's a brief overview of each metric being collected: + +### Request Size (`request_size`) + +Tracks the size of incoming requests to the FastAPI application, providing insights into the incoming data load the server is handling. Given the nature of our project this metric is very important + +### Response Size (`response_size`) + +Measures the size of responses sent from the FastAPI application, which is useful for understanding the amount of data being served to clients. + +### Latency (`latency`) + +Records the latency of requests, offering a direct measure of the response times that clients are experiencing when interacting with the application. + +### Total Number of Requests (`no_requests`) + +Counts the total number of requests received by the FastAPI application, segmented by handler, method, and status. This metric is crucial for observing the traffic patterns and the load on the application. + +## Environment Variables + +- `METRICS_NAMESPACE`: Configurable namespace for the metrics, defaulting to "fastapi". +- `METRICS_SUBSYSTEM`: Configurable subsystem for the metrics, defaulting to "app". + +These can be set to organize and distinguish metrics in environments where multiple applications or instances are being monitored. + +[Prometheus Local Config](../../prometheus.yml) + +[Prometheus Deploy Config](../../prometheus-deploy.yml) + +[Instrumentator](./prometheus/instrumentator.py) + +## Usage + +Once configured, Prometheus scrape the `/metrics` endpoint of our FastApi application to collect the defined metrics. +Metric scraping of the backend deployed is available at this link : [Metrics](https://itdisambiguation.azurewebsites.net/metrics) + +# Grafana +We used the Grafana tool to be able to graphically display some values ​​obtained from Prometheus metrics by performing queries. +The dashboard and metrics are displayed in the following readme: +[Main Readme](../../README.md) diff --git a/src/api/api.py b/src/api/api.py index e7d7167..7fc6ed2 100644 --- a/src/api/api.py +++ b/src/api/api.py @@ -128,7 +128,7 @@ def _get_model_infos(request: Request, model_name: str): metrics = ModelMetrics( mrr=mrr, hits1=hits1, - hits3=hits1, + hits3=hits3, ) elif model_name == "ViT-B-16": diff --git a/src/api/prometheus/instrumentator.py b/src/api/prometheus/instrumentator.py index 381c333..68ede65 100644 --- a/src/api/prometheus/instrumentator.py +++ b/src/api/prometheus/instrumentator.py @@ -1,23 +1,22 @@ -import os -from prometheus_fastapi_instrumentator import Instrumentator, metrics +import os # noqa:E402,E501 +from prometheus_fastapi_instrumentator import Instrumentator, metrics # noqa:E402,E501 +# pylint: enable=wrong-import-position NAMESPACE = os.environ.get("METRICS_NAMESPACE", "fastapi") -SUBSYSTEM = os.environ.get("METRICS_SUBSYSTEM", "app") +SUBSYSTEM = os.environ.get("METRICS_SUBSYSTEM", "model") -# Crea un oggetto Instrumentator instrumentator = Instrumentator( should_group_status_codes=True, should_ignore_untemplated=True, should_instrument_requests_inprogress=True, - excluded_handlers=["/metrics"], # Escludi l'endpoint /metrics dall'istrumentazione + excluded_handlers=["/metrics"], inprogress_name="inprogress", inprogress_labels=True, ) -# Aggiungi metriche personalizzate o standard al tuo instrumentator +# Additional Metrics -# Misura la dimensione delle richieste instrumentator.add( metrics.request_size( should_include_handler=True, @@ -27,10 +26,7 @@ metric_namespace=NAMESPACE, metric_subsystem=SUBSYSTEM, ) -) - -# Misura la dimensione delle risposte -instrumentator.add( +).add( metrics.response_size( should_include_handler=True, should_include_method=True, @@ -39,10 +35,7 @@ metric_namespace=NAMESPACE, metric_subsystem=SUBSYSTEM, ) -) - -# Misura la latenza delle richieste -instrumentator.add( +).add( metrics.latency( should_include_handler=True, should_include_method=True, @@ -51,10 +44,7 @@ metric_namespace=NAMESPACE, metric_subsystem=SUBSYSTEM, ) -) - -# Conta il numero totale di richieste -instrumentator.add( +).add( metrics.requests( should_include_handler=True, should_include_method=True, diff --git a/src/api/schemas.py b/src/api/schemas.py index b6955ae..dc38d44 100644 --- a/src/api/schemas.py +++ b/src/api/schemas.py @@ -1,27 +1,39 @@ -from pydantic import BaseModel, validator -from nltk.corpus import stopwords +""" Module used to define schemas """ + from http import HTTPStatus from typing import List +from pydantic import BaseModel, validator +from nltk.corpus import stopwords + class PredictContextPayload(BaseModel): + + """ Class used for the prediction of the context payload """ + target_word: str contexts: str @validator("target_word") def target_word_nonempty(cls, v): + + """ Method to verify that the target word is not empty """ + if not v.strip(): raise ValueError("target_word must be a valid string") return v @validator("contexts") def check_contexts(cls, contexts, values): + + """ Method to check the contexts """ + word = values["target_word"] contexts = contexts.split(",") if len(contexts) < 2: raise ValueError("You should send at least two contexts") - for i in range(len(contexts)): - c = contexts[i].strip() + for i, c in enumerate(contexts): + c = c.strip() if not c: raise ValueError("One of the contexts is empty") @@ -36,17 +48,26 @@ def check_contexts(cls, contexts, values): class PredictImagesPayload(BaseModel): + + """ Class used for the prediction of the images payload """ + target_word: str context: str @validator("target_word") def target_word_nonempty(cls, v): + + """ Method to verify that the target word is not empty """ + if not v.strip(): # Utilizza strip() per gestire anche le stringhe con solo spazi raise ValueError("target_word must be a valid string") return v @validator("context") def check_context(cls, context, values): + + """ Method to check the contexts """ + word = values.get("target_word", "").strip() # Gestisci il caso in cui target_word sia None o solo spazi c = context.strip() @@ -62,7 +83,11 @@ def check_context(cls, context, values): return context + class PredictContextResponseData(BaseModel): + + """ Class used for the prediction of the context response data """ + model_name: str target_word: str contexts: str @@ -70,44 +95,76 @@ class PredictContextResponseData(BaseModel): predicted_score: float predicted_context_index: int + class PredictContextResponseModel(BaseModel): + + """ Class used for the prediction of the context response model """ + message: str = HTTPStatus.OK.phrase status_code: int = HTTPStatus.OK.value data: PredictContextResponseData + class PredictImageResponseData(BaseModel): + + """ Class used for the prediction of the image response data """ + model_name: str target_word: str context: str predicted_image_index: int predicted_score: float + class PredictImageResponseModel(BaseModel): + + """ Class used for the prediction of the image response model """ + message: str = HTTPStatus.OK.phrase status_code: int = HTTPStatus.OK.value data: PredictImageResponseData + class ModelMetrics(BaseModel): + + """ Class that describes the model metrics """ + mrr: float hits1: float hits3: float + class GetModelInfosData(BaseModel): + + """ Class used to get infos on the model """ + model_name: str n_parameters: int description: str - typical_usage:str + typical_usage: str metrics: ModelMetrics + class GetModelInfosResponseModel(BaseModel): + + """ Class used to get infos on the model response """ + message: str = HTTPStatus.OK.phrase status_code: int = HTTPStatus.OK.value data: GetModelInfosData + class GetModelNamesData(BaseModel): + + """ Class used to get the model names data """ + model_names: List[str] + class GetModelNamesResponseModel(BaseModel): + + """ Class used to get the response of model names """ + message: str = HTTPStatus.OK.phrase status_code: int = HTTPStatus.OK.value data: GetModelNamesData diff --git a/src/features/.gitkeep b/src/features/.gitkeep deleted file mode 100644 index e69de29..0000000 diff --git a/src/features/__init__.py b/src/features/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/src/models/evaluate.py b/src/models/evaluate.py index cac43ad..26ffaf4 100644 --- a/src/models/evaluate.py +++ b/src/models/evaluate.py @@ -3,12 +3,14 @@ import sys sys.path.append('src') from torch.utils.data import DataLoader -from conf import config -from utils import VWSDDataset, Disambiguator import dagshub import mlflow import torch import open_clip +from utils import VWSDDataset, Disambiguator +from conf import config + + DEV = 'cuda' if torch.cuda.is_available() else 'cpu' disambiguator = Disambiguator(device=DEV) @@ -36,9 +38,9 @@ def compute_metrics(scores_tot, pos): def predict(model_1, words, contexts, images_1): - """ - Method used to make the prediction - + """ + Method used to make the prediction + images_1 must be of shape [batch_size, n_images, channels, height, width] """ @@ -51,17 +53,19 @@ def predict(model_1, words, contexts, images_1): imgs_emb.view(text_emb.size(0), n_images, -1))).softmax(-1) return scores_tot -def predict_context(model_1, word, contexts, image): + +def predict_context(model_1, target_word, contexts, image): """ Method used to predict the context given an image """ - word_expanded = [word for _ in range(len(contexts))] + word_expanded = [target_word for _ in range(len(contexts))] text = tokenizer([f"This is {c}, {exp}." for c, exp in zip(contexts, disambiguator(word_expanded, contexts))]).to(DEV) text_emb = model_1.encode_text(text, normalize=True) imgs_emb = model_1.encode_image(image.unsqueeze(0), normalize=True) - scores = (100 * torch.einsum("ij,kj->ik", imgs_emb, text_emb)).softmax(-1) - return scores + scores_tot = (100 * torch.einsum("ij,kj->ik", imgs_emb, text_emb)).softmax(-1) + return scores_tot + if __name__ == '__main__': with torch.no_grad(): diff --git a/src/models/train.py b/src/models/train.py index 0a65e1f..b11819c 100644 --- a/src/models/train.py +++ b/src/models/train.py @@ -1,12 +1,14 @@ """ Module used to train the model """ -import torch +import sys +sys.path.append('src') import open_clip -from src.conf import config -from src.utils import VWSDDataset, Disambiguator +import torch from torch.utils.data import DataLoader from torch.optim import AdamW from torch.nn import NLLLoss +from conf import config +from utils import VWSDDataset, Disambiguator # Function for training the model diff --git a/src/visualization/.gitkeep b/src/visualization/.gitkeep deleted file mode 100644 index e69de29..0000000 diff --git a/src/visualization/__init__.py b/src/visualization/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/tests/README.md b/tests/README.md index dbc80d0..35204cd 100644 --- a/tests/README.md +++ b/tests/README.md @@ -24,6 +24,8 @@ This folder contains all tests needed for code, model and data. The folder is st 4. Change in the learning rate - `preprocessing_testing` Check the correct behavior of the preprocessing code. +A Great Expectations suite has been also been used during the dataset testing, in combination with pytest by asserting the final outcome of the suite. + ## Usage Tests should only be ran using the command ``` diff --git a/tests/api_testing/test_predict_images.py b/tests/api_testing/test_predict_images.py index 93d63ba..798964f 100644 --- a/tests/api_testing/test_predict_images.py +++ b/tests/api_testing/test_predict_images.py @@ -1,6 +1,6 @@ import pytest from io import BytesIO -from urllib.request import urlopen +from urllib.request import Request,urlopen from src.api.api import app from fastapi import status from fastapi.testclient import TestClient @@ -13,7 +13,9 @@ def send_images_to_api(client, image_urls, target_word, context): """Scarica le immagini e le invia all'API.""" files = [] for url in image_urls: - res = urlopen(url) + request_site = Request(url, headers={"User-Agent": "Mozilla/5.0"}) + res = urlopen(request_site) + #res = urlopen(url) files.append(('images', BytesIO(res.read()))) # Send images to API @@ -29,8 +31,8 @@ def send_images_to_api(client, image_urls, target_word, context): return response def test_success(): - image_urls = ["https://www.magiacomputers.it/media/k2/items/cache/f710044bf79a4b1f5d8b085e5e5d9711_M.jpg", - "https://gdsit.cdn-immedia.net/2016/11/TOPOLINO-970x485.jpg"] + image_urls = ["https://support.content.office.net/it-it/media/e8384959-ad1a-1b95-762b-2861496b886e.png", + "https://i.pinimg.com/736x/7d/62/75/7d62759b503d9005892a592696055faf.jpg"] target_word = "mouse" context = "pc" @@ -49,8 +51,8 @@ def test_success(): ] ) def test_failure(target_word, context): - image_urls = ["https://www.magiacomputers.it/media/k2/items/cache/f710044bf79a4b1f5d8b085e5e5d9711_M.jpg", - "https://gdsit.cdn-immedia.net/2016/11/TOPOLINO-970x485.jpg"] + image_urls = ["https://support.content.office.net/it-it/media/e8384959-ad1a-1b95-762b-2861496b886e.png", + "https://i.pinimg.com/736x/7d/62/75/7d62759b503d9005892a592696055faf.jpg"] target_word = "mouse" context = "" @@ -62,18 +64,18 @@ def test_failure(target_word, context): "image_urls", [ # More then ten images - ["https://www.magiacomputers.it/media/k2/items/cache/f710044bf79a4b1f5d8b085e5e5d9711_M.jpg", - "https://gdsit.cdn-immedia.net/2016/11/TOPOLINO-970x485.jpg", - "https://www.magiacomputers.it/media/k2/items/cache/f710044bf79a4b1f5d8b085e5e5d9711_M.jpg", - "https://gdsit.cdn-immedia.net/2016/11/TOPOLINO-970x485.jpg", - "https://www.magiacomputers.it/media/k2/items/cache/f710044bf79a4b1f5d8b085e5e5d9711_M.jpg", - "https://gdsit.cdn-immedia.net/2016/11/TOPOLINO-970x485.jpg", - "https://www.magiacomputers.it/media/k2/items/cache/f710044bf79a4b1f5d8b085e5e5d9711_M.jpg", - "https://gdsit.cdn-immedia.net/2016/11/TOPOLINO-970x485.jpg", - "https://www.magiacomputers.it/media/k2/items/cache/f710044bf79a4b1f5d8b085e5e5d9711_M.jpg", - "https://gdsit.cdn-immedia.net/2016/11/TOPOLINO-970x485.jpg"], + ["https://support.content.office.net/it-it/media/e8384959-ad1a-1b95-762b-2861496b886e.png", + "https://i.pinimg.com/736x/7d/62/75/7d62759b503d9005892a592696055faf.jpg", + "https://support.content.office.net/it-it/media/e8384959-ad1a-1b95-762b-2861496b886e.png", + "https://i.pinimg.com/736x/7d/62/75/7d62759b503d9005892a592696055faf.jpg", + "https://support.content.office.net/it-it/media/e8384959-ad1a-1b95-762b-2861496b886e.png", + "https://i.pinimg.com/736x/7d/62/75/7d62759b503d9005892a592696055faf.jpg", + "https://support.content.office.net/it-it/media/e8384959-ad1a-1b95-762b-2861496b886e.png", + "https://i.pinimg.com/736x/7d/62/75/7d62759b503d9005892a592696055faf.jpg", + "https://support.content.office.net/it-it/media/e8384959-ad1a-1b95-762b-2861496b886e.png", + "https://i.pinimg.com/736x/7d/62/75/7d62759b503d9005892a592696055faf.jpg"], # Less then two images - ["https://www.magiacomputers.it/media/k2/items/cache/f710044bf79a4b1f5d8b085e5e5d9711_M.jpg"] + ["https://support.content.office.net/it-it/media/e8384959-ad1a-1b95-762b-2861496b886e.png"] ] ) def test_failure_images(image_urls):