Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding the pipeline for the task explanation and Llm #2190

Open
wants to merge 50 commits into
base: main
Choose a base branch
from

Conversation

Bepitic
Copy link
Contributor

@Bepitic Bepitic commented Jul 15, 2024

📝 Description

  • I added the Task explanation and built the pipeline for the future LLM that will categorize and explain the image.
  • I modified the visualization such the explanation can be seen in the image outputted.

✨ Changes

Select what type of change your PR is:

  • 🐞 Bug fix (non-breaking change which fixes an issue)
  • 🔨 Refactor (non-breaking change which refactors the code base)
  • 🚀 New feature (non-breaking change which adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • 📚 Documentation update
  • 🔒 Security update

✅ Checklist

Before you submit your pull request, please make sure you have completed the following steps:

  • 📋 I have summarized my changes in the CHANGELOG and followed the guidelines for my type of change (skip for minor changes, documentation updates, and test enhancements).
  • 📚 I have made the necessary updates to the documentation (if applicable).
  • 🧪 I have written tests that support my changes and prove that my fix is effective or my feature works (if applicable).

For more information about code review checklists, see the Code Review Checklist.

@Bepitic
Copy link
Contributor Author

Bepitic commented Jul 15, 2024

Hello, I'm looking to make this contribution as effective as possible, so please feel free to share any suggestions, concerns, or ideas you might have.
Thanks in advance for your time and input!

Copy link
Contributor

@samet-akcay samet-akcay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added my initial comments, will do another round of review

src/anomalib/__init__.py Outdated Show resolved Hide resolved
src/anomalib/callbacks/metrics.py Outdated Show resolved Hide resolved
src/anomalib/callbacks/metrics.py Outdated Show resolved Hide resolved
Copy link

codecov bot commented Jul 16, 2024

Codecov Report

Attention: Patch coverage is 63.49206% with 23 lines in your changes missing coverage. Please review.

Project coverage is 80.67%. Comparing base (e7e6cb8) to head (a463b5b).
Report is 2 commits behind head on main.

Files Patch % Lines
src/anomalib/utils/visualization/image.py 47.61% 22 Missing ⚠️
...anomalib/deploy/inferencers/openvino_inferencer.py 83.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2190      +/-   ##
==========================================
- Coverage   80.78%   80.67%   -0.12%     
==========================================
  Files         248      248              
  Lines       10859    10885      +26     
==========================================
+ Hits         8772     8781       +9     
- Misses       2087     2104      +17     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Bepitic Bepitic requested a review from samet-akcay July 18, 2024 16:00
Copy link
Contributor

@djdameln djdameln left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR.

I think it would be good to add the OpenAI model implementation to this PR, so that we can run the pipeline and generate some visualizations.

src/anomalib/__init__.py Outdated Show resolved Hide resolved
src/anomalib/callbacks/metrics.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@ashwinvaidya17 ashwinvaidya17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR looks good but I am not sure about the expected output. Maybe it would be a good idea to add tests for this is possible.
This is the code I used for visualization. Is this what you would expect? (ignore the noisy image)

from anomalib.utils.visualization.image import ImageResult, ImageVisualizer, VisualizationMode
import numpy as np
from anomalib import TaskType
import matplotlib.pyplot as plt

anomaly_map = np.zeros((100, 100), dtype=np.float64)
anomaly_map[10:20, 10:20] = 1.0
gt_mask = np.zeros((100, 100))
gt_mask[15:25, 15:25] = 1.0
image = np.random.randint(0, 255, size=(100, 100, 3), dtype=np.uint8)
image_result = ImageResult(
    image=image,
    pred_score=0.9,
    pred_label="abnormal",
    text_descr=("Some very long text to see how it is formatted in the image"
                " Some very long text to see how it is formatted in the image"),
    anomaly_map= anomaly_map,
    gt_mask = gt_mask,
    pred_mask = anomaly_map,
)

image_visualizer = ImageVisualizer(
    mode = VisualizationMode.FULL,
    task= TaskType.EXPLANATION,
)
result = image_visualizer.visualize_image(image_result)

plt.imshow(result)
plt.show()
afbeelding

src/anomalib/utils/visualization/image.py Show resolved Hide resolved
src/anomalib/utils/visualization/image.py Show resolved Hide resolved
Copy link
Collaborator

@ashwinvaidya17 ashwinvaidya17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Looks fine overall. I just have a few minor comments.

tests/integration/model/test_models.py Outdated Show resolved Hide resolved
src/anomalib/models/image/gptvad/lightning_model.py Outdated Show resolved Hide resolved
tests/unit/models/image/chatgpt_vision/test_api.py Outdated Show resolved Hide resolved
src/anomalib/models/image/gptvad/chatgpt.py Outdated Show resolved Hide resolved
src/anomalib/models/image/gptvad/chatgpt.py Outdated Show resolved Hide resolved
src/anomalib/models/image/gptvad/chatgpt.py Outdated Show resolved Hide resolved
src/anomalib/models/image/gptvad/readme.md Outdated Show resolved Hide resolved
Copy link
Collaborator

@ashwinvaidya17 ashwinvaidya17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks looks good to me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants