Added logic to fetch README files, documentation, commit messages, an… #2919

SahilDhillon21 · 2024-11-13T09:59:55Z

…d issue trackers from repository APIs.

Related issue: #2681

Updated Project model to include fields for README content, documentation links, commit summaries, and issue tracker counts.
Implemented API calls in update_projects.py to gather and save this data to the database.

Fixed the migration file issue that I was facing in the previous PR.

I had to remove the line "Authorization": f"token {settings.GITHUB_TOKEN}" from the header as it was giving an error saying 'Unable to fetch repository - 401'. I was unsure how to deal with it so I have removed it for now.

sentry-io · 2024-11-13T10:00:04Z

🔍 Existing Issues For Review

Your pull request is modifying functions with the following pre-existing issues:

📄 File: website/management/commands/update_projects.py

Function	Unhandled Issue
`handle`	SystemExit: 1 /project/{slug}/ `Event Count:` 1

_{Did you find this useful? React with a 👍 or 👎}

SahilDhillon21 · 2024-11-13T11:47:57Z

@DonnieBLT I believe for the CodeQL test the languages should be python, javascript; instead of being 'python javascript'. Though I am unsure about how to fix this.

DonnieBLT · 2024-11-13T14:09:30Z

The reason that the token is there is because it will help with the rate limiting, you can create a token from your github profile settings, page and can you please avoid changes to white space I’m not sure what linter settings you’re using, but maybe if we can standardize them we won’t see the white space changes

SahilDhillon21 · 2024-11-13T18:50:32Z

I've added the GitHub token and fixed the formatting. Please review and let me know.

DonnieBLT

Can you please check the comments?

DonnieBLT · 2024-11-15T05:29:54Z

website/management/commands/update_projects.py

+                    )
+
+                # Set Issue Tracker URL
+                project.issue_tracker_url = f"https://github.com/{repo_name}/issues"


We can remove this since it's universal

Understood.

website/management/commands/update_projects.py

…lon21/BLT into ai-project-enhancement

SahilDhillon21 · 2024-11-19T15:08:38Z

I have integrated a basic summary model "facebook/bart-large-cnn". However, due to the large variations in the readme files of the repositories, the summaries aren't too effective. I had thought of pre-processing the content to only pass relevant sections, but even that seems to be difficult since there is no particular structure followed. @DonnieBLT which direction should I look into to improve this? Though openai API is paid, it could do the job really well compared to the generic python models

DonnieBLT · 2024-11-19T15:09:58Z

We can use the OpenAI we already have a API key and it is set up in the code

…lon21/BLT into ai-project-enhancement

DonnieBLT · 2024-11-24T01:40:42Z

website/templates/website/project_list.html

@@ -53,130 +62,7 @@ <h3>Projects: {{ projects.count }}</h3>
                {% endfor %}
            </ul>
        {% endif %}
-        <ul class="project-list">


can we keep this in this file please

So I assume we don't want to create a separate template and have all the code here itself? Let us finalize what to do with the search function and I'll make the changes accordingly

Yes we can combine this into one template and adjust the global search to work as you have it.

DonnieBLT · 2024-11-24T01:41:34Z

website/label_generation.py

+openai.api_key = os.getenv("OPENAI_API_KEY")
+
+
+def generate_labels(readme_content, github_topics):


can you move this to utils.py too please

DonnieBLT · 2024-11-24T01:42:45Z

website/management/commands/update_projects.py

+                        project.readme_content = readme_content
+                        readme_text = markdown_to_text(readme_content)
+                        project.ai_summary = ai_summary(readme_text, project.topics)
+                        project.ai_labels = json.loads(generate_labels(readme_text, project.topics))


should we just add the labels verbatim from the topics?

I found the AI-generated labels to be more accurate and effective, but we can surely use these topics directly. I'll modify it

DonnieBLT

requesting a few changes, we're almost there

website/management/commands/update_projects.py

…e url from project model

SahilDhillon21 · 2024-11-24T14:09:58Z

Have made most of the changes, just need a final heads-up on what to do with the search functionality as it's quite buggy and limited. Should I create a new PR to improve its working if we go that route?

DonnieBLT

A few adjustments request

DonnieBLT · 2024-11-28T04:56:59Z

website/management/commands/update_projects.py

+                    )
+
+                # Check for Documentation URL (homepage)
+                project.documentation_url = repo_data.get("homepage")


I believe we have a homepage field

DonnieBLT · 2024-11-28T04:58:15Z

website/models.py

@@ -756,6 +756,10 @@ class Project(models.Model):
    closed_issues = models.IntegerField(default=0)
    size = models.IntegerField(default=0)
    commit_count = models.IntegerField(default=0)
+    readme_content = models.TextField(null=True, blank=True)
+    documentation_url = models.URLField(null=True, blank=True)


Use homepage_url

DonnieBLT · 2024-11-28T05:00:04Z

website/templates/website/project_list.html

@@ -22,6 +22,15 @@ <h3>Projects: {{ projects.count }}</h3>
                    <i class="fas fa-plus-circle"></i> Add Project
                </button>
            </form>
+            <form id="search-form" class="search-form">


Can you combine them into the one top search?

DonnieBLT · 2024-11-28T05:00:56Z

website/templates/website/project_list.html

@@ -53,130 +62,7 @@ <h3>Projects: {{ projects.count }}</h3>
                {% endfor %}
            </ul>
        {% endif %}
-        <ul class="project-list">


Yes we can combine this into one template and adjust the global search to work as you have it.

…ch function

SahilDhillon21 · 2024-11-30T11:23:45Z

I have combined the search bar into one and added all the remaining categories. I'll raise a new pr later to improve the UI of the search results, have kept it basic for now.

Added github token header

34add4f

SahilDhillon21 force-pushed the ai-project-enhancement branch from 95dc670 to 34add4f Compare November 13, 2024 17:32

SahilDhillon21 added 3 commits November 13, 2024 23:34

small format changes

71e2baf

whitespace edits

d7da328

ruff formatted

8e1f7db

SahilDhillon21 and others added 6 commits November 14, 2024 21:18

Merge branch 'main' into ai-project-enhancement

d147d9f

Merge branch 'main' into ai-project-enhancement

fb4c6c8

Fields for summary and labels added

80c1cbe

Summary and labels added in project list template

ba7adcd

Updated projectviewset to handle ai summary updates

fa6ff9c

added method to split and return ai labels

fa8501d

DonnieBLT requested changes Nov 19, 2024

View reviewed changes

SahilDhillon21 and others added 7 commits November 19, 2024 13:53

Merge branch 'OWASP-BLT:main' into ai-project-enhancement

0c56498

Merge branch 'ai-project-enhancement' of https://github.com/SahilDhil…

8d5b239

…lon21/BLT into ai-project-enhancement

Removed tracker url

56b7f62

Added summarizer model

ff445d0

fix formatting

968e7da

Double quotes on summary_text

d7b7c09

Merge branch 'main' into ai-project-enhancement

2028e01

SahilDhillon21 and others added 5 commits November 20, 2024 00:05

Created openai summarize functions

94a03b9

ruff formatted

95bbb84

Merge branch 'main' into ai-project-enhancement

07f8863

isort order

58642e8

Merge branch 'ai-project-enhancement' of https://github.com/SahilDhil…

c2b67e6

…lon21/BLT into ai-project-enhancement

DonnieBLT reviewed Nov 24, 2024

View reviewed changes

DonnieBLT requested changes Nov 24, 2024

View reviewed changes

DonnieBLT added the requested changes label Nov 24, 2024

DonnieBLT reviewed Nov 24, 2024

View reviewed changes

website/management/commands/update_projects.py Outdated Show resolved Hide resolved

SahilDhillon21 and others added 7 commits November 24, 2024 17:05

Merge branch 'OWASP-BLT:main' into ai-project-enhancement

431c60b

Shift summarizer and label generation to utils.py, remove issue track…

8400c87

…e url from project model

Remove 'generate labels' function from utils.py

21e71bd

Github topics/labels now get saved in the Tag model

54a2f4f

Merge migrations

03f05f1

remove json import

e997511

inline imports

465e344

DonnieBLT requested changes Nov 28, 2024

View reviewed changes

SahilDhillon21 and others added 14 commits November 29, 2024 22:48

Merge branch 'OWASP-BLT:main' into ai-project-enhancement

6626ab7

remove doc_url and use single project template

c79a0e4

Added options for issues, domains and labels in main search

d2af8c5

INcluded other categories in header and filtering logic added in sear…

64dcd28

…ch function

fix project query to return disticnt results and add ui for projects

28dbda2

finalise changes

a55fd71

remove redundant search bar from top

26c0e80

margin

a6a569c

margine added

481363c

retain the query and category in search bar

8b494da

fix formatting

ab3e879

final linting changes

a8fb7fd

remove project list template and fix inline styles

0211ea6

fix migrations

9b191e9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added logic to fetch README files, documentation, commit messages, an… #2919

Added logic to fetch README files, documentation, commit messages, an… #2919

SahilDhillon21 commented Nov 13, 2024

sentry-io bot commented Nov 13, 2024

SahilDhillon21 commented Nov 13, 2024

DonnieBLT commented Nov 13, 2024

SahilDhillon21 commented Nov 13, 2024

DonnieBLT left a comment

DonnieBLT Nov 15, 2024

SahilDhillon21 Nov 19, 2024

SahilDhillon21 commented Nov 19, 2024

DonnieBLT commented Nov 19, 2024

DonnieBLT Nov 24, 2024

SahilDhillon21 Nov 24, 2024

DonnieBLT Nov 28, 2024

DonnieBLT Nov 24, 2024

DonnieBLT Nov 24, 2024

SahilDhillon21 Nov 24, 2024

DonnieBLT left a comment

SahilDhillon21 commented Nov 24, 2024

DonnieBLT left a comment

DonnieBLT Nov 28, 2024

DonnieBLT Nov 28, 2024

SahilDhillon21 Nov 30, 2024

DonnieBLT Nov 28, 2024

DonnieBLT Nov 28, 2024

SahilDhillon21 commented Nov 30, 2024

		openai.api_key = os.getenv("OPENAI_API_KEY")


		def generate_labels(readme_content, github_topics):

Added logic to fetch README files, documentation, commit messages, an… #2919

Are you sure you want to change the base?

Added logic to fetch README files, documentation, commit messages, an… #2919

Conversation

SahilDhillon21 commented Nov 13, 2024

sentry-io bot commented Nov 13, 2024

🔍 Existing Issues For Review

SahilDhillon21 commented Nov 13, 2024

DonnieBLT commented Nov 13, 2024

SahilDhillon21 commented Nov 13, 2024

DonnieBLT left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SahilDhillon21 commented Nov 19, 2024

DonnieBLT commented Nov 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DonnieBLT left a comment

Choose a reason for hiding this comment

SahilDhillon21 commented Nov 24, 2024

DonnieBLT left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SahilDhillon21 commented Nov 30, 2024