tools: web scraper

[Struki84]

Added a simple web scraper tool, you can set the scraping depth when creating the tool, default is 1. The tool will scrap all the links it finds on the domain, will not follow outside links. Depending on the size of the website it might take a while.

Will return a string containing all the content it found, meta data, list of links and scraped links. Will comply with robots.txt

PR Checklist

• Read the Contributing documentation.
• Read the Code of conduct documentation.
• Name your Pull Request title clearly, concisely, and prefixed with the name of the primarily affected package you changed according to Good commit messages (such as memory: add interfaces for X, Y or util: add whizzbang helpers).
• Check that there isn't already a PR that solves the problem the same way to avoid creating a duplicate.
• Provide a description in this PR that addresses what the PR is solving, or reference the issue that it solves (e.g. Fixes #123).
• Describes the source of new concepts.
• References existing implementations as appropriate.
• Contains test coverage for new functions.
• Passes all golangci-lint checks.

[Struki84]

@tmc @FluffyKebab any feedback on this?

[tmc]

Can you make the options configurable (not just depth).

[Struki84]

@tmc is this better?

[Struki84]

@tmc bump ^^

[tmc]

LGTM