Crawler Integration #358

kamilczaja · 2024-10-24T14:48:21Z

Closes #347

#98) * feat: max data offers per connector * feat: max contract offers per connector * refactor: DataOfferFetcher * refactor: DataOfferFetcher * refactor: minor remarks * feat: DataOfferLimitsEnforcer * feat: DataOfferLimitsEnforcer * feat: DataOfferLimitsEnforcer * refactor: further refactorings * refactor: further refactorings * refactor: further refactorings * test: add DataOfferLimitsEnforcerTest * test: add DataOfferLimitsEnforcerTest * test: add DataOfferLimitsEnforcerTest * test: add DataOfferLimitsEnforcerTest * test: add DataOfferLimitsEnforcerTest * refactor: DataOfferLimitsEnforcer * test: no_limit_and_two_dataofffers_and_contractoffer_should_not_limit

Bumps org.flywaydb.flyway from 9.19.1 to 9.19.3. --- updated-dependencies: - dependency-name: org.flywaydb.flyway dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps org.flywaydb.flyway from 9.19.3 to 9.19.4. --- updated-dependencies: - dependency-name: org.flywaydb.flyway dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat: filtering of data offers * chore: reformat code and organize imports * chore: fix checkstyle

* feat: data offer pagination (and sorting fixes) * chore: add stubs for new endpoints * chore: test migrations so that non-empty tables will be migrated.

* feat: dataspace filter * test: add dataspace filter * chore: revert dataspace as asset prop * feat: add dataSpace to CatalogQueryFields * feat: add dataSpace to CatalogQueryFiaelds * feat: add dataSpace to CatalogQueryFields * feat: add dataSpace to CatalogQueryFields * chore: fix checkstyle * refactor: add DataSpaceConfig * refactor: buildDataSpaceField * refactor: BrokerServerSettings * feat: get known dataspaces from config * refactor: PR remarks * refactor: minor refactorings * refactor: minor refactorings * refactor: minor refactorings * refactor: minor refactorings * refactor: minor refactorings * refactor: minor refactorings * refactor: minor refactorings * test: test_available_filter_values_to_filter_by * test: test_available_filter_values_to_filter_by

…lues, but other filters should (#133)

Bumps org.flywaydb.flyway from 9.19.4 to 9.20.0. --- updated-dependencies: - dependency-name: org.flywaydb.flyway dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps org.jooq:jooq from 3.18.4 to 3.18.5. --- updated-dependencies: - dependency-name: org.jooq:jooq dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: fix api-wrapper integration * test: refactor assertEqualJson

* feat: dataOfferDetailPage * feat: connectorDetailPage * feat: detail pages * feat: detail pages * feat: detail pages * test: detail pages * refactor: minor refactorings * refactor: minor refactorings * refactor: minor test refactorings * refactor: minor test refactorings * refactor: minor test refactorings * refactor: minor test refactorings * refactor: DataOfferDetailPageQueryService * refactor: DataOfferDetailPageQueryService * refactor: DataOfferDetailPageQueryService * refactor: changed models slightly, improved tests --------- Co-authored-by: Tim Berthold <[email protected]> Co-authored-by: Tim Berthold <[email protected]>

* feat: permanently delete old offline connectors * feat: permanently delete old offline connectors * feat: permanently delete old offline connectors * feat: permanently delete old offline connectors * feat: permanently delete old offline connectors * feat: permanently delete old offline connectors * feat: permanently delete old offline connectors * feat: permanently delete old offline connectors * feat: permanently delete old offline connectors * feat: permanently delete old offline connectors * feat: permanently delete old offline connectors * feat: permanently delete old offline connectors * feat: permanently delete old offline connectors * feat: permanently delete old offline connectors * feat: permanently delete old offline connectors * feat: permanently delete old offline connectors * feat: permanently delete old offline connectors * feat: permanently delete old offline connectors * feat: permanently delete old offline connectors * feat: permanently delete old offline connectors * feat: permanently delete old offline connectors * chore: minor refactorings * chore: checkstyle * test: DeadConnectorRemovalTest * refactor: pr remarks * chore: checkstyle * chore: pr remarks * chore: checkstyle * refactor: test does not need full edc extension anymore --------- Co-authored-by: Richard Treier <[email protected]>

* chore: add path mapping to reverse proxy deployment documentation * chore: fix wording

Co-authored-by: Richard Treier <[email protected]>

* fix: api wrapper integration * fix: api wrapper integration * chore: checkstyle * chore: checkstyle

…client pipeline (#170)

Co-authored-by: Richard Treier <[email protected]>

--------- Co-authored-by: Kamil Czaja <[email protected]>

* fix: Remove duplicate database indices * fix: Improve documentation

--------- Co-authored-by: Kamil Czaja <[email protected]>

Co-authored-by: Tim Berthold <[email protected]>

This reverts commit a02b631ea9a04a0e256ded2e4448e7ebf6effbdb.

Co-authored-by: Kamil Czaja <[email protected]>

richardtreier · 2024-11-18T12:56:01Z

docs/deployment-guide/goals/production/README.md

+
+#### Catalog Crawler Configuration
+
+A productive configuration will require you to join a DAPS.


"For each dataspace environment you need one catalog crawler" (continue proper onboarding onto what this is and how to deal with it)

richardtreier · 2024-11-18T12:56:35Z

...ler/catalog-crawler/src/test/resources/migration-test-utils/V9999__Make_Columns_Nullable.sql

@@ -0,0 +1,25 @@
+do


could you please check if this is still required

richardtreier · 2024-11-18T12:57:53Z

docs/deployment-guide/goals/production/.env.catalog-crawler

+# Default ENV Vars
+
+# This file will be sourced as bash script:
+# - KEY=Value will become KEY=${KEY:-"Value"}, so that ENV Vars can be overwritten by parent docker-compose.yaml.
+# - Watch out for escaping issues as values will be surrounded by quotes, and dollar signs must be escaped.
+
+# ===========================================================
+# Available Catalog Crawler Config
+# ===========================================================
+
+# Environment ID
+CRAWLER_ENVIRONMENT_ID=missing-env-CRAWLER_ENVIRONMENT_ID
+
+# Fully Qualified Domain Name (e.g. example.com)
+MY_EDC_FQDN=missing-env-MY_EDC_FQDN
+
+# Postgres Database Connection
+CRAWLER_DB_JDBC_URL=jdbc:postgresql://missing-postgresql-url
+CRAWLER_DB_JDBC_USER=missing-postgresql-user
+CRAWLER_DB_JDBC_PASSWORD=missing-postgresql-password
+
+# Database Connection Pool Size
+CRAWLER_DB_CONNECTION_POOL_SIZE=30
+
+# Database Connection Timeout (in ms)
+CRAWLER_DB_CONNECTION_TIMEOUT_IN_MS=30000
+
+# CRON interval for crawling ONLINE connectors
+CRAWLER_CRON_ONLINE_CONNECTOR_REFRESH=*/20 * * ? * *
+
+# CRON interval for crawling OFFLINE connectors


wait a minute, don't we not need this anymore, since we have Config-as-Java-Code

richardtreier · 2024-11-18T12:58:06Z

docs/deployment-guide/goals/production/README.md

+| Caddy behind OAuth2 Proxy | caddy:2.7                                                                                         |
+| Authority Portal Backend  | authority-portal-backend, see [CHANGELOG.md](../../../../CHANGELOG.md) for compatible versions.   |
+| Authority Portal Frontend | authority-portal-frontend, see  [CHANGELOG.md](../../../../CHANGELOG.md) for compatible versions. |
+| Catalog Crawler           | authority-portal-crawler, see [CHANGELOG.md](../../../../CHANGELOG.md) for compatible versions.   |


Catalog Crawler (one per environment)

richardtreier · 2024-11-18T12:59:49Z

docs/deployment-guide/goals/production/README.md

+- The catalog crawler is meant to be served via TLS/HTTPS.
+- The catalog crawler is meant to be deployed with a reverse proxy terminating TLS / providing HTTPS.
+- All requests are meant to be redirected to the deployment's `11003` port.


"Each catalog crawler..." requires rework. Maybe the AP and the crawlers should have their own sections

third point is a bit vague

richardtreier · 2024-11-18T13:02:03Z

.github/workflows/ci.yml

+          registry-password: ${{ secrets.GITHUB_TOKEN }}
+          image-base-name: ${{ env.IMAGE_NAME_BASE }}
+          image-name: "authority-portal-crawler"
+          connector-name: "catalog-crawler-ce"


there's no dev image?

richardtreier · 2024-11-18T13:02:31Z

CHANGELOG.md

+  - To prevent versioning conflicts with the image from EDC CE up to version 10.4.1, the image is now named differently. See [compatible versions](#compatible-versions) below.
+


it's enough that this is in the deployment migration notes.

richardtreier and others added 30 commits June 6, 2023 15:24

chore: post-release cleanup (#120)

af1eb6c

feat: filtering of data offers (#125)

720d26e

* feat: filtering of data offers * chore: reformat code and organize imports * chore: fix checkstyle

feat: data offer pagination (and sorting fixes) (#127)

cbe0ea7

* feat: data offer pagination (and sorting fixes) * chore: add stubs for new endpoints * chore: test migrations so that non-empty tables will be migrated.

fix: filter should not be applied to itself when fetching possible va…

e322223

…lues, but other filters should (#133)

fix: npe from empty arrays returned by db query (#134)

519074a

fix: add vault-filesystem dependendcy

b10ca7c

fix: tasks not being run in parallel (#136)

15abb97

refactor: remove management-api (#145)

a54a4e5

fix: thread pool not executing in parallel (#147)

cfefbc8

fix: try fix pipeline (#148)

1793a98

feat: better logged messages (#149)

d0f51a7

chore: release prep + cleanup (#152)

00be7f0

chore: set edc-extensions version to 3.3.0 (#153)

907c7de

chore: post release cleanup (#154)

f1edad1

chore: fix api-wrapper integration (#156)

b4b394e

* chore: fix api-wrapper integration * test: refactor assertEqualJson

chore: add path mapping to reverse proxy deployment documentation (#161)

8c2b30c

* chore: add path mapping to reverse proxy deployment documentation * chore: fix wording

fix: missing required EDC daps config params (#163)

1fb1ea1

refactor: split api wrapper (#162)

72c2e89

Co-authored-by: Richard Treier <[email protected]>

fix: api wrapper integration (#169)

763227f

* fix: api wrapper integration * fix: api wrapper integration * chore: checkstyle * chore: checkstyle

chore: finalize broker server api client, improved READMEs, fixed ts …

18a9c73

…client pipeline (#170)

fix: broker server ts client not working (#173)

d1ec527

fix: PolicyDto after latest api-wrapper changes (#177)

3470746

kamilczaja and others added 28 commits November 18, 2024 13:37

feat: rework always-true policy definition (#1005)

15f2e6e

Co-authored-by: Richard Treier <[email protected]>

chore: prepare release (#1007)

1eb8fcb

chore: fix always-true policy test (#1008)

f7895ec

chore: fix test (#1009)

36d870b

fix: test (#1010)

75fd7cd

chore: Rework E2E test extension to be used in the EDC EE (#1011)

f8d349e

feat: fetch a single contract agreement by its ID (#1016)

727b9cf

--------- Co-authored-by: Kamil Czaja <[email protected]>

chore(deps): update LoggingHouse dependency to latest version (#1015)

87add37

New Always True policy should not cause errors (#1012)

303860e

chore: sync ap migrations (#1021)

37aff08

Release prep (#1022)

ba62af5

fix: Remove duplicate database indices (#1024)

1c24116

* fix: Remove duplicate database indices * fix: Improve documentation

docs: add faq.md (#1026)

1aa1e18

fix: Address update (#1029)

9b6ec3f

--------- Co-authored-by: Kamil Czaja <[email protected]>

fix: flyway validation failing in mds variant (#1025)

8af8ff8

chore: Release 10.2.0 preparation (#1032)

df510db

docs: add on-request data-source as example in asset-creation (#1034)

a450640

fix(postman): update collection to reflect recent API changes (#1037)

180332b

feat: log terminations to logging house (#1027)

9a5df50

chore: Release prep 10.3.0 (#1038)

76551e8

chore(workflows): remove unused workflow (#1039)

dbe965e

feat: add CreateDataOffer API endpoint (#1035)

2e4c9cb

chore: Prepare release 10.4.0 (#1042)

4ce6073

Co-authored-by: Tim Berthold <[email protected]>

Also add .env changes (#1044)

e06eb27

chore: ap flyway migration sync (#1045)

eea8425

Revert "chore: ap flyway migration sync (#1045)" (#1049)

edbfa31

This reverts commit a02b631ea9a04a0e256ded2e4448e7ebf6effbdb.

chore: Release 10.4.1 (#1050)

f44e219

feat: catalog crawler, moved here from edc-ce

0f1b609

Co-authored-by: Kamil Czaja <[email protected]>

richardtreier force-pushed the crawler-integration-temp-3 branch from cff59da to 0f1b609 Compare November 18, 2024 12:51

richardtreier requested changes Nov 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crawler Integration #358

Crawler Integration #358

kamilczaja commented Oct 24, 2024 •

edited by richardtreier

Loading

richardtreier Nov 18, 2024

richardtreier Nov 18, 2024

richardtreier Nov 18, 2024

richardtreier Nov 18, 2024

richardtreier Nov 18, 2024

richardtreier Nov 18, 2024

richardtreier Nov 18, 2024


		#### Catalog Crawler Configuration

		A productive configuration will require you to join a DAPS.

		- To prevent versioning conflicts with the image from EDC CE up to version 10.4.1, the image is now named differently. See [compatible versions](#compatible-versions) below.

Crawler Integration #358

Are you sure you want to change the base?

Crawler Integration #358

Conversation

kamilczaja commented Oct 24, 2024 • edited by richardtreier Loading

richardtreier Nov 18, 2024

Choose a reason for hiding this comment

richardtreier Nov 18, 2024

Choose a reason for hiding this comment

richardtreier Nov 18, 2024

Choose a reason for hiding this comment

richardtreier Nov 18, 2024

Choose a reason for hiding this comment

richardtreier Nov 18, 2024

Choose a reason for hiding this comment

richardtreier Nov 18, 2024

Choose a reason for hiding this comment

richardtreier Nov 18, 2024

Choose a reason for hiding this comment

kamilczaja commented Oct 24, 2024 •

edited by richardtreier

Loading