Resolve type id 'eclipse-link' as a subtype of MetaStoreManagerFactory

[fivetran-arunsuri]

Description

Resolve type id 'eclipse-link' as a subtype of MetaStoreManagerFactory. Right now with the main branch as it is when we try to configure and run polaris server for production use case with eclipse link as metaStoreManager and postgres as db, it fails with following error

Failed to parse configuration at: metaStoreManager; Could not resolve type id 'eclipse-link' as a subtype of org.apache.polaris.core.persistence.MetaStoreManagerFactory: known type ids = [in-memory] (for POJO property 'metaStoreManager')
at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain: org.apache.polaris.service.config.PolarisApplicationConfig["metaStoreManager"])

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• able to start server locally with eclipse link as metastore manager and hit requests to create catalog etc

Checklist:

Please delete options that are not relevant.

• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• If adding new functionality, I have discussed my implementation with the community using the linked GitHub issue

[MonkeyCanCode]

Hello @fivetran-arunsuri ,

That is expected as Polaris is not enable eclipse-link by default. It is possible to enable it as part of the build. See following for detail (same can be found in https://polaris.apache.org/in-dev/unreleased/metastores/):

Apache Polaris supports the following optional build options:

-PeclipseLink=true – Enables the EclipseLink extension.
-PeclipseLinkDeps=[groupId]:[artifactId]:[version],... – Specifies one or more additional dependencies for EclipseLink (e.g., JDBC drivers) separated by commas.

This is archived by the same file you put the PR with following snippet:

if (project.properties.get("eclipseLink") == "true") {
  dependencies { implementation(project(":polaris-eclipselink")) }
}

Reasoning for this is by default eclipselink is configured to use h2 database which is not very useful for prod graded deployment. Due to licensing concern as well as various diff backends end-users may preferred, none of the other database drivers are included as of now. There is a purpose to use quarkus to close this gap (#392), however, that is not yet completed as of today.

Thanks,
Yong Zheng

[fivetran-arunsuri]

Hello @fivetran-arunsuri ,

That is expected as Polaris is not enable eclipse-link by default. It is possible to enable it as part of the build. See following for detail (same can be found in https://polaris.apache.org/in-dev/unreleased/metastores/):

Apache Polaris supports the following optional build options:

-PeclipseLink=true – Enables the EclipseLink extension.
-PeclipseLinkDeps=[groupId]:[artifactId]:[version],... – Specifies one or more additional dependencies for EclipseLink (e.g., JDBC drivers) separated by commas.

This is archived by the same file you put the PR with following snippet:

if (project.properties.get("eclipseLink") == "true") {
  dependencies { implementation(project(":polaris-eclipselink")) }
}

Reasoning for this is by default eclipselink is configured to use h2 database which is not very useful for prod graded deployment. Due to licensing concern as well as various diff backends end-users may preferred, none of the other database drivers are included as of now. There is a purpose to use quarkus to close this gap (#392), however, that is not yet completed as of today.

Thanks, Yong Zheng

Hey Yong Zheng,
Thank you for the information. I agree that EclipseLink is currently configured to use the H2 database by default. However, I noticed that the steps to include EclipseLink as a build option are missing from the production documentation here. This also leads to issues because the configuration requires setting up the metastoremanager like this:
metaStoreManager:

type: eclipse-link
 conf-file: META-INF/persistence.xml
 persistence-unit: polaris

in polaris-server.yml, which currently pulls EclipseLink from the build options as you mentioned, creating confusion.

Adding the dependency directly in the PR, as I’ve done, should prevent any errors mentioned in the documentation and keep things straightforward. We could also update the documentation to specify that in case additional backends require dependencies for eg postgres here that can be added to the build file, as our team is currently handling it now. Since we’ll be modifying persistence.xml anyway, this approach seems practical.

I'm unclear on why making EclipseLink optional is necessary. Regardless of the database backend used, this property would still be required for production cases and not the in memory obe, wouldn’t it? Let me know your thoughts.

Thanks,
Arun

[MonkeyCanCode]

Hello @fivetran-arunsuri ,
That is expected as Polaris is not enable eclipse-link by default. It is possible to enable it as part of the build. See following for detail (same can be found in https://polaris.apache.org/in-dev/unreleased/metastores/):

Apache Polaris supports the following optional build options:

-PeclipseLink=true – Enables the EclipseLink extension.
-PeclipseLinkDeps=[groupId]:[artifactId]:[version],... – Specifies one or more additional dependencies for EclipseLink (e.g., JDBC drivers) separated by commas.

This is archived by the same file you put the PR with following snippet:

if (project.properties.get("eclipseLink") == "true") {
  dependencies { implementation(project(":polaris-eclipselink")) }
}

Reasoning for this is by default eclipselink is configured to use h2 database which is not very useful for prod graded deployment. Due to licensing concern as well as various diff backends end-users may preferred, none of the other database drivers are included as of now. There is a purpose to use quarkus to close this gap (#392), however, that is not yet completed as of today.
Thanks, Yong Zheng

Hey Yong Zheng, Thank you for the information. I agree that EclipseLink is currently configured to use the H2 database by default. However, I noticed that the steps to include EclipseLink as a build option are missing from the production documentation here. This also leads to issues because the configuration requires setting up the metastoremanager like this: metaStoreManager:

type: eclipse-link
 conf-file: META-INF/persistence.xml
 persistence-unit: polaris

in polaris-server.yml, which currently pulls EclipseLink from the build options as you mentioned, creating confusion.

Adding the dependency directly in the PR, as I’ve done, should prevent any errors mentioned in the documentation and keep things straightforward. We could also update the documentation to specify that in case additional backends require dependencies for eg postgres here that can be added to the build file, as our team is currently handling it now. Since we’ll be modifying persistence.xml anyway, this approach seems practical.

I'm unclear on why making EclipseLink optional is necessary. Regardless of the database backend used, this property would still be required for production cases and not the in memory obe, wouldn’t it? Let me know your thoughts.

Thanks, Arun

Hello,

So my first PR to Polaris is also making that a must: https://github.com/apache/polaris/pull/47/files. Later on this change got accidently rolled off due to PR #114. Then back to your question, I think it is more likes a preference thing regarding if EclipseLink should not enabled or not by default. IMO, is it not very useful when we can't use persistent backend for a catalog other than some simple testing etc. I took a quick look at the production doc you mentioned above, it does stated the following on how to enable it:

To use EclipseLink for metastore management, specify the configuration `metaStoreManager.conf-file` to point to an EclipseLink `persistence.xml` file. This file, local to the Polaris service, contains details of the database used for metastore management and the connection settings. For more information, refer to the [metastore documentation]({{% ref "metastores" %}}).

> [!IMPORTANT]
> EclipseLink requires
> 1. Building the JAR for the EclipseLink extension
> 2. Setting the `eclipseLink` gradle property to `true`.
>
> This can be achieved by setting `eclipseLink=true` in the `gradle.properties` file, or by passing the property explicitly while building all JARs, e.g.: `./gradlew -PeclipseLink=true clean assemble`

Then for sure we will need persistent backend. How to add additional libs for persistent backend is document in the main READ.me instead of production doc. I think there may be some documentation improvement needed (feel free to PR those or I can take care it later this weekend). It may be better to raise a discussion with community on weather to enable this by default. Based on my use case, it is part of build on my pipeline. So this is always included along with additional jar dependencies for the db client.

One potential use case people may have with not wanting this enabled by default is using their own version of EclipseLink with custom implementation and different backend requirements (e.g. MongoDB as backend). The current EcliseLink plugin won't work with document stores database.

Thanks,
Yong

[jbonofre]

I don't think it makes sense to have eclipselink dep defined here.
polaris-service and extensions should stay "decoupled".

[dimas-b]

polaris-service and extensions should stay "decoupled".

+1 to decoupling service classes and extensions.

Taking this one step further, we may want to consider build-time options for producing servers (e.g. docker images) for a particular combination of extensions. I think it is quite doable with gradle.