Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Upgrade Karaf to 4.4.7, Xtext/Xtend 2.37.0 #4406

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

holgerfriedrich
Copy link
Member

@holgerfriedrich holgerfriedrich commented Oct 5, 2024

This is WIP and prepares for next Karaf version 4.4.7 (currently still at SNAPSHOT level).

Fixes #4283
Fixes CVE in BouncyCastle 1.77

Upgrade Karaf from 4.4.6 to 4.4.7

  • Sync runtime dependencies with Karaf 4.4.7, most notably:
    • PaxWeb 8.0.29
    • Jetty 9.4.56.v20240826
    • BouncyCastle 1.78.1
    • CXF 3.6.4
    • Jackson 2.17.2
    • JNA 5.15.0
    • JAXB 2.3.9
    • commons-io 2.17.0
    • commons-lang3 3.17.0
    • XBean 4.26
    • ASM 9.7.1
  • Resolve itest runbundles

Upgrade Xtext/Xtend to 2.37.0

@andrewfg
Copy link
Contributor

@holgerfriedrich did you see the comments on #3315 about our dependency on Jetty 9.x whereby Jetty 9.x is EOL but upgrading to Jetty 10 or 11 or 12 apparently requires Karaf 4.5 ??

So in other words your incremental Karaf 4.4.x upgrades are all fine and good, but do not resolve the issue that we are over dependendent on Jetty 9.x which is now THREE major versions out of date from current Jetty code.

@J-N-K
Copy link
Member

J-N-K commented Oct 10, 2024

We can't do anything about that. Karaf 4.5 should bring at least Jetty 10, but TBH - looking at the current speed of development - I don't see that before next year.

@andrewfg
Copy link
Contributor

andrewfg commented Oct 10, 2024

@J-N-K is there a hard point in our Karaf 4.4.x configuration that absolutely forbids importing Jetty 10+ ?? .. I am wondering if there may be some approach whereby we can try to decouple these inter dependencies. The current tight interdependence put us (IMHO) in a very risky situation if one or other of the two code bases is eventually abandoned.

@holgerfriedrich
Copy link
Member Author

@andrewfg the last discussion about Jetty on the Karaf mailing list I have seen is there: https://lists.apache.org/thread/n30xb51gffwmfxs1ldw8zlvj21moszyh

I agree with @J-N-K, it is likely that a Jetty upgrade will not happen soon.
I cannot tell what our options are (besides waiting).

Karaf 4.4.x upgrades are still useful. IIRC with the last upgrades we got ready for Java 21 and closed a relevant CVE in Jetty. 4.4.7 will get bouncy castle to a recent version and upgrade a few more dependencies.

@J-N-K
Copy link
Member

J-N-K commented Oct 11, 2024

I wonder if Karaf is a dead-end. PaxWeb does not even support Jetty 12 (there is a version for Jetty 10 - PayWeb 9.x), but Karaf does not use that.

@andrewfg
Copy link
Contributor

I am still wondering of there is some way to build OH to import Jetty 12 directly into OH core, and exclude the older Jetty parts that would come indirectly via Karaf. That would give us much tighter control over versioning.

@openhab-bot
Copy link
Collaborator

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/unifi-binding-beta-3-2-0-3-6-0/131156/183

@J-N-K
Copy link
Member

J-N-K commented Oct 13, 2024

The difficult thing is that we also use both, Jetty server and Jetty client. Since they have common components it would be a lot easier if we use the same version for both. And that's where the problem is: For the server part (especially the servlets) we depend on PaxWeb and that currently doesn't work with the newer versions of Jetty (11/12).

@andrewfg
Copy link
Contributor

andrewfg commented Oct 14, 2024

Hmm.

It seems to me that we are in some jeopardy here..

  • OH depends on Karaf
  • OH depends (therefore) on Paxweb
  • OH depends (therefore) on Jetty
  • Current Paxweb (and therefore) Karaf use Jetty 9.x
  • Jetty 9.x is EOL and THREE major versions behind the current 12.x
  • Paxweb dev is resisting upgrading the Jetty dependency from 9.x to 12.x due to a) the amount of effort, and b) the risk of breaking changes
  • Karaf dev is (therefore) considering dropping Paxweb because (as mentioned above) it is not current
  • Karaf dev is (therefore) considering switching from Jetty to (say) Tomcat

It seems to me that OH architecture council needs to do some thinking. Do we accept our core to be so dependent on above mentioned weak points in dependency support? And/or do we have a vision of the future path?

Perhaps we need to contribute active PRs to Paxweb and (therefore) Karaf in order to bypass the bottleneck? And/or perhaps we need to make our own fork of Paxweb (and eventually Karaf)?

@kaikreuzer
Copy link
Member

Paxweb dev is resisting upgrading the Jetty dependency from 9.x to 12.x due to a) the amount of effort, and b) the risk of breaking changes
Karaf dev is (therefore) considering dropping Paxweb because (as mentioned above) it is not current
Karaf dev is (therefore) considering switching from Jetty to (say) Tomcat

Do you have links to these discussions in the respective communities? First step might be to involve ourselves there.
A decision by Paxweb not to upgrade essentially means that they have given up the project...

@andrewfg
Copy link
Contributor

Do you have links to these discussions in the respective communities?

@kaikreuzer see links below; note: you need to read it all, including opening the quoted text bubbles..

  • The Karaf discussion (already provided by @holgerfriedrich) is here
  • It references a prior discussion on Paxweb in Karaf here

@J-N-K
Copy link
Member

J-N-K commented Oct 14, 2024

I don't think that is what @grgrzybek said about future support. The last discussion regarding Jetty 11 (as Pax Web 10) that I remember is https://lists.apache.org/thread/4lwbt9g0w707zomnq6cfx179n8w90d3h: It's not ready (and as far as I can see there is still no release for Pax Web 10), but there is work on that. I can't find any comment about not supporting Jetty 12.

Changing to Jetty 11 (or 12) is also a big step for us because we need to move from javax to jakarta namespace. We should also upgrade JAXB to version 4 then.

@grgrzybek
Copy link

Yes - moving from javax to jakarta is a huge effort - that's why Pax Web 10 is still to be done...
And while WAR/WAB and even Whiteboard (which implies HttpService underneath) are actually ready, entire Pax Web 10 has other JavaEE deps to switch to Jakarta (like CDI or JSF) which is more problematic because CDI/JSF OSGi support is modest to say the least...

And because I feel that Pax Web 10 would be much less than Pax Web 8/9 (9 differs only by switching Jetty 9 to Jetty 10, which are still javax.servlet), I intentionally delay it until I have higher motivation (and more time).

And yes - Pax Web 10 is using Jetty 12 (but not in it's multi-servlet-api extent) - only jakarta.servlet package is supported.
Ideally Pax Web 10 should ship with Whiteboard tracker for javax.servlet based Whiteboard services that could be wrapped with jakarta.servlet API (servlets, filters, listeners, error pages, welcome files, JSPs, SCIs).

@andrewfg
Copy link
Contributor

I don't think that is what @grgrzybek said about future support.

Hmm. You may be right. He certainly has added support for Jetty 12 at least on this repo .. but I am not sure if that is the branch on which Karaf 4.5 would (eventually) be based ??

@J-N-K
Copy link
Member

J-N-K commented Oct 14, 2024

Pax Web 9 doesn't solve our issues with outdated Jetty (as Jetty 10 is also EoL for community support), though.

@grgrzybek
Copy link

My work on future Pax Web 10 based on Tomcat 10 (soon - 11), Undertow 2.3 and Jetty 12 is here: https://github.com/ops4j/org.ops4j.pax.web/commits/main-jakarta/

And I am aware that if Karaf 4.5 needs JakartaEE 10 support for web-related OSGi CMPN specifications (https://docs.osgi.org/specification/osgi.cmpn/8.1.0/) then either Felix HTTP is needed as default implementation or work on Pax Web 10 has to be hastened...

Mind that OSGi CMPN 8.1 is quite relentless for web-related specifications:

@shikousa
Copy link

@grgrzybek Thanks for the information. It seems that for the Unifi binding in Openhab we need partitioned cookies. The current Pax Web 8/Jetty 9 does not support the setPartitioned() feature. Is there some kind of workaround available or do we need to wait for >= Pax Web 9/Jetty 10 in Karaf?

https://community.openhab.org/t/unifi-binding-beta-3-2-0-3-6-0/131156/180

@grgrzybek
Copy link

Oh, I wasn't aware of partitioned cookies...
Jetty added an attribute in jetty/jetty.project@f82844e
I think not all browsers (Safari?) support it and definitely it's not part of Servlet API.
So please don't expect a unified (Jetty / Tomcat / Undertow) Pax Web support anytime soon...

@J-N-K
Copy link
Member

J-N-K commented Oct 15, 2024

Oh, I wasn't aware of partitioned cookies... Jetty added an attribute in jetty/jetty.project@f82844e I think not all browsers (Safari?) support it and definitely it's not part of Servlet API. So please don't expect a unified (Jetty / Tomcat / Undertow) Pax Web support anytime soon...

I believe we need that on client side, but I vaguely remember that using Jetty 9 server and Jetty 10 client in the same OSGi container is difficult. I might be wrong on that.

@shikousa
Copy link

@grgrzybek I think partitioned cookies are already supported in Jetty 10. So isn't it possible that Pax Web 9 also supports it?

@grgrzybek
Copy link

@grgrzybek I think partitioned cookies are already supported in Jetty 10. So isn't it possible that Pax Web 9 also supports it?

May I ask you please to create an issue here? https://github.com/ops4j/org.ops4j.pax.web/issues

I may look at it, as it sounds interesting and important ;) But I can't promise any release date with this implemented... Please provide some context and links there.

@openhab-bot
Copy link
Collaborator

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/unifi-and-2fa/107352/19

* Sync runtime dependencies with Karaf 4.4.7, most notably:
   * PaxWeb 8.0.29
   * Jetty 9.4.56.v20240826
   * BouncyCastle 1.78.1
   * CXF 3.6.4
   * Jackson 2.17.2
   * JNA 5.15.0
   * JAXB 2.3.9
   * commons-io 2.17.0
   * commons-lang3 3.17.0
   * XBean 4.26
   * ASM 9.7.1
 * Resolve itest runbundles

Signed-off-by: Holger Friedrich <[email protected]>
* Upgrade Xtext/Xtend from 2.36.0 to 2.37.0, see release notes:
  https://eclipse.dev/Xtext/releasenotes.html#/releasenotes/2024/11/19/version-2-37-0
  https://eclipse.dev/Xtext/xtend/releasenotes.html#/releasenotes/2024/11/19/version-2-37-0
* Upgrade dependencies
  * ecj from 3.36.0 to 3.37.0
  * gson from 2.10.1 to 2.11.0
  * classgraph to 4.8.176
  * jna from 5.14.0 to 5.15.0
  * guava from 3.33.0 to 3.33.1

Signed-off-by: Holger Friedrich <[email protected]>
@holgerfriedrich holgerfriedrich changed the title [WIP] Upgrade Karaf from 4.4.6 to 4.4.7 [WIP] Upgrade Karaf to 4.4.7, Xtext/Xtend 2.37.0 Nov 24, 2024
@holgerfriedrich
Copy link
Member Author

Short update:

Update to Xtext/Xtend 2.37.0 has been added. It relies on ASM 9.7.1, introduced with Karaf 4.4.7. Could not get the feature resolution to succeed on current master, this is why I added it here.

PRs for all repos are ready and up to date. Compilation and tests are running fine.

Once Karaf 4.4.7 is released, we are ready to go.

This is quite a risk as the planned winter release is soon and only one milestone build planned in between.
On the other hand, we should include the patch to Jetty 9.4.56 included in the release, if possible.

@holgerfriedrich
Copy link
Member Author

My test build starts up fine, discovers and installs add-ons. On startup, it shows the following warning in cmd;

WARNING: package org.apache.karaf.specs.locator not in java.base
Error starting karaf activator org.apache.karaf.specs.activator.Activator: org/apache/karaf/specs/locator/OsgiLocator

In the log there is the following error:

20:35:36.592 [ERROR] [Events.Framework                     ] - FrameworkEvent ERROR
java.lang.NoClassDefFoundError: org/apache/karaf/specs/locator/OsgiLocator
        at org.apache.karaf.specs.activator.Activator.register(Activator.java:125) ~[org.apache.karaf.specs.activator-4.4.7-SNAPSHOT.jar:4.4.7-SNAPSHOT]
        at org.apache.karaf.specs.activator.Activator.bundleChanged(Activator.java:97) ~[org.apache.karaf.specs.activator-4.4.7-SNAPSHOT.jar:4.4.7-SNAPSHOT]
        at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:949) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEventPrivileged(EquinoxEventPublisher.java:229) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:138) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:130) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor.publishModuleEvent(EquinoxContainerAdaptor.java:217) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.eclipse.osgi.container.ModuleContainer.applyDelta(ModuleContainer.java:857) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.eclipse.osgi.container.ModuleContainer.resolveAndApply(ModuleContainer.java:560) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.eclipse.osgi.container.ModuleContainer.resolve(ModuleContainer.java:503) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.eclipse.osgi.container.ModuleContainer.resolve(ModuleContainer.java:492) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.eclipse.osgi.container.ModuleContainer$ContainerWiring.resolveBundles(ModuleContainer.java:1451) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.apache.karaf.features.internal.service.BundleInstallSupportImpl.resolveBundles(BundleInstallSupportImpl.java:244) ~[?:?]
        at org.apache.karaf.features.internal.service.FeaturesServiceImpl.resolveBundles(FeaturesServiceImpl.java:1175) ~[?:?]
        at org.apache.karaf.features.internal.service.Deployer.deploy(Deployer.java:1027) ~[?:?]
        at org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvision(FeaturesServiceImpl.java:1069) ~[?:?]
        at org.apache.karaf.features.internal.service.FeaturesServiceImpl.lambda$doProvisionInThread$13(FeaturesServiceImpl.java:1004) ~[?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: java.lang.ClassNotFoundException: org.apache.karaf.specs.locator.OsgiLocator

To be followed up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awaiting external dependency Depends on a new version of an external dependency work in progress A PR that is not yet ready to be merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

SLF4J(W) console output when logging into Karaf console
7 participants