[FEATURE] drop git submodules and switch to Zephyr's "west" multi-repo tool

[marc-hb]

Rationale, pros and cons: https://docs.google.com/document/d/1TuJ8Z4W9rr1vUuKQ0O3Vh5dPPnGw5dKizylaEPtiPcg

I suggest having questions and discussions about implementation details in this Google doc and higher level discussions here. Whatever works.

Related:

• Continuous Integration with multiple git repos? (using west or similar) zephyrproject-rtos/zephyr#34713
• Transfer rimage git history to subdirectory tools/rimage/ in the sof git repo #8297

[lgirdwood]

@marc-hb lets make sure this can work with Zephyr west and without. i.e. when we are using Zephyr or using xtos. Lets also mark this for v1.7

[marc-hb]

This should solve the problem of synchronizing multi-repo submissions like this kconfig one: thesofproject/sof-test#458

[marc-hb]

Let's make sure this can work with Zephyr west and without. i.e. when we are using Zephyr or using xtos.

This shouldn't be a problem.

I think the next step is to propose a hierarchy of (optional) manifests. In addition to the above requirement, we also don't want to force every single firmware developer to download a Linux kernel repo that is 50 times bigger than the firmware repo they need and that most people will never use. This would also slow down many CI checks; CI code frequently git clones from scratch because for small repos it doesn't matter.

Note shallow --depth support in west is... complicated: zephyrproject-rtos/west#319

[lgirdwood]

@andyross @nashif fyi

[lgirdwood]

Moving t v1.9 as this will occur over time as Zephyr is integrated.

[aborisovich]

Initial draft done in PR6005.

[marc-hb]

Thanks @aborisovich

[lgirdwood]

@marc-hb part 2 of this work will be deleting the submodules from the repo - how do you want to proceed here. Does Jenkins need any work ?
@wszypelt is Internal CI ready to use west instead of submodules ?

[aborisovich]

@lgirdwood most of the work to do is in the CMake scripts.

[marc-hb]

I don't think deleting submodules is worth the effort and the migration pain for XTOS users. It would also force them to use (a very limited subset of) west for not much value. "better the devil you know", especially for a legacy option.

[andyross]

To be fair, and the Zephyr docs don't make this clear, west is actually a much easier tool for the "multiple gits" problem than git submodule (or Android repo) ever was. It seems like a pretty minor inconvenience even for the most conservative of downstream users.

And as far as downstreams[1], I'm really hopeful we can get development work on all the existing xtos platforms moved over to Zephyr by the end of the year or a few months after. The submodules can always live on in maintenance branches as needed.

[1] While recognizing that I'm notoriously optimistic about development schedules, and that this represents my personal desire as much as it does my employer's!

[marc-hb]

To be fair, and the Zephyr docs don't make this clear, west is actually a much easier tool for the "multiple gits" problem than git submodule (or Android repo) ever was.

To clarify: I totally agree with this west vs submodules comparison (grepo is different and off-topic)

This is impossible to prove but I came to the conclusion that most "git submodule hate" is due to the "manifest" not being in plain text. This makes it easier for git: fewer indirections and moving parts getting out of sync with each other and fewer manifest-rev complications like #548. But the buried/hidden manifest makes things less flexible and much more cryptic for the end-user.

It seems like a pretty minor inconvenience even for the most conservative of downstream users.

It's a "minor inconvenience" but it is a NEW inconvenience for developers, continuous integrations and other workflows that have been using git submodules for years and years.

The submodules can always live on in maintenance branches as needed.

My point exactly.