Speed up builds by working CMake cache system

[aborisovich]

Project's current state

SOF project incorrectly uses Cmake build system, and every time a change to the code is done a fresh build needs to be executed by deleting build-<platform_name> directory. Attempts to rebuild the project without directory deletion cause undefined behavior - sometimes it builds successfully, sometimes it fails at the linking state, and sometimes fails at compilation.
This is caused by changes to header files which are not tracked by the underlying build system (Ninja) so every change to a header file does not cause dependent compilation units to rebuild. Normally, header files should be included to CMake sources just like source files. This creates dependency by the underlying build system that recognizes which header files are included by compiled source files and if any header file is changed, the dependent source files is recompiled.
Moreover, CMake version 3.23 introduced target_sources(FILE_SETS <...>) command that can be used to organize header files better. If the inclusion of all header files looks like a big chunk of code, we may use regexes and include all files from the directory
with .h extension. This can be achieved by getting all files from the directory with file(GLOB <...>) and filtering using generator expression - regex on the list of files.

Motivation

The subject is worth investigating due to prolonged build times with Xtensa native toolchain.
Xtensa already announced deprecation of offline licensing leaving only floating license.
Moreover, new toolchain xt-clang which we are likely to introduce soon is about to prolong build times by ~4x.

Idea

Use Cmake cache properly and do not delete every time build directories when rebuilding the code.

1. Add all header files to the cmake target_sources command.
2. Convince Zephyr project to do the same.

Some proofs:
https://stackoverflow.com/questions/13703647/how-to-properly-add-include-directories-with-cmake

[marc-hb]

every time change to the code is done a fresh build needs to be executed by deleting build-<platform_name> directory.

I think that's a bit extreme / too conservative. For small, day to day edits incremental builds work fine in my experience.

Attempts to rebuild the project without directory deletion causes undefined behavior - sometimes it builds successfully, sometimes it fails at linking state, sometimes fails at compilation.

I have observed issues like this too but only when making "big" changes like when switching to a significantly different git branch.

This is caused by changes to header files which are not tracked by the underlying build system (Ninja) so every change to a header file does not cause dependent compilation units to rebuild.

I'm a bit surprised by this, I naively assumed CMake was able to automatically extract .h dependencies. Can you share a demo? If you can share a reproduction then by all means file a new issue upstream at https://github.com/zephyrproject-rtos/zephyr. It may or may not be fixed but for sure some experts will look at it and shed light on it.

If inclusion of all header files looks like the big chank of code, we may use regexes and include all files from directory with .h extension.

Mmm.... https://stackoverflow.com/questions/32411963/why-is-cmake-file-glob-evil

Moreover, CMake version 3.23 introduced target_sources(FILE_SETS <...>) command

I see target_sources() is already used in lot in Zephyr. So maybe we need different Github issues for different problems? At the very least one in Zephyr and one in SOF for SOF-specific problems.

The subject is worth investigating due to prolonged build times with Xtensa native toolchain.

Fixed! by commit 9a3a518 xtensa-build-zephyr.py: pass -j option to west -> ninja. I recommend -j 200.

Xtensa already announced deprecation of offline licensing leaving only floating license.
Moreover, new toolchain xt-clang which we are likely to introduce soon is about to prolong build times by ~4x.

Can you share more info about this?

Related:

• Improve header structure and usage zephyrproject-rtos/zephyr#41543

[aborisovich]

Fixed! by commit 9a3a518 xtensa-build-zephyr.py: pass -j option to west -> ninja. I recommend -j 200

This is amazing! Wonder if it works similarly with xt-clang.
I'll prepare some "hello world" presentation showing the issue.

[aborisovich]

Ok, performed some tests using cmake version 3.23.2 and it looks like I was wrong about it:

Attempts to rebuild the project without directory deletion cause undefined behavior - sometimes it builds successfully, sometimes it fails at the linking state, and sometimes fails at compilation.
This is caused by changes to header files which are not tracked by the underlying build system (Ninja) so every change to a header file does not cause dependent compilation units to rebuild. Normally, header files should be included to CMake sources just like source files. This creates dependency by the underlying build system that recognizes which header files are included by compiled source files and if any header file is changed, the dependent source files is recompiled.

Changing a function defined in a header that is not added by target_sources() does cause recompilation of the .c source file that includes it. This must had been fixed in some version of CMake...
Tested (on Windows OS)

• Generator: Ninja + GCC compiler (provided by MSYS2)
• Generator: MsBuild + MSVC compiler

Anyway, I am glad it works as intended now!
Here is dummy project I've tested if anybody wants to dig the subject further
hello_cmake_headers.zip