Behavior change in building bottles against bottles from previous macOS versions

[scpeters]

I maintain the osrf/simulation tap and have noticed a recent behavior change in test-bot. It used to be that an _or_later suffix could be appended to the macOS version describing the sha256 data in a bottle do block, which would allow a bottle to be used with newer versions of macOS. At some point, the _or_later suffix was removed as any bottle could be used with a newer version of macOS. Our jenkins instance at build.osrfoundation.org currently has Catalina and Big Sur nodes for building bottles. Until fairly recently, we were building bottles for both of these versions. Starting on Nov 3rd (first noticed in osrf/homebrew-simulation@fb7743c) any formula that didn't have all dependencies bottled for a given version started using --build-from-source instead of --build-bottle in test-bot. One of our formulae (ignition-cmake2) without a big_sur bottle does not have compiled code, just text files, so it is truly reusable on any macOS version. When _or_later suffixes were part of brew, I used that suffix with this package. Now, since it doesn't have a big_sur bottle, test-bot will not build big_sur bottles for any of its dependents.

I'm not asking for anything to be reverted; this has mainly spurred me to be more proactive about building big_sur bottles. I did want to mention it though because it feels like a behavior change to me, and I haven't noticed any explicit discussion about it.

As a side note, it would be really helpful to be able to generate a console message from the output of build_bottle? in lib/tests/formulae.rb to indicate which dep is preventing bottles from building built. Perhaps an API similar to unsatisfied_requirements_messages would work.

[MikeMcQuaid]

@scpeters Thanks for the clear write-up!
@carlocab Could you have a look at this and give some thoughts? Thanks!

[carlocab]

This behaviour grew out of a discussion from several places, but one place to look is here: #683

In Homebrew/core we don't want to bottle a formula using dependencies built on an earlier version of macOS to avoid subtle compatibility issues. We already required this informally when building bottles: Homebrew/homebrew-core#87708. However, it's too easy to accidentally dispatch a bottle job for a formula with an unbottled dependent, so we decided it was better to have test-bot enforce this requirement.

If ignition-cmake2 is just a bunch of text files, then you can build an :all bottle instead (you'll see a few hundred of these in Homebrew/core) to indicate that it is OS-independent, and the build_bottle? method will recognise that as a valid dependency to use for bottling.

I agree that it would be a good idea to generate a console message to indicate which dependency is preventing bottles from being built. You can, however, already see this for yourself by using brew unbottled.

[MikeMcQuaid]

@carlocab do you think this is logic we could or should either 1) limit to homebrew/core or 2) provide a flag to change behaviour?

[carlocab]

A flag would be more useful, I think. We could add it if external tap maintainers want it, but I'm not under the impression that they do (yet -- I can easily be persuaded otherwise).

Thoughts, @scpeters?

[scpeters]

If ignition-cmake2 is just a bunch of text files, then you can build an :all bottle instead (you'll see a few hundred of these in Homebrew/core) to indicate that it is OS-independent, and the build_bottle? method will recognise that as a valid dependency to use for bottling.

thanks for the tip about :all bottles! I don't see an option for specifying this, but I looked at dev-cmd/bottle.rb and see that it will automatically declare a bottle as :all if the bottles are identical. I just ran diffoscope and the only differences I see are in the brew metadata. I'm guessing that I would need --only-json-tab if order for the bottles to match. I'll look into this further.

I agree that it would be a good idea to generate a console message to indicate which dependency is preventing bottles from being built. You can, however, already see this for yourself by using brew unbottled.

thanks I'll experiment with brew unbottled

A flag would be more useful, I think. We could add it if external tap maintainers want it, but I'm not under the impression that they do (yet -- I can easily be persuaded otherwise).

Thoughts, @scpeters?

I agree that a flag would be more useful, but I don't need the flag. I'd rather just keep our bottles up-to-date, and if I can figure out the :all bottles, that will be helpful as well. The main thing that caught me out was the behavior change, but I don't mind it.

I'll close this, but people can speak up if they would prefer that we add the aforementioned flag.

Thanks @carlocab and @MikeMcQuaid!

[MikeMcQuaid]

thanks for the tip about :all bottles! I don't see an option for specifying this, but I looked at dev-cmd/bottle.rb and see that it will automatically declare a bottle as :all if the bottles are identical. I just ran diffoscope and the only differences I see are in the brew metadata. I'm guessing that I would need --only-json-tab if order for the bottles to match. I'll look into this further.

Yup 👍🏻

[ipatch]

not trying to hijack this issue/thread, but i noticed this same behavior in a different context. i've been working with the homebrew-freecad tap, and am finally getting around at attempting to bottle freecad itself. however when i attempted to build freecad yesterday evening. the CI/CD gha i've setup using gha i noticed the below durning the CI run.

https://github.com/FreeCAD/homebrew-freecad/runs/4234958897?check_suite_focus=true#step:13:22

so i came across this issue doing a search for has dependencies without a compatible bottle!

and then ran,

brew unbottled --dependents --verbose freecad

which gave me no useful output related to which dependency of freecad that doesn't have a bottle 🤷‍♂️

so at least could it possible for the homebrew test-bot to print a list of formulae that don't have bottle associated with it so the author / tap maintainer knows which formula requires bottling in order to bottle subsequent formula?

[Bo98]

In your case it's [email protected], which raises a valid issue that we should probably not be skipping formulae with deps that don't have bottle blocks at all, since there are taps which don't provide bottles but still use test-bot.

[carlocab]

Won't formulae with deps that don't have bottles at all will error out on a test-bot run in this step?

homebrew-test-bot/lib/tests/formulae.rb

Lines 320 to 322 in 8ed8d35

	# Don't care about e.g. bottle failures for dependencies.
	test "brew", "install", "--only-dependencies", *install_args, formula_name,
	env: { "HOMEBREW_DEVELOPER" => nil }

I vaguely remember seeing complaints about no bottle being available when I was building formulae in my tap.

In any case, I've opened #717 to display the unbottled dependencies when skipping.

[Bo98]

Won't formulae with deps that don't have bottles at all will error out on a test-bot run in this step?

No, it should not for non-core taps. Having no bottle block is a valid and common scenario for third-party taps.

[carlocab]

I've merged #717 which should display unbottled dependencies when we skip a formula from these. I think this behaviour is useful (I'd want it in my own tap, for instance), but I'm open to making this customisable with flags if there's demand for it, as mentioned above:

A flag would be more useful, I think. We could add it if external tap maintainers want it, but I'm not under the impression that they do (yet -- I can easily be persuaded otherwise).

[scpeters]

so i came across this issue doing a search for has dependencies without a compatible bottle!

and then ran,

brew unbottled --dependents --verbose freecad

which gave me no useful output related to which dependency of freecad that doesn't have a bottle 🤷‍♂️

there's a line in brew unbottled that skips non-core formulae:

https://github.com/Homebrew/brew/blob/3.3.4/Library/Homebrew/dev-cmd/unbottled.rb#L126

I'll open a pull request to remove it, and we can discuss further there

[scpeters]

there's a line in brew unbottled that skips non-core formulae:

https://github.com/Homebrew/brew/blob/3.3.4/Library/Homebrew/dev-cmd/unbottled.rb#L126

I'll open a pull request to remove it, and we can discuss further there

Homebrew/brew#12450