-
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prevent extensions from blocking parallel pre-compilation #55910
Conversation
fcdd78b
to
55b40ed
Compare
Backported PRs: - [x] #55849 <!-- Mmap: fix grow! for non file IOs --> - [x] #55863 <!-- Update TaskLocalRNG docstring according to #49110 --> - [x] #54433 <!-- Root globals in toplevel exprs --> - [x] #55828 <!-- Fix some corner cases of `isapprox` with unsigned integers --> - [x] #55890 <!-- Profile: fix order of fields in heapsnapshot & improve formatting --> - [x] #55884 <!-- inference: add missing `TypeVar` handling for `instanceof_tfunc` --> - [x] #55881 <!-- Install terminfo data under /usr/share/julia --> - [x] #55909 <!-- do not intentionally suppress errors in precompile script from being reported or failing the result --> - [x] #55355 <!-- relocation: account for trailing path separator in depot paths --> - [x] #55917 <!-- fix rawbigints OOB issues --> - [x] #55892 <!-- TOML: Avoid type-pirating `Base.TOML.Parser` --> - [x] #55798 <!-- Broadcast binary ops involving strided triangular --> - [x] #55919 <!-- Limit `@inbounds` to indexing in the dual-iterator branch in `copyto_unaliased!` --> Contains multiple commits, manual intervention needed: - [ ] #54009 <!-- allow extensions to trigger from packages in [deps] --> - [ ] #55509 <!-- Fix cong implementation to be properly random and not just cycling. --> - [ ] #55569 <!-- Add a docs section about loading/precomp/ttfx time tuning --> - [ ] #55824 <!-- Replace regex package module checks with actual code checks --> Non-merged PRs with backport label: - [ ] #55932 <!-- REPL: make UndefVarError aware of imported modules --> - [ ] #55910 <!-- Prevent extensions from blocking parallel pre-compilation --> - [ ] #55908 <!-- add logic to prefer loading modules that are already loaded --> - [ ] #55886 <!-- irrationals: restrict assume effects annotations to known types --> - [ ] #55871 <!-- lowering: don't reverse handler order in `(pop-handler-list ...)` --> - [ ] #55870 <!-- fix infinite recursion in `promote_type` for `Irrational` --> - [ ] #55867 <!-- update `hash` doc string: `widen` not required any more --> - [ ] #55851 <!-- [REPL] Fix #55850 by using `safe_realpath` instead of `abspath` in `projname` --> - [ ] #55813 <!-- Check for conflicting `@ccallable` name before JIT registration --> - [ ] #55743 <!-- doc: heap snapshot viewing --> - [ ] #55741 <!-- Change annotations to use a NamedTuple --> - [ ] #55534 <!-- Set stdlib sources as read-only during installation --> - [ ] #55499 <!-- propagate the terminal's `displaysize` to the `IOContext` used by the REPL --> - [ ] #55458 <!-- Allow for generically extracting unannotated string --> - [ ] #55457 <!-- Make AnnotateChar equality consider annotations --> - [ ] #55220 <!-- `isfile_casesensitive` fixes on Windows --> - [ ] #55169 <!-- `propertynames` for SVD respects private argument --> - [ ] #54457 <!-- Make `String(::Memory)` copy --> - [ ] #53957 <!-- tweak how filtering is done for what packages should be precompiled --> - [ ] #51479 <!-- prevent code loading from lookin in the versioned environment when building Julia --> - [ ] #50813 <!-- More doctests for Sockets and capitalization fix --> - [ ] #50157 <!-- improve docs for `@inbounds` and `Base.@propagate_inbounds` -->
Bump |
Bump? This is waiting for review |
Is that counting as a review? |
yes |
This change is needed for correct-ness, as it turns out. The existing behavior where we lift to just under the parent tightens the scheduling constraints on triggers too much, leading to false cycles. For example
but after injection of
which has put
Basically the way that |
Makes sense to me from the description. |
This change is a slimmed-down version of JuliaLang#55910, without any of the re-factoring. This improves the parallelism of the pre-compile job and fixes a correctness bug, where unconditionally injecting a dependency after its parent could be in conflict with dependencies in between triggers, causing false extension cycles. For example adding `AMDGPU.jl` and `Tracker.jl` in your project reports a false cycle (without this change).
This change is a slimmed-down version of JuliaLang#55910, without any of the re-factoring. This improves the parallelism of the pre-compile job and fixes a correctness bug, where unconditionally injecting a dependency after its parent could be in conflict with dependencies in between triggers, causing false extension cycles. For example adding `AMDGPU.jl` and `Tracker.jl` in your project reports a false cycle (without this change).
This change is a slimmed-down version of JuliaLang#55910, without any of the re-factoring. This improves the parallelism of the pre-compile job and fixes a correctness bug, where unconditionally injecting a dependency after its parent could be in conflict with dependencies in between triggers, causing false extension cycles. For example adding `AMDGPU.jl` and `Tracker.jl` in your project reports a false cycle (without this change).
Previously our precompilation code was causing any dependencies of a package A to wait on all of A's weakdeps to finish pre-compiling, even if it can't actually load those weakdeps (or the extension itself) This would lead to a pre-compile ordering like: ``` A B \ / \ Ext AB \ | / C / \ / D ``` Here, extension `C` cannot pre-compile in parallel with `Ext {A,B}` and `B`, because it has to wait for `Ext {A,B}` to finish pre-compiling. That happens even though `C` has no way to load either of these. This change updates the pre-compile ordering to be more parallel, reflecting the true place where `Ext {A,B}` can be loaded: ``` A B / \ / \ C Ext AB | \ | / \-- D --/ ``` which allows `C` to compile in parallel with `B` and `Ext{A,B}`
55b40ed
to
ccd969b
Compare
…ies (#56624) This is a slimmed-down version of #55910, without any of the re-factoring. This improves the parallelism of the pre-compile job and fixes a correctness bug, where unconditionally injecting a dependency after its parent could be in conflict with dependencies in between triggers, causing false extension cycles. Without this fix, adding `AMDGPU.jl` and `Tracker.jl` reports a false cycle on 1.11
These are the main correctness fix from JuliaLang#55910, so it's important that we have test coverage for it.
These are the main correctness fix from JuliaLang#55910, so it's important that we have test coverage for it.
…56666) It is possible for an extension `ExtAB` to be loadable by one of its triggers, e.g. if `A` loads `B`. However, this loading is not supposed to happen during pre-compilation of `A`. Getting this wrong means disagreeing with the scheduled pre-compile jobs (`A` is not scheduled to depend on or generate a cache file for `ExtAB` but accidentally attempts both) and leads to confusing errors about missing cache files. We used to cover up this bad behavior w/ an erroneous cycle warning (fixed by #55910), but now we need to be sure this works.
…56666) It is possible for an extension `ExtAB` to be loadable by one of its triggers, e.g. if `A` loads `B`. However, this loading is not supposed to happen during pre-compilation of `A`. Getting this wrong means disagreeing with the scheduled pre-compile jobs (`A` is not scheduled to depend on or generate a cache file for `ExtAB` but accidentally attempts both) and leads to confusing errors about missing cache files. We used to cover up this bad behavior w/ an erroneous cycle warning (fixed by #55910), but now we need to be sure this works.
…uliaLang#56666) It is possible for an extension `ExtAB` to be loadable by one of its triggers, e.g. if `A` loads `B`. However, this loading is not supposed to happen during pre-compilation of `A`. Getting this wrong means disagreeing with the scheduled pre-compile jobs (`A` is not scheduled to depend on or generate a cache file for `ExtAB` but accidentally attempts both) and leads to confusing errors about missing cache files. We used to cover up this bad behavior w/ an erroneous cycle warning (fixed by JuliaLang#55910), but now we need to be sure this works.
…n extensions (#56666) (#56676) It is possible for an extension `ExtAB` to be loadable by one of its triggers, e.g. if `A` loads `B`. However, this loading is not supposed to happen during pre-compilation of `A`. Getting this wrong means disagreeing with the scheduled pre-compile jobs (`A` is not scheduled to depend on or generate a cache file for `ExtAB` but accidentally attempts both) and leads to confusing errors about missing cache files. We used to cover up this bad behavior w/ an erroneous cycle warning (fixed by #55910), but now we need to be sure this works.
Previously our precompilation code was causing anything with package A as a dependency to wait on all of A's extensions and weakdeps to finish before starting to pre-compile, even if it can't actually load those weakdeps (or the extensions themselves)
This would lead to a pre-compile ordering like:
Here
C
cannot pre-compile in parallel withExt {A,B}
andB
, because it has to wait forExt {A,B}
to finish pre-compiling. That happens even thoughC
has no way to load either of these.This change updates the pre-compile ordering to be more parallel, reflecting the true place where
Ext {A,B}
can be loaded:which allows
C
to compile in parallel withB
andExt{A,B}
Please review 55b40ed (that's the only commit w/ important changes)