Make world-age increments explicit #56509
Conversation
|
The SparseArrays test failure is known - |
|
@nanosoldier |
|
I’m not opposed to this PR, but what’s the specific reason for introducing
(plus an increment after a That said, I think introducing |
test/reflection.jl
Outdated
| @@ -777,6 +777,7 @@ faz1(x) = foo1(x) | |||
| @test faz1(1.0) == 1 | |||
| m = first(methods(bar1, Tuple{Int})) | |||
| Base.delete_method(m) | |||
There was a problem hiding this comment.
Does this mean that the result of Base.delete_method (and Base.delete_binding) won’t take effect until the user explicitly calls Core.@worldinc? That sounds like it would be a pretty breaking change. I agree with the idea of restricting the effect of :worldinc to the top level (for helping inference), but it seems like Base.delete_method might need some adjustments, such as calling @eval internally or something similar.
There was a problem hiding this comment.
Does this mean that the result of Base.delete_method (and Base.delete_binding) won’t take effect until the user explicitly calls Core.@WorldInc?
It takes effect immediately (in a new world age), but the effect is not visible in the current task until it raises it world age.
Calling @eval internally will not raise the world age, because it's not at top level. delete_method is mostly used internally by Revise which can take an appropriate adjustment. There could also be a delete_method macro that does the world age raising implicitly.
Two reasons:
|
|
@vtjnash suggests implicit world-age increments between the args of |
Forgot to mention that @vtjnash pointed out that I already made world age increment for macro purposes in #53515, so it would be inconsistent for it not to increment the world age for execution, which I found convincing. |
Keno
force-pushed
the
kf/explicitageinc
branch
from
d085375
to
4330bad
Compare
|
The package evaluation job you requested has completed - possible new issues were detected. |
|
@nanosoldier |
|
The package evaluation job you requested has completed - possible new issues were detected. |
|
Alright, a fair number of tests with dependencies on JuliaInterpreter or Revise, which need to be taught about
|
Split out from #56509 to facilitate adjusting downstream packages.
This was part of #56509, but is an independent bugfix. The basic issue is that these macro were using `do` block internally. This is undesirable for test macros, because we would like them not to affect the behavior of what they're testing. E.g. right now: ``` julia> using Test julia> const x = 1 1 julia> @test_nowarn const x = 1 ERROR: syntax: `global const` declaration not allowed inside function around /home/keno/julia/usr/share/julia/stdlib/v1.12/Test/src/Test.jl:927 Stacktrace: [1] top-level scope @ REPL[3]:1 ``` This PR just writes out the try/finally manually, so the above works fine after this PR.
This was part of #56509, but is an independent bugfix. The basic issue is that these macro were using `do` block internally. This is undesirable for test macros, because we would like them not to affect the behavior of what they're testing. E.g. right now: ``` julia> using Test julia> const x = 1 1 julia> @test_nowarn const x = 1 ERROR: syntax: `global const` declaration not allowed inside function around /home/keno/julia/usr/share/julia/stdlib/v1.12/Test/src/Test.jl:927 Stacktrace: [1] top-level scope @ REPL[3]:1 ``` This PR just writes out the try/finally manually, so the above works fine after this PR.
Split out from #56509 to facilitate adjusting downstream packages.
stdlib/REPL/src/REPL.jl
Outdated
| @@ -367,6 +367,7 @@ function eval_user_input(@nospecialize(ast), backend::REPLBackend, mod::Module) | |||
| for xf in backend.ast_transforms | |||
| ast = Base.invokelatest(xf, ast) | |||
| end | |||
| toplevel_eval_with_hooks(mod, Expr(:worldinc)) | |||
There was a problem hiding this comment.
This is a no-op (if the implementation is not broken)
| toplevel_eval_with_hooks(mod, Expr(:worldinc)) |
There was a problem hiding this comment.
Overall SGTM (though needs a rebase against the parts that are merged already).
Looks like a lot of the needed added Core.@worldinc also will be already solved by making each toplevel / module statement continue to imply an updated world (since they already imply an updated macro world, and the statements are supposed to run in the world after that macro expansion, which wouldn't be possible to express with these explicit markers anyways)
One other remaining todo item is adding this to the Compiler verifier code. I don't know if we run any debug tests on PRs, but I believe we run those on nightly, and it likely should error on the unexpected Expr head.
These were added in JuliaLang/julia#56523. This simply adds tracking for the world age to the interpreter, but does not use it for anything. JuliaInterpreter's current model of world age does not match Base anyway. We should fix that eventually, but it's probably easier to do that once JuliaLang/julia#56509 and binding partitions are merged.
Keno
force-pushed
the
kf/explicitageinc
branch
from
4330bad
to
3bbfc51
Compare
* Handle `:latestworld` expr These were added in JuliaLang/julia#56523. This simply adds tracking for the world age to the interpreter, but does not use it for anything. JuliaInterpreter's current model of world age does not match Base anyway. We should fix that eventually, but it's probably easier to do that once JuliaLang/julia#56509 and binding partitions are merged. * 1.6 compat
Upcoming base PR JuliaLang/julia#56509 makes precise when exactly world age increments automatically at top-level. Packages are mostly not supposed to notice, but of course Revise is a bit patholgical in that it does basically expect asynchronous changes to have top level effects. After that PR, that requires an explicit opt-in. This PR makes the appropriate adjustments to the tests.
Upcoming base PR JuliaLang/julia#56509 makes precise when exactly world age increments automatically at top-level. Packages are mostly not supposed to notice, but of course Revise is a bit patholgical in that it does basically expect asynchronous changes to have top level effects. After that PR, that requires an explicit opt-in. This PR makes the appropriate adjustments to the tests.
|
@nanosoldier |
|
(And just to complete example, on this PR, all of those example cases will return |
Keno
added
the
minor change
|
Looking good on CI. Planning to merge shortly. |
|
Can we give some time to let the ecosystem changes percolate through (so that we don't break PkgEval too much tonight), or is this in the way of followup work? |
|
I don't think it's extremely urgent or anything, but I also don't want things to get stale and I do want to get the next PR up. We already got TestExtras and Legolas which were the two packages that had downstream deps. I'll ask for tags on those. I think for the 10 other ones, they'll just need to come in as they come in, many of them are not super actively developed. |
Keno
force-pushed
the
kf/explicitageinc
branch
from
d47b8ec
to
c693de3
Compare
This test currently relies on implicit world age increments at top level. We're re-evaluating where these go because julia is currently inconsistent about it in the interpreter, compiler and inference. To make sure this test keeps working on 1.12, add an explicit world age increment. See JuliaLang/julia#56509.
This PR introduces a new, toplevel-only, syntax form
:worldincthat semantically represents the effect of raising the current task's world age to the latest world for the remainder of the current toplevel evaluation (that context being an entry toevalor a module expression). For detailed motivation on why this is desirable, see #55145, which I won't repeat here, but the gist is that we never really defined when world-age increments and worse are inconsistent about it. This is something we need to figure out now, because the bindings partition work will make world age even more observable via bindings.Having created a mechanism for world age increments, the big question is one of policy, i.e. when should these world age increments be inserted.
Several reasonable options exist:
callexpressionAs an example, case, consider
a == aat toplevel. Depending on the semantics that could either be the same as in local scope, or each of the four world age dependent lookups (three binding lookups, one method lookup) could (potentially) occur in a different world age.The general tradeoff here is between the risk of exposing the user to confusing world age errors and our ability to optimize top-level code (in general, any
:worldincstatement will require us to fully pessimize or recompile all following code).This PR basically implements option 2 with the following semantics:
:worldincexprs or after:moduleexprs.:worldincafter all struct definitions, method definitions,usingand `import.@evalmacro inserts a worldinc following the call toevalif at toplevelincludegains an implicitworldinc.Of these the fourth is probably the most questionable, but is necessary to make this non-breaking for most code patterns. Perhaps it would have been better to make
includea macro from the beginning (esp because it already has semantics that look a little like reaching into the calling module), but that ship has sailed.Unfortunately, I don't see any good intermediate options between this PR and option #3 above. I think option #3 is closest to what we have right now, but if we were to choose it and actually fix the soundness issues, I expect that we would be destroying all performance of global-scope code. For this reason, I would like to try to make the version in this PR work, even if the semantics are a little ugly.
The biggest pattern that this PR does not catch is:
We could apply the same
includespecial case to eval, but given the existence of@evalwhich allows addressing this at the macro level, I decided not to. We can decide which way we want to go on this based on what the package ecosystem looks like.