builds breaking because analyzers fail with Value cannot be null. (Parameter 'type')

[radical]

Failing rolling build hit for browser-wasm windows Release LibraryTests_AOT:

CSC : error CS8032: An instance of analyzer Microsoft.NetCore.CSharp.Analyzers.Resources.CSharpMarkAssembliesWithNeutralResourcesLanguageAnalyzer cannot be created from D:\a\_work\1\s\.packages\microsoft.codeanalysis.netanalyzers\8.0.0-preview1.23117.2\analyzers\dotnet\cs\Microsoft.CodeAnalysis.CSharp.NetAnalyzers.dll : Value cannot be null. (Parameter 'type'). [D:\a\_work\1\s\src\libraries\System.ComponentModel.Composition.Registration\src\System.ComponentModel.Composition.Registration.csproj::TargetFramework=net8.0]

Changes since the last passing rolling build - f6bed51...2819a94

cc @lewing @steveisok

Known Issue Error Message

Fill the error message using known issues guidance.

{
  "ErrorPattern": ".*An instance of analyzer Microsoft.NetCore.CSharp.Analyzers.Resources.CSharpMarkAssembliesWithNeutralResourcesLanguageAnalyzer cannot be created .*Value cannot be null.*",
  "BuildRetry": false
}

Report

Build	Definition	Step Name	Console log	Pull Request
180513	dotnet/runtime	Restore and Build Product	Log	#82476
180257	dotnet/runtime	Restore and Build Product	Log	#82350
179712	dotnet/runtime	Build product	Log

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	3	3

[lewing]

This isn't specific to wasm is it?

[lewing]

https://dev.azure.com/dnceng-public/public/_build/results?buildId=180566&view=logs&j=f3b7e53d-5a04-5317-3e8c-82ae7488ad7d&t=b32ae8c0-c4d6-5d0a-6327-bd5df8cb1d69&l=1376

not mono or wasm specific

[lewing]

cc @pavelsavara @RikkiGibson @jkoritzinsky

[stephentoub]

Seems pretty much every PR is hitting this on at least one leg. It's also not always the same analyzer failing.

Did we recently take updates to anything to do with Roslyn, compiler dependencies, etc?

[jkoritzinsky]

Yes, we recently updated Roslyn. We have a PR out to revert the update as it's also causing timeouts.

[stephentoub]

cc: @jaredpar

[lewing]

revert pr is #82466

[hoyosjs]

Merged to unblock

[jaredpar]

@jmarolf, @mavasani FYI, looks like an analyzer break here.

[steveisok]

Should we keep this open now that #82466 went in?

[lewing]

probably at least until we have a roslyn issue tracking the regression here.

we have #82583 tracking performance

and dotnet/roslyn#66900 tracking the parameter regression

but nothing tracking these seemingly random analyzer failures

[RikkiGibson]

@jaredpar have we ruled out the cause possibly being recent analyzer loader changes?

[jaredpar]

@RikkiGibson seems doubtful it's related. Generally analyzer load failures will result in missing method exception, assembly load, file not found, etc ... The failures would also be very deterministic. This is a random-ish failure on a null value. My suspicion would be race condition in the analyzer.

But @stephentoub mentioned

It's also not always the same analyzer failing.

So probably not in the specific analyzer. May need to grab a debug compiler and push it through the paces here. One where we can add some more diagnostics / trigger fail fasts so we can see what is going on.

Another potential reason is the underlying CLR bug causing #82583 is manifesting here as well.

[jaredpar]

@lewing my investigation into this is proving difficult until we get global.json to point to a runtime with the fix for #80822. Without that the compiler is crashing 300+ times a build. That's making it difficult to separate out the noise. Keeping an eye on #82599 which will move us to preview1 which has the fix.

[jaredpar]

@lewing

Have PR #82734 where I'm simply pushing the compiler forward now that runtime has moved to 8.0.100-preview.1.23115.2. The reason that is important is it has the fix for #80822 inside it. That deals with issues around collectable assemblies and the roslyn update in question moved our entire analyzer loading into collectable assemblies. It's not an explicit consequence of that bug but it seems like a possible one.

The PR #82734 is complicated because of the high number of test failures. Looking through the individual failures though I don't see the analyzer failure anymore. It's consistently popped up at least once in all my test runs against older CLR versions. It's undeniably a race condition of sorts though so even a clean run isn't good evidence of success.

I'm going to force another run here. If we don't see any more instances of this failure then I think the best path forward is to assume #80822 was the root of the failure and move forward with a new Roslyn drop. If that proves wrong we can back it out.

[jaredpar]

Had multiple successful builds with the patched runtime. At this point reached the conclusion that it was a side effect of #80822. If this problem returns after we move to a new Roslyn compiler in the future please let me know and I will dig into it further.