Huge performance degradation between 3.2 and 3.6.3

[RobSwDev]

Test execution time and memory usage both surge after upgrading MSTest.TestAdapter from 3.2 to 3.6.3

Steps To Reproduce

I can't share our code base, but I did do some poking around with windbg:

It looks like all the test class instances are held onto permanently.
Each of our test classes has member variables that hold several MB of data (This may be dubious practice, but it's how our tests been built)
If these instances are not garbage collected then memory keeps growing.

ExecutionContextService.InstancesExecutionContexts in Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices seems to be the culprit. For whatever reason, none of the test instances are being cleaned up - I still have 1000s of test instances rooted.

Expected behavior

Performance should not degrade

Actual behavior

A set of tests that previously took 5 minutes now takes 1 hour.

Additional context

Command used to run tests has a rather large test case filter, in case that is relevant

vstest.console test.dll 
/Testcasefilter:"(TestCategory!=A & TestCategory!=B & .. & TestCategory!=Z)" 
/Settings:.runsettings
/inIsolation 
/Logger:trx;LogFileName="Log.trx" 
/ResultsDirectory:"c:\temp"

[RobSwDev]

Our RunSettings file contains the following

  <RunConfiguration>
    <MaxCpuCount>1</MaxCpuCount>
  </RunConfiguration>

I imagine the reasons we had for setting this originally are lost in the mists of time.
In a sample to repro the issue, I've found this makes a huge difference.

And we're using Server GC