Add support for a test name format

[Evangelink]

Summary

Introduce a new CLI argument allowing to change the way tests are displayed on console.

Background and Motivation

As of today, when doing --list-tests or when using the --output detailed the tests are displayed using their display name which is matching behavior of VSTest.

There are many cases where this is not ideal and it would be great to be able to change that. For example, if you want to discover the tests in a given assembly or through a given filer, you will see only display names when it would be useful to see the fully qualified name. When having multiple tests with a same display name, having the fully qualified name would also be useful.

Related issues or discussions: #4111 #3221

Proposed Feature

Add a new CLI option named --test-format (I am open to different naming suggestions) allowing the users to specify a format in which they want the test to be displayed. This argument would be in an open format with some predefined variable replacements (similar to git log pretty format).

To get started, I think it would be enough to support display name, fully qualified name and test ID as they are common to most test frameworks. I would like to suggest using long form names instead of few char symbols (unlike git log pretty) to have something self explanatory.

We will have to define the good pattern for variable replacement, possibilities from existing tools are:

• $<variable-name> like unix
• %<variable-name> like git log
• %<variable-name>% like cmd
• $<variable-name>$ like some templating systems
• ${<variable-name>} like unix/PS (although that's not the only syntax)

Example:

For the example below, I'll use the git log like syntax

// default
--test-format %displayname

--test-format "test: %displayname (%fullyqualifiedname)"

Alternative Designs

We could remove the flexible part and only allow some enum based presets: DisplayName (default), FQN, ID. This is probably simpler but could remove a big chunk of flexibility to user (is this so bad?).

One note is that my ideal solution would allow me to have something like <display name> (<fqn>) but we could do some preset for that.

[nohwnd]

It would be nice to have more use cases for this.

To me it seems much easier and more flexible to change the default to <display name> (<fqn>). The new dotnet test experience does this, with counts per assembly+tfm, and then on the end shows the total count.

And for more flexibility add option to render the the info in JSON with all the info we have about the test, and let the consumer process it.

It also seems more future proof, as long as we get the basic json format correctly.

This could also then be similar for run, where acceptance tests often parse out console output, and break every time we change that.

(we took similar path in Pester where we return an object that describes the test run, for the reason that we have no idea how people will want to process the result)