Use TestNG API

[sormuras]

The discovery and execution is implemented in very raw way.

• The TestNG API should be used to discover test classes and test methods to create TestDescriptors.
• The TestNG API should be used to execute test methods and report the result accordingly.

[juherr]

• The TestNG API should be used to discover test classes and test methods to create TestDescriptors.

TestNG doesn't provide an api for test discovery. Instead, we use the suite file and TestNG will find tests accordingly to the suite.
In that situation, is it possible to generate the right suite from the engine api?

• The TestNG API should be used to execute test methods and report the result accordingly.

This part should not be a problem.

[sormuras]

Instead, we use the suite file and TestNG will find tests accordingly to the suite.

Is there a way to inspect such suite files and list all tests that will be executed w/o executing them? How is a TestNG suite file passed to the TestNG runtime? Via an explicit option? Does TestNG scan the current directory for suite files?

In that situation, is it possible to generate the right suite from the engine api?

No, it is up to the engine implementation to tell the Platform which tests it is able to execute -- based on the selectors the user (of the Platform, read: build tool) passed. I implemented the handling of 3 selectors in DiscoveryHelper that scan class files for @Test-annotated methods in this spike:

testng-junit5/src/main/java/org/testng/junit5/DiscoveryHelper.java

Lines 27 to 53 in 78eb11d

	// class-path root
	engineDiscoveryRequest
	.getSelectorsByType(ClasspathRootSelector.class)
	.stream()
	.map(ClasspathRootSelector::getClasspathRoot)
	.map(uri -> findAllClassesInClasspathRoot(uri, classFilter))
	.flatMap(Collection::stream)
	.forEach(candidate -> handler.accept(engine, candidate));
	// package
	engineDiscoveryRequest
	.getSelectorsByType(PackageSelector.class)
	.stream()
	.map(PackageSelector::getPackageName)
	.map(packageName -> findAllClassesInPackage(packageName, classFilter))
	.flatMap(Collection::stream)
	.forEach(candidate -> handler.accept(engine, candidate));
	// class
	engineDiscoveryRequest
	.getSelectorsByType(ClassSelector.class)
	.stream()
	.map(ClassSelector::getJavaClass)
	.filter(classFilter)
	.forEach(candidate -> handler.accept(engine, candidate));
	// TODO Add missing selector types...

The interesting part is the // TODO Add missing selector types... line. There 10 concrete classes of DiscoverySelector from which the user may pass a set of:

Each TestEngine should handle as many selector types as possible.

[sormuras]

Open API. TestNG exposes its engine with a convenient set of listeners which allow you to plug-in your own reporting mechanism or for any other purpose (as a matter of fact, the JUnitReport code is such a listener). It is also trivial to invoke TestNG programmatically, thus skipping the XML phase and providing TestNG with direct Java objects.

The later part sounds promising. Copied from http://beust.com/weblog/2004/09/01/announcing-testng-1-0/

The Surefire TestNG Provider does something similar:

https://github.com/apache/maven-surefire/blob/99083b26bad8a6bf5305aaad08573850623b19d2/surefire-providers/surefire-testng/src/main/java/org/apache/maven/surefire/testng/TestNGProvider.java#L211-L231

[sormuras]

Current iteration already uses TestNG.run() with a listener that informs the Platform about which test is executed and what the result is:

testng-junit5/src/main/java/org/testng/junit5/TestNGine.java

Lines 73 to 85 in 1655fc0

	for (TestDescriptor classDescriptor : engine.getChildren()) {
	engineListener.executionStarted(classDescriptor);
	UniqueId classId = classDescriptor.getUniqueId();
	Class<?>[] testClasses = {((ClassDescriptor) classDescriptor).getTestClass()};
	TestNG testNG = new TestNG(false);
	testNG.addListener((ITestNGListener) new TestListener(engineListener, classId));
	testNG.setTestClasses(testClasses);
	testNG.run();
	engineListener.executionFinished(classDescriptor, TestExecutionResult.successful());
	}

[sormuras]

@juherr Anything we can improve here?

[juherr]

I'm really sorry, I didn't take the time to have a look on it.

But I don't forget it!

[juherr]

@sormuras Any advice to run tests in debug mode from IDE?

Running a simple test in IntelliJ

package org.testng.junit5;

import org.testng.Assert;
import org.testng.annotations.Test;

public class TestNGTest {

    @Test
    void test() {
        Assert.assertNull(null);
    }
}

produces

!!! JUnit version 3.8 or later expected:

java.lang.ClassNotFoundException: junit.framework.ComparisonFailure
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:315)
	at com.intellij.rt.execution.junit.JUnitStarter.junitVersionChecks(JUnitStarter.java:221)
	at com.intellij.rt.execution.junit.JUnitStarter.canWorkWithJUnitVersion(JUnitStarter.java:207)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:61)

And I don't find how to force JUnit5 runner in IntelliJ. Any idea?

[sormuras]

And I don't find how to force JUnit5 runner in IntelliJ. Any idea?

Afaik, an annotation has to be meta-annotated with @Testable to be routed to the JUnit Platform within IDEA. Correct, @akozlova?

Any advice to run tests in debug mode from IDE? Running a simple test in IntelliJ...

Mh, isn't TestNG supported by IDEA? Or do you need a plugin? Strange, that the JUnit 3/4 Starter kicks in...

[juherr]

Ok, I will try with @Testable. I saw some params in the plugin code but I didn't find how to pass them.

TestNG is supported by IDEA which runs the TestNG plugin. That's why I force a JUnit runner on the TestNG class which runs JUnit 4 by default.

[akozlova]

If you annotate your Test annotation with Testable, then JUnit 5 should be used. At the same time, the test could be treated as testng test as the qualified name of the annotation was not changed, so users would see the chooser which is not that good to my taste...

[sormuras]

I agree with Anna.

Why do you want to IDE execute your TestNG tests as JUnit 5 tests?

[juherr]

I'm just trying to find the easier way to use the debugger from the IDE.
There is no objective to have a selection later :)

[sormuras]

Ah, I see.

Mh... give me second. It should work best within this very project.

[sormuras]

Here we go on Travis CI:

https://travis-ci.com/testng-team/testng-junit5/jobs/172970155#L694

With a break point set in IDEA

...leads to a halt while debugging:

[sormuras]

fyi, I added public to the test() method. Otherwise the current TestNGine implementation doesn't find the test method.

This is due to using Class#getMethods() here:

https://github.com/testng-team/testng-junit5/blob/master/src/main/java/org/testng/junit5/TestNGine.java#L57

[sormuras]

@juherr Does it work on your box now too?

[juherr]

@sormuras Yes, it is working. Thank!

[sormuras]

You're welcome!