TestNG HTML and XML Reports Example
In this article, we will go through the TestNG HTML and XML reports.
We will be doing the following:
- Start with a TestNG project which we will later run to generate reports. We will be running a main suite containing a couple of child suites, this will create enough data for us to review the reports generated.
- Go through the TestNG report model.
- Examine TestNG provided default reports that are created by default in the directory
./test-output
. - In the end, implement a custom report using
Reporter
API.
Let’s start with the setup:
- I am using using Maven as the build tool and Eclipse as the IDE, version Luna 4.4.1.
- TestNG Maven Project Example will guide you on how to setup a Maven based project and run the TestNG tests.
Table Of Contents
1. TestNG Report Project
The main motivation behind the design of our example is to come with an optimum view of the report. It consists of the following:
- Suites: A main suite that contains a couple of child suites.
- Tests: Most of them pass and some fail.
- Data-driven testing: One of the test uses
@DataProvider.
- Groups: Some of the tests are grouped.
- Ignored Methods: One of the tests is disabled, so it will be ignored during the test run.
- Dynamic Test Name: We will use a factory instantiated test which implements
ITest
so that each test instance gets a dynamic name.
main-suite.xml
is our main suite. It contains child suites suite1.xml
and suite2.xml
.
suite1.xml
suite2.xml
main-suite.xml
has one test main-suite-test1
and a couple of test classes TestClass
and TestClass1
.
main-suite.xml:
<?xml version="1.0" encoding="UTF-8"?> <suite name="main-suite" parallel="false"> <suite-files> <suite-file path="./suite1.xml" /> <suite-file path="./suite2.xml" /> </suite-files> <test name="main-suite-test1"> <classes> <class name="com.javacodegeeks.testng.reports.TestClass" /> <class name="com.javacodegeeks.testng.reports.TestClass1" /> </classes> </test> </suite>
Test methods of TestClass
:
a1()
– will passa2()
– expects parameterparam
, will fail as we are not passing the valuea3()
– disabled so will show up in ignored methods
TestClass:
package com.javacodegeeks.testng.reports; import org.testng.annotations.Parameters; import org.testng.annotations.Test; public class TestClass { @Test public void a1() { } @Parameters("param") @Test public void a2(String param) { } @Test(enabled=false) public void a3() { } }
Test methods of TestClass1
:
t1()
andt2(),
both belong to groupmyGroup
and are expected to fail.t3()
belongs to groupmyGroup
, will pass.t4()
will pass.t5()
expects a parameter and depends on aDataProvider
calleddp()
for its parameter values.
TestClass1:
package com.javacodegeeks.testng.reports; import org.testng.Assert; import org.testng.annotations.DataProvider; import org.testng.annotations.Test; public class TestClass1 { @Test(groups="myGroup") public void t1() { Assert.assertTrue(false); } @Test(groups="myGroup") public void t2() { Assert.assertTrue(false); } @Test(groups="myGroup") public void t3() { } @Test public void t4() { } @Test(dataProvider="dp") public void t5(String param) { } @DataProvider private Object[][] dp() { return new Object[][]{{"one"}, {"two"}}; } }
suite1.xml
has a couple of tests:
suite1-test1
suite1-test2
Test suite1-test1
is made up of below classes:
TestClass1
TestClass2
And suite1-test2
is made up of:
TestClass3
FactoryInstantiatedTestClass
suite1.xml:
<?xml version="1.0" encoding="UTF-8"?> <suite name="suite1" parallel="false"> <test name="suite1-test1"> <classes> <class name="com.javacodegeeks.testng.reports.TestClass1" /> <class name="com.javacodegeeks.testng.reports.TestClass2" /> </classes> </test> <test name="suite1-test2"> <classes> <class name="com.javacodegeeks.testng.reports.TestClass3" /> <class name="com.javacodegeeks.testng.reports.FactoryInstantiatedTestClass"/> </classes> </test> </suite>
Now we will go through test methods of each class.
Test methods of TestClass2
.
- Methods
c1()
,c2()
andc3()
are expected to pass. c4()
is expected to fail
TestClass2:
package com.javacodegeeks.testng.reports; import org.testng.Assert; import org.testng.annotations.Test; public class TestClass2 { @Test public void c1() { } @Test public void c2() { } @Test public void c3() { } @Test public void c4() { Assert.assertTrue(false); } }
Test methods of TestClass3
.
d1()
andd2
are expected to pass.d1()
belongs to groupmyGroup
.
TestClass3:
package com.javacodegeeks.testng.reports; import org.testng.annotations.Test; public class TestClass3 { @Test(groups="myGroup") public void d1() { } @Test public void d2() { } }
Points to note about FactoryInstantiatedTestClass
:
FactoryInstantiatedTestClass
acts as a test class as well as a factory class that instantiates itself using static methodcreate()
.- If you notice it implements
ITest
so thatgetTestName()
can return a distinct name for each test instance. FactoryInstantiatedTestClass
creates two test instances. Test methodf()
will fail for one of them.
FactoryInstantiatedTestClass:
package com.javacodegeeks.testng.reports; import org.testng.Assert; import org.testng.ITest; import org.testng.annotations.Factory; import org.testng.annotations.Test; public class FactoryInstantiatedTestClass implements ITest { private String param; public FactoryInstantiatedTestClass(String param) { this.param = param; } public String getTestName() { return getClass().getSimpleName() + "-" + param; } @Factory public static Object[] create() { return new Object[]{new FactoryInstantiatedTestClass("TestNG"), new FactoryInstantiatedTestClass("Reports")}; } @Test public void f() { if (param.equals("Reports")) { Assert.assertTrue(false); } } }
Our final suite suite2.xml
contains just one test suite2-test1
which in turn contains just one test class TestClass4
.
suite2.xml:
<?xml version="1.0" encoding="UTF-8"?> <suite name="suite2" parallel="false"> <test name="suite2-test1"> <classes> <class name="com.javacodegeeks.testng.reports.TestClass4" /> </classes> </test> </suite>
Test methods of TestClass4
.
e1()
will passe2()
will also pass. Note that it callsReporter.log()
, this will print message passed-in to the HTML reports.
TestClass4:
package com.javacodegeeks.testng.reports; import org.testng.Reporter; import org.testng.annotations.Test; public class TestClass4 { @Test public void e1() { } @Test public void e2() { Reporter.log("Method name is e2"); } }
2. Report Model
ISuite defines a Test Suite. It provides access to the full information on the result of a suite. It contains methods that are applicable at suite level. Below are some of the methods that will help you in building custom reports.
getName()
returns name of the suite.getInvokedMethods()
returns a list of the methods that were invoked. Each item in the list is of typeIInvokedMethod.
getResults()
returns a map of the test results, key being the test name and value anISuiteResult
object.ISuiteResult
represents the result of a suite run.getTestContext()
returns the testing context objectITestContext.
ITestContext
contains all the information for a given test run, for example,getPassedTests()
returns result of passed tests in form ofIResultMap
object.IResultMap
contains result of the tests run. For example,getAllResults()
returns a set ofITestResult
object.ITestResult
describes the result of a test.getMethod()
returns anITestNGMethod
object which is the test method this result represents.ITestNGMethod
describes a TestNG annotated method.getMethodName()
returns name of the method.
Below is diagram that shows the POJO model of the TestNG report.
3. TestNG Default Reports
TestNG comes with certain predefined listeners and by default they are added to the test execution. Whenever TestNG is run, HTML and XML reports are generated by default in the directory ./test-output
.
For implementing a reporting class, the class has to implement an org.testng.IReporter
interface. TestNG has its own reporter objects and these objects are called when the whole suite run ends. The object containing the information of the whole test run is passed on to the report implementations. The default implementations are:
Main
– This is the main HTML reporter for suites. You can access the main report by openingindex.html
fromtest-output
directory.FailedReporter
– This reporter is responsible for creatingtestng-failed.xml
that contains just the test methods failed.XMLReporter
– TestNG offers an XML reporter capturing TestNG specific information that is not available in JUnit reports, createstestng-results.xml
.EmailableReporter2
– A reporter that generates one big HTML fileemailable-report.html
that’s easy to email to coworkers. If system propertynoEmailableReporter
is set to some non-null value, the emailable report will not be generated.JUnitReportReporter
– It creates JUnit compatible XML fileSuiteHTMLReporter
– This is the older version of HTML reporter for suites.
If we want only the custom reporters run then we make use of the flag –useDefaultListeners
.
For example:
java org.testng.TestNG –useDefaultListeners false testng.xml
4. Main Report Layout
test-output
directory contains an index.html
file that is the entry point to the TestNG HTML report.
The main layout is composed of top pane, left pane and the details pane.
- Top Pane – Summary of test results.
- Left Pane – Suite based information. All the suites will be listed here.
- Details Pane – Once we click on a piece of information in the left pane, the details of it will be shown here.
4.1. Test Results Summary
The top-level report gives us a list of all the suites that were just run, along with an individual and compound total for each passed, failed, and skipped test.
4. 2. Suite Pane
In our example, we have three suites. You can see all of the three suites listed here:
- main-suite
- suite2
- suite1
Clicking on ‘All Suites’ will expand the suites.
Each suite has two sections.
- Info – Suite information
- Result – Test results
The information section will have the following details:
testng.xml
– Contents oftestng.xml
file.- Tests run
- Groups Involved
- Test Timings
- Reporter Output – If the test logs a message by calling
Reporte.log()
then those messages can be seen here - Any ignored methods, for example, a disabled method
- Chronological view – List of test methods executed in chronological order
4. 3. Suite Details Pane
The detail panel will show the details of whatever is selected on the left panel. By default, if there are any failures, the detail panel will show those failures.
Click on ‘Test’ to see the summary of the tests that suite has run for.
Click on ‘Groups’ to see the groups involved.
Click on ‘Times’ to see how long each case took.
Click on ‘Reporter Output’ to see the report log.
Click on ‘Ignored Methods’ to see the ignored methods if any.
If a data provider is used to inject parameter values, you can see them in the test method.
5. Emailable Report
A reporter that generates one big HTML file that’s easy to email to other team members. Open test-output/emailable-report.html
to see the report.
The first section contains a summary of all the tests.
The next section contains details of each test run.
The last section contains details of each test method.
6. Old Suite Html Reporter
These are the old styled suite reports, you will find them in folder test-output/old
.
The top-level report gives us a list of all the suites that were just run, along with an individual and compound total for each passed, failed, and skipped test. Click on ‘link’ to take a look at the testng.xml
file used for each suite.
Clicking on the suite link brings up the main suite report.
The left pane shows a list of all the tests found in this suite. Clicking on any link in the left pane will show the details in the right pane.
7. TestNg.xml just for the failed Tests
testng-failed.xml
file contains a subset of testng.xml
to rerun just the failed tests.
testng-failed.xml:
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd"> <suite name="Failed suite [suite1]"> <listeners> <listener class-name="com.javacodegeeks.testng.reports.Reporter"/> </listeners> <test name="suite1-test1(failed)"> <classes> <class name="com.javacodegeeks.testng.reports.TestClass1"> <methods> <include name="t1"/> <include name="t2"/> </methods> </class> <class name="com.javacodegeeks.testng.reports.TestClass2"> <methods> <include name="c4"/> </methods> </class> </classes> </test> <test name="suite1-test2(failed)"> <classes> <class name="com.javacodegeeks.testng.reports.FactoryInstantiatedTestClass"> <methods> <include name="f"/> </methods> </class> </classes> </test> </suite>
8. TestNG Results in XML
Right click on TestClass2
and run TestNG. This will produce the XML results just for TestClass2
.
testng-results.xml
is the TestNG report of test results in XML.
testng-results.xml:
<?xml version="1.0" encoding="UTF-8"?> <testng-results skipped="0" failed="1" total="4" passed="3"> <reporter-output> </reporter-output> <suite name="Default suite" duration-ms="20" started-at="2015-03-10T06:11:58Z" finished-at="2015-03-10T06:11:58Z"> <groups> </groups> <test name="Default test" duration-ms="20" started-at="2015-03-10T06:11:58Z" finished-at="2015-03-10T06:11:58Z"> <class name="com.javacodegeeks.testng.reports.TestClass2"> <test-method status="PASS" signature="c2()[pri:0, instance:com.javacodegeeks.testng.reports.TestClass2@1c2c22f3]" name="c2" duration-ms="0" started-at="2015-03-10T11:41:58Z" finished-at="2015-03-10T11:41:58Z"> <reporter-output> </reporter-output> </test-method> <!-- c2 --> <test-method status="PASS" signature="c3()[pri:0, instance:com.javacodegeeks.testng.reports.TestClass2@1c2c22f3]" name="c3" duration-ms="10" started-at="2015-03-10T11:41:58Z" finished-at="2015-03-10T11:41:58Z"> <reporter-output> </reporter-output> </test-method> <!-- c3 --> <test-method status="PASS" signature="c1()[pri:0, instance:com.javacodegeeks.testng.reports.TestClass2@1c2c22f3]" name="c1" duration-ms="0" started-at="2015-03-10T11:41:58Z" finished-at="2015-03-10T11:41:58Z"> <reporter-output> </reporter-output> </test-method> <!-- c1 --> <test-method status="FAIL" signature="c4()[pri:0, instance:com.javacodegeeks.testng.reports.TestClass2@1c2c22f3]" name="c4" duration-ms="0" started-at="2015-03-10T11:41:58Z" finished-at="2015-03-10T11:41:58Z"> <exception class="java.lang.AssertionError"> <message> <![CDATA[expected [true] but found [false]]]> </message> <full-stacktrace> <![CDATA expected [true] but found [false] at org.testng.Assert.fail(Assert.java:94) at org.testng.Assert.failNotEquals(Assert.java:494) at org.testng.Assert.assertTrue(Assert.java:42) at org.testng.Assert.assertTrue(Assert.java:52) at com.javacodegeeks.testng.reports.TestClass2.c4(TestClass2.java:22) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84) at org.testng.internal.Invoker.invokeMethod(Invoker.java:714) at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901) at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231) at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111) at org.testng.TestRunner.privateRun(TestRunner.java:767) at org.testng.TestRunner.run(TestRunner.java:617) at org.testng.SuiteRunner.runTest(SuiteRunner.java:334) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291) at org.testng.SuiteRunner.run(SuiteRunner.java:240) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224) at org.testng.TestNG.runSuitesLocally(TestNG.java:1149) at org.testng.TestNG.run(TestNG.java:1057) at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:111) at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:204) at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:175) ]]> </full-stacktrace> </exception> <reporter-output> </reporter-output> </test-method> </class> </test> </suite> </testng-results>
9. JUnit XML Reports
TestNG by default generates the JUnit XML reports for any test execution.
TEST-com.javacodegeeks.testng.reports.TestClass4.xml:
<?xml version="1.0" encoding="UTF-8"?> <!-- Generated by org.testng.reporters.JUnitReportReporter --> <testsuite hostname="INMAA1-L1005" name="com.javacodegeeks.testng.reports.TestClass4" tests="2" failures="0" timestamp="8 Mar 2015 13:27:14 GMT" time="0.001" errors="0"> <testcase name="e1" time="0.001" classname="com.javacodegeeks.testng.reports.TestClass4"/> <testcase name="e2" time="0.000" classname="com.javacodegeeks.testng.reports.TestClass4"/> </testsuite> <!-- com.javacodegeeks.testng.reports.TestClass4 -->
We can use these XML report files as input for generation of a JUnit HTML report. Below is the ant build configuration XML file that uses Junit’s target junitreport
to generate an HTML report for the test execution.
Once executed a JUnit HTML report will be generated in the configured directory ./test-output/junit-html-report
.
build.xml:
<?xml version="1.0" encoding="UTF-8"?> <project name="JUnit Report in HTML" default="junit-htm-report" basedir="."> <property name="junit-xml-reports-dir" value="./test-output/junitreports" /> <property name="report-dir" value="./test-output/junit-html-report" /> <target name="junit-htm-report"> <!-- Delete and recreate the html report directories --> <delete dir="${report-dir}" failonerror="false" /> <mkdir dir="${report-dir}" /> <junitreport todir="${report-dir}"> <fileset dir="${junit-xml-reports-dir}"> <include name="**/*.xml" /> </fileset> <report format="noframes" todir="${report-dir}" /> </junitreport> </target> </project>
Open file named junit-noframes.html
to see the results.
10. Progressive Html Report for Individual Tests
TestHTMLReporter
is a ITestListener
reporter that generates HTML report for individual tests. There will be one folder for each suite with the same name as the suite’s name. For each test, there will be one HTML and XML report. Since it implements ITestListener
, one can see the report progressing as the tests progress in their run.
suite1
contains two test ssuite1-test1
and suite1-test2
so you can see one set of HTML and XML files for each test. There is also one testng-failed.xml
that contains the failed test methods.
Open suite1-test1.html
to see the report.
The first section shows the summary of the test results.
In the second section, you see the details of failed tests.
In the final section, you will see the details of passed tests.
11. Custom Report
In this example, we will see the method of writing your custom reporter and attaching it to TestNG.
To write a custom reporter class, our extension class should implement the IReporter
interface.
It implements the definition for the method generateReport
of the IReporter
interface. The method takes three arguments :
- The first is the
XmlSuite
object, which is the list suites mentioned in the testng XML being executed. - The second one being suites which contains the suite information after the test execution; this object contains all the information about the packages, classes, test methods, and their test execution results.
- The third being the outputDirectory, which contains the information of the output folder path where the reports will be generated.
The custom report writes to an HTML file as well as prints in the console. Once custom report is run, it will generate HTML file test-output/custom-report.html
.
Reporter:
package com.javacodegeeks.testng.reports; import java.io.BufferedWriter; import java.io.File; import java.io.FileWriter; import java.io.IOException; import java.io.PrintWriter; import java.util.List; import java.util.Map; import java.util.Set; import org.testng.IReporter; import org.testng.IResultMap; import org.testng.ISuite; import org.testng.ISuiteResult; import org.testng.ITestContext; import org.testng.ITestResult; import org.testng.xml.XmlSuite; public class CustomReporter implements IReporter { private PrintWriter mOut; public void generateReport(List xmlSuites, List suites, String outputDirectory) { new File(outputDirectory).mkdirs(); try { mOut = new PrintWriter(new BufferedWriter(new FileWriter(new File( outputDirectory, "custom-report.html")))); } catch (IOException e) { System.out.println("Error in creating writer: " + e); } startHtml(); print("Suites run: " + suites.size()); for (ISuite suite : suites) { print("Suite>" + suite.getName()); Map<String, ISuiteResult> suiteResults = suite.getResults(); for (String testName : suiteResults.keySet()) { print(" Test>" + testName); ISuiteResult suiteResult = suiteResults.get(testName); ITestContext testContext = suiteResult.getTestContext(); print(" Failed>" + testContext.getFailedTests().size()); IResultMap failedResult = testContext.getFailedTests(); Set testsFailed = failedResult.getAllResults(); for (ITestResult testResult : testsFailed) { print(" " + testResult.getName()); print(" " + testResult.getThrowable()); } IResultMap passResult = testContext.getPassedTests(); Set testsPassed = passResult.getAllResults(); print(" Passed>" + testsPassed.size()); for (ITestResult testResult : testsPassed) { print(" " + testResult.getName() + ">took " + (testResult.getEndMillis() - testResult .getStartMillis()) + "ms"); } IResultMap skippedResult = testContext.getSkippedTests(); Set testsSkipped = skippedResult.getAllResults(); print(" Skipped>" + testsSkipped.size()); for (ITestResult testResult : testsSkipped) { print(" " + testResult.getName()); } } } endHtml(); mOut.flush(); mOut.close(); } private void print(String text) { System.out.println(text); mOut.println(text + " "); } private void startHtml() { mOut.println(""); mOut.println(""); mOut.println("TestNG Html Report Example"); mOut.println(""); mOut.println(""); } private void endHtml() { mOut.println(""); } }
The Reporter
class is added as a listener to the test suite using the listeners
and listener
tag as deined in the testng.xml
file.
main-suite.xml:
<?xml version="1.0" encoding="UTF-8"?> <suite name="main-suite" parallel="false"> <listeners> <listener class-name="com.javacodegeeks.testng.reports.CustomReporter" /> </listeners> <suite-files> <suite-file path="./suite1.xml" /> <suite-file path="./suite2.xml" /> </suite-files> <test name="main-suite-test1"> <classes> <class name="com.javacodegeeks.testng.reports.TestClass" /> <class name="com.javacodegeeks.testng.reports.TestClass3" /> </classes> </test> </suite>
Output:
Suites run: 3 Suite>main-suite Test>main-suite-test1 Failed>3 a2 org.testng.TestNGException: Parameter 'param' is required by @Test on method a2 but has not been marked @Optional or defined in C:\javacodegeeks_ws\testngReports\src\test\resources\main-suite.xml t1 java.lang.AssertionError: expected [true] but found [false] t2 java.lang.AssertionError: expected [true] but found [false] Passed>5 a1>took 0ms t5>took 0ms t3>took 0ms t4>took 1ms t5>took 0ms Skipped>0 Suite>suite2 Test>suite2-test1 Failed>0 Passed>2 e2>took 0ms e1>took 0ms Skipped>0 Suite>suite1 Test>suite1-test1 Failed>3 c4 java.lang.AssertionError: expected [true] but found [false] t2 java.lang.AssertionError: expected [true] but found [false] t1 java.lang.AssertionError: expected [true] but found [false] Passed>7 c3>took 0ms t5>took 0ms t5>took 0ms c2>took 0ms t3>took 0ms t4>took 0ms c1>took 0ms Skipped>0 Test>suite1-test2 Failed>1 FactoryInstantiatedTestClass-Reports java.lang.AssertionError: expected [true] but found [false] Passed>3 FactoryInstantiatedTestClass-TestNG>took 0ms d1>took 0ms d2>took 0ms Skipped>0
Download the Eclipse Project
This was an article about TestNG HTML and XML reports.
You can download the full source code of this example here: testngReports.zip
Thanks a lot for the detailed explanation.
Great post. Even I created a custom html report “testng-metrics” generated using testng listener
Highlights:
1. No code changes required in your automation scripts
2. Export test results (csv,pdf,excel)
3. Sort/Search results
4. Visualize test results
5. Can be shared without any content loss
Sample report – https://testng-metrics.netlify.com
Repo – https://github.com/adiralashiva8/testng-metrics/