TestNG Listeners Example

This article aims to introduce you to TestNG listeners and show you an example for each of the listeners.

In TestNG, a listener is represented by the marker interface org.testng.ITestNGListener. TestNG provides you with many more interfaces that extend org.testng.ITestNGListener. Each interface defines one aspect of TestNG. In order to extend TestNG behavior one needs to implement the TestNG-provided listener interface and then integrate it with TestNG.

First, I will introduce you to each of these listeners and then we will look into the different ways of integrating the listeners. Now a bit about my set up:

• I am using Eclipse as the IDE, version Luna 4.4.1.
• I will be running the tests using eclipse TestNG plugin so you need to install the TestNG Eclipse Plugin.

1. Introduction to TestNG Listeners

A TestNG listener always extends the marker interface org.testng.ITestNGListener. Using listeners, one can extend TestNG in their dealings with notifications, reports and test behavior. Below are the listeners that TestNG provides:

• IExecutionListener
• IAnnotationTransformer
• ISuiteListener
• ITestListener
• IConfigurationListener
• IMethodInterceptor
• IInvokedMethodListener
• IHookable
• IReporter

1.1. Example of IExecutionListener

IExecutionListener is a listener that monitors the beginning and end of a TestNG run. It has two methods, onExecutionStart() and onExecutionFinish(). Method onExecutionStart() is called before the TestNG starts running the suites and onExecutionFinish() is called after TestNG is done running all the test suites.

In the below example, I have two IExecutionListener listeners, ExecutionListener1 and ExecutionListener2. In class ExecutionListener1, in method onExecutionStart(), I record the start time and in method onExecutionFinish(), I print the time TestNG takes to run all the suites.

ExecutionListener1:

package com.javacodegeeks.testng;

import org.testng.IExecutionListener;

public class ExecutionListener1 implements IExecutionListener {
	private long startTime;

	@Override
	public void onExecutionStart() {
		startTime = System.currentTimeMillis();
		System.out.println("TestNG is going to start");		
	}

	@Override
	public void onExecutionFinish() {
		System.out.println("TestNG has finished, took around " + (System.currentTimeMillis() - startTime) + "ms");
	}
}

In my second listener, ExecutionListener2, in onExecutionStart(), I notify the interested parties that the TestNG is going to start. Likewise, in onExecutionFinish(), I notify them that TestNG has finished running the suites. For simplicity’s sake, I haven’t used any mail related code and instead you will just see simple messages as the intention is only to show you the possibilities.

ExecutionListener2:

package com.javacodegeeks.testng;

import org.testng.IExecutionListener;

public class ExecutionListener2 implements IExecutionListener {

	@Override
	public void onExecutionStart() {
		System.out.println("Notify by mail that TestNG is going to start");		
	}

	@Override
	public void onExecutionFinish() {
		System.out.println("Notify by mail, TestNG is finished");
	}
}

I also have a test class TestClass, it has a @BeforeSuite, a test and an @AfterSuite method.

TestClass:

package com.javacodegeeks.testng;

import org.testng.annotations.AfterSuite;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.Test;

public class TestClass {
	@BeforeSuite
	public void beforeSuite() {
		System.out.println("beforeSuite");
	}
	
	@Test
	public void t() {
		System.out.println("test");
	}
	
	@AfterSuite
	public void afterSuite() {
		System.out.println("afterSuite");
	}
}

My test configuration has the <listeners> element where each <listener> represents one listener. You need to specify the listener implementation’s fully qualified name in class-name attribute.

executionListenerTestng.xml:

<?xml version="1.0" encoding="UTF-8"?>
<suite name="Suite" parallel="false">
	<listeners>
		<listener class-name="com.javacodegeeks.testng.ExecutionListener1" />
		<listener class-name="com.javacodegeeks.testng.ExecutionListener2" />
	</listeners>

	<test name="Test">
		<classes>
			<class name="com.javacodegeeks.testng.TestClass" />
		</classes>
	</test>
</suite>

In the output, you can see one set of messages are printed before TestNG starts running the suites and the other set of messages are printed once all the suites have been run.

Output:

TestNG is going to start
Notify by mail that TestNG is going to start
[TestNG] Running:
  C:\javacodegeeks_ws\testNgListeners\test\com\javacodegeeks\testng\executionListenerTestng.xml

beforeSuite
test
afterSuite

===============================================
Suite
Total tests run: 1, Failures: 0, Skips: 0
===============================================

TestNG has finished, took around 83ms
Notify by mail, TestNG is finished

1.2. Example of IAnnotationTransformer

Annotations are static in nature by design, so any change in the values require recompilation of source files. Since TestNG relies heavily on annotations, it would be nice if one can override its behavior at runtime. This is exactly what TestNG allows you to do using its annotation transformation framework.
IAnnotationTransformer is a TestNG listener which allows you to modify TestNG annotation and configure it further.

1.2.1. Example of @Test annotation transformer

In the below example, we configure the @Test annotation.
TestAnnotationTransformerExample is our test class. It contains test methods t1, t2 and t3. Methods t1 and t2 accept a string parameter but we haven’t provided any DataProvider. The DataProvider will be set on-the-fly in the annotation transformer, based on the method. We also would want to disable method t3.

TestAnnotationTransformerExample:

package com.javacodegeeks.testng;

import org.testng.annotations.Test;

public class TestAnnotationTransformerExample {
	
	@Test
	public void t1(String param) {
		System.out.println("Method is t1, parameter is " + param);
	}
	
	@Test
	public void t2(String param) {
		System.out.println("Method is t2, parameter is " + param);
	}
	
	@Test
	public void t3() {
		System.out.println("Method is t3");
	}		
}

TestAnnotationTransformerListener is our test annotation transformer listener. It implements IAnnotationTransformer. Method transform transforms the annotation.
It takes four parameters. First parameter is of type ITestAnnotation and it represents @Test annotation. Most common use of @Test annotation is at method level but it can also be placed at class or constructor level. The last three parameters tell us, on which Java element the annotation was found: a class, a constructor, or a method. Only one of them will be non-null.

You can change the annotation values by calling any of the setters on the ITestAnnotation interface. In the below example, we dynamically set the data provider for test method t1 and t2. We also disable the test method if it is t3.

TestAnnotationTransformerListener:

package com.javacodegeeks.testng;

import java.lang.reflect.Constructor;
import java.lang.reflect.Method;

import org.testng.IAnnotationTransformer2;
import org.testng.annotations.IConfigurationAnnotation;
import org.testng.annotations.IDataProviderAnnotation;
import org.testng.annotations.IFactoryAnnotation;
import org.testng.annotations.ITestAnnotation;

public class TestAnnotationTransformerListener implements IAnnotationTransformer {

	@Override
	public void transform(ITestAnnotation annotation, Class testClass,
			Constructor testConstructor, Method testMethod) {		
		if (testMethod.getName().equals("t1")) {
			System.out.println("set data provider for " + testMethod.getName()); 
			annotation.setDataProviderClass(DataProviderFactory.class);
			annotation.setDataProvider("getDp1");
		} else if (testMethod.getName().equals("t2")) {
			System.out.println("set data provider for " + testMethod.getName()); 
			annotation.setDataProviderClass(DataProviderFactory.class);
			annotation.setDataProvider("getDp2");
		} else if (testMethod.getName().equals("t3")) {
			System.out.println("Disable " + testMethod.getName()); 
			annotation.setEnabled(false);
		}
	}
}

testAnnotationTransformerTestng.xml:

<?xml version="1.0" encoding="UTF-8"?>
<suite name="Suite" parallel="false">
	<listeners>
		<listener class-name="com.javacodegeeks.testng.TestAnnotationTransformerListener" />
	</listeners>

	<test name="Test">
		<classes>
			<class name="com.javacodegeeks.testng.TestAnnotationTransformerExample" />
		</classes>
	</test>
</suite>

Output:

set data provider for t2
set data provider for t1
Disable t3
[TestNG] Running:
  C:\javacodegeeks_ws\testNgListeners\test\com\javacodegeeks\testng\testAnnotationTransformerTestng.xml

Method is t1, parameter is one
Method is t1, parameter is two
Method is t1, parameter is three
Method is t2, parameter is 1
Method is t2, parameter is 2
Method is t2, parameter is 3

===============================================
Suite
Total tests run: 6, Failures: 0, Skips: 0
===============================================

IAnnotationTransformer only lets you modify a @Test annotation. If you need to modify other TestNG annotations like a configuration annotation, @Factory or @DataProvider you may have to use the enhanced interface IAnnotationTransformer2. I will demonstrate this in my next examples that transform annotations other than @Test.

1.2.2. Example of @DataProvider annotation transformer

Use this interface instead of IAnnotationTransformer if you want to modify any TestNG annotation besides @Test. In this example, based on the dataProvider, we decide whether it should be used in parallel. If the dataProvider returns a large data set, we run it in parallel.

DataProviderAnnotationTransformerExample:

package com.javacodegeeks.testng;

import org.testng.annotations.Test;

public class DataProviderAnnotationTransformerExample {	
	
	@Test(dataProvider="largeDataSet", dataProviderClass=DataProviderFactory.class)
	public void largeDataTest(String param) {
		System.out.println("Method is t3, parameter is " + param + " threadId: "
				+ Thread.currentThread().getId());
	}		
}

If the annotation name is “largeDataSet”, the dataProvider annotation is modified to run on a parallel.

DataProviderAnnotationTransformerListener:

package com.javacodegeeks.testng;

import java.lang.reflect.Constructor;
import java.lang.reflect.Method;

import org.testng.IAnnotationTransformer2;
import org.testng.annotations.IConfigurationAnnotation;
import org.testng.annotations.IDataProviderAnnotation;
import org.testng.annotations.IFactoryAnnotation;
import org.testng.annotations.ITestAnnotation;

public class DataProviderAnnotationTransformerListener implements IAnnotationTransformer2 {
	
	@Override
	public void transform(IDataProviderAnnotation annotation, Method method) {
		if (annotation.getName().equals("largeDataSet")) {
			System.out.println("Large data set, run parallely");
			annotation.setParallel(true);
		}
	}	
	
	@Override
	public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) {		
	}

	@Override
	public void transform(IFactoryAnnotation annotation, Method method) {
	}
	
	@Override
	public void transform(IConfigurationAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) {
	}
}

dataAnnotationTransformerTestng.xml:

<?xml version="1.0" encoding="UTF-8"?>
<suite name="Suite" parallel="false">
	<listeners>
		<listener class-name="com.javacodegeeks.testng.DataProviderAnnotationTransformerListener" />
	</listeners>

	<test name="Test">
		<classes>
			<class name="com.javacodegeeks.testng.DataProviderAnnotationTransformerExample" />
		</classes>
	</test>
</suite>

You can see in the output, each invocation of t3 results in a different threadId, as it is configured to run parallely.

Output:

[TestNG] Running:
  C:\javacodegeeks_ws\testNgListeners\test\com\javacodegeeks\testng\dataAnnotationTransformerTestng.xml

Large data set, run parallely
Method is t3, parameter is Data threadId: 13
Method is t3, parameter is Set threadId: 14
Method is t3, parameter is Large threadId: 12

===============================================
Suite
Total tests run: 3, Failures: 0, Skips: 0
===============================================

1.2.3. Example of @Factory annotation transformer

In this example, we transform a factory annotation.

FactoryAnnotationTransformerExample is a test class which depends on @Factory annotation for its creation. We will modify the annotation dynamically to set its source to a DataProvider.

FactoryAnnotationTransformerExample:

package com.javacodegeeks.testng;

import org.testng.annotations.Factory;
import org.testng.annotations.Test;

public class FactoryAnnotationTransformerExample {
	private String name;

	@Factory
	public FactoryAnnotationTransformerExample(String name) {
		this.name = name;
		System.out.println("In constructor: " + name);
	}
	
	@Test
	public void t1() {
		System.out.println("Method is t1, name is " + name);
	}		
}

FactoryAnnotationTransformerListener is the factory annotation transformer. In the transform method, we set the DataProvider name and its class.

FactoryAnnotationTransformerListener:

package com.javacodegeeks.testng;

import java.lang.reflect.Constructor;
import java.lang.reflect.Method;

import org.testng.IAnnotationTransformer2;
import org.testng.annotations.IConfigurationAnnotation;
import org.testng.annotations.IDataProviderAnnotation;
import org.testng.annotations.IFactoryAnnotation;
import org.testng.annotations.ITestAnnotation;

public class FactoryAnnotationTransformerListener implements IAnnotationTransformer2 {

	@Override
	public void transform(IFactoryAnnotation annotation, Method method) {
		annotation.setDataProvider("constructorParams");
		annotation.setDataProviderClass(DataProviderFactory.class);
	}
	
	@Override
	public void transform(IConfigurationAnnotation annotation, Class testClass,	Constructor testConstructor, Method testMethod) {
	}

	@Override
	public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) {		
	}

	@Override
	public void transform(IDataProviderAnnotation annotation, Method method) {
	}
}

DataProviderFactory contains the static data providers.

DataProviderFactory:

package com.javacodegeeks.testng;

import org.testng.annotations.DataProvider;

public class DataProviderFactory {
	@DataProvider
	public static Object[][] getDp1() {
		return new Object[][]{{"one"}, {"two"}, {"three"}};
	}
	
	@DataProvider
	public static Object[][] getDp2() {
		return new Object[][]{{"1"}, {"2"}, {"3"}};
	}
	
	@DataProvider(name="largeDataSet")
	public static Object[][] getLargeDataSet() {
		return new Object[][]{{"Large"}, {"Data"}, {"Set"}};
	}
	
	@DataProvider(name="constructorParams")
	public static Object[][] getConstructorParams() {
		return new Object[][]{{"a"}, {"b"}, {"c"}};
	}
}

factoryAnnotationTransformerTestng.xml:

<?xml version="1.0" encoding="UTF-8"?>
<suite name="Suite" parallel="false">
	<listeners>
		<listener class-name="com.javacodegeeks.testng.FactoryAnnotationTransformerListener" />
	</listeners>

	<test name="Test">
		<classes>
			<class name="com.javacodegeeks.testng.FactoryAnnotationTransformerExample" />
		</classes>
	</test>
</suite>

Output:

In constructor: Default test name
In constructor: a
In constructor: b
In constructor: c
[TestNG] Running:
  C:\javacodegeeks_ws\testNgListeners\test\com\javacodegeeks\testng\factoryAnnotationTransformerTestng.xml

Method is t1, name is a
Method is t1, name is b
Method is t1, name is c

===============================================
Suite
Total tests run: 3, Failures: 0, Skips: 0
===============================================

1.2.4. Example of Configuration annotation transformer

In this example, we will alter the configuration based annotations like @BeforeSuite, @BeforeTest etc.

ConfigurationAnnotationTransformerExample is the test class. It contains some configuration methods with a description attribute.

ConfigurationAnnotationTransformerExample:

package com.javacodegeeks.testng;

import org.testng.annotations.AfterMethod;
import org.testng.annotations.AfterSuite;
import org.testng.annotations.AfterTest;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.BeforeTest;
import org.testng.annotations.Test;

public class ConfigurationAnnotationTransformerExample {
	
	@BeforeSuite(description="before suite annotation")
	public void beforeSuite() {
		System.out.println("in beforeSuite");
	}
	
	@BeforeTest(description="before test annotation")
	public void beforeTest() {
		System.out.println("in beforeTest");
	}
	
	@BeforeMethod(description="before method annotation")
	public void beforeMethod() {
		System.out.println("in beforeMethod");
	}
		
	@Test(description="test method annotation")
	public void t() {
		System.out.println("test method");
	}	
	
	@AfterMethod(description="after method annotation")
	public void afterMethod() {
		System.out.println("in afterMethod");
	}
	
	@AfterTest(description="after test annotation")
	public void afterTest() {
		System.out.println("in afterTest");
	}
	
	@AfterSuite(description="after suite annotation")
	public void afterSuite() {
		System.out.println("in after suite");
	}

}

The listener is very simple. It just prints the annotation description but one can also do some concrete configuration here like adding the method to a new group, or increasing the timeOut value, if one is already set and is not sufficient, or change the dependencies. One can even disable the method.

ConfigurationAnnotationTransformerListener:

package com.javacodegeeks.testng;

import java.lang.reflect.Constructor;
import java.lang.reflect.Method;

import org.testng.IAnnotationTransformer2;
import org.testng.annotations.IConfigurationAnnotation;
import org.testng.annotations.IDataProviderAnnotation;
import org.testng.annotations.IFactoryAnnotation;
import org.testng.annotations.ITestAnnotation;

public class ConfigurationAnnotationTransformerListener implements IAnnotationTransformer2 {

	@Override
	public void transform(IConfigurationAnnotation annotation, Class testClass,
			Constructor testConstructor, Method testMethod) {
		System.out.println("Configure annotation " + annotation.getDescription());
	}
	
	@Override
	public void transform(ITestAnnotation annotation, Class testClass,
			Constructor testConstructor, Method testMethod) {		
	}

	@Override
	public void transform(IDataProviderAnnotation annotation, Method method) {
	}	

	@Override
	public void transform(IFactoryAnnotation annotation, Method method) {
	}		
}

configurationAnnotationTransformerTestng.xml:

<?xml version="1.0" encoding="UTF-8"?>
<suite name="Suite" parallel="false">
	<listeners>
		<listener class-name="com.javacodegeeks.testng.ConfigurationAnnotationTransformerListener" />
	</listeners>

	<test name="Test">
		<classes>
			<class name="com.javacodegeeks.testng.ConfigurationAnnotationTransformerExample" />
		</classes>
	</test>
</suite>

You can see from the output that the annotations are transformed first and then the configuration methods are invoked.

Output:

Configure annotation before suite annotation
Configure annotation after test annotation
Configure annotation after suite annotation
Configure annotation before test annotation
Configure annotation before method annotation
Configure annotation after method annotation
[TestNG] Running:
  C:\javacodegeeks_ws\testNgListeners\test\com\javacodegeeks\testng\configurationAnnotationTransformerTestng.xml

in beforeSuite
in beforeTest
in beforeMethod
test method
in afterMethod
in afterTest
in after suite

===============================================
Suite
Total tests run: 1, Failures: 0, Skips: 0
===============================================

1.3. Example of ISuiteListener

We also have a listener for the suite called ISuiteListener. It has two methods, onStart and onFinish. Method onStart is invoked before TestNG starts running the suite and onFinish is invoked after TestNG has run the suite.

The listener is called for each suite, if the parent suite contains child suites then the child suites are first run before running the parent suite. This is done so that the results for parent suite can reflect the combined results of the child suites.

In the below test configuration, we have a parent suite containing child suites.

suiteListenerTestng.xml:

<?xml version="1.0" encoding="UTF-8"?>
<suite name="SuiteListenerExample">
	<listeners>
		<listener class-name="com.javacodegeeks.testng.SuiteListener" />
	</listeners>	
  <suite-files>
  	 <suite-file path="./childSuite.xml"/>
  </suite-files>
</suite>

SuiteListenerExample is the test class. Its @BeforeSuite method depends on parameter ui. Imagine the parameter containing values like JSF, web etc. This parameter value will be set before the suite is started.

SuiteListenerExample:

package com.javacodegeeks.testng;

import org.testng.annotations.AfterSuite;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.Parameters;
import org.testng.annotations.Test;

public class SuiteListenerExample {
	
	@Parameters("ui")
	@BeforeSuite
	public void beforeSuite(String parm) {
		System.out.println("before suite, ui value: " + parm);
	}
	
	@Test
	public void t() {
		System.out.println("test method");
	}
	
	@AfterSuite
	public void afterSuite() {
		System.out.println("after suite");
	}
}

In SuiteListener.onStart, we set the parameter ui to value web.

SuiteListener:

package com.javacodegeeks.testng;

import java.util.HashMap;
import java.util.Map;

import org.testng.ISuite;
import org.testng.ISuiteListener;
import org.testng.xml.XmlSuite;

public class SuiteListener implements ISuiteListener {

	@Override
	public void onStart(ISuite suite) {
		System.out.println("Start suite " + suite.getName());
		XmlSuite xmlSuite = suite.getXmlSuite();
		if (!xmlSuite.getTests().isEmpty()) {
			Map parms = new HashMap();
			parms.put("ui", "web");
			System.out.println("Set ui param value");
			xmlSuite.setParameters(parms);
		}		
	}

	@Override
	public void onFinish(ISuite suite) {
		System.out.println("Finish suite " + suite.getName());
	}
}

The SuiteListener fires once for the child suite and then the parent suite.

Output:

[TestNG] Running:
  C:\javacodegeeks_ws\testNgListeners\test\com\javacodegeeks\testng\childSuite.xml

Start suite Child Suite
Set ui param value
before suite, ui value: web
test method
after suite
Finish suite Child Suite

===============================================
Child Suite
Total tests run: 1, Failures: 0, Skips: 0
===============================================

[TestNG] Running:
  C:\javacodegeeks_ws\testNgListeners\test\com\javacodegeeks\testng\suiteListenerTestng.xml

Start suite SuiteListenerExample
Finsh suite SuiteListenerExample

===============================================
SuiteListenerExample
Total tests run: 1, Failures: 0, Skips: 0
===============================================

1.4. Example of ITestListener

ITestListener is the listener for test running. You can either implement ITestListener or extend the TestNG provided implementation TestListenerAdapter as it has many convenient methods and we don’ have to re-invent the wheel.

ITestListener has methods on following events:

• onStart is invoked after the test class is instantiated and before any configuration method is called
• onTestSuccess is invoked on success of a test
• onTestFailure is invoked on failure of a test
• onTestSkipped is invoked whenever a test is skipped
• onTestFailedButWithinSuccessPercentage is invoked each time a method fails but is within the success percentage requested.
• onFinish is invoked after all the tests have run and all their Configuration methods have been called.

TestListenerExample is our test class. It has a @BeforeTest and an @AfterTest method. It has four test methods:

1. t1() is expected to run fine
2. t2() is expected to fail as it doesn’t throw the expected exception
3. t3() receives a parameter but since we haven’t set a DataProvider, it is skipped
4. t4() is invoked five times, of which, twice its going to fail. We have set the expected successPercentage to 80.

TestListenerExample:

package com.javacodegeeks.testng;

import org.testng.Assert;
import org.testng.annotations.AfterSuite;
import org.testng.annotations.BeforeTest;
import org.testng.annotations.Test;

public class TestListenerExample {
	
	@BeforeTest
	public void beforeTest() {
		System.out.println("before test");
	}
	
	@Test
	public void t1() {
		System.out.println("t1 test method");
	}
	
	@Test(expectedExceptions=RuntimeException.class)
	public void t2() {
		System.out.println("t2 test method will fail");
	}
	
	@Test
	public void t3(String p) {
		System.out.println("t3 test method will skip as parameter p is not set");
	}
	
	@Test(successPercentage=80, invocationCount=5)
	public void t4() {
		i++;		
		System.out.println("t4 test method, invocation count: " + i);
		if (i == 1 || i == 2) {
			System.out.println("fail t4");
			Assert.assertEquals(i, 10);
		}
	}
	
	@AfterSuite
	public void afterTest() {
		System.out.println("after test");
	}
	
	private int i;
}

TestListener is our implementation class for ITestListener. Each callback method prints a message so that we know whether the method is called.

TestListener:

package com.javacodegeeks.testng;

import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;

public class TestListener implements ITestListener {

	@Override
	public void onTestStart(ITestResult result) {
		System.out.println("on test method " +  getTestMethodName(result) + " start");
	}

	@Override
	public void onTestSuccess(ITestResult result) {
		System.out.println("on test method " + getTestMethodName(result) + " success");
	}

	@Override
	public void onTestFailure(ITestResult result) {
		System.out.println("on test method " + getTestMethodName(result) + " failure");
	}

	@Override
	public void onTestSkipped(ITestResult result) {
		System.out.println("test method " + getTestMethodName(result) + " skipped");
	}

	@Override
	public void onTestFailedButWithinSuccessPercentage(ITestResult result) {
		System.out.println("test failed but within success % " + getTestMethodName(result));
	}

	@Override
	public void onStart(ITestContext context) {
		System.out.println("on start of test " + context.getName());
	}

	@Override
	public void onFinish(ITestContext context) {
		System.out.println("on finish of test " + context.getName());
	}
	
	private static String getTestMethodName(ITestResult result) {
		return result.getMethod().getConstructorOrMethod().getName();
	}
}

testListenerTestng.xml:

<?xml version="1.0" encoding="UTF-8"?>
<suite name="TestListenerExample Suite">
    <listeners>
		<listener class-name="com.javacodegeeks.testng.TestListener" />
	</listeners>	
  <test name="TestListenerExample">  	
    <classes>
      <class name="com.javacodegeeks.testng.TestListenerExample"/>
    </classes>
  </test>
</suite>

From the output, we can observe the below:

• onStart is invoked first.
• onTestStart is called once for each test before it is invoked.
• onTestSuccess is invoked whenever a test passes. In our example, t1 always passes whereas, t4 passes three times.
• onTestFailure is called for t2 as t2 will always fail. It is also called for t4 as it fails twice out of five times that it is invoked.
• onTestSkipped is called once for t3 as it is bound to skip.
• onTestFailedButWithinSuccessPercentage is called once for t4, the first time it fails. It is not called again as it doesn’t match the requested successPercentage of 80
• Finally onFinish is called once when the tests are all run.

Output:

[TestNG] Running:
  C:\javacodegeeks_ws\testNgListeners\test\com\javacodegeeks\testng\testListenerTestng.xml

on start of test TestListenerExample
before test
on test method t1 start
t1 test method
on test method t1 success
on test method t2 start
t2 test method will fail
on test method t2 failure
test method t3 skipped
on test method t4 start
t4 test method, invocation count: 1
fail t4
test t4 failed but within success
on test method t4 start
t4 test method, invocation count: 2
fail t4
on test method t4 failure
on test method t4 start
t4 test method, invocation count: 3
on test method t4 success
on test method t4 start
t4 test method, invocation count: 4
on test method t4 success
on test method t4 start
t4 test method, invocation count: 5
on test method t4 success
on finish of test TestListenerExample
after test

===============================================
TestListenerExample Suite
Total tests run: 8, Failures: 3, Skips: 1
===============================================

Process finished with exit code 0

1.4. Example of IConfigurationListener

IIConfigurationListener is the listener interface for events related to configuration methods.

In the below test class MyConfigListenerExample , we have a @BeforeSuite, @AfterSuite and a @Test method.
We can use @Listeners annotation to specify the listener class. Note that this is another way of providing listeners to TestNG other than the testng.xml way.

MyConfigListenerExample:

package com.javacodegeeks.testng;

import java.util.Arrays;

import org.testng.TestNG;
import org.testng.annotations.AfterSuite;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(value=MyConfigListener.class)
public class MyConfigListenerExample {
	@BeforeSuite
	public void beforeSuite() {
		System.out.println("before suite");
	}
	
	@Test
	public void t() {
		System.out.println("test method t");
	}
	
	@AfterSuite
	public void afterSuite() {
		System.out.println("after suite");
	}
	
	public static void main(String[] args) {
		TestNG testNG = new TestNG();
		testNG.setTestSuites(Arrays.asList("test/com/javacodegeeks/testng/configurationListenerTestng.xml"));
		testNG.run();
	}
}

We have kept the listener class simple, just printing messages, so we know when a callback method gets called.

MyConfigListener:

package com.javacodegeeks.testng;

import org.testng.IConfigurationListener2;
import org.testng.ITestResult;

public class MyConfigListener implements IConfigurationListener2 {

	@Override
	public void onConfigurationSuccess(ITestResult tr) {
		System.out.println("on configuration success");		
	}

	@Override
	public void onConfigurationFailure(ITestResult tr) {
		System.out.println("on configuration failure");	
	}

	@Override
	public void onConfigurationSkip(ITestResult tr) {
		System.out.println("on configuration skip");			
	}

	@Override
	public void beforeConfiguration(ITestResult tr) {
		System.out.println("called before the configuration method is invoked");
	}
}

configurationListenerTestng.xml:

<?xml version="1.0" encoding="UTF-8"?>
<suite name="ConfigurationListenerExample Suite">
	<listeners>
		<listener class-name="com.javacodegeeks.testng.MyConfigListener" />
	</listeners>
	<test name="ConfigurationListenerExample">
		<classes>
			<class name="com.javacodegeeks.testng.MyConfigListenerExample" />
		</classes>
	</test>
</suite>

From the output, we can see that beforeConfiguration is called before the invocation of the configuration method. onConfigurationSuccess gets called on the success of a configuration method.
Output:

[TestNG] Running:
  C:\javacodegeeks_ws\testNgListeners\test\com\javacodegeeks\testng\configurationListenerTestng.xml

called before the configuration method is invoked
before suite
on configuration success
test method t
called before the configuration method is invoked
after suite
on configuration success

===============================================
ConfigurationListenerExample Suite
Total tests run: 1, Failures: 0, Skips: 0
===============================================

1.6. Example of IMethodInterceptor

IMethodInterceptor interface is used to modify the list of test methods that we want TestNG to run. It will be invoked right before TestNG starts invoking test methods.
It has just one method to implement intercept which returns the altered list of methods.

Lets being with our test class. MethodInterceptorListenerExample has two test methods. One of the test methods t1 is to test performance so we grouped it in “perf”.
Suppose we want to only run the performance based tests and not the other tests, we will have to provide a IMethodInterceptor listener that can filter out the other tests and return only performance based tests.

MethodInterceptorListenerExample:

package com.javacodegeeks.testng;

import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners({com.javacodegeeks.testng.MethodInterceptorListener.class})
public class MethodInterceptorListenerExample {
	@Test(groups="perf")
	public void t1() {
		System.out.println("test method: t1");
	}
	
	@Test
	public void t2() {
		System.out.println("test method: t2");
	}
}

MethodInterceptorListener is our listener class. You can see we are returning an altered method list, filtering methods other than methods belonging to “perf” group.

MethodInterceptorListener:

package com.javacodegeeks.testng;

import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;

import org.testng.IMethodInstance;
import org.testng.IMethodInterceptor;
import org.testng.ITestContext;
import org.testng.annotations.Test;

public class MethodInterceptorListener implements IMethodInterceptor {

	@Override
	public List intercept(List methods,
			ITestContext context) {
		List result = new ArrayList();
		for (IMethodInstance m : methods) {
			Test test = m.getMethod().getMethod().getAnnotation(Test.class);
			Set groups = new HashSet();
			for (String group : test.groups()) {
				groups.add(group);
			}
			if (groups.contains("perf")) {
				result.add(m);
			} else {
				String testMethod = m.getMethod().getMethod().getName();
				System.out.println(testMethod
						+ " not a performance test so remove it");
			}
		}
		return result;
	}
}

methodInterceptorListenerTestng.xml:

<?xml version="1.0" encoding="UTF-8"?>
<suite name="Suite" parallel="false">
	<listeners>
		<listener class-name="com.javacodegeeks.testng.MethodInterceptorListener" />
	</listeners>
	<test name="Test">
		<classes>
			<class name="com.javacodegeeks.testng.MethodInterceptorListenerExample" />
		</classes>
	</test>
</suite>

From the output, we see only t1 has run.

Output:

[TestNG] Running:
  C:\javacodegeeks_ws\testNgListeners\test\com\javacodegeeks\testng\methodInterceptorListenerTestng.xml

t2 not a performance test so remove it
test method: t1

===============================================
Suite
Total tests run: 1, Failures: 0, Skips: 0
===============================================

1.7. Example of IInvokedMethodListener

IInvokedMethodListener is listener that gets invoked before and after a method is invoked by TestNG. It will be invoked for all methods, both test and the configuration methods.

InvokedMethodListenerExample:

package com.javacodegeeks.testng;

import org.testng.annotations.AfterSuite;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.Test;

public class InvokedMethodListenerExample {
	@BeforeSuite
	public void beforeSuite() {
		System.out.println("before suite");
	}
	
	@Test
	public void t1() {
		System.out.println("t1 test method");
	}
	
	@AfterSuite
	public void afterSuite() {
		System.out.println("after suite");
	}	
}

InvokedMethodListener:

package com.javacodegeeks.testng;

import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestResult;

public class InvokedMethodListener  implements IInvokedMethodListener {
    @Override
    public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
        System.out.println("before invocation of " + method.getTestMethod().getMethodName());
    }

    @Override
    public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
        System.out.println("after invocation " + method.getTestMethod().getMethodName());
    }
}

invokedMethodListenerTestng.xml:

<?xml version="1.0" encoding="UTF-8"?>
<suite name="Suite" parallel="false">
	<listeners>
		<listener class-name="com.javacodegeeks.testng.InvokedMethodListener" />
	</listeners>
	<test name="Test">
		<classes>
			<class name="com.javacodegeeks.testng.InvokedMethodListenerExample" />
		</classes>
	</test>
</suite>

Output:

[TestNG] Running:
  C:\javacodegeeks_ws\testNgListeners\test\com\javacodegeeks\testng\invokedMethodListenerTestng.xml

before invocation of beforeSuite
before suite
after invocation beforeSuite
before invocation of t1
t1 test method
after invocation t1
before invocation of afterSuite
after suite
after invocation afterSuite

===============================================
Suite
Total tests run: 1, Failures: 0, Skips: 0
===============================================

1.8. Example of IHookable

If a test class wants to do something more, like a JAAS authentication, before invoking the test method, it needs to implement IHookable. If a test class implements this interface, its run() method will be invoked instead of each @Test method found.
The test method being invoked is passed in, encapsulated in a IHookCallBack object so one can run it by invoking IHookCallBack.runTestMethod().

In the below example, I skip running the test, based on the test method’s parameter value. If the parameter value is “dummy” client, the test is skipped but run for other valid clients.

HookableExample:

package com.javacodegeeks.testng;

import org.testng.IHookCallBack;
import org.testng.IHookable;
import org.testng.ITestResult;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class HookableExample implements IHookable {

	@Override
	public void run(IHookCallBack callBack, ITestResult testResult) {
		Object[] parms = callBack.getParameters();
		if (parms[0].equals("dummy")) {
			System.out.println("Skip for parameter dummy");			
		} else {
			callBack.runTestMethod(testResult);
		}
	}
		
	@BeforeSuite
	public void beforeSuite() {
		System.out.println("before suite");
	}
	
	@Test(dataProvider="getDp")
	public void t(String p) {
		System.out.println("test method t called with parameter " + p);
	}
	
	@DataProvider
	public Object[][] getDp() {
		return new Object[][]{{"client1"}, {"client2"}, {"dummy"}};
	}	
}

hookableTestNg.xml:

<?xml version="1.0" encoding="UTF-8"?>
<suite name="HookableExample Suite">
	<test name="HookableListenerExample">
		<classes>
			<class name="com.javacodegeeks.testng.HookableExample" />
		</classes>
	</test>
</suite>

Output:

[TestNG] Running:
  C:\javacodegeeks_ws\testNgListeners\test\com\javacodegeeks\testng\hookableTestng.xml

before suite
test method t called with parameter client1
test method t called with parameter client2
Skip for parameter dummy

===============================================
HookableExample Suite
Total tests run: 3, Failures: 0, Skips: 0
===============================================

1.9. Example of IReporter

IReporter is the listener you need to implement if you want to generate a report after all the suites are run.

In my test class, ReporterListenerExample, I have grouped three method t1, t2 and t4 in “perf”. Method t3 is not in any group.
Suppose I want to generate a report that contains test results of the tests belonging to “perf” group, I need to implement IReporter and implement generateReport method.

ReporterListenerExample:

package com.javacodegeeks.testng;

import org.testng.Assert;
import org.testng.annotations.Test;

public class ReporterListenerExample {
	@Test(groups="perf")
	public void t1() {
		System.out.println("in t1");
	}

	@Test(groups="perf", expectedExceptions=RuntimeException.class)
	public void t2() {
		System.out.println("in t2");
	}

	@Test
	public void t3() {
		System.out.println("in t3");
	}

	@Test(groups="perf", invocationCount=5)
	public void t4() {
		System.out.println("in t4");
		i++;
		if (i==1 || i==3) {
			Assert.assertEquals(i, 10);
		}
	}
	
	private int i;
}

ReporterListener is my class that implements IReporter. In the generateReport, I get methods belonging to “perf” group and then print its test results.
Few points to note regarding the implementation:

• ISuite.getMethodsByGroups() returns a map of group and collection of ITestNGMethod objects as value
• ITestNGMethod is TestNG’s view of the test method.
• The ITestResult interface gives access to the start and end times of test method.

ReporterListener:

package com.javacodegeeks.testng;

import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.Set;

import org.testng.IReporter;
import org.testng.IResultMap;
import org.testng.ISuite;
import org.testng.ISuiteResult;
import org.testng.ITestContext;
import org.testng.ITestNGMethod;
import org.testng.ITestResult;
import org.testng.xml.XmlSuite;

public class ReporterListener implements IReporter {

	@Override
	public void generateReport(List xmlSuites, List suites,
			String outputDirectory) {
		System.out.println("*****Custom Report******");
		ISuite suite = suites.get(0);
		Map<String, Collection> methodsByGroup = suite.getMethodsByGroups();
		Map<String, ISuiteResult> tests = suite.getResults();
		for (String key : tests.keySet()) {
			System.out.println("Key: " + key + ", Value: " + tests.get(key));
		}
		Collection suiteResults = tests.values();
		ISuiteResult suiteResult = suiteResults.iterator().next();
		ITestContext testContext = suiteResult.getTestContext();
		Collection perfMethods = methodsByGroup.get("perf");
		IResultMap failedTests = testContext.getFailedTests();
		for (ITestNGMethod perfMethod : perfMethods) {
			Set testResultSet = failedTests.getResults(perfMethod);
			for (ITestResult testResult : testResultSet) {
				System.out.println("Test " + testResult.getName() + " failed, error " + testResult.getThrowable());
			}
		}
		IResultMap passedTests = testContext.getPassedTests();
		for (ITestNGMethod perfMethod : perfMethods) {
			Set testResultSet = passedTests.getResults(perfMethod);
			for (ITestResult testResult : testResultSet) {
				System.out.println("Test " + testResult.getName() + " passed, time took " + 
			(testResult.getStartMillis() - testResult.getEndMillis()));
			}
		}
		System.out.println("*****End of Report******");
	}
}

reportListenerTestng.xml:

<?xml version="1.0" encoding="UTF-8"?>
<suite name="TestListenerExample Suite">
	<listeners>
		<listener class-name="com.javacodegeeks.testng.ReporterListener" />
	</listeners>
	<test name="TestListenerExample">
		<classes>
			<class name="com.javacodegeeks.testng.ReporterListenerExample" />
		</classes>
	</test>
</suite>

Output:

[TestNG] Running:
  C:\javacodegeeks_ws\testNgListeners\test\com\javacodegeeks\testng\reportListenerTestng.xml

in t1
in t2
in t3
in t4
in t4
in t4
in t4
in t4

===============================================
TestListenerExample Suite
Total tests run: 8, Failures: 3, Skips: 0
===============================================

*****Custom Report******
Key: TestListenerExample, Value: [SuiteResult context=TestListenerExample]
Test t2 failed, error org.testng.TestException: 
Expected exception java.lang.RuntimeException but got org.testng.TestException: 
Method ReporterListenerExample.t2()[pri:0, instance:com.javacodegeeks.testng.ReporterListenerExample@46f5f779] should have thrown an exception of class java.lang.RuntimeException
Test t4 failed, error java.lang.AssertionError: expected [10] but found [3]
Test t4 failed, error java.lang.AssertionError: expected [10] but found [1]
Test t1 passed, time took -6
Test t4 passed, time took 0
Test t4 passed, time took 0
Test t4 passed, time took -1
*****End of Report******

2. Adding TestNG listeners

We have already seen few ways of adding listeners. I will summarize here the different ways of adding listeners and will show you an example of each method:

1. Using <listeners> element in testng.xml
2. Using @Listeners annotation at class level
3. Adding listeners through TestNG addListener()API
4. Through java.util.ServiceLoader mechanism

Let me now show you an example of each method.

2.1. Adding listeners in testng.xml

One can add listeners using <listeners> element in testng.xml, where each listener will be defined as a child element using <listener>. The fully qualified class name of listener will be specified in class-name attribute.

For example,

invokedMethodListenerTestng.xml:

<?xml version="1.0" encoding="UTF-8"?>
<suite name="Suite" parallel="false">
	<listeners>
		<listener class-name="com.javacodegeeks.testng.InvokedMethodListener" />
	</listeners>
	<test name="Test">
		<classes>
			<class name="com.javacodegeeks.testng.InvokedMethodListenerExample" />
		</classes>
	</test>
</suite>

2.2. Adding listeners using TestNG @Listeners annotation

One can also define the listeners in the java code itself using class level @Listeners annotation. The listener classes will be specified commas separated as its attributes.

For example,

MethodInterceptorListenerExample:

package com.javacodegeeks.testng;

import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners({com.javacodegeeks.testng.MethodInterceptorListener.class})
public class MethodInterceptorListenerExample {
	@Test(groups="perf")
	public void t1() {
		System.out.println("test method: t1");
	}
	
	@Test
	public void t2() {
		System.out.println("test method: t2");
	}
}

2.3. Adding listeners using TestNG API

If you are running TestNG programmatically, you can add the listeners using TestNG.addListeners() API.

For example, in the below class, we create a TestNG object. We then set the test classes that we want to run, add a SuiteListener listener and invoke run method.

TestNGListenersAPI:

package com.javacodegeeks.testng;

import org.testng.TestNG;

public class TestNGListenersAPI {
	public static void main(String[] args) {
		TestNG testNG = new TestNG();
		testNG.setTestClasses(new Class[] { TestClass.class });
		testNG.addListener(new SuiteListener());
		testNG.run();
	}
}

Output:

[TestNG] Running:
  Command line suite

Start suite Command line suite
Set ui param value
before suite
in test method t
Finsh suite Command line suite

===============================================
Command line suite
Total tests run: 1, Failures: 0, Skips: 0
===============================================

One can also call the specific listener methods instead of the generic addListener as listed below

• setAnnotationTransformer to set annotation transformer.
• setMethodInterceptor to set method interceptor.
• addInvokedMethodListener to add IInvokedMethodListener object.
• setHookable to set IHookable object.
• addExecutionListener to add IExecutionListener object.

2.4. Adding listeners using java.util.ServiceLoader

You can also add the listeners using ServiceLoader mechanism.

• First create your own listener.

ServiceLoaderExampleSuiteListener:

package com.javacodegeeks.serviceloader;

import org.testng.ISuite;
import org.testng.ISuiteListener;

public class ServiceLoaderExampleSuiteListener implements ISuiteListener {

	@Override
	public void onStart(ISuite suite) {
		System.out.println("on Start " + suite.getName());
	}

	@Override
	public void onFinish(ISuite suite) {
		System.out.println("on finish " + suite.getName());		
	}
}

• Next, compile your listener. If you are using eclipse, it would automatically compile into bin dir.
• Create a dir META-INF/services, add a file with name org.testng.ITestNGListener in it.
• Open the file in an editor and add the fully qualified listener class name, in our case it is com.javacodegeeks.testng.ServiceLoaderExampleSuiteListener
• Create jar of META-INF and the listener class.

I have combined all of the above steps into a bat file.

run_external_listeners.bat:

cd listener
mkdir com\javacodegeeks\serviceloader
copy ..\bin\com\javacodegeeks\serviceloader\ServiceLoaderExampleSuiteListener.class com\javacodegeeks\serviceloader
jar cvf ../serviceLoader.jar .
cd..
echo run %1%
java -classpath serviceLoader.jar;testng.jar;bin  org.testng.TestNG %1%

When you run the bat file, you need to specify the xml testng file you want to run.

run_external_listeners testServiceLoader.xml

Output:

[TestNG] Running:
  C:\javacodegeeks_ws\testNgListeners\testServiceLoader.xml

on Start ServiceLoader
before suite
in test method t
on finish ServiceLoader

===============================================
ServiceLoaderExample Suite
Total tests run: 1, Failures: 0, Skips: 0
===============================================

Download the Eclipse Project

In this article, I have shown you several examples of TestNG listeners.

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you our best selling eBooks for FREE!

1. JPA Mini Book

2. JVM Troubleshooting Guide

3. JUnit Tutorial for Unit Testing

4. Java Annotations Tutorial

5. Java Interview Questions

6. Spring Interview Questions

7. Android UI Design

and many more ....

I agree to the Terms and Privacy Policy

Sign up

Thank you!

We will contact you soon.