Monthly Archives: April 2014

Selenium TestNG – Customizing tests at runtime with IAnnotationTransformer

Recently I had a request to have the possibility to choose which @Test to run from an external file that is human readable and easily configurable thus testng.xml was out of the question.

My first thought was: this should be easy, we can just pass a true or false value from an external file, to the “enabled” test method attribute. This approach quickly failed since the annotation’s attributes seem to accept only constants, so for “enabled” it’s either true or false, not a “superposition” of both :)

So I started doing a little digging and ultimately found the ITestAnnotation Transformer interface:

1
2
3
4
public interface IAnnotationTransformer {
public void transform(ITest annotation, Class testClass,
Constructor testConstructor, Method testMethod);
}

The transform() method will be executed before any of the annotated methods/classes in our code and when we implement it we have the opportunity to use setters to alter the annotation attributes.

Let’s take the following example with 3 tests:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
package testthis.selenium;
import org.testng.annotations.Test;
public class SelectTestToRun {
@Test(testName = "test1", enabled = false)
public void testing() {
System.out.println("test1 run");
}
@Test(testName = "test2", enabled = false)
public void testing2() {
System.out.println("test2 run");
}
@Test(testName = "test3", enabled = false)
public void testing3() {
System.out.println("test3 run");
}
}

As you can see we initialize the tests with the enabled attribute set to false and also give them a testName. In order to have a test running, we will have to set it to true from the transform() method.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
package testthis.selenium;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
import java.util.Properties;
import org.testng.IAnnotationTransformer;
import org.testng.annotations.ITestAnnotation;
public class AnnotationTest implements IAnnotationTransformer {
Properties props = new Properties();
public void transform(ITestAnnotation annotation, Class testClass,
Constructor testConstructor, Method testMethod) {
try {
props.load(new FileInputStream("configure.properties"));
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
if (props.values().contains(annotation.getTestName())) {
annotation.setEnabled(true);
}
}
}

Now here was the tricky part. This needed a way to differentiate between our test methods, because annotation.setEnabled(true) will cause all the @Test methods to run.
So we had to perform a check if among the values read from the properties file, there are any testNames matching. If so, then that @Test will have its enabled attribute set to “true”.

To make this easier, we will name the keys inside the properties file with the same value as the testNames and if we have the same value, then the test is run.

1
2
3
4
key=value
test1=test
test2=test2
test3=test3

For this to work, we have to Run As TestNG Suite, from the testng.xml and also pass a listener class, in our case, it is the fully qualified name of the AnnotationTest class.

Now let’s have a look at the results:

1
2
3
4
5
6
7
8
[TestNG] Running:
C:\Users\mdima\workspace\SeleniumTest\testng.xml
test2 run
test3 run
===============================================
Suite
Total tests run: 2, Failures: 0, Skips: 0
===============================================

Since we passed something else rather than the testName of the first test, test1() method was not run.

So we can now easily customize which tests to run from the properties file, without touching the Test methods, as long as we’re using the testng.xml and therefore this is also useful for the maven-surefire-plugin in a maven project.

Enjoy TestNG customization,
-M.

Test execution priority in Selenium TestNG

Hello everyone,

Few days ago I was taking a look over a class which contained many @Test methods (from org.testng.annotations of course :) ) and whenever it was running, I was getting a different order of the tests in the results’ summary.

So this got me thinking and a quick solution came to mind: test dependency (dependsOnMethods / dependsOnGroups) but this would then imply that if the initial test fails, the execution would stop right then and there. I then found out about test method priority and it seems to be exactly what I was looking for and very easy to use.

Given the following example let’s see how we can obtain a desired order of execution: A > B > C

1
2
3
4
5
6
7
8
9
10
11
12
13
14
    @Test(priority = 1)
    public void testC() {
        System.out.println("Test C");
    }

    @Test(priority = 2)
    public void testA() {
        System.out.println("Test A");
    }

    @Test
    void testB() {
        System.out.println("Test B");
    }

In the above case, we get this order of execution: B > C > A which isn’t exactly what we wanted, let’s see how we can change that.

A few things need to mentioned about test method priorities:

  • lowest priority (can also be negative) will be executed first, thus -5 will run before priority 1
  • when no priority is specified, the default priority is 0, in our example, testB has the default priority

Since we’re aiming to obtain the A -> B ->C flow, we should use, the lowest priority for testA and set B and C to 1 and 2 respectively:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
    @Test(priority = 2)
    public void testC() {
        System.out.println("Test C");
    }

    @Test
    public void testA() {
        System.out.println("Test A");
    }

    @Test(priority = 1)
    void testB() {
        System.out.println("Test B");
    }