Test Faster with the Next Generation of Parallel Automation Execution Framework – TestNg
Since the inception of automated testing we’ve seen many platforms and iterations, each with their own unique set of pros-and-cons; however, a common issue regardless of the platform has been an inherently slow execution as tests had to be executed linearly.
Enter TestNG, the next-generation testing framework which solves this long-standing performance problem by providing the ability to execute tests in parallel.
With parallel execution in TestNG, you can significantly reduce execution time and better validate real-world scenarios by simulating parallel traffic; no site has just one visitor at a time.
Leveraging parallel testing via TestNG has allowed us to reduce execution time by 50%, saving time and money for both our team and clients. In this blog post I discuss how your organization and clients can realize this benefit and many others using TestNG.
Before you Get Started with TestNG
Assuming you have an existing test suite you’ll want to convert to TestNG to take advantage of parallel execution, the conversion of your existing test suite for use with TestNG can be considered a two-part effort:
- A fairly simple exercise of updating the test suite to run within the TestNG framework.
- Updating the test suite logic to be thread-safe in support of parallel execution. This will require careful review and consideration of how your tests are executed. For example moving shared variables from base class to test-level, leveraging thread-local variables, etc.
I will discuss in-depth about preparing your test suite to support parallel execution using TestNG.
But first, let’s discuss the additional benefits TestNG will provide aside from parallel execution.
TestNG benefits:
In my 10-years managing an enterprise-grade team of Quality Assurance and CI/CD Engineers, here are the key benefits we’ve realized from switching to TestNG:
- Annotations. By using annotations you can define functions as part of the runner without the need to rename them or keep their name common between all tests in the automation suite.
Also, you can simply pass parameters through these annotations to the functions you annotate.
Here’s a simple example of how the annotation can simplify your logic:
In the example above we have a complete test suite abstraction, where we used @test annotation to annotate GeneralTest and GeneralTest2 as tests. |
- Achieving data-driven testing. TestNG supports data-driven testing by leveraging data parameterization which allows us to test a piece of software against different inputs as parameters. This is considered best-practice for test suites to ensure our software is running with all possible user inputs.
TestNG can pass all requisite parameters through either an XML runner or via DataProviders. As an example, passing browser type to one test. I will talk in this article more about the parameterization ways using TestNG.
- Allows QA engineers to execute test cases based on the group and priority. As an example, you can create two groups – ‘Regression’ and ‘Sanity’. If you want to execute the test cases under the Sanity group, then you can do so easily with the TestNG framework. TestNG even allows you to indicate execution priority.
Continuing the example, you would be able to run Sanity tests prior to Regression.
Many of the benefits we’ve discussed with TestNG are dependent on a concept called Annotations.
TestNG Annotations:
Hierarchy annotations:
A TestNG annotation is a keyword starting with “@” symbol and added before the annotated method. In TestNG, @test must be in the test suite.
TestNG builds atop standard Java Annotations, providing the ability to reuse annotations as syntactic metadata that can be added to any method. TestNG can identify each annotated method and execute it through its runner via several different annotations.
Below is the basic set of annotations by TestNG :
- @BeforeSuite: The annotated method will be run before all tests in this suite have run.
- @AfterSuite: The annotated method will be run after all tests in this suite have run.
- @BeforeTest: The annotated method will be run before any test method.
- @AfterTest: The annotated method will be run after all the test methods.
- @BeforeClass: The annotated method will be run before the first test method in the current class is invoked.
- @AfterClass: The annotated method will be run after all the test methods in the current class have been run.
- @BeforeMethod: The annotated method will be run before each test method.
- @AfterMethod: The annotated method will be run after each test method.
- @Test: The annotated method is a part of a test case.
Example:
Here’s an example suite using several of the annotations mentioned above, in particular how annotations determine the hierarchy of execution:
Here’s a detailed example of an annotation which shows how you can use annotations within a function as part of each run cycle:
Annotations allow the TestNG framework to execute methods in exactly the same order mentioned in the code above, resulting in the following output:
All annotations, albeit @test, support an extensive list of parameters which are documented here.
I would like to call out two parameters worth special mention:
- alwaysRun: If set to true, this configuration method will be run regardless of what groups it belongs to.
- groups: The list of groups this class/method belongs to.
Regarding the @test annotation, these are the most important parameters to consider:
- dataProvider: The name of the data provider for this test method.
- enabled: Whether methods on this class/method are enabled.
- expectedExceptions: The list of exceptions that a test method is expected to throw. If no exception or a different than one on this list is thrown, this test will be marked a failure.
- groups: The list of groups this class/method belongs to.
- priority: The priority for this test method. Lower priorities will be scheduled first.
Data Provider Annotation:
Of special mention is the @DataProvider annotation which affords a great deal of flexibility to import test data:
- @DataProvider: Marks a method of supplying data for a test method. The annotated method must return an Object[][] where each Object[] can be assigned the parameter list of the test method. The @Test method that wants to receive data from this DataProvider needs to use a dataProvider name equal to the name of this annotation. It takes two parameters:
- name The name of this data provider. If it’s not supplied, the name of this data provider will automatically be set to the name of the method.
- parallel If set to true, tests generated using this data provider are run in parallel. The default value is false.
For more information about annotations and their parameters please visit the following TestNG documentation.
Annotations best practices:
- Avoid using any annotation that will be ignored, for example using invocationTimeOut while invocationCount is not defined.
- Avoid using two hierarchy annotations for the same function.
- Use only the annotations you absolutely need.
TestNG XML Test Suite configurations:
Standard operating procedure for software validation is testing against a set of scenarios. TestNG provides you the ability to group several scenarios into a single entity named a test suite. A TestNG test suite is driven by an XML file which contains the following information:
- Document type definition as the doctype document type.
- XML version , encoding.
- Parameters.
- Test classes/ Packages.
- Run configurations (Name, Parallel level, parallel factor, data provider threads,..)
Run configurations:
XML suite configurations are very flexible and can provide a wide range of options to suit your particular needs. Below are some tags I would like to mention which are heavily used in TestNG XML:
- Name: The name of this suite.
- parallel: Whether TestNG should use different threads
- thread-count: An integer giving the size of the thread pool to use if you set parallel.
- time-out: The time to wait in milliseconds before aborting the method (if parallel=”methods”) or the test (parallel=”tests”)
- data-provider-thread-count: An integer giving the size of the thread pool to use for parallel data providers.
Still, need to know more about all XML run configuration tags? Please check TestNG documentation
XML suite sample with parameterization:
Parametrization Drives Data-Driven Testing:
As we create software, the goal is to author behavior which is consistent regardless of the data input and environment. As such, we need to verify our software is properly handling many permutations of data and robust enough to gracefully handle the unexpected.
Insert Parameterization which allows us to provide multiple, disparate sets of data to our software during runt-time, also known as data-driven testing.
Type of Parameterization in TestNG:
There are two ways you can achieve parameterization with TestNG
- Parameter annotation via the TestNG XML file.
- DataProvider annotation.
With this method you provide an Object[][] to the data provider which is subsequently consumed by the @test function. You can also use any library that can handle an XML file format to pull all required data from an Excel Worksheet, for example.
I like to use the Apache POI library for this purpose. Apache POI creates and maintains Java APIs for manipulating various file formats per the OpenOffice XML standards. In short, you can read and write MS Excel files using Java.
Using Apache POI allows you to maintain large sets of testing data within Excel directly which certainly has its advantages.
Both methods provide a solid approach to achieve Data-Driven testing, but I personally prefer to use the POI library as it greatly simplifies test data management and maintenance.
Assertions
Assertions can be considered the most important part of your test suite lifecycle as they allow you to confirm successful execution of tests. TestNG provides us with a wide variety of Assertion types which allows us to develop robust validation criteria.
A simple example of a TestNG Assertion is:
- Navigating to a website.
- Logging into the website.
- Verifying site title.
- Verifying a particular input field accepts integers only, like Zip Code.
In TestNG we have two, primary Assertion types; note there are several varieties within each type.
- Hard Assertion (the default type in TestNG)
- Soft Assertion
Hard Assertion in TestNG
These types of assertion are validation criteria critical to the successful execution of your test suite, in particular, they will halt execution if a failure is encountered. Hard Assertions play a vital role for projects where you have functionality which is absolutely critical to the success of your application.
A good example is the login functionality. If I want to see my past orders, for example, then what is the point of checking this test case when the login validation already failed?
Soft Assertions
Soft Assertions are to be used for functionality which is not critical to the success of your application. In the event a Soft Assertion fails, validation continues and subsequent Assertions will be executed.
So why would you ever want to use Soft Assertions in TestNG? Let’s consider the example of an Assertion which checks to see the price displayed to customers is in the correct format – $XX.XX. In this case, we want to know if the application is not displaying the correct format but don’t want to stop test suite execution if the validation fails.
TestNG provides us with several Assertion methods, the following of which are the most commonly used. Please note additional Assertion methods are listed in the following TestNG documentation:
- Assert.assertEqual(X, Y): Pass the actual string, boolean, or Objects value and the expected values are identical.
- Assert.assertTrue(condition): This method asserts if the condition is true or not. If not, TestNG assertion will throw the exception error.
- Assert.assertFalse(condition): This method asserts if the condition is false or not. If not, then it throws an exception error.
Here’s an example of how these Assertion methods can be used:
While both Assertions will fail, GeneralTest will continue execution and will not make the test abort until assertAll() is executed..
GeneralTestHardAssert on the other hand will abort test execution immediately as it’s a Hard Assert.
Dependent Tests:
In order to properly design your TestNG suite ensure you build small, unit-level tests to exercise specific areas. You can then use the dependsOnMethods qualifier to run your unit-level tests in any order desired. This promotes reusability and consistency in your test suite.
In the example below, execution of SignIn will first run Register:
Parallel Execution:
As previously discussed the primary advantage of TestNG is the ability for parallel execution;however, with more power comes more responsibility. In particular, you must take great care to ensure thread-safe tests.
From my experience creating reliable, thread-safe test frameworks which efficiently utilize shared resources without wasting memory, duplicating code, etc. is dependent on the following best practices:
- Avoid using public shared variables in base classes.
- Use the ThreadLocal class to create variables, so each thread can own its variable value.
- Avoid writing data to datastores/database without closing connection each time, keeping it open will block other threads from using it.
- Implement the concept of the singleton in your suite (one instance from class).
- Don’t compromise Object-oriented concepts, this will protect your code from being unsafe while using TestNG.
Although parallel execution has significant benefits it also has some drawbacks:
- Requires significant increase in effort from QA Engineers to write and maintain sustainable, parallel tests. This includes ensuring your TestNG libraries are continually updated.
- Requires a significant investment in hardware if you’d like to execute more than five (5) concurrent threads.
- An extension of my first point, an Increase in difficulty for debugging issues.
- TestNG still has some issues which need to be resolved, especially related to reporting on parallel tests.
While these drawbacks are certainly not trivial, they can readily be overcome with dedicated, hard-working QA engineers who are passionate about maximizing efficiency and effectiveness of automated test frameworks for your customers.
Conclusion
TestNG is a framework I highly recommend as it allows you to build scalable, robust testing solutions with relatively minimal effort. TestNG is also constantly updated with new features to maintain its competitive advantage over other automation frameworks.
With a bit of effort, TestNG can add tremendous value to your automation framework offering and increase confidence in your application.
