7 Open-Source Test Automation Frameworks

This article was originally published by TestProject. Thank you for supporting the partners who make SitePoint possible.

As we enter the last quarter of 2017, TestProject’s team decided to round up the best open-source test automation frameworks out there, to help you choose the right one for you! Here are the pros and cons of 7 different open-source test automation frameworks.

1. Robot Framework

Robot Framework (RF) is a test automation framework for acceptance testing and acceptance test-driven development (ATDD). This framework is written in Python, but can also run on Jython (Java) and IronPython (.NET), and therefore is cross-platform (Windows, Linux, or MacOS). Pros:

• It simplifies the test automation process by utilizing the keyword-driven testing (KDT) approach, which helps testers create readable tests that are easily made.
• Has easy to use test data syntax.
• Has a rich ecosystem around it consisting of various generic test libraries and tools that are developed as separate projects.
• Has many APIs that make it highly extensible.
• Although it’s not a built-in ability, RF enables to execute parallel tests via pabot library or Selenium Grid.

Cons:

• It’s not easy to customize HTML reports.

Bottom line: This cross-platform framework is highly recommended if you are targeting for KDT automation with a wide range of libraries and extensions. If you want to add new keywords (via RF test library APIs), a basic knowledge in Java/Python/C programming language is required.

2. JUnit

JUnit is a framework for unit testing of Java applications, used to write and run repeatable tests. Pros:

• Tests are written in pure Java which is known as the leading programming language worldwide.
• Supports test-driven development (TDD).
• Enables you to create your own unit test case suite.
• Integrates very well with other tools (for example, Maven) and with IDEs (for example, IntelliJ).
• Has history – so it has a large user base that makes it easy to find documentation on it.

Cons:

• If a mocking ability is required, one needs to add Mockito (or some other mocking library).
• Tests are not readable by non-technical people, since for example the methods names in JUnit are constrained by Java conventions.

Bottom line: If you’re looking to write unit testing for your Java application, it’s probably the best choice. However, for functional testing or non-Java applications, you should consider other solutions.

3. Spock

Spock is a testing and specification framework for Java and Groovy applications. It is based on JUnit. Pros:

• Creates readable tests and supports plain English sentences, making it easy to read.
• Provides the surrounding context, so it easily enables you to understand how to fix a failure.
• Has built-in mocking and stubbing capabilities.
• Supports data-driven-tests (DDT).

Cons:

• Requires a basic knowledge of the Groovy programming language.

Bottom line: If your application is based on JVM and you are targeting for BDD test automation with DSL, this framework is just for you!

4. NUnit

NUnit is a unit testing framework for all .Net languages. Originally inspired by Junit, it is written entirely in C#, and has been completely redesigned to take advantage of many .NET language features. Pros:

• Quick initiation and test execution.
• Comes with assertions and annotations.
• Enables parallel testing.
• Supports test-driven development (TDD).

Cons:

• It is not cross-platform as it is used only for .Net languages.
• It doesn’t integrate into the Visual Studio ecosystem, so using it means more maintenance.

Bottom line: A nice open-source framework for C# unit testing, with a long history and good reputation. However, in case you’re already using .NET languages, you may consider MSTest.

5. TestNG

TestNG is a test automation framework for Java that is inspired by JUnit and NUnit, but includes improved and new functionalities (NG – Next Generation). It is designed to cover all the test automation categories: unit testing, functional test, end-to-end, integration testing, etc. Pros:

• It is easily integrated into the Maven cycle.
• Gives the developer the ability to write flexible and powerful tests.
• Supports Data Driven testing (DDT).
• Annotations are easy to understand.
• Test cases can be grouped easily.
• Allows you to create parallel tests.

Cons:

• Only supports Java, so you need to have at least a basic knowledge of the Java programming language.
• You have to invest time in framework setup & design.

Bottom line: If you use Java, are looking for end-to-end test automation framework and willing to invest some time in framework setup – you should definitely consider using TestNG.

6. Jasmine

Jasmine is a JavaScript unit testing framework. It is also known as a Behavior Driven Development (BDD) testing framework for JavaScript. It’s suited for websites, Node.js projects, or anywhere JavaScript can run. It is mainly paired with AngularJS. Pros:

• Additionally to JavaScript, it can run in Python and Ruby, which can greatly assist you if you want to run your client-side tests alongside your server-side ones.
• Is supported by many CIs (Codeship, Travic, etc.).
• Has built-in syntax for assertions.

Cons:

• In most scenarios it requires a test runner (such as Karma).
• Has difficulties with asynchronous testing.

Bottom line: Jasmin might be a perfect fit for your needs if you’re searching for a unified (client-server) unit testing solution.

7. Mocha

Mocha is a JavaScript unit testing framework, that runs tests on Node.js. It is mainly paired with ReactJS. Pros:

• Has its own test runner built-in.
• Supports asynchronous testing.
• Allows flexibility since you can use any assertion library (Chai, expect.js, Must.js, etc.) that fits your needs (as a replacement to Node’s standard ‘assert’ function).

Cons:

• Relatively new to the field (developed in 2012), which means it’s still changing and its user base and support might be lacking in some aspects.
• Provides just the base test structure, thus requires additional setup and configuration (may be an advantage for some).

Bottom line: If you are looking for JavaScript stand alone unit testing framework, Mocha is your go-to framework!

Frequently Asked Questions (FAQs) about Open Source Test Automation Frameworks