How to Order Test cases – Guidelines
Today’s article demonstrates how to Order Test cases execution using a testing framework. Today in this article, We shall cover this requirements using Test frameworks like MSTest, xUnit, and NUnit.
We will cover below aspects of test case ordering in today’s article,
As a good practice, we generally don’t order Unit test cases, because we always create them as stateless as possible and each unit test should run independently of each other.
But test case execution orders using Alphabetical or Display names still can be used for UnitTest cases if needed for better organizing the Test cases.
Priority orders can be used for functional test cases or Integration test cases where we need to perform the operation in certain orders.
This could be legitimate use cases for Integration test cases and Functional Test cases . When you are dealing with the testing of various workflow and functional scenarios it might be necessary to plan certain action in an order ex.by Priority order to execute the business workflow.
Example: Transcation in Banking
1.Login to Login page of Banking website
2.Showing default summary page
3.Performing fund transfer
4.Veryfing balance
Ordering on test cases can be achieved by any of the below mechanisms,
- Alphabetical – Order test method based on test name’s Alphabetical order
- DisplayName – Order test method based on test name’s DisplayName order
- Priority– Order test method based on test name’s using priority order set by the user providing better control on execution.
Understanding ITestCaseOrderer interface
- ITestCaseOrderer is the user interface provided by the XUnit ordering the execution of test cases. A class implements this interface to participate in ordering tests for the test runner.
- Test case orderers are applied using the Xunit.TestCaseOrdererAttribute.
- Test case orderer can be applied at the below level,
- Assembly,
- Test collection
- Test class level.
XUnit Test case Order by Priority – custom attribute
To order xUnit tests with custom attributes like below we have used TestPriority.
Define a TestPriorityAttribute and PriorityOrderer as follows.
Please see below an example of TestPriorityAttribute definition.
[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class TestPriorityAttribute : Attribute
{
public int Priority { get; private set; }
public TestPriorityAttribute(int priority) => Priority = priority;
}
PriorityOrderer class is defined as below using ITestCaseOrderer
public class PriorityOrderer : ITestCaseOrderer
{
public IEnumerable<TTestCase> OrderTestCases<TTestCase>(
IEnumerable<TTestCase> testCases) where TTestCase : ITestCase
{
string assemblyName = typeof(TestPriorityAttribute).AssemblyQualifiedName!;
var sortedMethods = new SortedDictionary<int, List<TTestCase>>();
foreach (TTestCase testCase in testCases)
{
int priority = testCase.TestMethod.Method
.GetCustomAttributes(assemblyName)
.FirstOrDefault()
?.GetNamedArgument<int>(nameof(TestPriorityAttribute.Priority)) ?? 0;
GetOrCreate(sortedMethods, priority).Add(testCase);
}
foreach (TTestCase testCase in
sortedMethods.Keys.SelectMany(
priority => sortedMethods[priority].OrderBy(
testCase => testCase.TestMethod.Method.Name)))
{
yield return testCase;
}
}
private static TValue GetOrCreate<TKey, TValue>(
IDictionary<TKey, TValue> dictionary, TKey key)
where TKey : struct
where TValue : new() =>
dictionary.TryGetValue(key, out TValue result)
? result
: (dictionary[key] = new TValue());
}
Once you define the classes above, Attribute your test class with the TestCaseOrdererAttribute to the PriorityOrderer.
[TestCaseOrderer("OrechstrationService.Project.Orderers.PriorityOrderer", "OrechstrationService.Project")]
public class HttpHelperTests
{
public static bool Test1Called;
public static bool Test2Called;
public static bool Test3Called;
[Fact, TestPriority(3)]
public async Task AccountController_OnGetAccount_Valid_HTTPClient()
{
//Arrange
//Act
//Assert
Test3Called = true;
Assert.True(Test1Called);
Assert.True(Test2Called);
}
}
Below are examples of other unit test cases which execute based on priority.
[Fact, TestPriority(2)]
public async Task AccountController_OnGetAccount_Valid_HTTPClientFactory()
{
//Arrange
//Act
//Assert
Test2Called = true;
Assert.True(Test1Called);
Assert.False(Test3Called);
}
[Fact, TestPriority(1)]
public async void AccountController_OnGetAccount_Valid_Typed_HTTPClient()
{
//Arrange
//Act
//Assert
Test1Called = true;
Assert.False(Test2Called);
Assert.False(Test1Called);
}
With the above arrangements now once we execute the test cases, we shall see test cases are getting executed based on the priority set.
Alternatively – Assert statements for the above 3 Tests using Test1Called, Test2Called, and Test3Called also confirm the order of the execution is correct.
Order Test cases in Alphabetical Order
Implement the ITestCaseOrderer and provide an ordering mechanism. With this, we will order the Test cases using the method name.
using System.Collections.Generic;
using System.Linq;
using Xunit.Abstractions;
using Xunit.Sdk;
namespace OrechstrationService.Project
{
public class AlphabeticalOrderer : ITestCaseOrderer
{
public IEnumerable<TTestCase> OrderTestCases<TTestCase>(
IEnumerable<TTestCase> testCases) where TTestCase : ITestCase =>
testCases.OrderBy(testCase => testCase.TestMethod.Method.Name);
}
}
TestCaseOrderer now can be defined using AlphabeticalOrderer as below,
[TestCaseOrderer("OrechstrationService.Project.Orderers.AlphabeticalOrderer", "OrechstrationService.Project")]
public class HttpHelperTests
{
public static bool Test1Called;
public static bool Test2Called;
public static bool Test3Called;
[Fact]
public async Task AccountController_OnGetAccount_Valid_HTTPClient()
{
//Arrange
//Act
//Assert
Test3Called = true;
Assert.True(Test1Called);
Assert.True(Test2Called);
}
[Fact]
public async Task AccountController_OnGetAccount_Valid_HTTPClientFactory()
{
//Arrange
//Act
//Assert
Test2Called = true;
Assert.True(Test1Called);
Assert.False(Test3Called);
}
[Fact]
public async void AccountController_OnGetAccount_Valid_Typed_HTTPClient()
{
//Arrange
//Act
//Assert
Test1Called = true;
Assert.False(Test2Called);
Assert.False(Test3Called);
}
}
Once executed you can see test cases are now run based on the method name,
Order XUnit test cases by Collection Name Order
Please see the below article for more details,
References:
Do you have any comments or ideas or any better suggestions to share?
Please sound off your comments below.
Happy Coding !!
Please bookmark this page and share it with your friends. Please Subscribe to the blog to receive notifications on freshly published(2024) best practices and guidelines for software design and development.