'Introducing' Mantle Testkit
I’ve already written twice about problems Mantle Testkit solves without properly introducing it. Here’s what our WordPress testing framework is, why we built it, and why we use it for WordPress testing at Alley.
I’ve now written two posts about problems we’ve solved in Mantle Testkit. One was about WordPress tests leaking state between test cases. The other was about testing code that calls exit().
At some point while writing those, it occurred to me that I’d skipped a fairly important post: I never actually introduced Mantle Testkit.
So, hi. This is Mantle Testkit. It's a project we've been working on at Alley since 2020.
It’s the testing framework we use for WordPress development at Alley, and it grew out of a fairly simple frustration: testing WordPress code shouldn’t require accepting a worse developer experience than we expect everywhere else in PHP.
First, a naming clarification
There are two closely related things here.
Mantle Testing Framework is the actual testing framework. It contains the test case, WordPress bootstrapping, factories, HTTP request testing, assertions, state management, installation tooling, and the other pieces that make up the testing experience.
Mantle Testkit is the standalone Composer package that lets you use that testing framework without building your application on Mantle. That distinction is important because you do not need to use Mantle as your application framework. You do not need Mantle models, routing, service providers, Blade, or anything else.
You can have a completely ordinary WordPress plugin, theme, or site and use Testkit purely for testing. That’s actually one of its primary uses.
At Alley, we use it as our WordPress testing layer. New WordPress projects and packages don’t need to invent their own testing setup or copy the traditional WordPress test bootstrap around. They get Testkit.
The thing we were trying to fix
WordPress has a very good test suite, but it is a test suite built primarily for testing WordPress itself.
For years, the WordPress ecosystem has repurposed the test infrastructure from wordpress-develop to test plugins, themes, and applications. I’ve done that plenty of times too, and it works. But there are some odd consequences when you use a test suite designed to test WordPress as the foundation for testing an application built on WordPress.
Traditionally, your tests install or check out another copy of WordPress and its test library somewhere outside your project. Your test bootstrap then has to know where all of that lives.
That means the WordPress executing your code during a test may not actually be the WordPress sitting in front of you in your editor. That sounds like a small implementation detail until you’re debugging something: your IDE thinks a function lives here, Xdebug is stepping through a different copy over there, PHPUnit knows about classes that your IDE can’t quite understand, and CI needs an installation shell script before the tests can even start.
We’ve collectively gotten used to this because that’s how WordPress testing has worked for a very long time. Testkit takes a different approach.
When you’re testing an existing WordPress project, it uses the WordPress codebase you’re already working in and gives it a separate test database. Same code, disposable data.
For standalone plugins and themes, Testkit can install WordPress automatically and put your project into that installation for the test run. You don’t need to maintain the usual install-wp-tests.sh dance yourself.
The goal isn’t to make WordPress disappear. Quite the opposite: I want my integration tests running against as much real WordPress as practical.
It’s still WordPress
This is probably the most important part of the design: Testkit isn’t a collection of mocks pretending to be WordPress.
If your test creates a post, WordPress creates a post. If your code registers a post type, WordPress registers the post type. If you make a request, the framework sets up the WordPress request state, runs the main query, loads the template, captures the response, and lets you make assertions against what happened.
That lets tests look like this:
1<?php
2
3declare(strict_types=1);
4
5namespace Tests\Feature;
6
7use Mantle\Testkit\TestCase;
8
9class PostTest extends TestCase {
10 public function test_post_can_be_viewed(): void {
11 $post = static::factory()->post->create_and_get(
12 [
13 'post_title' => 'Hello from Testkit',
14 ]
15 );
16
17 $this->get( get_permalink( $post ) )
18 ->assertOk()
19 ->assertQueriedObject( $post )
20 ->assertSee( 'Hello from Testkit' );
21 }
22}
There’s a real WP_Post there and a real WordPress query. We’re exercising the request through WordPress rather than testing a pile of mocks that happen to have the same function names.
The HTTP testing API was heavily inspired by Laravel’s testing framework, because Laravel gets something important right here: tests should describe behavior without making the mechanics of executing that behavior painful. Mantle adapts that idea to WordPress rather than trying to hide WordPress underneath it.
We didn’t want migration to mean rewriting every test
There was another constraint when we started building this: we already had a lot of WordPress tests, and so did everyone else.
A replacement testing framework that required rewriting them all would be interesting academically and useless practically. Testkit deliberately keeps compatibility with familiar WordPress test APIs.
Existing code using things like:
1$this->go_to( home_url( '/' ) );
or:
1$post_id = static::factory()->post->create();
can keep doing that.
In many projects, the most interesting part of the migration is changing the base test case:
1-use WP_UnitTestCase;
2+use Mantle\Testkit\TestCase;
3
4-abstract class ExampleTest extends WP_UnitTestCase {
5+abstract class ExampleTest extends TestCase {
There is obviously more configuration involved for non-trivial projects, but we intentionally made compatibility a constraint rather than treating old tests as something users should throw away. That gives us room to improve the experience around those APIs instead of inventing new names simply because we can.
The bootstrap is part of the framework
One of my least favorite parts of traditional WordPress integration testing has always been the amount of infrastructure surrounding the tests themselves: download WordPress, download the test library, generate configuration, create the database, put the plugin in the correct place, and tell PHPUnit where everything went.
Testkit has an installation manager because I don’t think every WordPress project should independently solve that problem.
For a straightforward project, setup can be as small as:
1<?php
2
3\Mantle\Testing\install();
For something more involved:
1<?php
2
3\Mantle\Testing\manager()
4 ->maybe_rsync_plugin()
5 ->install();
The installation manager also gives us one place to handle the increasingly weird environments that real WordPress projects live in. Standalone plugins are different from monorepos. A WordPress VIP project is different from a small plugin. Some projects have custom wp-content paths. Some need an object-cache drop-in. Some need a specific URL or additional setup before WordPress boots.
Those shouldn’t each require a bespoke shell script held together by environment variables and institutional memory. CI should ideally get to the boring part:
1composer install
2composer test
That’s where we’ve tried to get it.
Factories should make setup cheap
Testkit also inherits and extends one of the best ideas in WordPress’s existing test suite: factories.
Need a post?
1$post = static::factory()->post->create_and_get();
Need a user?
1$user = static::factory()->user->create_and_get();
Need ten posts?
1$post_ids = static::factory()->post->create_many( 10 );
The easier fixture setup becomes, the less incentive there is to share giant database dumps or construct half-valid WordPress objects manually. This becomes especially useful once you’re writing feature tests instead of trying to isolate every five lines of application code from WordPress.
And that’s probably where my testing philosophy has moved the most while we’ve built Testkit.
I want integration tests to be cheap
There are plenty of pieces of PHP code that deserve small, isolated unit tests. But an enormous amount of interesting WordPress application code exists specifically to interact with WordPress: hooks, queries, REST endpoints, templates, metadata, authentication, rewrite rules, and registered content types.
You can mock all of those boundaries, and sometimes you should. But at a certain point you’ve created a beautifully isolated test proving that your code works with your imitation of WordPress.
I’d rather make running WordPress cheap enough that we don’t have to reach for that approach by default. That’s a big part of what Testkit is really about: not making tests cleverer, but making realistic tests inexpensive enough that people actually write them.
And then WordPress makes things interesting
Running real WordPress creates its own problems.
WordPress was never designed around the idea that the entire application would repeatedly boot, mutate itself, make assertions, and then somehow return to exactly the same state for another test a few milliseconds later. There is global state everywhere: post types, taxonomies, rewrite rules, registered metadata, scripts, styles, roles, capabilities, and query variables.
That is why I ended up writing an entire post about state leaking between tests. It’s also why Testkit has gradually accumulated code whose entire job is making WordPress behave like a system under test.
We snapshot things. We restore things. We intercept requests. We prevent accidental remote HTTP calls. We make otherwise terminal behavior observable. We keep finding pieces of WordPress state that everybody assumed the existing suite reset but actually didn’t.
That’s not incidental work around the testing framework. At this point, I think it’s some of the most important work in it, because a testing framework isn’t particularly useful if test 83 behaves differently depending on whether test 47 ran first.
This is what we use
Testkit started as part of Mantle, but I don’t really think of it as a Mantle-only feature anymore. At Alley, it’s how we test WordPress.
We use it in applications, standalone plugins, and packages. Our WordPress project tooling is built around it so a new project starts with the same testing foundation instead of every team deciding how to bootstrap WordPress again.
The standalone package is intentional. I’d rather someone use Testkit in an otherwise completely conventional WordPress project than feel like they need to adopt an application framework just to get better tests.
You can install it with:
1composer require --dev mantle-framework/testkit
The testing framework keeps the useful parts of WordPress’s existing testing API and adds the pieces we’ve wanted while testing real WordPress applications: request testing, better assertions, installation management, state isolation, factories, HTTP fakes, time manipulation, and a growing collection of smaller utilities that mostly exist becapuse we needed them ourselves.
I’m sure I’ll keep writing about individual pieces of it. There are enough strange corners of testing WordPress to keep that going for quite a while.
But now, at least, I’ve formally introduced the thing. I'm trying to write more and share.