Your WordPress tests are leaking state

WordPress was never designed to run this way.

A normal WordPress request has a convenient cleanup mechanism: PHP exits. WordPress boots, plugins register post types, taxonomies, metadata, rewrite rules, and everything else they need, and the request is handled. Then the process ends and all of that in-memory state disappears with it.

A test suite breaks that assumption. We boot WordPress once, then run hundreds or thousands of tests against the same process. Every test is supposed to behave as if it starts from a known state, but much of WordPress was built with the assumption that the process itself would provide that isolation.

The database is the obvious state, and WordPress's test suite handles it well. The runtime is harder, which is how you end up with one of my least favorite classes of test failure:

The test passes by itself, but fails when I run the whole suite.

There is a worse version, too:

The test passes in the suite, but fails by itself.

The first one wastes your time. The second one should worry you because it means your test is green only because another test happened to prepare part of its environment. That isn't a well-written test, though it probably isn't your fault. Either way, you end up with tests conflicting with each other and plenty of setUp()/tearDown() cleanup madness.

The database is clean. WordPress isn't.

Consider a fairly unremarkable test:

 1public function test_something_with_books(): void {
 2	register_post_type(
 3		'book',
 4		[
 5			'public'   => true,
 6			'supports' => [ 'title', 'editor' ],
 7		]
 8	);
 9
10	$this->assertTrue( post_type_exists( 'book' ) );
11
12	// Test something interesting...
13}

register_post_type() doesn't write a row to the database. It changes WordPress's in-memory state. Specifically, book now exists in the global post type registry. (Using globals for everything was a great idea, right?!)

So a later test can do this:

1public function test_book_is_not_registered(): void {
2	$this->assertFalse( post_type_exists( 'book' ) );
3}

Run this test alone and it passes. Run it after the first test and it fails.

Nothing flaky happened. Both tests behaved perfectly consistently. They just weren't isolated, and post types are only one example.

WordPress keeps a surprising amount of its runtime configuration in process-wide state: post types, taxonomies, post statuses, registered meta, rewrite state, public query variables, sitemap state, post type features, roles, capabilities, and more.

That architecture makes sense for WordPress's normal lifecycle. Registration generally happens once during bootstrap, and the process disappears shortly afterward. Things get more interesting when the process sticks around for another 2,000 tests.

WordPress core knows about this problem

For a long time, I missed an important detail in the WordPress test framework: WordPress already has methods for resetting post types, taxonomies, and post statuses between tests.

But look at when it uses them:

1if ( defined( 'WP_RUN_CORE_TESTS' ) && WP_RUN_CORE_TESTS ) {
2	$this->reset_post_types();
3	$this->reset_taxonomies();
4	$this->reset_post_statuses();
5
6	// ...
7}

Those resets happen when testing WordPress core. For non-core tests, WordPress deliberately skips them for a completely reasonable reason: plugins register post types and taxonomies during init.

Imagine booting your plugin, letting it register its book post type, and then having the testing framework blindly reset the post type registry before your first assertion. Your test would be isolated, but it would also be testing an environment that no longer resembles your application.

Cleaning global state isn't enough. You need to know which state belongs to the application and which state belongs to the previous test. WordPress core can't determine that for your plugin, but your testing layer can.

Cleanup works until you forget something

The first instinct is usually to make every test clean up after itself.

1public function tearDown(): void {
2	unregister_post_type( 'book' );
3
4	parent::tearDown();
5}

For simple cases, that works. We've done this one before a few (or more) times. I've come to dislike this model for framework-level isolation, though, because it asks the test that mutated the environment to also know every piece of state that mutation affected. WordPress's state is rarely that simple.

We found a good example of this recently in Mantle. Its testing framework was already preserving $wp_post_types, so a test could unregister a post type and we'd restore the post type registry before the next test. That sounded correct, but it wasn't.

WordPress also stores post type features such as title, editor, and thumbnail in a separate global: $_wp_post_type_features.

After restoring the post type, you could get this wonderfully inconsistent result:

1$this->assertTrue( post_type_exists( 'book' ) );
2
3$this->assertFalse( post_type_supports( 'book', 'title' ) );

The post type existed again, but part of the state describing it didn't. We had restored what we knew to restore and leaked what we didn't. That's why I don't think teardown is the right abstraction for this problem.

Restore a known state instead

Mantle's Testkit takes the opposite approach. Instead of asking every test:

What did you change, and how do we undo it?

we ask:

What did WordPress look like before you changed it?

Then we restore that state.

Today, the set of globals Mantle snapshots includes:

1protected const GLOBALS_TO_BACKUP = [
2	'_wp_post_type_features',
3	'wp_meta_keys',
4	'wp_post_statuses',
5	'wp_post_types',
6	'wp_rewrite',
7	'wp_sitemaps',
8	'wp_taxonomies',
9];

Public query variables are preserved separately as well.

There's another detail here that matters: we don't simply snapshot WordPress before the entire test suite and reset to that state every time. A test class may legitimately establish state in setUpBeforeClass().

So the lifecycle is roughly:

Capture WordPress's original state.
Allow the test class to perform its setup.
Capture that as the baseline for the class.
Restore that baseline before each test.
Restore WordPress's original state when the class is finished.

Objects in those globals have to be deep-copied, too. Otherwise your "snapshot" can contain references to the same objects the test is about to mutate.

The goal isn't an empty WordPress installation before every test. It's a known WordPress installation before every test. That distinction is what makes the technique useful for plugins and applications.

You can find these leaks without Testkit

You don't need Mantle to discover whether your test suite has this problem. Randomize your test order:

1vendor/bin/phpunit --order-by=random

Then run it again. And again.

If different orders produce different results, start looking for shared state. When you find a suspicious test, run it independently:

1vendor/bin/phpunit tests/Feature/BooksTest.php

Then run its entire test class, followed by the suite. You're looking for a test whose result depends on what executed before it.

Test-order randomization isn't only a way to find flaky tests. It tests one of the assumptions underneath your entire suite:

Does this test actually establish everything it needs?

That's a useful question in any test suite, and it's particularly useful in WordPress.

We're still finding state

I would love to say that after maintaining Testkit for years we've found every WordPress global that can leak between tests. I don't believe that.

$_wp_post_type_features was added to our preservation logic because we found a real case our previous implementation didn't handle. We still have an open issue around roles and capabilities, another particularly awkward piece of WordPress runtime state. Tests need to be able to modify roles, while application-defined roles need to survive between tests.

I'm sure it won't be the last one. WordPress wasn't designed around the idea that every operation had to be reversible so another test could immediately reuse the same PHP process. It was designed around requests, where process termination gives you that reset for free.

Testing removes that reset. Once you recognize that, a lot of mysterious WordPress test behavior stops being mysterious. A green test doesn't only need a clean database. It needs a known runtime, and in WordPress those are very different things.

We welcome pull requests, too!