Test Data Management: Synthetic, Masked, or Seeded?

Test data management is the practice of getting the rows, files, and accounts a test suite runs against into a test environment: safe to use outside production, structurally correct, and available before the test that reads them runs. It covers three separate problems that get treated as one in most write-ups: where the data comes from, whether it's safe to use somewhere other than production, and whether it still matches the schema by the time a test reads it.

A suite that logs in as a seeded user, submits a seeded order, and checks a seeded inventory count can break in three unrelated ways when any one of those rows goes stale. A login returns 401 because the fixture's password hash uses an algorithm the app dropped last sprint. An order references a product ID that got deleted from the schema's foreign key target but never removed from the fixture. An inventory count fails an assertion because a different test consumed a unit first and nobody isolated the two runs. None of these are the feature failing. They're the data failing quietly underneath it, and they cost exactly as much debugging time as a real defect before anyone realizes the code was fine.

Where does test data come from?

Three sources, and they solve different problems rather than competing for the same job.

Most teams that get this right don't pick one. They mask a production subset for the tests that need real-world shape, generate synthetic rows for volume and for cases production hasn't hit yet, and keep a small hand-seeded set for the handful of tests that need one exact, unchanging row.

Free and open sourceCasely writes these cases for youAttach a spec and one file of your team's existing test cases in Claude. Casely copies your columns, names the gaps it found in the spec, and exports a single Excel file your tracker imports in one pass.Read the install docs→

How does masking actually protect the data?

Masking replaces a sensitive value with a realistic one, not a blank or a repeated placeholder. A masked email still looks like an email and still routes through validation. A masked name still passes a "must contain a space" rule. The two properties that make this work are determinism and referential integrity: the same source value should always mask to the same output value, and every table that references a masked row needs the mask applied consistently, or a join that worked in production returns nothing in the test environment.

The step most teams skip is discovery. Masking a field nobody flagged as sensitive isn't a technical failure, it's a step that never ran. A schema grows a new column with a customer's phone number in it, nobody updates the masking rule set, and that column ships unmasked into a test database that a wider group of people can query than could ever see production. Rebuilding the sensitive-field list after every schema change matters more than the choice of masking algorithm.

Why generate synthetic data instead of just masking?

Masking only works on data that already exists. A feature that shipped last week has no production rows to mask, so the only way to test it under load, or test the account that has 10,000 orders instead of the three a manual tester created, is to generate rows that never existed. Synthetic generation is also the practical answer for isolation: a suite that generates its own fresh rows per run doesn't collide with a parallel run consuming the same seeded account, which is the source of a specific kind of flaky test that looks intermittent but is actually two tests sharing one row.

The tradeoff is realism. A generator that fills a "signup date" column with any date in a valid range will happily produce an account that signed up after its most recent order, a combination production could never actually contain. Good synthetic generation encodes those relationships on purpose. A generator that doesn't gets tests that pass against combinations a real user would never produce, and misses the ones they would.

What breaks when the schema changes?

Hand-written fixtures are the most common source of this failure because nothing forces them to change when the schema does. Someone adds a NOT NULL column or a new foreign key, runs the migration, and every fixture file that predates it is now wrong: it fails outright, or worse, it seeds an incomplete row that only fails three tests downstream in a way that looks unrelated to the migration that caused it.

Two things reduce this. First, generate fixtures from the schema instead of writing them by hand, so a schema change means regenerating rather than hunting through files for every reference to the changed table. Second, treat a fixture file the same way you'd treat generated code: never hand-edit it to patch around a schema change, regenerate it, because a hand patch is exactly the kind of drift that caused the problem in the first place.

Where this breaks

Masking is only as good as the field inventory behind it, and that inventory goes stale the same way fixtures do, quietly, after a schema change nobody re-audited. Synthetic data is only as realistic as the correlations someone bothered to encode, and a generator with no domain rules produces data that passes tests a real account would fail. Seeded fixtures are fast and deterministic right up until two tests run in parallel against the same seed and one mutates a row the other depends on, which produces a failure that looks like a flaky test but is actually a data ownership bug.

None of these three approaches decides what a test suite needs to cover. They decide how the data underneath that coverage gets there safely and stays correct. Casely generates the test cases themselves, the titles, steps, and expected results derived from a requirement, and stays out of the layer below that: it doesn't provision, mask, or seed the data those cases run against. That's a separate discipline with its own tooling, and treating it as an afterthought to test case writing is how a suite ends up debugging its fixtures instead of its features.