Comparison · target: schemathesis vs json schema faker
Schemathesis vs json-schema-faker for Contract test fixture
User compares a property-based API tester with a schema-based data generator.
Direct answer
For the Contract test fixture, Schemathesis and json-schema-faker are built for solving different halves of the problem, not for 10 constraints that decide whether the output loads.
The comparison holds for a schema of this size — 2 tables — and does not claim anything about runtime, community size or pricing.
A compatibility table below scores Schemathesis and json-schema-faker against the same Contract test fixture; the foreign key row is decisive, because a document carries a dangling reference without any tool objecting until load time.
Data as of 2026-11-18 · source: 0 Stack Overflow questions, json-schema-faker at 4,376,957 monthly downloads · negative-case messages reproduced by executing the fixture against a local SQL engine (Node.js v24.22.0, engine 3.53.4) at 2026-11-18T11:53:54.233Z
The scorecard — Contract test
The scorecard is scored against one shared fixture: Contract test fixture, 2 tables and 7 columns. Each row states what a tool has to do for the output to load, and the note column says which property of the fixture decided the row. Rows are not weighted by preference; a row is decided by whether the output would load.
Where Schemathesis wins
Schemathesis is the better choice when a running service exists: it checks live responses against the contract, which catches drift that no offline fixture can see. The Contract test fixture is the shared reference for every row in the table above.
Where json-schema-faker wins
json-schema-faker is the better choice when the output is a single document: it reads the schema properties and generates a value for each one, which is what a unit test needs when the data carries no relationships. Neither of the two reads 10 constraints of this fixture as a database contract, which is the part a seed has to satisfy before a single row can be inserted.
How the rows were decided
Every row in the scorecard is decided by a property of the fixture, not by reading documentation. For Contract test fixture, the deciding fixture properties were the endpoints to specs reference, the specs.format range declaration, the NOT NULL columns. A tool that misses any one of those produces output that a database will refuse.
A note on what this comparison does not cover
This comparison measures output correctness on one fixture. It does not measure runtime, community size, documentation quality or long-term maintenance, and it does not claim that either tool is wrong for your project. It also does not compare pricing, because every row is decided by the schema rather than by a price list: what a tool charges says nothing about whether its output loads.
| Criterion | SchemaSeed | Document generation | What decided it |
|---|---|---|---|
| Reads a multi-file OpenAPI document | yes | partial | both resolve local $ref chains |
| Honours NOT NULL | yes | partial | both read the required array |
| Honours enum, range and length declarations | yes | partial | the fixture declares specs.format with format IN ('openapi','swagger'); the declaration is read out of the schema, the DDL carries it as a CHECK constraint and the value that breaks it ships as a negative case |
| Honours enum members | yes | partial | the enum array is copied verbatim |
| Produces SQL, not just JSON | yes | no | output format is the difference |
| Resolves foreign keys across schemas | yes | no | the fixture has endpoints.spec_id pointing at specs, and the second tool never reads that table |
| Emits deliberately invalid instances | yes | no | negative cases are a separate output |
| Same seed produces the same bytes | yes | yes | both are seedable |
| Check | Result on this fixture | Consequence |
|---|---|---|
| More than one table references another | yes — endpoints → specs | the generator must resolve cross-table references |
| Output has to be SQL, not JSON | yes — the target is PostgreSQL | a document generator produces the wrong artefact |
| Tests must assert the API rejects bad input | yes — a negative case for every declaration in the schema | negative instances have to be generated separately |
| Same seed must reproduce the same bytes | yes — seed 42 is fixed | fixtures can be committed and diffed |
npx --yes @wenlu8848/schemaseed generate --input ./cmp_contracts.openapi.yaml --out ./seed --formats pytest --count 10 --seed 42
Download cmp_contracts.openapi.yaml
# Generated pytest suite
Seed: 42 — mode: nested
Run it with:
```
pytest pytest/
```
The suite is self contained: `_validator.py` implements the JSON Schema
keywords the generated data uses, so only pytest is needed.
- `test_specs.py`: 10 valid rows, 11 negative cases (enum-violation, format-violation, missing-required, null-violation, type-mismatch)
- `test_endpoints.py`: 10 valid rows, 24 negative cases (enum-violation, format-violation, missing-required, null-violation, type-mismatch)
Frequently asked questions
- Which one should I pick?
- If the output has to load into a database and the schema has relationships, take the schema-to-SQL tool. On Contract test fixture the deciding constraint was endpoints.spec_id.
- How many criteria were compared?
- Eight, each decided by a property of the Contract test fixture: 2 tables and 7 columns.
- Was this a benchmark?
- No. It is one 2-table fixture of 7 columns measured on whether the produced rows satisfy the schema, not on speed, memory or price.
- Do both tools produce deterministic output?
- Both are seedable, which is why the Contract test fixture rows can be committed and diffed; on this fixture that is the one property the two share.
Run it against your own schema
SchemaSeed is MIT licensed and free: unlimited local generation, no account, no telemetry and no usage limit. It reads your schema file on your own machine, so nothing is uploaded anywhere. If it saves you time, you can support the project.
Get SchemaSeed — free Support the project
Questions: support@schemaseed.drz8848.win