Why We're Building Open-Source Test Reporting

If you’ve set up test reporting for a real team, you already know the problem. You spend an afternoon evaluating tools, discover the ones that look good are expensive SaaS products, and the ones that are free haven’t had a commit in three years. You pick one anyway, wire up your CI pipeline, and eventually accept that your test results now live on someone else’s servers under pricing that will change the moment you scale.

We built SpecHive because we were tired of that trade-off.

The Problem With Existing Tools

The SaaS options are polished, and some of them are genuinely good products. But they come with a set of structural problems that don’t go away no matter how good the UX is.

First, the pricing. Most platforms charge per seat or per test run, which means your bill grows every time your team or test suite grows. That’s a perverse incentive. You should be adding tests and engineers without worrying about the reporting tab in your AWS Cost Explorer.

Second, your test data leaves your network. For most teams this is an acceptable risk. For teams in regulated industries, or teams that simply care about what data they expose to third parties, it is not. Test results contain more information than most people realize: endpoint URLs, database identifiers, internal service names, error messages that describe the shape of your system. Sending all of that to an external service is a choice that should be made deliberately.

Third, when the vendor decides to change their pricing, deprecate a feature, or shut down, you have no recourse. Your historical data, your dashboards, your integrations — all of it is hostage to a relationship you have no control over.

The open-source alternatives have the opposite problem. They are self-hosted, transparent, and free, but most of them are abandoned. The issues are unanswered, the UX hasn’t been touched in years, and critical features like flaky test detection simply don’t exist.

Our Approach

SpecHive is licensed under AGPL-3.0. The full platform — reporting, analytics, artifact storage, team management — is in the open-source repository with no features removed or paywalled.

We also offer a managed cloud tier for teams that don’t want to run their own infrastructure. The cloud version is additive: you get zero-ops setup, managed storage, and automatic upgrades. You do not get features that aren’t in the OSS version. If you ever want to migrate from cloud to self-hosted, you can export your data and move. We have no interest in building lock-in into the product.

This model works because the value of the cloud offering is operational, not functional. Running PostgreSQL, handling storage at scale, keeping the service up — that is genuinely useful work that we can do on your behalf. But it is not a reason to hold features hostage.

What SpecHive Does Today

The current release focuses on Playwright, which is where we started because it is the test framework we use ourselves.

When you connect your Playwright suite to SpecHive, you get a full test run history with pass/fail breakdowns, duration tracking, and environment metadata. Flaky tests are detected automatically through retry analysis — if a test passes on retry, SpecHive tracks that pattern across runs and surfaces tests that are failing intermittently before they become a reliability problem.

Artifacts are stored alongside results. Screenshots, videos, and Playwright traces are attached to each failing test and accessible directly from the run report, without needing to dig through CI logs or download a zip from your artifact store.

For teams, there is organization-level analytics: aggregate pass rates, slowest tests, most frequently failing tests across projects. CI environment is detected automatically for GitHub Actions, GitLab CI, Jenkins, CircleCI, and Azure DevOps, so your runs are tagged with branch, commit, and pipeline information without manual configuration.

What Is on the Roadmap

We want to be direct about what is planned versus what exists today.

Multi-framework support is coming. Vitest and Jest are the next two on the list. The reporter protocol is designed to be framework-agnostic, but the per-framework integrations are not yet shipped.

GitHub integration for pull request comments — posting a test result summary directly to the PR — is planned. This is one of the most-requested features and we are actively working on it.

Advanced analytics, including test suite health scoring and regression detection across branches, is further out on the roadmap. The data model supports it, but the surface has not been built yet.

Why Open Source Matters for Test Infrastructure

Test infrastructure occupies an unusual position in your stack. It sits outside your application, so it rarely gets the same scrutiny. But it touches every part of your development process and accumulates detailed information about how your system behaves.

That is exactly why it matters that you can read the code. When SpecHive collects a test result, you can inspect what data is captured, how it is stored, and what leaves your environment. You do not have to trust a privacy policy.

It also means you are not locked in. If SpecHive stops meeting your needs, you can fork it, modify it, or migrate your data to something else. Self-hosting with Docker Compose means the software runs on your servers under your terms. There is a single docker-compose.yml that brings up the full stack.

Community contributions are welcome and the project is structured to make them straightforward. If you need a feature and it fits the scope of the project, we would rather you open a PR than build a private fork.

Getting Started

The cloud tier is free up to 5,000 test results per month. Self-hosted is free with no limits, forever.

If you want to try the cloud version, you can create an account at app.spechive.dev. If you want to self-host, the repository has everything you need: SpecHive on GitHub.

For a practical walkthrough of wiring Playwright into SpecHive, see Setting Up Playwright Test Reporting in 5 Minutes.

We are building this in public. If something does not work the way you expect, open an issue. If you have a question about the architecture or the roadmap, start a discussion. The whole point of building in the open is that the conversation happens in the open too.