Four ways a before-and-after goes wrong
- The rule is chosen after the numbers. Pick the window and the threshold once you have seen the chart, and any release can be shown to have worked.
- The release day is counted. On the day a version goes out, some people use the old one and some the new. That day belongs to neither side.
- A busy week reads as a worse one. More traffic means more errors. A count with no denominator says nothing about the rate.
- Thin data reads as a trend. 3 events against 1 is a tripling. Below a minimum volume, the honest answer is “not enough to say”.
How Forge reads a release
- The plan comes first. Each work spec carries how the outcome will be measured — the metric, the two windows, the minimum volume and the change that counts — written by Forge from the issue’s counts and editable by the team, before anything ships.
- The release is read, not marked. Your tracker sends a release value with every row. The first production row under a new value is the moment that version reached your users; nobody has to remember to mark it. A deployment pipeline can also record it.
- The release day is in neither window. The baseline is the days before the release day, the after-window the days after it — 7 and 7 by default.
- Every figure carries its denominator. A share is stated as “91 of 9,800”, never as a bare percentage, and a window Forge does not fully hold is withheld with a sentence, never counted as zero.
- The answer is one of six readings. Observed improvement, no observed improvement, regression, mixed (parts that moved in opposite directions), insufficient data, or cannot verify — the one a person sets when a window never reaches its volume.
npm install @ambera/forge-tracking
import { createForgeTracker } from '@ambera/forge-tracking'
const forge = createForgeTracker({
token: 'ftk_…', // your product's tracking key, safe in page source
environment: 'production',
release: COMMIT_SHA, // how Forge knows which version sent each row
})
forge.track('signup_completed')
forge.captureException(error)An example
A release, 4f0a3c1, reaches production on 18 October. One error on the sign-up page was the reason for it. Forge’s reading a week later: “TypeError (reading ’email’): 91 of 9,800 (1 %) in 2026-10-19 to 2026-10-25, against 28 of 9,800 (0.3 %) in 2026-10-11 to 2026-10-17; +225 % relative; observed regression.”
The release day is in neither window, both windows hold the same number of sessions, and the sentence ends with what it is: an observation. The team reads it, reopens the issue and decides what to do next.
What a before-and-after cannot tell you
Forge says this beside every outcome, because it is true of every before-and-after:
- It is not proof the release caused the change. Something else may have moved in the same week.
- It is not an experiment. When you need to know a change caused something, split your users and test it.
- It is never “the fix worked”. Forge says observed improvement, and leaves the verdict to the team.
Questions
- Do I need to change how I deploy?
- No. Pass the release — a version or a commit SHA — to the tracker when you create it. Forge reads every new value in production as a release.
- What if traffic changed a lot between the two weeks?
- That is why every figure is a share with its denominator: errors per session, completions per entry. When a window falls under the plan’s minimum volume, the reading is insufficient data, not a number.
- Can I change the plan after the release?
- No. The plan is locked when the release is read, the way a test’s rule is fixed before its first count. A threshold chosen after the numbers decides nothing.
- What does it cost?
- Free for 1 product, no card: the tracking, the issues, the releases and their outcomes, up to 20,000 rows a day. Solo is $29 a month, flat, for 3 products and 100,000 rows a day each. Nothing is metered and there are no credits; the one paid addition to oversight is the model’s draft of a work spec.