Tracking Schema Change Lead Time from CMS to Production

This guide, part of DX & Developer Experience Metrics, shows how to measure one number that says a lot about a headless integration: how long it takes for a change to the content model to be usable in production. It covers what to count, how to collect the timestamps automatically from the repository and the deploy pipeline, and which steps usually dominate.

Schema change lead time measures coupling. When the model lives in code, types are generated, migrations are scripted and CI checks compatibility, a new field can be in production within the hour. When the model is changed by hand in a web UI, types are updated manually and someone clicks through preview to check nothing broke, the same change takes days and blocks other work while it waits. Tracking the number over time shows whether the integration is getting easier or harder to change.

The Problem

A team building a product catalogue felt that model changes were slow but had no numbers. Some changes seemed to go out the same day; others took a week. Discussions about improving the process went in circles, because everyone remembered a different recent change. The engineering manager wanted to know whether investing in type generation and scripted migrations would pay off, and how to tell afterwards whether it had.

How to Measure It

Define a model change as a pull request that touches the model definition, for example the migrations folder, schema files or a generated types file. For each such pull request, record four timestamps, all available from systems the team already uses:

1. Opened, from the git host.
2. Merged, from the git host.
3. Migration applied to production, from the CI job that runs migrations, or the time the model was changed through the CMS’s API.
4. Live, from the deployment platform, the first production deploy containing the merge commit.

Lead time is live minus opened. The intermediate timestamps split it into review time, pipeline time and release time, which is where the insight lies. A long review time points at unclear ownership or large changes; a long pipeline time at slow codegen, tests or manual steps; a long release time at batching or release schedules.

Implementation

A small script run after each production deploy finds merged model changes included in the deploy and records their timestamps. The example uses the GitHub API and a deploy webhook; other git hosts and deploy platforms have equivalent data.

// scripts/record-schema-lead-time.ts: run by the deploy pipeline after a successful production deploy
import { Octokit } from "@octokit/rest";

const MODEL_PATHS = [/^migrations\//, /^cms\/schema\//, /^src\/generated\/cms-types\.ts$/];

interface DeployInfo { sha: string; previousSha: string; liveAt: string }

export async function recordLeadTimes(deploy: DeployInfo, emit: (row: Record<string, unknown>) => Promise<void>) {
  const gh = new Octokit({ auth: process.env.GITHUB_TOKEN });
  const [owner, repo] = (process.env.GITHUB_REPOSITORY ?? "").split("/");

  // Commits in this deploy that were not in the previous one.
  const { data: diff } = await gh.repos.compareCommits({ owner, repo, base: deploy.previousSha, head: deploy.sha });

  for (const commit of diff.commits) {
    const { data: prs } = await gh.repos.listPullRequestsAssociatedWithCommit({ owner, repo, commit_sha: commit.sha });
    const pr = prs.find((p) => p.merged_at);
    if (!pr) continue;

    const { data: files } = await gh.pulls.listFiles({ owner, repo, pull_number: pr.number, per_page: 100 });
    if (!files.some((f) => MODEL_PATHS.some((re) => re.test(f.filename)))) continue;

    await emit({
      metric: "schema_change_lead_time",
      pr: pr.number,
      opened_at: pr.created_at,
      merged_at: pr.merged_at,
      live_at: deploy.liveAt,
      lead_time_h: (Date.parse(deploy.liveAt) - Date.parse(pr.created_at)) / 3_600_000,
      review_h: (Date.parse(pr.merged_at!) - Date.parse(pr.created_at)) / 3_600_000,
      release_h: (Date.parse(deploy.liveAt) - Date.parse(pr.merged_at!)) / 3_600_000,
    });
  }
}

The migration timestamp comes from the CI job itself: have the migration step emit its own event with the pull request number, so the dashboard can split release time into pipeline and waiting. Store the rows wherever the team keeps metrics, and chart the median and the 90th percentile per month. Medians show the normal experience; the 90th percentile shows the changes that got stuck.

Shortening the stages

Each stage has a typical fix. Review time shrinks with smaller changes and a named owner per content type who is expected to review within a day. Pipeline time shrinks with generated types instead of hand-written ones, scripted migrations run by CI against a branch environment, and automated visual checks instead of manual preview clicks. Release time shrinks with continuous deployment of the frontend, so a merged change goes live with the next deploy rather than a weekly release. The migration guide describes the expand and contract sequence that makes continuous deployment of model changes safe.

Presenting the metric to the team

Show lead time as a monthly chart of the median and 90th percentile, with a stacked bar underneath splitting the median into review, pipeline and waiting time. Annotate the chart with the changes made to the process, such as “types generated in CI” or “continuous deployment enabled”, so the effect of each is visible. Link each data point to its pull request, so anyone can open the slowest changes of the month and see what held them up. In practice, the slowest changes are the most useful part of the dashboard: they are usually large changes that should have been split, changes waiting on an unavailable reviewer, or changes that needed a manual step nobody had automated yet. Discussing the three slowest changes at a monthly retrospective keeps the conversation concrete and avoids arguments about averages.

Configuration Reference

Gotchas & Edge Cases

• Multi-step migrations. An expand and contract migration spans several pull requests. Count each separately; the expand step’s lead time is what matters for unblocking frontend work.
• Model changes made in the UI. Changes made directly in the CMS’s web interface bypass the repository and are invisible to the script. Treat that as a finding: the first improvement is usually moving the model into code.
• Draft pull requests. A pull request opened as a draft days before it is ready inflates lead time. Start the clock when it is marked ready for review, if your git host records that.
• Rollbacks. A deploy that is rolled back and redeployed should count the redeploy as live. Record the final successful deploy, not the first attempt.

Worked Example

The catalogue team recorded lead times for six weeks before changing anything: the median was 31 hours and the 90th percentile 4.5 days, with the manual type update and preview check as the largest stage. They generated types in CI, replaced the manual preview check with visual diffs of twenty key pages against a branch environment, and switched the frontend to continuous deployment. Six weeks later the median was 1.4 hours and the 90th percentile 9 hours, mostly review time on larger changes. The numbers settled the investment question, and the review stage became the next target.

Rollout Checklist

• Define model changes by file paths in the repository.
• Record opened, merged, migrated and live timestamps automatically.
• Chart monthly median and 90th percentile, split by stage.
• Fix the largest stage first: usually manual types, manual checks or release batching.
• Keep measuring after changes to confirm the improvement.

Frequently Asked Questions

Related

• DX & Developer Experience Metrics
• Measuring Preview Latency for Editors
• Migrating Content Models Without Breaking the Frontend