Generating XML Sitemaps from Headless CMS Routes

Sitemap generators that traverse the filesystem or server routing table don’t work in a headless stack — there’s no filesystem to walk, only paginated API endpoints whose locale variants rarely match the final URLs. The sitemap becomes a compiled artifact built during the build or revalidation phase. This guide, part of Dynamic Sitemap Generation, covers the aggregation pipeline: cursor-paginated fetch with backoff, status filtering, and stream-based XML serialization that won’t exhaust memory on a large content graph.

The Core Challenge

Asynchronous publication is the root problem. You must filter unpublished nodes before route compilation, resolve slug collisions programmatically, and normalize trailing slashes before serialization. Extraction starts with a recursive content-type query over cursor-paginated GraphQL or REST endpoints, then a transformation layer maps internal IDs to canonical paths. A strict routing contract is what keeps orphaned URLs out and ensures crawlers only see valid endpoints.

Route Extraction & State Filtering

Draft and scheduled content pollutes production sitemaps unless you filter at the query level. Trigger regeneration only on a publish transition — CMS webhooks or ISR hooks, not routine deploys. Naive scripts break on rate limits, so batch aggressively, back off exponentially on transient failures, and cache intermediate route arrays. Parallelize locale fetches but serialize the final XML assembly to avoid race conditions.

Multilingual Routing & Hreflang

Each locale variant resolves independently, and fallback chains must evaluate before hreflang mapping. Inject alternate language tags into the <url> block as <xhtml:link> elements — appending them as query parameters violates the Sitemaps XML Protocol and dilutes crawl budget. Only list alternates for locales where the page is actually translated; fallback URLs that serve default-locale content belong neither in the sitemap nor in any hreflang set. This is core to Localization & SEO Optimization.

Implementation

This TypeScript pipeline handles cursor pagination, filters unpublished content, and enforces XML schema compliance. It uses the sitemap library for stream-based serialization, which avoids memory exhaustion on large content graphs.

import { SitemapStream, streamToPromise, EnumChangefreq } from 'sitemap';
import { Readable } from 'stream';

export interface CMSRoute {
  id: string;
  slug: string;
  locale: string;
  updatedAt: string;
  alternateLocales: { locale: string; url: string }[];
}

interface FetchOptions {
  apiEndpoint: string;
  token: string;
  maxRetries?: number;
}

async function fetchWithBackoff(url: string, headers: Record<string, string>, maxRetries = 3): Promise<Response> {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    const res = await fetch(url, { headers });
    if (res.ok) return res;
    
    const delay = Math.pow(2, attempt) * 1000;
    console.warn(`API rate limit or error. Retrying in ${delay}ms...`);
    await new Promise((resolve) => setTimeout(resolve, delay));
  }
  throw new Error(`Failed to fetch routes after ${maxRetries} attempts.`);
}

export async function fetchPublishedRoutes({ apiEndpoint, token, maxRetries = 3 }: FetchOptions): Promise<CMSRoute[]> {
  const routes: CMSRoute[] = [];
  let cursor: string | null = null;

  do {
    const params = new URLSearchParams({
      limit: '100',
      status: 'published',
      ...(cursor && { cursor })
    });

    const res = await fetchWithBackoff(`${apiEndpoint}?${params}`, {
      Authorization: `Bearer ${token}`,
      'Content-Type': 'application/json'
    }, maxRetries);

    const data = await res.json();
    const normalized = data.results.map((entry: any) => ({
      id: entry.id,
      slug: entry.slug.replace(/\/+$/, ''), // Normalize trailing slashes
      locale: entry.locale,
      updatedAt: entry.updated_at,
      alternateLocales: entry.alternateLocales || []
    }));

    routes.push(...normalized);
    cursor = data.pagination?.next_cursor || null;
  } while (cursor);

  return routes;
}

export async function generateSitemapXML(routes: CMSRoute[], baseUrl: string): Promise<string> {
  const stream = new SitemapStream({ hostname: baseUrl });
  
  // Transform CMS routes into sitemap-compatible objects
  const urlObjects = routes.map((route) => ({
    url: `/${route.locale}/${route.slug}`,
    lastmod: route.updatedAt,
    changefreq: EnumChangefreq.WEEKLY,
    priority: 0.7,
    links: route.alternateLocales.map((alt) => ({
      lang: alt.locale,
      url: alt.url.startsWith('http') ? alt.url : `${baseUrl}${alt.url}`
    }))
  }));

  const readable = Readable.from(urlObjects);
  readable.pipe(stream);

  const xmlBuffer = await streamToPromise(stream);
  return xmlBuffer.toString();
}

Integration Flow

The pipeline runs as an event-driven sequence from publish event to a purged, regenerated artifact.

1. Webhook Listener: Deploy a lightweight serverless function or Next.js API route that listens for content.published or content.updated events from your headless provider.
2. Route Aggregation: Trigger fetchPublishedRoutes() to pull the latest published state. The exponential backoff wrapper ensures resilience against provider-side throttling.
3. XML Serialization: Pass the aggregated array into generateSitemapXML(). The sitemap library serializes entries as a stream; for very large shards, pipe the stream straight to object storage or the HTTP response instead of collecting it with streamToPromise, so the whole document is never held in memory.
4. Cache Invalidation: Write the resulting XML string to your CDN origin or object storage (e.g., AWS S3, Vercel Blob). Purge the CDN cache for /sitemap.xml immediately after write completion.
5. Framework Hook: If using Next.js, tie this pipeline to revalidatePath('/sitemap.xml') or leverage on-demand ISR to keep the artifact synchronized without full rebuilds.

Pipeline Integration & Performance

Treat the sitemap as a compiled artifact, as in Dynamic Sitemap Generation, and run the pipeline in isolation from the frontend build to avoid deployment bottlenecks. Health-check XML validity before publishing — a schema validator or regex assertion on <urlset> compliance and namespace declarations.

Past 50,000 URLs or 50MB uncompressed, Google Search Central requires sharding. Shard by content type or locale and generate a sitemap index referencing each shard, with referential integrity across all of them.

Building alternates from translation groups

Hreflang alternates need to know, for each page, which other locales hold a genuine translation and at which URL. Depending on the localization model, that information comes from different places. With field-level localization, a page technically exists in every locale, so check whether its key fields, such as title and body, have values in each locale before listing that locale. With entry-level localization, translations are separate entries linked by a translation group or document id; fetch the group once and build the alternates from its members. In both cases, generate the alternates for every member from the same set, so the hreflang set is reciprocal by construction: if the German page lists the French one, the French page lists the German one. Store the resolved set per page during extraction, with the URL for each member, and use it for the sitemap, the page head and any audit, so all three agree.

Configuration Reference

Gotchas & Edge Cases

• Retrying client errors. The backoff wrapper above retries every non-OK response; a 401 or 404 will not fix itself. Retry only 429 and 5xx, and fail fast otherwise.
• Slug collisions. Two entries with the same slug in one locale produce duplicate URLs. Detect collisions during generation and fail loudly.
• Trailing slash policy. Normalize to the site’s canonical form, with or without a trailing slash, matching the router and canonical tags exactly.
• Timezones in lastmod. Use full ISO 8601 timestamps with timezone, as the CMS provides them. Dates without time are valid but lose precision.

Worked Example

A documentation site with 12,000 pages in three locales replaced a build-time sitemap plugin, which walked the output directory and listed preview pages by accident, with the event-driven pipeline. Shards per locale and section kept each file under 3,000 URLs, and only translated pages carried hreflang alternates. Regeneration after a publish took four seconds for the affected shard instead of a full rebuild, and the next audit found no drafts, previews or fallback URLs in any sitemap.

Rollout Checklist

• Fetch published routes with cursor pagination and targeted retries.
• Normalize slugs and detect collisions before serializing.
• Add hreflang alternates only for translated locales.
• Stream XML into shards below the size limits, with an index.
• Regenerate affected shards on publish and purge their cache.
• Validate XML and audit URLs before publishing a new sitemap.

Frequently Asked Questions

Related

• Dynamic Sitemap Generation
• Incremental Sitemap Regeneration for Dynamic CMS Routes
• Noindexing and Canonicalizing Fallback Pages