Publisher architecture guide
Media Edge Architecture: Traffic Spikes, Ad Revenue and Crawl Budget
A breaking-news event tests cache freshness, origin capacity, paywall boundaries, third-party dependencies, and crawler policy at once. Design them as one controlled system.
On this page
Publisher resilience is not a single CDN configuration. Hot articles, cache misses, personalised paywalls, third-party ad tags, images, bots, and crawlers compete for the same delivery and origin path. Separate public availability from customer-specific state, then make the dependencies observable.
Overview
Outcome
Define a media delivery architecture that protects public article availability during spikes without leaking entitlement, hiding content from intended crawlers, or making an ad-tech outage the article outage.
Classify editorial traffic before caching it
Use separate contracts for immutable assets, public article body, fast-changing public elements, and authenticated or entitlement-dependent responses. RFC 9111 permits request collapsing and carefully controlled stale content for compatible public representations; it does not permit an undifferentiated “cache the site” rule.
- Reader and crawler
Public article and assets request the delivery path.
- Edge cache
Reusable article representations absorb the hot path.
- Entitlement boundary
Paywall, account, and consent state remain private.
- Origin and third parties
Publishing, ads, analytics, and media fail independently where possible.
Public articles should degrade independently from personalised, third-party, and origin-dependent components.
| Component | Delivery rule | Failure behaviour |
|---|---|---|
| Public article and image | Explicit public cache contract | Serve approved fresh or bounded stale representation |
| Paywall and account | Private, authorisation-aware response | Do not reuse across readers |
| Ad and analytics tags | Governed external dependency | Preserve readable editorial page when dependency fails |
robots.txt and sitemap | Stable, monitored public path | Detect crawler access failures quickly |
Isolate paywalls and ad-tech failures
Keep a cacheable article representation distinct from a reader-specific entitlement decision where the product permits it. For indexed paywalled content, use Google's documented paywall markup rather than serving different content deceptively. Third-party JavaScript can change outside a publisher's release process; inventory it, assign owners, control tag changes, and monitor safe fallback states.
article=/news/event
cache_state=warm origin_fetch_rate=within-budget
paywall_response=private ad_dependency=fallback-ready
googlebot_path=verified-and-allowed sitemap=200
stop_condition=origin-errors or incorrect-entitlement responsePreserve crawl access without trusting user agents
Google says crawl capacity reacts to server health; slow responses and server errors can reduce it. Keep the URL inventory, canonical state, redirects, sitemaps, and response quality legible. robots.txt controls crawling, not reliable removal from search. Verify claimed Googlebot by reverse and forward DNS or published IP ranges, not a User-Agent string.
Image delivery needs its own checks: Google supports image sitemap URLs on CDN domains and recommends ordinary HTML images, stable URLs where practical, useful context, and descriptive alt text. CSS background images are not an image-indexing substitute.
Troubleshooting
Publisher incident traps
- Expiring a hot article before the replacement is cacheable and verified.
- Caching entitlement, consent, or personalised ad decisions in a shared response.
- Treating
robots.txtas access control or a de-indexing mechanism. - Allowlisting a crawler from user agent alone or letting a WAF silently challenge it.
Related guides
Authoritative references
- RFC 9111: HTTP Caching
- Google: Manage crawl budget for large sites
- Google: Paywalled content structured data
- Google: Image SEO best practices
- OWASP Third-Party JavaScript Management Cheat Sheet
Prepare the publisher path before the headline breaks
Optimi can help align caching, origin protection, crawler evidence, and third-party dependency controls around media events.
Review media delivery