Humans, bots, agents. Who owns this layer?

Published

Sep 2, 2026

Reading time

9 min

The morning looks normal. Conversion is down without a human-traffic spike. A “official” crawler has been collecting 403s for two days. A catalogue API is busy at an hour when the mobile app is asleep. Three tickets open. None of them contains the other two.

Security sees a challenge that “held.” SEO sees a crawl that went quiet. Product sees stock moving faster than real baskets. Each team has a console. Each has part of the truth. Nobody has the layer.

That is not a tooling gap. It is a structure gap.

Three faces, one door

A premium site no longer serves one visitor. It serves three, on the same edge.

Hostile. Scrapers on product pages, stuffing on login, inventory bots on stock. OWASP’s Automated Threats project names these as distinct events — OAT-011 Scraping, OAT-008 Credential Stuffing, OAT-021 Denial of Inventory — not as one “bot” blob. Mix them up and you punish the wrong flow.

Useful. Googlebot, crawlers that feed answers, shopping agents completing a journey a mobile app was designed for a person. Block them “to be safe” and you disappear from an index, then from an answer. Since 1 July 2026, Cloudflare classifies this traffic as Search, Agent, and Training: three intents, not one checkbox.

Internal. Unapproved tools, agents on your APIs, calls to a model that never made the security register. The edge already sees what arrives at the door. What leaves toward a model API often does not — until the invoice. Two doors. Still one missing owner.

Treat agents like attacks and the site goes dark in search. Treat attacks like customers and stock, accounts, and trust leave through the same door. The question is not “bot or not.” It is who decides allow, challenge, block — for all three.

The platforms shipped the knobs

Cloudflare Bot Management, the Search / Agent / Training controls, AI Gateway. Fastly Bot Management, AI Bot Management, inspection before cache and challenges after. Google is testing Web Bot Auth so some agents can sign requests instead of leaning on a User-Agent string.

This is real. It is useful. It is also another console, another line item, another team that will not pick up at 2 a.m.

Knobs are not policy. A rule that reads User-Agent contains bot → challenge is a knob. A signature is another: it proves who, not allow, challenge, or block. Trusted names get borrowed. Telling a signed agent from a headless browser wearing the same mask, letting a search crawler through while holding stuffing on /login, living with the result when SEO and security disagree: that is a job.

On 15 September, new Cloudflare defaults land for new domains. Training and Agent blocked on pages that show ads; Search allowed. Mixed-purpose crawlers take the strictest rule — block Training and search can go with it. The platform just made a policy choice. Someone in the account still has to live with it.

Platforms will keep shipping features. The rentrée question is not which box to tick. It is who is accountable when those features disagree.

What we claim

Optimi is not a new console. We do not host models. We do not replace Cloudflare or Fastly: they are partners, supply we operate.

We are the owner. One intelligent entry point in front of origin. DDoS, WAF, bot, API — managed, not merely licensed. MYO to see non-human traffic on non-sampled logs. A written policy: allow, challenge, block, by face. And someone who is still there when a rule that looked clean in staging meets peak.

We do not sell configuration. Platforms are making that self-serve. We sell judgment, and a throat to choke.

What to do now

Not a software demo. A read of your non-human traffic: what is allowed, what is challenged, what is blocked, and who owns those three verbs today.

If the answer is “a bit of everyone, so no one,” you have named the gap. That is where a POV starts.