Bot mitigation
Telling automated traffic apart from real people, then blocking, slowing, or challenging the bad half without punishing everyone else.
See it
What it is
Bot mitigation is two jobs stapled together: deciding whether a request came from a person, then deciding what to do about it. Detection reads signals a script struggles to fake: TLS fingerprints (JA3, JA4), header order and casing, missing Accept-Language, headless browser tells, IP reputation and ASN (datacenter ranges are not where customers live), and behavior over time (300 product pages in 40 seconds, no mouse movement, signup completed in 0.4s). Responses range from silent block, to a challenge (Cloudflare Turnstile, hCaptcha), to tarpitting, to quietly serving a cached or slightly wrong response so the scraper never learns it was caught.
Reach for it on the endpoints where automation actually costs you: signup (fake accounts, free-tier abuse), login (credential stuffing against leaked password lists), password reset and OTP send (SMS pumping fraud), checkout (carding), search and pricing pages (scrapers), and any expensive AI endpoint. Not on your marketing pages, where you want Googlebot and preview crawlers to sail through, which is why any real setup keeps a verified good-bot allowlist.
Gotchas: CAPTCHAs tax your real users hardest, break for screen reader users, and cost attackers about a tenth of a cent per solve at a solver farm, so treat them as friction, not a wall. Blocking by user agent string is theater, since it is one line to change. robots.txt is a polite request, not enforcement, and the current wave of AI crawlers is the reason that stopped being funny. Watch your false positive rate as closely as your block rate: a blocked signup is a lost customer who never tells you.
Ask AI for it
Add layered bot mitigation to the signup, login, and password reset endpoints of this app. Layer 1: rate limit per IP and per account, with exponential backoff after failures. Layer 2: an invisible Cloudflare Turnstile challenge that only escalates to interactive when the risk score is high. Verify every challenge token server-side before the action runs, checking the expected hostname and action, the expiry, and that the token has not already been redeemed; a token that only gets checked in the browser is decoration. Layer 3: server-side heuristics scoring datacenter ASNs, missing Accept-Language, form-fill time under one second, and disposable email domains. Route high scores to a challenge rather than a hard block, and log every decision with the reason so I can audit false positives. Allowlist Googlebot and our uptime monitor by their published IP ranges, or by forward-confirmed reverse DNS where the PTR result is resolved back to an IP that matches. Never trust a user agent string, and never trust a PTR lookup on its own.