One line in an access log rarely tells you much. A request for /pricing at 3:14 a.m. from an address with no referer and a string that looks like a browser could be a search engine, a scraper, a monitoring probe, or somebody's forgotten cron job. The pattern around that line is what identifies the caller. Years of tooling taught us to read logs for search engine crawlers; the traffic arriving now follows different rules, and the old filters will mislabel a good chunk of it.

Two families of traffic, one log file

Legacy bots arrived with a contract, more or less: announce who you are, fetch robots.txt first, respect Disallow and Crawl-delay, spread requests so you don't knock a small server over, and let the site owner verify your identity through reverse DNS. Search engine crawlers still largely behave this way, because their value depends on not getting blocked.

The crawlers now collecting pages for answer engines and model training inherited some of those manners and dropped others. Some check robots.txt; some read it and ignore the parts they dislike; a few never request it at all. Many identify themselves honestly in the User-Agent string and publish their IP ranges. Others arrive wearing a Chrome user agent. That inconsistency is the defining trait: identity and behaviour no longer line up the way they used to.

Check the user agent, then verify it

Names worth knowing in the User-Agent field: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, Applebot-Extended, meta-externalagent, Bytespider, CCBot, Amazonbot. On the traditional side you already have Googlebot, bingbot, DuckDuckBot, YandexBot, plus the SEO tools — AhrefsBot, SemrushBot, MJ12bot, DotBot.

Treat the string as a claim, not a fact. For Googlebot and bingbot, the standard test is reverse DNS followed by forward DNS confirmation: the PTR record should resolve to a hostname in the operator's domain, and that hostname should resolve back to the same IP. For newer crawlers, the reliable method is the operator's published IP list, which several vendors now serve as a plain-text or JSON file you can pull into a firewall rule or log filter. If an address claims to be a named crawler but sits outside the published ranges, you have found something worth blocking.

Header-level tells

Headers often expose a crawler that is pretending to be a browser. A real Chrome request over HTTPS typically carries Accept, Accept-Language, Accept-Encoding with br and zstd, a set of Sec-Fetch-* fields, and cookies on repeat visits. A bare HTTP client sends a thin set instead: often just Accept and Accept-Encoding, sometimes no Accept-Language, no Sec-Fetch headers, no cookies, and a Referer that is missing or points to a URL that does not exist on your site.

Other clues: HTTP/1.1 where your real users negotiate HTTP/2 or HTTP/3; a header order that does not match the browser named in the string; a TLS fingerprint inconsistent with the claimed client. None of these is a conviction on its own. Three together usually are.

Behaviour is the stronger signal

Headers can be copied. Request patterns are harder to fake, particularly at scale. Compare the profiles you will actually see:

• Traditional crawlers: fetch robots.txt first, then the sitemap; request a modest number of URLs per minute; revisit known pages on a schedule; honour crawl-delay where they support it; often request CSS and JavaScript when rendering; come from a stable, verifiable IP.
• Answer-engine and training crawlers: pull the full HTML of thousands of URLs in a short window; skip assets entirely; request pages with no inbound links; refetch the same documents weeks later; arrive in bursts from a rotating pool of addresses; occasionally ask for /llms.txt or similar files.
• Scrapers posing as browsers: spoofed user agent, no robots.txt request, sequential IDs walked in order, aggressive concurrency, and a cheerful disregard for 429 responses.

The tell that matters most is what the crawler asks for compared with what a visitor needs. A client that fetches every article but never an image, stylesheet, or script is not a person.

A step-by-step triage for a suspicious entry

1. Copy the IP and run reverse DNS. No PTR record plus a browser user agent is an early warning sign.
2. Check that IP against the published ranges for whatever crawler it claims to be.
3. Pull every request from that IP in the last 24 hours and look at volume, paths, and status codes.
4. Check whether the session requested robots.txt, and whether it respected Disallow.
5. Look for asset requests. Zero assets across thousands of page views means a parser, not a reader.
6. Compare the header set with a real browser's. Missing Accept-Language plus missing Sec-Fetch-* is telling.
7. Decide — allow, throttle, or block — then write the rule down so the next incident takes minutes instead of hours.

Where the grey areas live

Uptime checkers, link previewers, accessibility scanners, search-console verification fetches, and your CDN's own health probes all look faintly robotic. Block on user agent alone and you will eventually break something you rely on, which is why the order above matters: verify the IP, check the behaviour, then act.

One more complication. Some collectors deliberately fetch only a handful of pages per domain and spread those requests across many addresses over months. Those are nearly invisible in a single day's log and show up only in monthly aggregates. If you are deciding what to allow, review a full month rather than a busy afternoon.

Turning findings into rules

Once you know which names and ranges you care about, keep a small, boring routine going:

• Maintain an allowlist of verified crawler ranges and refresh it monthly; operators do change them.
• Alert on the shape of traffic, not just volume. A spike in 200s to article URLs with no asset requests is far more interesting than a generic traffic jump.
• Log the full User-Agent and Referer, and keep logs long enough to compare month against month.
• Block by IP range where you can. User agent blocking is trivial to bypass.
• Re-check your rules after every CDN or WAF change; bot policies often live in one layer and get dropped in another.

Do that and the next 3 a.m. request for /pricing will take you two minutes to explain.

Photo: Laura Musikanski / Pexels