A daily question lands in most ops inboxes: how much of yesterday's traffic was actually people? Your access logs already know. They also arrive faster than anyone can read them — a few hundred thousand lines on a quiet site, tens of millions on a busy one. The fix is not another dashboard nobody opens. It is a small, boring script that runs before you wake up, asks Elasticsearch a handful of precise questions, and mails you the answers. Here is how to build that, from log shipping to the morning digest.

Start with the questions, not the code

A useful bot report is defined by the decisions it supports, not by how many charts it contains. Agree on the output before writing a single query. For most teams, six lines of numbers are enough:

• Bot share of total requests, with the previous seven days beside it for context
• Top user-agent strings among bot traffic, grouped into families rather than listed one by one
• Most-targeted paths, which usually means login pages, search endpoints, checkout, and API routes
• Bots that received 4xx or 5xx responses, and which hosts served them
• Distinct source IPs per family — five IPs causing most of the load is a very different problem from five hundred
• An unclassified bucket, which you should read every day for the first month

Write these down as a list of questions in a text file. The script becomes a translation of that file into aggregations, and the report stays honest because you can still explain every number in it.

Get the fields you need into Elasticsearch

If you already ship nginx or Apache logs with Filebeat's built-in module, most of the work is done: url.path, http.response.status_code, source.ip, user_agent.original, and @timestamp will be there. Two decisions matter more than the rest.

First, keep the raw user-agent string. Parsed fields are convenient, but when a new crawler shows up you will want the exact text to build a rule from. Second, decide where classification happens. Doing it at ingest time with an ingest pipeline and a lookup file keeps queries cheap and the report fast. Doing it at query time with the client or a runtime mapping lets you reclassify a year of history after you fix a bad rule. Teams that backfill often should classify at query time; teams querying billions of documents should not.

Keep index names predictable

Daily indices such as logs-nginx-2025.04.12 let the report pin itself to yesterday's data instead of scanning the whole cluster. If you use data streams or ILM rollover, use a date range filter instead and make sure the report is explicit about the timezone it is filtering in.

Classify bots without fooling yourself

Substring matching on the user-agent is the whole classification strategy for a lot of teams, and it works right up until it doesn't. Honest crawlers announce themselves; scrapers copy a browser string and move on. Split traffic into three buckets and label them clearly in the report:

• Verified — search and cloud crawlers confirmed by reverse DNS plus a published IP range
• Declared — the string says bot, but nothing confirms the source
• Unknown — everything else, including empty user-agents and headless browsers

Verification is a short sequence, and it is worth automating rather than eyeballing:

1. Resolve the source IP to a hostname.
2. Resolve that hostname back to an IP and check that it matches the original.
3. Confirm the hostname ends in the domain you expect for that crawler.
4. Check the IP against the operator's published range list.

Do this for a sample of traffic, cache the results in a small lookup index keyed by IP, and refresh the cache daily. Never verify every request at query time; you will hammer DNS and slow the report to a crawl. Keep the rules in a versioned file — one line per rule, with the date added and a short note about who added it. Six months from now that note is the only thing that will save you.

The Python side: four aggregations and a template

One dependency, the official elasticsearch client, and one file is enough. Structure it as separate functions: one returns request counts by family, one returns top paths, one returns error counts, one returns IP cardinality. Each sends size=0 with an aggs block and ignores the hits entirely.

The aggregations themselves are ordinary. A terms aggregation on the classification field gives the headline split. A date_histogram with an hourly interval shows whether the spike was a steady crawl or a two-minute hammering. A terms aggregation on url.path, wrapped in a filter for non-human traffic, gives the target list. A cardinality aggregation on source.ip tells you how concentrated the traffic is. Add a filter aggregation for status codes above 399 and you have the whole report.

Two habits pay off quickly. Save the raw aggregation response to disk as JSON before rendering anything, because when a number looks wrong you will want the original response rather than a formatted email. And use a composite aggregation if you need more than a couple of dozen families; raising the size parameter into the thousands is a good way to make the cluster unhappy.

Render with a template engine or plain string formatting into HTML mail. Keep it to one screen: a table, a short bar for the seven-day trend, and a couple of sentences of plain commentary written by the script, such as the largest single source and the most-hit path. The commentary is what people actually read.

Schedule it, and make it repeatable

A systemd timer or a cron entry at 06:15 is plenty. Pass the date as an argument — --date=yesterday — so backfills are a command rather than a code change. Store timestamps in UTC and convert once, explicitly, with a named timezone; timezone bugs are the most common reason a report quietly reports the wrong day.

Make the job idempotent: the same date produces the same output file, overwritten each run. Send it to one place. A digest that arrives in three channels is a digest that gets filtered.

Watch for these failures

• An empty or missing index that produces a cheerful report full of zeros — alert on document count, not on exit code
• Index rotation or a renamed field breaking the query silently
• Clock drift and daylight saving changes shifting the reporting window
• Mapping growth from adding a new field per rule instead of reusing one

A first week plan

Day one, confirm the fields exist with one small query and print ten documents. Day two, write the classification file with the thirty rules you already know by heart. Day three, add the four aggregations and dump JSON to stdout. Day four, render the HTML and send it only to yourself. Day five, schedule it and backfill the previous week so you can see whether the numbers move the way you expect. Then spend a week reading the unclassified bucket and adding rules as new crawlers appear.

The goal is a report so predictable you would notice immediately if it stopped arriving. Boring, on time, and correct beats clever every single morning.

Photo: Daniil Komov / Pexels