Datadog bills your logs on two separate meters, and the second one is the one that empties the account.

Everybody budgets for the first meter. You send logs, you pay per gigabyte, roughly ten cents, and you can do that math in your head. The second meter is indexing, and indexing is priced completely separately from sending, which means that the privilege of actually being able to search the logs you already paid to ingest costs you again, per million events, for as long as you keep them queryable, and the default retention window of 15 days is the one almost nobody ever changes. Last time I priced it that second meter was around $1.70 per million events. So you can ship the exact same volume you shipped last quarter, change nothing, touch nothing, and still watch the line item climb, because the cost lives in indexing and nobody on your team is looking at indexing.

I had a client last year, Series C, fintech, who pinged me at 11pm on a Sunday because logs had hit 60% of their Datadog bill and finance had started asking pointed questions in the eng channel. They swore they hadn’t changed their logging. They were right. They hadn’t. What they’d done was add about a dozen services over six months, every one of them indexing 100% of its output by default, because that is the default, and no one ever went back to turn it down. The bill never spiked. It crept. Creeping is worse, honestly, because there’s no single moment to point at and nothing obvious to roll back.

Here’s the rough shape, and I’m rounding:

50M log events/day, indexed at 15 days:
  50M × $1.70/M   ≈ $85/day   ≈ $2,550/mo   (indexing)
  + ingest                     ≈ $900/mo
  -------------------------------------------
  ≈ $3,450/mo, most of which you will never open

That “never open” part is the whole game. Teams index everything and then search maybe 5% of it. You are paying full freight to keep the other 95% instantly queryable so that on the rare day you need it you don’t have to wait thirty seconds. It feels responsible. It’s the most expensive insurance policy in your stack and the claim almost never gets filed.

What to actually do about it

You don’t have a logging problem. You have an indexing problem, and the fix is to get ruthless about what earns an index.

Split sending from indexing, first in your head and then in the config. Send everything, ingestion is cheap. Then use Datadog’s index filters and exclusion rules to keep the junk out of the searchable tier: health checks, readiness probes, the 200s from your load balancer, every line that exists purely so some dashboard can claim full coverage.

If you do one thing, do this: pull the per-service log volume report and look at the top three talkers. It is always two or three services generating most of the volume, and it’s almost always something stupid. A health check logging every second. A retry loop writing each attempt at INFO. My favorite, from an actual incident, was a service that logged the entire request body on every upload, base64-encoded PDFs and all, until a single endpoint was producing more log volume than the rest of the platform combined. Fix the top three and the bill drops without anyone losing a log they’d have missed.

Then tier whatever’s left. Hot and searchable, 3 to 7 days, for the logs you actually reach for mid-incident. Everything else gets archived to cheap object storage (S3, usually) and rehydrated on the rare occasion you need to go spelunking through last month. Datadog will sell you Flex Logs for the middle ground, and whether that’s worth it comes down to how often you really query old data, which for most teams is “almost never, but very loudly when they do.” And sample the high-volume low-value stuff while you’re at it: you don’t need every access log, you need a representative slice plus every single error, because the errors are the only ones anyone greps at 2am.

Why it happens at all

None of this is hard, which should tell you the problem isn’t technical. The log bill is overspent at basically every team I walk into for one reason: it belongs to nobody. An engineer adds a log line to debug something on a Tuesday, ships it, forgets it exists, and it runs forever. Multiply that by every engineer and every Tuesday for two years. Finance watches the total go up and to the right, asks engineering what’s happening, and engineering shrugs, because no single person can see the whole picture and no single person’s name is on it.

So put a name on it. One person, responsible for the ratio of what you ingest to what you index, looking at it once a month the way they’d look at any other cloud line that’s climbing. That’s the entire intervention. It’s boring and it works, which is true of most of the things that actually save money.

(One caveat before you take any of these numbers into a board deck: Datadog reshuffles its pricing and packaging often, so confirm the current rates yourself. The mechanic, pay to send and then pay again to search, is the part that hasn’t changed and isn’t going to.)

If your observability bill is outrunning your traffic, I went deeper on where the money actually goes in my reliability costs guide.

— Youn