The line item is called “custom metrics overage” and I have watched grown engineering managers stare at it like it’s written in a language nobody in the room speaks.
Everything else on a Datadog invoice at least sounds like a thing. Hosts, fine, you have hosts, you can count them. Log ingestion, sure, you know you ship logs. APM, you can picture the traces. But “custom metrics: $6,400” just sits there with nothing behind it you can picture, no host count to hold onto, no gigabytes to feel in your gut, and the person who owns the budget has no idea whether that’s three metrics or three million, or which team made them, or whether cutting it in half would blind an alert that someone’s sleep depends on. So it survives every single budget review, year after year, purely because nobody in the room can see inside it.
Let me tell you what’s inside it. It’s almost always one tag.
A metric is a name times its tags
Here’s the mechanic, and I promise once you see it you can’t unsee it on any bill you’ll ever look at again.
In Datadog a custom metric is not “a thing you measure.” It’s one unique combination of a metric name, a host, and every tag value hanging off it. So checkout.latency is not one metric. checkout.latency broken out by region (say four values) and service (say ten) is forty metrics, one per combination, and that is completely fine, forty is a rounding error. Datadog’s Pro plan includes 100 custom metrics per host and Enterprise gives you 200, pooled across the whole fleet, so with fifty hosts you’re sitting on a bucket of five or ten thousand before a single cent of overage, and forty barely scratches it. Trouble starts the afternoon someone adds a tag whose values don’t stop coming. Tag that same checkout.latency with user_id, mid-incident, because for twenty stressful minutes you genuinely needed to see latency per customer, and if you have eighty thousand users you have just turned forty metrics into 3.2 million, each one billed as its own custom metric at five dollars per hundred per month, and that tag is still quietly sitting there in October because nobody remembered to pull it back out in April, by which point it has quietly cost you more every single month than the engineer who added it takes home in a week.
That’s the whole disease. Cardinality is just the math word for “how many unique values does this tag take,” and the unbounded ones, the user_id, the request_id, the pod_name that gets a fresh hash every deploy, the container_id, the raw URL path with the order number baked right into it, are the tags that turn a rounding error into a mortgage payment while you’re looking the other way.
The ones that get everyone
Some tags are cardinality landmines, and you can learn the list in about thirty seconds, so here it is.
Anything that’s an identity: user_id, customer_id, session_id, trace_id. Anything ephemeral that Kubernetes regenerates on you: pod_name, container_id, the replica hash baked into a deployment. Anything with a raw number stuck into it, like a URL path of the form /orders/558213/items, or an email address, or a full SQL query string shoved in as a tag value. Every one of these feels totally harmless the moment you add it, because in that moment you’re staring at one incident and one user and it is, right then, genuinely useful. The cost never arrives in the moment. It arrives as a monthly average, quietly, three weeks later, on a bill that will not tell you which tag did it to you.
I once traced a client’s entire custom-metrics overage, low five figures every month, to a single tag some well-meaning engineer had bolted onto a request-duration metric: the full request path, order IDs and all. One tag. It had been sitting there for seven months. We deleted it in about four minutes and the next invoice dropped by more than a junior engineer’s monthly pay, and to this day I don’t think the person who originally added it ever found out they’d done it.
Finding yours
You don’t need a project for this, or a task force, or a quarter. You need one afternoon and Datadog’s own metrics summary page, which will cheerfully rank every metric you have by cardinality if you sort the column, and the top of that list is exactly where all your money went.
Go there and sort by cardinality, descending. Look hard at the top ten, and for each one ask a genuinely simple question: do we ever actually query, alert, or dashboard on this at this level of granularity? Not “could it conceivably be useful someday,” because everything could conceivably be useful someday, that mindset is exactly how you got here. Do you, in real observed practice, page a human or draw a single graph that needs latency split out per individual user? For the expensive ones the honest answer is almost always no. So you drop the tag, or you reach for Metrics Without Limits, which lets you keep ingesting the raw high-cardinality thing while only indexing, and paying standard overage on, the specific tag combinations you truly use, and everything else falls back to a flat ten cents per hundred metrics ingested. Configured with care it’s a real lever. Configured carelessly it’s just a second confusing line on the same bill, so choose the handful of tags you genuinely query and index only those.
The reason this works, and the reason it’s honestly a little satisfying every time, is that cutting cardinality costs you almost nothing operationally. You are not switching off monitoring. You’re deleting a per-user breakdown of a metric that no living person has looked at per-user since the 3am incident that spawned it ten months ago, from an engineer who has since changed teams and forgotten the tag exists. Reliability doesn’t drop a single millimeter. But the bill drops hard.
Sort your metrics by cardinality this week, stare hard at the top ten, and for every one you can’t immediately justify keeping at that granularity, either drop the tag or hide it behind Metrics Without Limits, and you’ll very likely knock a four-figure number off next month’s invoice without a single alert going quiet.
The high-cardinality cost trap is one of three OTel pitfalls I break down here.
— Youn