Tools I stole from work — New Relic
Second in the Tools I Stole From Work series: putting work-grade observability under a home network, and trimming the free tier when it hit 100GB.
Second in the series. Last time it was Playwright. This time it’s New Relic, which has been sitting underneath my homelab for the better part of a year and is almost invisible — which is what I want from an observability stack.
What I stole§
New Relic’s free tier gives you 100 GB of ingest a month and a single full-user seat. At work I’d been using it for backend services where the ingest volume was managed by a careful sampling plan; at home I assumed the volume problem would solve itself, because — how much data could a homelab possibly emit?
The answer, of course, is a great deal of it, if you let every process on every host phone home every fifteen seconds.
The shape§
The setup is unfussy:
- The infrastructure agent runs on every host that does meaningful work. Six of them, mostly: the Unraid box, two Mac-shaped servers, a Pi backup target, a small Ubuntu box, and a Debian arm64 thing I keep around for builds.
- Application data — request rates, error rates, request durations — feeds in from the handful of services I run that bother to emit them. The blog you’re reading is one of them; the gym-coach service for my son’s routine is another.
- Custom events I care about — backups completed, restore drills run, alerts that fired — go in through the events API.
Dashboards are deliberately boring. One per host, one per service, one for “the house” that draws from a handful of Home Assistant–backed metrics through a TimescaleDB middle layer. If a panel needs me to think about what it means, I rework it.
The week the free tier bit§
A few weeks in, the ingest dial started running hot. Closer to the 100 GB limit than I’d ever expected. The culprit wasn’t the services I’d been deliberate about — it was the infrastructure agent, doing what infrastructure agents do by default: emitting process-level metrics for every process on every host, every fifteen seconds.
At work you’d notice this because cost dashboards exist and someone owns them. At home I noticed it because the New Relic dashboard told me, in the same UI it tells me everything else, that I was about to fall off the free tier.
The fix was two changes, applied fleet-wide:
# /etc/newrelic-infra.yml
enable_process_metrics: false
metrics_process_sample_rate_seconds: 30Process-level metrics off entirely; the remaining sample rate halved. Ingest dropped to a comfortable steady-state inside the 100 GB cap with no perceptible loss of useful signal. The process-level data is the kind of thing you want when you’re debugging a specific incident, and you can turn it back on per-host for the duration. Carrying it always-on, for every process, for every host, was a vanity setting.
What I stole, more precisely§
The free tier didn’t teach me to do this. Cost-aware observability taught me to do this, at work, on a much bigger pipeline. The instincts that ported across:
- Default-off the noisy things. Process metrics, JVM internals, per-thread anything. They’re cheap to turn on when you need them.
- Sample, don’t drop. Halving the rate is almost always better than disabling a stream entirely. You keep the shape of the data.
- Know your ingest budget at all times. Either the platform gives you a dashboard or you build one. Don’t let cost be the thing that surprises you.
- Distinguish steady-state from incident. Steady-state telemetry is what you ship every day. Incident telemetry is what you turn on to investigate. Conflating them is how you blow your budget without making any decisions any better.
None of that is exotic. All of it would have been written down on the wall of any platform team I’ve worked on. At home, with no platform team, I am the wall.
What didn’t port across§
Two things I’d insist on at work, I haven’t bothered with at home:
- A formal alert taxonomy. The household alerts are a short list. I keep them in a Jira project, not in a runbook. Good enough.
- Cross-account ingest segregation. At work, different services live in different accounts so a noisy one can’t starve a quiet one. At home there’s one account. If a noisy thing starts spending the budget, I find out about it in a week and turn the knob.
The general point§
The free tier of a serious observability tool is genuinely good. The thing that lets you stay inside it isn’t the price — it’s the habit, learned at work, of treating telemetry as a budget rather than a default. Tools are cheap. Judgement about what to measure is what makes them work.
Next: a tool I started using at work in the last few months and brought straight home. Claude Code.