Dyno Sniper

Dyno Sniper is our solution to the “noisy neighbor” problem that’s common when running apps on shared hardware. It complements autoscaling rather than replacing it: autoscaling sizes your fleet to handle traffic, while sniping addresses per-dyno problems on your web processes.

How sniping works

Judoscale already collects per-dyno metrics for every web app it autoscales. You can see per-dyno metrics for queue time or response time (depending on which metric you are autoscaling with) by switching the chart:

With sniping enabled, we keep watching those per-dyno metrics for outliers — dynos whose queue time or response time stays well above the rest of the fleet for a sustained period. When a dyno crosses that bar, Judoscale restarts it via the Heroku API — which typically gives you a fresh dyno on a different physical host, hopefully away from any noisy neighbor that could be causing the trouble.

Some design choices are worth calling out:

• We compare each dyno to its peers, not to an absolute number. A dyno is only an outlier if it’s noticeably worse than the fleet’s median. This means sniping adapts to your app — a healthy 5ms queue time and a healthy 30ms queue time both have their own “normal”, and sniping only fires on dynos that drift far above that normal.
• The signal has to be sustained. A single bad reading isn’t enough; the dyno has to stay above the sniping threshold across most of the recent window before we restart it. This filters out noise from short, one-off spikes.
• Same cadence as autoscaling. Sniping checks for outliers every 10 seconds, matching our autoscaling loop.
• No separate configuration needed. The sniping algorithm factors in your upscale threshold, so it inherits the sensitivity you’ve already tuned for autoscaling.

When sniping won’t trigger

Sniping is intentionally conservative. There are a few situations where Judoscale will hold off on restarting a dyno even if it looks like an outlier:

• Autoscaling is disabled. Sniping won’t fire if autoscaling is not enabled on the process.
• The app is in maintenance mode. Sniping is paused entirely while maintenance mode is on.
• Fewer than 3 dynos are running. Outlier detection needs peers to compare against, so the process must have at least 3 dynos before sniping can fire. If you typically run only one or two dynos, a noisy-neighbor scenario will usually play out like this: Judoscale scales your app up due to increased queue time, the third dyno reveals the outlier, Judoscale restarts it, and queue time drops back down. This is much better than the alternative of continuing to scale up due to a single poor-performing dyno.
• Judoscale just scaled the process up. When autoscaling has recently added dynos, queue time is usually expected to improve. Sniping defers until that scale-up has settled to have a more stable reading of the metrics before firing any restarts.
• Many dynos look bad at once. If a large share of your dynos are over the sniping threshold simultaneously, that’s almost certainly a broader problem rather than poorly performing dynos. Sniping might still detect potential outliers, but it skips the restart in that case so it doesn’t churn your fleet during a potential incident. Note that autoscaling might still trigger as usual.
• The dyno is too new. A dyno that just started up doesn’t have enough history to be a fair comparison yet, so it’s excluded from the sniping check until it’s been reporting metrics for the recent window.
• The dyno was just sniped. After a snipe, we hold off on restarting the same dyno again for a few minutes — long enough for the replacement to warm up and start reporting stable metrics.

Using Dyno Sniper

You’ll find the sniping option in your web process’s advanced settings:

Once enabled, sniped restarts show up as a row of markers on the scaling chart for that process, alongside the usual scale events. Hover over any marker to see which dyno was sniped:

Each snipe takes a single dyno offline while it restarts and your app boots back up. Your other dynos continue serving traffic in the meantime.

You can uncheck the toggle at any time to stop sniping for that process.

Tell us what you think!

Do you have ideas for how we can make Dyno Sniper better for you? Or maybe you want to share a success story? Email [email protected]. We love to hear from you!