Now, before you jump straight to “abandon ship!!”, there are real questions we should think about when looking ahead. Heroku is still an excellent platform, runs very stably, and, to this day, has the smoothest DX for getting an application into production. For those of us with production apps currently running on Heroku, we need to be pragmatic about what this announcement means for our present, our future, and our time!

Salesforce’s announcement should ultimately drive a calm, collected conversation around both timing and execution. Heroku isn’t a sinking ship, it’s just done shipping new features.

Let’s Be Honest About Urgency

Urgency itself is a function of two inputs: having a thing to do and believing that you must do that thing soon. The sooner you believe you must do it, the more urgent it will feel. So allow me to reiterate the point I made above and mix in some urgency:

Heroku is not dying today, tomorrow, next month, or next year.

It is not urgent that you migrate away from Heroku.

The Salesforce announcement might serve to give you the first component of urgency: we’ll all have a ‘thing to do’ at some point: migrate to another platform. But it certainly does not give the second component (‘do that thing soon’). Heroku isn’t going anywhere. And, if you recall the late two-thousand-teens, this isn’t even the first time that Heroku will spend some years running without major feature improvements! We sincerely believe it’ll be a few years before there’s any real pressing need to migrate off Heroku if you’re already successfully running your production app there.

I don’t want to come off like a Heroku shill here, so let me clarify why I’m pushing back against the hype and panic. It has nothing to do with Heroku’s bottom line or expensive servers. It has to do with your team’s time spent shipping useful features that will grow the value of your app and/or business.

Even in the best of circumstances and setups, migrating platforms takes time. It requires testing, planning, mapping, and careful execution to ensure that you’re not dropping traffic or upsetting customers along the way. It’s work. All of this work has opportunity cost: you won’t be building and shipping the features and enhancements that your customers want. You won’t be improving your application or business. At the end of the day, your customers don’t care how or where you host your app. They just want it to work and provide them value!

Okay fine but give me an actual recommendation here?

Sure. Deep breath. Let the panic subside: most applications currently running on Heroku shouldn’t worry about migrating until next year (2027) at the earliest. If you have an enterprise contract, you should renew it in 2026.

You already chose Heroku, you’re already setup on Heroku, your app is already running fine on Heroku. You should try to capitalize on those gains as long as possible (especially if you have enterprise/discount pricing!). “Heroku isn’t going to get any new major features” doesn’t actually prevent you from realizing the value of your initial investment into “I want managed hosting I don’t have to worry about”. Moving to another PaaS would still satisfy the “I want managed hosting…” but the migration itself is an additional investment and cost that you simply don’t need to make yet. Take a deep breath and go build your app / business! That does reap value today.

😮‍💨

Looking at the Alternatives

Nonetheless, I know many readers are still going to queue up migrations in the coming months. Maybe that’s discomfort, simply having time available to migrate, or a bad taste in the mouth. I get it! Even as I wrote the paragraphs above I felt some of those same tensions. Honoring those thoughts (and knowing that the future will come eventually) it feels worthwhile to talk through some of the migration paths an existing Heroku app has ahead.

We’re going to evaluate each option in three primary lenses:

• Migration effort: how painful it would be to migrate a full production Heroku app to this new setup
• Ongoing operational load: how it feels (subjectively) to use over time — things like CLI, “hop into prod console”, control and tweakability, etc.
• Cost structure: how expensive is this new setup compared to Heroku, and how is it billed differently?

Then we’ll give our general take on each path outside of those three parameters. Today’s challengers:

• Render
• Fly.io
• Railway
• Run-it-Yourself Systems

But today’s look isn’t our one time “here’s the truth” post, it’s just a preview. We’ll give you our opinions here today based on our work integrating with most of these platforms and running various apps on them over the last three years, but we’re planning on going deeper in the coming months: Judoscale is going on tour. More on that below, but we’ll be moving our 3,000+RPS production app to each of these platforms to really feel out what it looks like for a production app that can’t go down!

Render: The Obvious Choice

If you asked me for a simple, single-sentence recommendation for most teams, it’s going to be Render. Many folks have described Render as, essentially, “the natural progression of Heroku” — perhaps what Heroku could’ve become had it never been acquired by Salesforce. I think this is mostly due to Render sharing many of the same philosophies as Heroku (fully managed PaaS, auto build detection, etc.) but just having been built fresh many years after Heroku: the Render devs had the chance to reimagine the Heroku UX from the ground up with plenty of Heroku experience to draw from.

Migration effort. Any migration is going to take effort, but things are pretty smooth here. Heroku to Render is a well-trod path at this point and Render’s own team offers migration assistance for those coming from Heroku. The mental model is broadly the same and you’ll feel at home within a few minutes of logging into the Render dashboard. The only gotcha to keep in mind is around buildpacks and system dependencies. Render does supply some base-level buildpacks that should cover most apps, but if your app requires specific system dependencies beyond their included set, you may need to build out a Dockerfile. Where on Heroku buildpacks themselves can be composable, Render’s approach is simply, “stay on the rails or bring your own Dockerfile” (more here).

Ongoing operational load. Again here, this one’s going to feel just like Heroku. They handle the infrastructure, you just merge to main. Metrics and web dashboard UI are all friendly and available, logs can be pushed wherever you need, manual rollbacks are simple and accessible, there’s a broad CLI for control if you prefer that style, you can take your favorite autoscaler with you… the list goes on. Essentially everything you love about Heroku exists in Render in parallel or enhanced form.

Cost structure. Of all that platforms and paths we’ll look at today, Render’s cost structure and setup matches Heroku’s the most. Like Heroku, their pricing revolves around pre-set, per-month pricing depending on which instance types (e.g. “dyno type”) you need. Unlike Heroku, they’re actually clear about how many vCPU cores you’re paying to hold (🎉). In terms of real cost, our rough estimate is that, depending on the composition of your app and resources you need, you’ll likely save 20-30% off your current Heroku bill for similar resources on Render.

Our general takeaway on Render is that it’s the right choice for the grand majority of currently-on-Heroku apps. It’s a near-seamless transition, the billing operates the same, the operational overhead for engineers learning the new platform is very low, and most apps will be able to get up-and-running within a day.

Fly.io: A Little More Complicated, A Little More Interesting

Still mostly on the high-level-PaaS layer, Fly.io was built to accomplish a different goal. Fly’s whole thing is distributing your app geographically so that your users will always hit an application server close-by, and doing so with “Fly machines” — micro VM’s with much smaller footprints than full-on Docker containers. Fly is also heavily optimized for its powerful CLI and config tooling. Fly is tremendously flexible and configurable, but comes with the cost of complexity: a steep learning curve!

Migration effort. Like Render, Fly has written guides specifically for those migrating from Heroku, including framework specific guides in many cases (Rails, Django, FastAPI, Flask, etc.) to help explain nuances. And these guides are certainly helpful, but there’s no getting around the paradigm shift: Fly is a fundamentally different platform from Heroku and doesn’t operate quite the same. There is going to be a learning lift as you get familiar with its UI tooling and flyctl CLI tool — the latter of which you absolutely will want to become highly familiar with.

Ongoing operational load. Like other PaaS’s, Fly can absolutely be configured to do the simple deploy-on-main thing and includes built in metrics dashboards, logging basics, and standard machine health checks, but you’ll find a lot of utility in flyctl. Restarting instances, changing environment variables, spinning up secondary production instances… all simple flyctl commands once you learn them! If you’re not already a heavy terminal user, dive on in. Fly exposes more primitives and control around lower-level constructs than most PaaS’s (think: direct VM controls, volumes, storage, regions, etc) and most of that is controlled via flyctl. So there’s more flexibility, but again, a steeper learning curve. Oh, also, you can still take your favorite autoscaler with you!

Cost structure. Fly walks a sort of middle-ground between resource tier-based pricing and metered usage, which makes it easy to jump around to difference scale sizes, tweak your RAM levels, and scale vertically as needed. Prices are per second of machine runtime, extra RAM can be added wherever you want (very cool), and Fly offers everyone a (massive) 40% discount when you opt to pre-reserve compute time — no enterprise contract required. If that sounds like a lot of levers to pull and tweak, that’s because it is. Again, Fly’s schtick here is configurability.

My take: if you’re the kind of person that was driving an automatic Honda Civic and already felt for years like you just wanted more of a car-person’s kind of car, then it’s probably true that Heroku’s recent announcement didn’t change anything for you — your Civic is still a Civic. But it’s understandable that Salesforce has, in some way or another, shaken you into realizing your dream. If you’re after that ‘69 Big Block Mustang with a four-barrel carb that you can tune juuuuust right… then Fly might be for you. This metaphor may have gone too far. Fly is complex. There are neat value-adds with that complexity, but it comes at the cost of complexity — there’s more to learn, more to understand, and more to manage.

Railway: Not Exactly Our Way

We’re not out to bash any hosting providers, especially ones that we support autoscaling on, but we also need to be honest: our experience with Railway has been pretty lackluster. All other bells and whistles aside, we had the worst actual system performance on Railway. Not because of dependent services or database latencies or anything like that, we just found our real, pure compute performance to be worse on Railway than any other platform. It was just plain slower.

We can’t tell you why that’s the case, and at the same time, we love that Railway’s schtick is running their own metal in datacenters rather than reselling metal they rent from the big three. That’s awesome! But we suspect that economies of scale are a relevant factor here.

Overall, we would not recommend Railway at this time. We love the mission and the goal, but we had a less-than-great-time. For the sake of being positive-outlook community members, we’ll simply leave it at that!

Oh, and we do still plan on taking another full crack at Railway when we go on tour — see more below.

The More HIY Stuff!

We live in a wonderful time of options! There are so many great options in the Host-it-Yourself world, in many flavors, and at many levels of even hosting-it-yourself. The bring-your-own VPS tooling like Dokku, HatchBox, Coolify, and CapRover offer lightweight PaaS-like experiences with great flexibility, each with their own distinct tradeoffs and workflows. Going even more complex, container orchestrators (e.g. they coordinate the Kubernetes for you) like Northflank, Porter, and Qovery can allow you to “bring your own cloud” (be it your own metal, rented Hetzner boxes, or AWS API keys, etc.) while still handling most of the complexities of Kubernetes cluster orchestration for you. And, of course, the big world of AWS itself — “Hop onto ECS Fargate!” or “Elastic Beanstalk, baby!” among other choices. There’s truly never been so many ways to run the “Heroku experience” yourself!

Honestly, there are a dizzying number of ways to make the technologies at this level of hosting control work. For the sake of this article not turning into a book, we’re going to mostly leave them unmentioned here. The reality is that if you’ve been a happy Heroku customer, you shouldn’t go looking down this path. I know that’s a strong statement that might make a few of the “come to the DIY-side!” folks upset, but it’s a pragmatic truth. These are two wholly different worlds with different levels of time and skill involved. Going ‘down’ a single layer in the hosting stack (as we perceive it) and getting into Fly’s ecosystem is already going to add overhead to your workflow as you need to learn to understand and handle their config complexity. Going all the way down to the HIY tooling is only going to add more ops time (or people!) to your app’s needs. If you’re happy with your PaaS-level at Heroku, stay up there!

The Real Answer

Let’s zoom out and take a deep breath. I still fully stand by my original sentiment above: Heroku isn’t going anywhere and will remain stable for years to come. There’s no urgency to move, and doing so will only detract from the hours you could be spending on your product itself at this point. Moving takes work. We can’t ignore that reality amidst the hype here.

Then, of course, conceding to those who are for sure going to move soon out of principle, spite, or otherwise disdain for Salesforce (which… I get), we covered some options. Render is the clearest, clean-cut, easy choice. Fly is more complex but more complicated. Railway isn’t recommended at the moment. Host-it-yourself and bring-your-own-cloud solutions are way more effort than a Heroku team should look at.

So…. move to Render and call it a day? Not exactly.

As your resident auto-scaling experts for the last decade, who have integrated deeply with and provide autoscaling services for nearly all of the platforms previously mentioned, we have some opinions.

But our opinions are from last year (or prior). And they’re based on integration work with Judoscale. And, who knows, they might just be wrong. So we’re going to do something that we haven’t seen done before: we’re going on tour.

Judoscale On Tour

As much as I wish that meant a music tour around the US with our kazoos, we actually hatched up a better idea. Judoscale is a 24/7 real-time reactive production application. We receive well over 3,000 RPS every moment of every day. Our downtime is exceedingly rare (generally only when Cloudflare or Heroku themselves have issues), but then, it darn well should be! We’re an auto-scaler! We need to be online, regardless of traffic load, so that we can reactively scale our clients’ applications correctly and appropriately any time of day.

Sounds like the perfect app to move to each of these platforms / services to test some things out.

To be clear: our “going on tour” means that we’re going to migrate the Judoscale production application, including all traffic, DNS, configs, background workers, etc, to each of Heroku’s competitors, one at a time, and document every step along the way for you all.

So, again, our real recommendation here is simply to hang tight on Heroku. We’re going to take the plunge for you (many times over) and move our real-time, high traffic application ourselves. We’re going to find the rough edges. We’re going to feel the performance bottlenecks. We’re going to foot the literal bill and feel the DX each of these new platforms provides compared to ol’ purple.

If that sounds exciting to you, make sure you subscribe to our newsletter below. We’ll start with a full breakdown of all the things we love and use on Heroku, which will set forth our rubric for how to evaluate other platforms.

Or maybe you’ve built out a tidy set of queues only to find your high-priority jobs are getting stuck behind slow (and unimportant) ones. Celery gives you powerful tools, but few guardrails.

These pain points usually stem from queue planning problems. Most teams slap labels like high_priority or emails on queues without defining what those mean.

If you plan your Python task queues around latency, you’ll have more predictable (and scalable) results. Ready to get started?

The basics of Celery Queues

Before we get into queue planning, let’s clarify some Celery terminology. If you already have a great understanding of how Celery works, feel free to skip to the next section.

Celery tasks

In Celery, a task is a single unit of work. For example, send_email_task might send a welcome email.

Celery queues

A queue in Celery refers to a named channel on the broker (like a Redis list or RabbitMQ queue) where tasks wait to be processed. By default, Celery uses a queue named "celery" (if you don’t specify one).

Celery workers

A worker is a Celery process that runs tasks. A worker can run multiple tasks concurrently, depending on its concurrency setting.

Celery concurrency

Concurrency refers to the number of tasks a worker can process at the same time. In prefork mode, this is the number of child processes (often defaults to the number of OS-reported CPUs).

Decisions you have to make when using Celery

In a typical deployment, you must decide how many queues to use and what they are called, which tasks go to each queue, and how many worker processes will consume each queue.

You also choose how many threads/processes each worker has (concurrency) and how many total containers to run (horizontal scaling). That’s a lot of decisions!

So let’s dig into how you can make these decisions with scaling in mind.

Why Celery queues run into problems at scale

Out of the box, Celery will use a single queue (usually named "celery" by default). If a task doesn’t specify a queue, it goes to the default queue. If you start a worker without specifying -Q, it will consume the default queue.

Could you build an app with just one queue? Sure. But please don’t.

Not every task is created equal

For a brand-new project, one queue might work fine for a short while. But very soon, you’ll encounter scenarios that push you to create additional queues:

• You have a task that needs to run quickly (a high-priority job), so you want it processed before other tasks.
• You have a task that takes a long time to run (perhaps several seconds or minutes), and you want it to have lower priority or even separate handling so it doesn’t block faster tasks.

In response, teams might eventually create ad-hoc queues like "urgent" for high priority and "low" for slow tasks.

Ambiguous queue names

However, there’s a big problem. Those queue names are ambiguous.

How urgent is “urgent”? What does “low” mean, exactly? As your application grows, you’ll find there are varying degrees of priority. One developer might add very_urgent or critical queues; another might introduce a queue for a specific feature like reports or emails.

Before you know it, you have a sprawl of Celery queues without a clear hierarchy or expectations.

Latency-based queues

Take a step back and consider what metrics define the “health” of a task queue. Three key metrics are commonly used:

• Worker CPU: How taxed is the CPU for worker processes?
• Queue depth: How many tasks are waiting in the queue (queue length).
• Queue latency: How long a task waits in the queue before a worker starts processing it (sometimes called queue time).

CPU can be used, but it doesn’t actually tell everything about the queue. It simply gives an indication (and often a trailing indication) of the worker process during an individual task. And task queues often back up without spiking CPU at all, giving a false sense of worker health.

Queue depth is easy to visualize (a simple count of jobs), so many people focus on it. Queue depth can be very misleading. The number of tasks doesn’t tell you how long they’ll take to clear.

For example, imagine two queues, each handled by one worker process:

• Queue A has 10 jobs enqueued, and each job takes ~1 second to run.
• Queue B has 10,000 jobs enqueued, but each job takes ~1 millisecond to run.

Queue B might look “backed up” at a glancem, but in reality, both queues will finish their work in about 10 seconds. The latency (wait time) for jobs in both queues is the same ~10 seconds, which is the metric that truly matters.

So, is a 10-second wait time good or bad? It depends.

The acceptable latency for a queue is a business decision. It depends on what the tasks are doing and how quickly that work needs to begin. This brings us back to the notion of “urgency”, but now we can quantify it. Instead of calling a queue “urgent” in a vague sense, we decide what latency is acceptable for that queue’s tasks.

Latency SLA queue names

If you’re convinced queue latency is the right metric to measure performance, you should fix the ambiguity in your queue names. Naming your queues after their latency targets (SLAs) is a great way to set yourself up for success.

For example:

• “urgent” becomes within_5_seconds (tasks should start within 5 seconds)
• “default” becomes within_5_minutes (tasks should start within 5 minutes)
• “low” becomes within_5_hours (tasks should start within 5 hours)

If I push a task to the within_5_seconds queue, I’m explicitly saying I expect that job to begin processing within five seconds. The name of the queue communicates the expectation.

You can choose whatever latency thresholds make sense for your app, the specifics aren’t as important as the explicitness of the naming.

By communicating latency expectations in the queue names, we get a few important things.

First, you’ll end up with fewer queues. You’re far less likely to create a new queue per feature or whim. Almost every new task will fit into an existing latency category. This should remove the temptation of one-off queues that don’t serve a strategic purpose.

Second, each queue now has a performance target (its name). This gives clarity for monitoring. If the within_5_minutes queue starts seeing 10-minute latencies, you have an unambiguous problem.

Of course, naming queues “within_X” doesn’t magically make tasks start within X time – you have to ensure enough worker capacity to meet those targets. That’s where scaling comes in.

Fortunately, this strategy makes it crazy easy to decide when to spin up more (or fewer) workers to scale, but we’ll talk more about that later.

Simple ways to scale Celery queues

Typically, scaling a Celery worker pool is with the goal of avoiding a queue backlog.

Now that our queue names encode latency expectations, we can define a clear scaling goal for each queue:

For most people, traffic and job volumes fluctuate too much to maintain this manually. You’ll want to autoscale your workers based on queue latency. With autoscaling in place, meeting those latency targets becomes trivial.

When jobs start waiting too long, spin up more workers; when the queues are empty, spin them down.

For example, if the within_5_seconds queue’s jobs are waiting >5 seconds, your autoscaler should add another worker process (or increase concurrency) for that queue. If the queue’s latency stays under 5 seconds, you can maybe scale down. We’ll talk about how to assign workers to queues next, which affects how you set up autoscaling triggers.

Speaking of queue assignment, how should we split up queues across Celery workers? I have a few opinions!

Your options for matching workers to queues

When it comes to queue-to-worker assignment, you have a couple of options. On one hand, you have one set of workers pulling from all queues. On the other hand, you have dedicated workers for each queue.

In between these two extremes, you might run some workers that each handle a subset of queues.

Running a single worker pool for all queues

Running a single worker pool for all queues is the simplest setup. It’s resource-efficient since any free worker can work on any task, and you don’t need to worry about balancing workers between queues.

However, the downsides are significant. You risk long-running tasks blocking high-priority tasks, plus it’s harder to autoscale effectively for all latency goals at once.

For example, suppose one Celery worker (with concurrency 4) is consuming within_5_seconds, within_5_minutes, and within_5_hours queues. If it picks up several very slow within_5_hours tasks (say tasks that each take minutes to execute) on all its worker processes, and then a bunch of new within_5_seconds tasks arrive, those fast tasks can’t start until a process is free.

All processes are busy churning on slow jobs, so even though the within_5_seconds queue is the highest priority, it’s effectively blocked. This defeats the purpose of having a fast queue!

Dedicated workers per queue

In this setup, each queue gets its own Celery worker process (or pool).

For example, you might start one set of workers with -Q within_5_seconds, another with -Q within_5_minutes, and so on. This completely isolates each latency tier.

The slow jobs in the 5-hour queue can never block the 5-second jobs, because they’re handled by different workers on possibly different machines.

Autoscaling becomes much cleaner because you can scale each worker deployment based on that queue’s latency threshold. The within_5_minutes workers only care about keeping that queue under 5 minutes latency, and if they’re idle, you can scale them down without affecting the queue time of unrelated queues.

The mental model is simpler, and each queue’s performance can be managed separately. The primary downside is the cost of running more separate processes.

The cost difference between one big worker vs. multiple smaller dedicated workers is often minor, and it’s far outweighed by the performance improvements. With dedicated per-queue workers, you also avoid starving out fast tasks with long-running ones.

A bit of both

One strategy is to try to group certain queues together on workers and isolate others. For example, maybe combine the within_5_seconds and within_5_minutes queues on one worker type, but keep the within_5_hours queue separate.

While this can work, any time you put multiple latency tiers on one worker, you reintroduce the possibility of interference. It also complicates autoscaling (which latency do you scale on for that combined worker?).

My recommendation

In summary, I recommend dedicated Celery workers per latency-based queue. It makes it straightforward to maintain each queue’s SLA.

If you’re on an autoscaling platform, set each worker deployment to scale up whenever its queue latency exceeds the target. To mitigate the potentially higher resource usage of this setup, I also recommend autoscaling your lower-priority workers (5 minutes, 5 hours, etc.) down to zero when the queues are idle. (Of course Judoscale makes this super easy 😁.)

If you’re doing this manually, you still benefit from clarity: you can monitor each queue’s wait time and add resources accordingly without guessing which queue is starved.

You should also look into other ways to effectively scale Python task queues, like fanning out large jobs.

One thing to keep in mind for Celery queues

One Celery-specific consideration that doesn’t apply to every queuing system is task acknowledgment timing. By default, Celery acknowledges a task as “received” when a worker picks it up. If the worker crashes mid-task, that task is dropped.

Setting acks_late=True (either globally or per-task) delays acknowledgment until the task completes. This means crashed tasks get redelivered, but it also means your tasks need to be idempotent, since they might run more than once.

If you’re using acks_late with Redis as your broker, pay attention to the visibility_timeout setting. This controls how long Redis waits before assuming a task was lost and redelivering it. The default is one hour. If you have tasks that need to run longer than your visibility timeout, they’ll get redelivered while still running.

For latency-based queue planning, the practical advice is that tasks in your fast queues (like within_5_seconds, within_5_minutes) should be short enough that the visibility timeout is irrelevant. For your slow queue, make sure your longest-running tasks finish well under the visibility timeout, or increase the timeout accordingly.

Shipping performant Celery queues

This opinionated guide for setting up your Celery queues is very much inspired by the strategies we know work well in the Sidekiq world. I hope this gives you some fresh ideas and a solid game plan for taming your Celery queues.

Remember, planning your queues boils down to:

• Name queues by expected latency.
• Isolate latency tiers on separate workers to avoid cross-interference.
• Monitor and autoscale by latency.

Follow these steps, and you’ll avoid most of the common background job headaches that plague teams as they scale up.

For example, a fully managed platform can eliminate server maintenance at the cost of less flexibility and more money, whereas running your own server gives maximum control but demands more operational work.

Your goal in deciding on where to host a node app is to align your hosting choice with your app’s technical requirements and your team’s capacity to manage the underlying infrastructure.

Different types of Node apps have different needs

APIs built with Node are stateless request/response services and are a good fit for most hosting models. A Node.js API can run on anything from a cheap VPS to serverless functions, since each request is independent and typically short-lived.

Real-time apps (like those with WebSockets), on the other hand, need persistent connections. Things like chat apps or live dashboards require hosting that supports long-lived network sockets. Traditional servers or container-based platforms are often necessary here as pure serverless platforms often don’t allow WebSockets or constant connections. For example, Vercel’s serverless functions cannot hold always-on WebSocket connections, but they do support WebSockets through their Edge Runtime.

Server-rendered apps (think Next.js) are certainly a special case. Frameworks like Next.js generate (most) pages server-side and often do well with serverless deployment. Next.js is tightly integrated with Vercel, which offers zero-configuration deployment, serverless functions for API routes, and edge caching for static assets. Many teams choose serverless platforms for these SSR apps to leverage features like automatic CDN distribution and on-demand scaling without managing servers. However, this serverless approach comes with tradeoffs in execution time limits and statefulness, which we’ll discuss later.

First, let’s talk about the option that demands the most of you.

Hosting Node apps on a VPS (or similar cloud service)

Running a Node.js app on a VPS (Virtual Private Server), Amazon EC2, or cloud virtual machine gives you maximum control over the environment. But with that comes maximum responsibility.

On a VPS, you get root access to install any OS packages, configure the stack exactly as you want, and run any background processes you need. This flexibility is powerful for custom setups, but the maintenance burden on you or your team is high. You are in charge of everything under the hood.

Applying OS security patches, monitoring disk and CPU usage, setting up firewalls, managing backups, and handling scaling manually are all things you should be prepared to manage if you go this route.

Using infrastructure-as-code and containers can ease some pain, but won’t eliminate ops work. Tools like Kamal can simplify deploying a containerized app to a VPS. However, Kamal doesn’t handle the surrounding infrastructure needs. You still need to set up things like load balancers, databases with backups, log aggregation, and system monitoring yourself.

Containers help by packaging your Node.js app with its dependencies, making it portable and consistent across environments. But the VPS still needs to have everything the container needs. You’ll still be responsible for orchestrating containers, scaling them, and managing the host VM’s health.

Hosting on a VPS or cloud VM is fine if you need fine-grained control or have specialized requirements that platforms don’t support. But it’s not an option I can recommend unless you have a dedicated ops team (or you just really love that sort of thing). I’ve hosted small projects on a VPS, and it’s always been more headache than the cost savings I faced.

Hosting your node app on a PaaS

Platform-as-a-Service (PaaS) offerings strike a middle ground by handling most infrastructure concerns while still letting you run a “server-like” app. Platforms like Heroku, Render, Amazon ECS Fargate, and Fly.io are PaaS leaders.

They allow you to push your Node.js code (via Git or container image) and then they build, run, and serve your application in a managed environment. Platforms abstract away the server (or VPS) management.

Most platforms give you the option between using containers or not, so the above image could be even simpler, with you only managing the app itself.

With platforms, there’s very little manual configuration and management. You get a deployment platform that automates scaling, security updates, and (some) monitoring, usually through a web dashboard or CLI. Developers can focus on code and let the platform handle the “ops” heavy lifting.

Using a PaaS still provides you with the flexibility to run long-running processes and async job queues like BullMQ or Bee-Queue, which are things that pure serverless platforms don’t support.

The general-purpose nature of PaaS means it doesn’t matter whether you’re deploying a frontend, a Node API, or a background worker. This makes platforms the best option for most Node apps.

You get persistent Node.js processes that can maintain state in memory, hold database connection pools, handle WebSocket connections, and even schedule cron jobs without worrying about hitting an execution timeout or some vendor constraint. Essentially, it offers the convenience of managed hosting without the severe limitations on process lifespan that come with serverless function environments.

You get a managed environment that dramatically reduces your operations overhead, but you keep quite a bit of control.

But serverless is right for some apps! Let’s look into that next.

Hosting serverless Node apps on Vercel or Netlify

Serverless platforms like Vercel and Netlify have gained popularity, especially for frontend-oriented and Jamstack applications. Vercel hired much of the React core team away from Meta and has stewarded the development of both React and Next.js, which positions them well to support Next apps in particular.

In a serverless model, you don’t maintain a running server process. Instead, your Node.js code is deployed as functions that execute on demand in response to requests (or events) and then terminate. This model brings automatic scaling per request – every incoming request can spin up a new isolated function instance if needed, so capacity can increase seemingly without bound, and you never pay for idle time.

Vercel and Netlify both provide an experience where you connect a Git repo, and they build and deploy your site with serverless functions backing any dynamic endpoints or API routes. This gives a fantastic developer experience for certain use cases. Frontend-heavy apps get static hosting plus dynamic capabilities without ever thinking about servers, and things like CI/CD, CDN distribution, and SSL are handled for you out of the box.

I host my personal site and a few simple projects on Vercel and am quite happy with how hands-off it’s made hosting. For my simple Next.js app, Vercel is a very good fit and also free.

That being said, if I want to expand this application to include more functionality, I’d probably run into some limitations.

The first major limitation is that serverless functions on these platforms have hard time limits. This means you cannot do long processing jobs directly. If your Node app needs to generate a large report or process a big file, you’ll likely exceed these limits and the platform will kill the function.

Long-running tasks have to be offloaded to external services or broken into much smaller jobs. But Vercel and Netlify do not allow running arbitrary background worker processes. You can’t have a worker listening to a queue or a scheduler that continuously runs in the background. “Background Functions” on Netlify simply allow a single function invocation to run longer (up to 15 minutes) asynchronously, but they are not equivalent to a always-on worker process.

Vercel recently introduced scheduled functions, which are cron-like triggers, but these are just periodic invocations of serverless functions, not persistent jobs. Any asynchronous or delayed work in a serverless architecture has to be handed off to another system (using an external job queue service, or triggering an AWS Lambda via event).

This is a fundamental design difference. Traditional platforms (like Heroku, Render, etc) let you run a worker indefinitely, whereas on Netlify/Vercel, you might schedule a function to run every few minutes, but it will start fresh and then terminate each time.

Both Vercel and Netlify abstract away containers and don’t let you deploy a custom Docker image to their platform. You are limited to the runtimes and languages they support and the build process they provide. While the support is often sufficient, the platform’s provided environment is the only environment. Vercel and Netlify focus on source-based deployment and static assets, not running arbitrary containers.

They are great at what they do (fast frontend deployments), but aren’t general-purpose hosting for any kind of app.

Autoscaling a Node app

Scalability is a big question for web developers, and different platforms scale Node apps in different ways. Understanding your autoscaling options and their implications for performance and cost matters a bit for choosing a host.

On traditional setups like VPS or self-managed servers, scaling is usually manual unless you build your own scripts or use cloud vendor tools to spin up new VMs. By contrast, PaaS platforms typically offer some form of horizontal autoscaling for Node apps, but the responsiveness to load can vary.

Heroku, for example, has a built-in autoscaler (available on certain tiers) that can add or remove dynos based on response time thresholds. The caveat with this metric is that they might react sluggishly or scale at the wrong times.

This is why third-party solutions like Judoscale have emerged. Judoscale focuses on request queue time as the metric to decide scaling, which directly measures if requests are backing up due to a lack of capacity.

Judoscale will add more web processes as capacity demands it, and we also watch your job queues to autoscale worker processes. If you want reliable autoscaling on a PaaS, you want Judoscale.

Scaling on serverless is weird

Serverless platforms scale very differently.

Essentially, they scale per request by default. There’s no “instance” for you to add.

Every incoming event will find capacity by the provider launching more copies of your function as necessary. This leads to effectively unlimited concurrency out of the box, which is great for absorbing traffic spikes without any configuration. The flip side is limited control over this scaling.

Normally, every request that comes in will result in a new Node.js runtime starting if the existing ones are all busy. This is an awesome way to ensure reliability in a scenario where you traffic increases quickly.

However, there are two big tradeoffs: cold starts and cost unpredictability.

When serverless scales, many of those new function invocations might incur a cold start delay (a few hundred milliseconds or more to initialize a Node environment). In a high-traffic scenario, you could have lots of functions cold-starting, which might cause latency for some requests. More importantly, from a cost perspective, serverless billing is usually metered by time and memory per execution, plus any external service calls (like database or bandwidth).

If you get 1000 concurrent requests frequently, you pay for 1000 function runs in parallel, which can add up quickly. I see developers on X and Reddit all the time complaining that their Vercel bills ballooned under heavy load.

This isn’t to say serverless can’t be cost-effective. For super volatile but low-average traffic, it can be the cheapest option.

If you require tight control and predictability, a PaaS with the right autoscaling tool might be preferable. If you need to handle unpredictable surges and are okay with the stateless function model, serverless will do it out of the box. Just keep an eye on those usage metrics!

Picking your hosting option based on developer experience

I’ve thrown a bunch of information at you, but I don’t want to make my opinion unclear.

I think you should prioritize developer experience. Whether you’re trying to decide where to host a solo project or influence a decision for an enterprise, put real weight behind the developer cost that comes with the “cheaper” options.

Beyond that, the decision comes down to your application’s type and its traffic profile.

Ask yourself a few questions about your Node.js app:

Does your app require persistent connections or background processes? If it does, then a serverless platform (Vercel/Netlify) likely won’t serve you well. You’d lean towards a PaaS or even your own VPS if you’re okay being pretty hands-on.

How much ops work are you (or your team) willing to take on? If you have a strong DevOps skillset or an ops team, hosting on VPS or some pure cloud solution might be a good fit. You’ll get full flexibility to tailor the environment and potentially save on high-volume costs by squeezing more out of each server. But if you’d rather not deal with server management, then PaaS or serverless is attractive.

What are your scaling and traffic patterns? For relatively steady, predictable traffic, it can be more cost-effective and simpler to run a fixed number of servers (or dynos) on a PaaS or VPS. You won’t get surprises in the bill, and you can ensure they’re always warm and performant. For spiky or highly variable traffic, serverless is an option.

Choose the platform that fits the shape of your app and your team. For a typical web API or monolithic Node app that has a mix of web requests and background jobs, a PaaS will provide the least friction. If you’re building a highly interactive frontend-heavy app (especially with Next.js), deploying the frontend on Vercel or Netlify can be great for the static+serverless benefits, possibly complemented by a separate backend for any heavy lifting.

Offloading these jobs (sometimes called tasks) to a job queue ensures your web process remains responsive and keeps latency down. A typical setup is to have your web processes enqueue jobs to an external system, and one or more worker processes consume and execute those jobs asynchronously.

This works well for keeping your web processes free and performant.

So you’ve got a Node.js app, and you know what needs to be passed off to a job queue. But do you know what job queueing system to use?

If you’re looking for a quick answer, I won’t make you wait. BullMQ is right most of the time. But let’s take a look at our options!

Bull and BullMQ for job queues

BullMQ is definitely the most popular Node.js job queue (especially if you also consider Bull).

It is a powerful queue library backed by Redis, known for its high performance and rich feature set. Bull can process a large volume of jobs quickly by leveraging Redis and an efficient implementation under the hood.

Jobs are persisted in Redis, so they won’t be lost if a worker crashes. Bull provides job persistence, automatic retries, error handling, and priority queues. Together, this gives you an unbeatable expectation of reliability.

BullMQ also supports multiple workers consuming the same queue, and you can configure concurrency (the number of jobs a single worker can process in parallel). This horizontal scaling ability means BullMQ can handle a lot of load and is also perfect for autoscaling, which we’ll get into later.

BullMQ is essentially a new (major) version of Bull, with mostly the same API and using Redis, but with improved internals. If you’re already using Bull, that’s fine. But if you’re starting fresh, consider BullMQ so you get long-term support and benefit from the improvements.

Since they’re Redis-based, Bull and BullMQ are naturally suited for modern web apps that may run across multiple processes. It’s no surprise Ruby’s Sidekiq uses Redis too. All workers connect to the same Redis instance, so adding more worker processes (whether permanently or by autoscaling) increases the throughput of job processing. Jobs will be pulled by any available worker.

BullMQ includes mechanisms to detect stalled jobs, such as requeueing failed jobs. For most web applications, a single Redis-backed queue can coordinate dozens of workers reliably. If your app already uses Redis, BullMQ fits in nicely. If not, you’ll need to introduce Redis just for the queue, which is probably a worthwhile tradeoff for the reliability it provides in most cases.

Bee-Queue for job queues

Bee-Queue is another popular Redis-backed job queue for Node. It’s designed with a focus on simplicity and speed, inspired by the shortcomings of older libraries. Like BullMQ, Bee-Queue requires a Redis instance to operate, a common theme we’ll continue to see.

Bee-Queue intentionally has a smaller feature set than BullMQ, trading breadth of features for low complexity and high performance. It gives us all of the core job queueing capabilities, but leaves out some of the advanced features of BullMQ.

This tradeoff is right for some people, as it’s notably easier to get started.

The library’s API is relatively straightforward. You create a queue, define a job processor function, and enqueue jobs. My time reading Bee-Queue’s examples and documentation has been stress-free as they’re very easy to understand. This can translate to faster initial setup and less overhead in learning the tool, something that’s really underrated in medium-sized software projects.

Despite being lightweight, Bee-Queue does include essentials for production. You get persistence in Redis, job completion callbacks, and even rate limiting and retry logic. It supports job timeouts, retry attempts, and will handle “stalled job” detection.

What it lacks is some features of Bull and BullMQ, like built-in priority levels or repeatable (scheduled) jobs.

Multiple Bee-Queue worker processes can consume from the same queue even if they’re on different machines, making scaling as simple as running more workers. This makes it a great fit for autoscaling scenarios.

In practice, you’d run one or more worker processes with Bee-Queue. If you need more throughput, just increase the number of workers, and jobs will be distributed across them. If you’re okay with using Redis (and most Node apps can add Redis via a managed service fairly easily), Bee-Queue provides a nice balance of simplicity and performance.

Still, it’s been 2 years since the last release of Bee-Queue, and the lack of recent maintenance/development may put off a lot of developers.

Agenda for job queues

Agenda is a different breed of job queue for Node when compared to BullMQ and BeeQueue. It is primarily a job scheduler built on MongoDB, not Redis! It focuses on scheduling jobs (think cron jobs and delayed jobs), but it also supports immediate job queuing with concurrency control.

Agenda is a popular choice, especially for teams already using MongoDB, since it uses your MongoDB database to store job information. If I were in a project not already using MongoDB, this wouldn’t be my first choice.

Agenda’s features overlap with BullMQ and Bee-Queue in some areas, but it has its own philosophy. Agenda stores jobs in a MongoDB collection, so if your application already uses MongoDB, you don’t need an extra infrastructure component for the queue. Jobs are persisted to the database, which ensures durability.

Agenda can also work with other databases (it supports a few Mongo-like interfaces), giving some flexibility in persistence. Still, it shines in scheduling future or recurring jobs. It offers a human-readable syntax (but still supports cron syntax) and the ability to schedule jobs at specific dates or intervals.

For example, you can schedule a job to run every day at 8 am, or run once a week, all using cron patterns or (close to) plain English. This makes Agenda ideal for background jobs that need to run on a schedule.

Agenda runs as a single process scheduler. It pulls jobs from Mongo and processes them in the same process. It does support concurrency (multiple jobs at once in one process) and can be scaled to multiple processes using MongoDB’s locking mechanism (to ensure two processes don’t run the same job).

However, scaling horizontally with Agenda is not as straightforward as with Redis queues. Agenda is generally single-master, meaning one instance should be scheduling to avoid duplicate scheduling of recurring jobs, though multiple workers can cooperate on different jobs. It’s not impossible to scale horizontally, of course, but the path isn’t as straightforward.

Agenda is probably best suited for applications that need cron-like scheduling and already use MongoDB. If you have a Node app in production that’s already using Mongo, you can use Agenda to schedule jobs without introducing Redis. It’s great for things like daily reports, periodic cleanup jobs, or any job that must run X times a day/week without needing to support another infrastructure piece.

Using a message broker like RabbitMQ

Instead of using a Node-specific library, you can opt for a message broker service such as RabbitMQ, Amazon SQS, or Google Cloud Tasks. These are not Node.js libraries. They’re external systems that Node can interface with through their APIs or client libraries.

For example, RabbitMQ is a robust open-source message queue that many large systems use. In a Node app, you might use a package to publish and consume messages from RabbitMQ.

The advantage of brokers like RabbitMQ is primarily reliability and advanced messaging patterns like acknowledgments and dead-letter queues.

Similarly, cloud services like AWS SQS or even Google Cloud Tasks are fully managed queues. They remove the need to run Redis or RabbitMQ yourself, which is attractive to a lot of people. These can scale virtually indefinitely and handle autoscaling scenarios by design.

The trade-off with using external cloud queues is that you’ll have to implement some features in your application code, like deciding how to schedule jobs or doing retries. Also, there’s a bit more latency as calls go over the network. Developer experience might not be as seamless as using a Node library, but if you prefer not to manage any infrastructure, they are a very reasonable option.

Autoscaling your workers

Scaling Node job queues is a necessary part of running them in production. Offloading intensive jobs to queues doesn’t do much for the performance of the queue processing itself, which isn’t that performant.

There are two big levers you can pull to scale your Node job queues. Vertical scaling means using more powerful workers with more threads/processes. Meanwhile, horizontal scaling increases the number of worker processes or machines. Comprehensive solutions require attention to both.

As we talked about above, the major Node job queues support horizontal scaling without too much hassle, so it’s worth putting some effort into. You can do this manually, but it’s best practice to set up an autoscaler.

This lets you keep your hands off, adding worker processes when your existing processes can’t keep up with demand, and removing them when demand allows, which saves you costs. Still, most autoscalers leave much to be desired. Heroku’s autoscaler doesn’t work for workers, and other major platforms that have support use CPU as the autoscaling metric, which is not an optimal way to measure demand on asynchronous worker processes.

Judoscale is a powerful autoscaler that you can add to most any hosting setup. The autoscaling algorithm scales based on queue latency, which is a much better indicator of queue well-being than CPU usage. If you’re running a Node app in production, try Judoscale’s free plan to see if it’s right for you.

Comparing Node job queue options and making a decision

My opinion here is somewhat controversial in that I think you should value developer experience a lot in your decision-making. That means using BullMQ unless you really need a ton of extra features, in which case use a message broker like RabbitMQ.

If your app environment already includes a certain datastore, leaning into that can simplify setup. For instance, if you use Redis, Bull or BullMQ will be straightforward to add. If you use MongoDB, Agenda might integrate more naturally. A solution that fits your existing stack usually means less friction for you, which I think you should place a premium on.

The goal here wasn’t to host a cage match between the various PaaS vendors currently on the market. It was to setup a scenario:

Let’s frame this conversation as a thought experiment: if you were starting a new startup today — something like Judoscale, but fresh — would you still choose Heroku? We’ll look at that decision through the lens of a founder building a real business, not a hobby app — meaning time to profitability, team velocity, cost structure, and technical tradeoffs all matter.

This isn’t a bashing session; I want to explore how the landscape has evolved and changed over the years, and what you might do today.

Then simply chat through it. I think we ended up with some interesting and valuable insights at both the technical layer as well as the business-leadership layer (e.g. solo dev trying to start a profitable app).

That said, I didn’t want to post a typical back-and-forth style Q&A article. Instead you’ll find concepts grouped together below, each with a little context beforehand. Enjoy!

Okay, let’s dive into this thing!

The Black-Box Dividend

Possibly the most important thing when spinning up a new bootstrapped business is actually making money. That is, getting your product running and live — providing value for people that are willing to pay for it — as soon as possible. When it comes to your application architecture and hosting, then, paved roads will get you to your destination faster than carving out your own from scratch.

The Glass-Box Leverage

Of course, here in 2026 the landscape isn’t simply Heroku vs. run-your-own-hardware-at-home. Fly, Render, Railway, and several other platform-based hosting services exist now. There’s competition! And there’s nuance. Many of these platforms are more open to complexity: bringing your own Docker images, choosing far more granular server resource tiers, and selecting geographical constraints, among so many other choices. That transparency (and complexity) can be good or bad.

Unbundle the Risk

One thing I know Adam’s been a pretty big advocate for the last few years is using disparate third-party service providers detached from your hosting solution. So I wanted to dive into that here with a historical view: what he did previously vs. today.

On Leaving the Black Box

We’ve covered some of the nuances of the “black box” and “glass box”, but I’m still curious what might drive people to actually migrate across the chasm, auxiliary services aside…

Interjecting here for a moment — Adam’s referencing the lack of options between Heroku’s std-2x dyno type and their perf-m. For many users, std-2x dynos lead to headaches when trying to process large files and/or data, while jumping to perf-m feels like overkill both in terms of capacity and cost.

If that’s something that resonates with you, we actually just published a strategy for getting the best of both worlds: “Dealing With Heroku Memory Limits and Background Jobs”.

Compute Is Commodity; DX Is Not?

One thing that all PaaS’s obviously have in common, regardless of what we call them or how we pay for them, is raw compute power. But how we developers can efficiently leverage that compute, and how fast we can do so might be a different question altogether.

Support, Sales, and the Human Stuff

We’d be remiss to ignore the soft edges (support and sales) because they become hard edges during incidents and procurement.

Simple Rules We Actually Use

Let’s start wrapping this whole thing up! I wanted to ask Adam to summarize some of the topics above into a straightforward path… specifically how he might go about starting Judoscale today if he was starting Judoscale today:

Wrap Up

I started this interview assuming we’d land on a winner. I thought for sure, after all these years, Adam would still land on Heroku! But Adam nudged me to a better question: How much of the machine do you need to control right now? Early on, “black box” hosting buys momentum you can’t afford to lose. As traffic grows and dollars stay stubborn, “glass box” hosting might make the math worth looking at again… especially if you’re already unbundling other services and can spin up a Docker image quickly.

Anyway, thanks for joining us for this candid conversation with Adam, and we hope it lends some clarity as you navigate your own hosting choices and business journeys! As always, keep building and keep questioning, because sometimes the best answers come from challenging the assumptions we hold most dear.

Totally disagree with us? Think Adam’s way off base about something? Let us know over on Reddit, here.

This post is the nerdy sequel to the latter: the actual boots-on-the-ground / nuts-and-bolts of how we attempted to track process utilization, how that proved to be a bad setup, and the clever idea that lead us to a way better v2. This is the story of low-level measurement with sampling, thread safety, and lackluster results leading to new ideas 😅.

The job to be done

As per our second post in this saga, our definition of ‘utilization’ is based around an idle-state. Paraphrased, it’s essentially:

Measure the fraction of time a web-server process is handling at least one request, then aggregate that across all processes over time.

Two constraints forced us to think carefully:

1. Extremely low overhead. Judoscale is a performance tool; it’s an autoscaler that’s intended to help your application soar. It is not something whose client code should impact your application! The Judoscale package should have a perceivably invisible performance impact on the app running it. Full stop. No compromises.
2. Correct values in a multi-threaded world. While Ruby, Python, and Node can operate in an asynchronous fashion, and that asynchronosity can be valuable for serving many web requests at once, we need to be very careful in collecting values. It’s easy to accidentally collect thread-level metrics which then overlap and become very confusing. We need to be careful to stay up at the process level.

So… now we need to actually write some code: how do you actually capture the idyllic “idle time” of a process in a real application receiving real traffic?

Attempt 1: Background Sampling

Our first proof-of-concept was built around running a mostly dormant background thread. It would essentially wake up every few hundred milliseconds, ask “is this process handling any requests right now?”, record that yes-or-no, then go back to sleep. Voilá: utilization!

It was easy to ship, but it had issues. Notably…

Aliasing difficulties. Bursty traffic and short requests can fall between samples. Imagine a process that handles a flurry of 30–50 ms requests. With a 250 ms sample rate, many bursts are invisible; you under‑count busyness simply because you looked away at the wrong moments. Whoops!

Jitter vs. overhead trade‑off. If we increased the sampling rate to reduce aliasing, we immediately hike CPU wakeups, heap churn, and lock contention (on every process, 24/7!) even when your app is idle. Oof ☹️

Low signal‑to‑noise. Inherently, sampling produces a staircase approximation of a curve. Real utilization is a smooth “busy/idle timeline.” Our samples were a blurry thumbnail of a scene that actually mattered.

I personally tend to visualize this, oddly enough, as a mathematical curve on a chart (oh how my high-school math teacher would be proud). Imagine we have some real curve of data, perhaps like this:

Okay, great. Now let’s pretend we don’t actually know what that curve looks like and we’re taking a sampling-based approach to figuring it out. What we end up with is a bunch of samples. That might look like this:

Which might be fine for some cases, but we’ve clearly lost several details from the original curve — the fast spikes and drops, in particular. Thus the issue of sampling rates is seen: sample too slowly relative to how fast your data actually changes and you won’t capture a high-detail image. Sample too quickly…

You end up with a great representation of the curve, but you took up way too much horsepower constantly waking up and reading those samples. It’s hard for an app to actually serve its requests when the thread scheduler is constantly switching back to a background thread asking “HEY ARE YOU SERVING A REQUEST?!” (“I’M FREAKING TRYING TO, THANK YOU VERY MUCH!!!”).

When we’re talking about requests that might take 5ms, 50ms, or 150ms to fully handle and deliver, a sample rate of 250+ms just doesn’t capture the details. And a faster sample rate feels heavy-handed. This wasn’t going to work…

Attempt #2: Event edges + a tiny counter

Okay, to be fair, the line curve I gave above was a little disingenuous to the actual type of data we’re trying to track. Utilization, as we’ve defined it, isn’t a curve with smooth radii and roller-coaster-esque waves. As we’ve defined it, instantaneous utilization is either a zero or a one. A process is either busy, or it is not. If we were to plot that on a chart, it would actually look more like this:

That is, a square wave representing a binary signal. Unfortunately, a square wave signal can actually make sampling results even worse. Check out how wrong an ill-timed sampling pattern can get:

If you believed your sample data in that case, you’d think the signal is almost always “on”, but that’s not true.

Anyway, the novel idea ended up being beautifully boring: don’t poll at all, just record state transitions cleverly. If we simply track the timestamps of when a process leaves and returns to idle, we can realize the real, true value of “how much time was it non-idle”? That looks like this:

And once we have the blue blocks, we can simply add them all together for a given timespan, then say active_time = (blue_block_total) / total_time. Sum the rectangles! Boom!

The Benefits of Edge-Tracking

Tracking the state-changes (we’ll call them “edges” for math’s sake) has some really fantastic benefits over polling.

• Computational cost: instead of constantly waking up a thread to check in on current requests (which requires stack shifting, single-threaded locking switches, etc.), we instead can simply read and/or write against a process-global timestamp register when any request starts or ends.
• Correctness: instead of hoping a reasonable sample rate provides a decent guess at the actual curve being modeled, we instead know the exact amount of time that a given process is non-idle! There’s no guess.
• Reliable for all traffic shapes: Sudden request waves, thin bursts, long I/O waits — they all work. If a worker is non‑idle, it gets counted correctly and appropriately.

Once we realized this route, we quickly understood that it was all upside. There’s no catch here! A purely better approach born of a realization that we’re tracking binary signals, not actual curves.

Let’s See Some Code

The great news with this new approach is that it’s so simple I can share the real code that implements it here in a blog post. This code is taken straight from the judoscale-ruby Github repository, which houses all of the Ruby packages Judoscale publishes.

👀 Note

One caveat in this code: while my diagram and example above focused on showing that we track “busy time”, our actual implementation is inverted: we track “idle time” rather than “busy time”.

Tracking “busy time” is slightly easier to grok (and build diagrams for!), but in reality our code does this:

It’s the inverse, so the math still all checks out, but understanding both “busy time” and “idle time” are useful for us! We just went with idle-side tracking for our code because it ended up slightly simpler. Check it out!

First, we have a Judoscale::UtilizationTracker class. It has a few methods and helpers in it, but the important parts start with the incr method (short for “increment”):

module Judoscale
  class UtilizationTracker
    # ...
    def incr
      @mutex.synchronize do
        if @active_request_counter == 0 && @idle_started_at
          # We were idle and now we're not - add to total idle time
          @total_idle_time += get_current_time - @idle_started_at
          @idle_started_at = nil
        end

        @active_request_counter += 1
      end
    end
    # ...
  end
end

First, keep in mind that this method is going to run every time a request comes in (starts). So, since we’re going to be incrementing a request counter and idle-time timer across multiple threads, we do need to use a simple Mutex (@mutex is simply a Mutex.new from the Ruby standard library). Once we’re certain that we can safely update our process-level variables, we need to do two things: mark that our “idle time” has ended, and increment our active-requests counter.

Pretty straightforward, there! Since this block may run as a multi-threaded application server picks up a request on thread #2 or #3, we’re careful to only end our “idle” timer if there aren’t already any requests being processed (if @active_request_counter == 0).

On the flip side, we have a decr method that runs every time a request finishes (ends):

module Judoscale
  class UtilizationTracker
    # ...
    def decr
      @mutex.synchronize do
        @active_request_counter -= 1

        if @active_request_counter == 0
          # We're now idle - start tracking idle time
          @idle_started_at = get_current_time
        end
      end
    end
    # ...
  end
end

This one’s even simpler: decrement the count of active requests by one and, if that was the last request in flight, mark that our “idle time” has begun — the process is now idle!

The end result of these two functions working together is an accurate value stored into @total_idle_time which, in real time, tells us the number of milliseconds the process was idle.

The last piece of the puzzle, then, is to report that ratio and reset that variable/register! We do that in one last method on Judoscale::UtilizationTracker:

module Judoscale
  class UtilizationTracker
    # ...
    def get_idle_ratio
      @mutex.synchronize do
        total_report_cycle_time = current_time - @report_cycle_started_at

        # Capture remaining idle time
        if @idle_started_at
          @total_idle_time += current_time - @idle_started_at
          @idle_started_at = current_time
        end

        idle_ratio = @total_idle_time / total_report_cycle_time
        @total_idle_time = 0.0
        idle_ratio
      end
    end
    # ...
  end
end

Some background here: Judoscale packages report back to Judoscale servers every 10 seconds (using a zero-performance-impact background POST) with a handful of capacity metrics about the application. In this case, @report_cycle_started_at represents the timestamp at the start of that 10-second bucket. Since we’re trying to figure out the idle ratio, we need to divide the idle time over the total time. “The beginning of the bucket until now” is that “total time”.

Once we have that, we have a special case for when this code runs while the process is actively idle as to prevent over-counting or under-counting idle time. Since our “report cycle” observation window might start/end during an idle period, we need to handle that carefully. Visually, that’d look like this:

Finally, we compute the idle ratio (a decimal, like 0.88 or 0.37), reset the @total_idle_time back to 0.0, and yield that idle ratio as the result. ✨

The last piece of code I’ll highlight is a layer up — the request middleware itself. This class, Judoscale::RequestMiddleware, is essentially what wraps every Rack request before and after it’s handed down to the Rack application itself. I’m chopping out a lot here, but the bits pertinent to our discussion remain:

module Judoscale
  class RequestMiddleware
    # ...
    def call(env)
      # ...
      tracker = UtilizationTracker.instance # Singleton
      tracker.incr

      # ... lots of other code

    ensure
      tracker.decr
    end
    # ...
  end
end

Essentially we’ve created a two-part contract:

1. Every time a request starts, we guarantee we’re going to call #incr on the Process-level singleton instance of UtilizationTracker
2. Every time a request ends, regardless of how or why it ends, we guarantee we’re going to call #decr on that same singleton instance (thanks, ensure!)

This is the glue that ensures our data inside of UtilizationTracker is consistent and accurate over the lifespan of the process. Isn’t it great?!

Aggregate It Together

Zooming out a little bit, we’ll conclude the deep-dive with a sense of how the aggregation works beyond a single process. Let’s say that you’ve got 2 production web services/dynos/containers/etc. running, and each runs 4 web processes. Since each process POST’s back its own metrics every 10 seconds, that means our back-end is going to get 8 data-points about your application’s overall web-process idleness/busyness. Maybe for a given 10-second bucket Process #1 on server #1 showed an idle ratio of 0.66 (that is, it was idle for two-thirds of that 10-second window), while process #4 on server #2 read a ratio of 0.22 (meaning it was handling at least one request almost the whole bucket).

Once we have all of the data points, the aggregate is actually simple: we average them together. For example, then, if we received these data points:

For that bucket, our average idle ratio would be:

(0.56 + 0.77 + 0.48 + 0.39 + 0.81 + 0.44 + 0.52 + 0.62)/8

Which is 0.57. So then, that application was idle 57% of the time (for that bucket) and, inversely, busy 43% of the time. Thus, that’d be a 43% utilization metric for that bucket, as we’ve defined it. Gathered, collected, and aggregated simply.

Wrapping It Up

If there’s a theme to this little blog-post saga, it’s that the simplest model that matches reality tends to win. We started by trying to guess at busyness with background sampling, only to discover all the usual traps: aliasing, jitter, and overhead. Then we reframed the problem to match the truth on the ground: a process is either idle or it isn’t. Record the edges. Sum the rectangles. Report the ratio. Done.

That shift gave us three things you actually feel in production: lower overhead, correctness across weird traffic shapes (long I/O, tiny bursts, mixed workloads), and numbers you can trust enough to automate against. When an autoscaler acts on a metric, the worst feeling in the world is, “ehh, it’s probably fine.” Edge-tracking turns “probably” into confidence.

And the aggregation story is intentionally boring, too. Each process tells us how idle it was in the last 10 seconds; we average those into an application-level picture. No fancy weighting, no black-box magic. If your fleet spends 57% of a bucket idle, that’s 43% utilized. That’s a number you can reason about, chart, alert on, and scale from.

So if you’ve been skeptical of utilization-based autoscaling because it felt hand-wavey or weird, we hope this demystifies it. The implementation is small on purpose, tested in the sharp edges of real apps (including our own!), and designed to vanish into the background until you need it. Watch your utilization settle into patterns you recognize, set the thresholds that reflect your own tolerance for headroom vs. cost, then enable utilization autoscaling.

In other words: measure what matters, measure it honestly, and keep the math simple enough that you’ll actually use it.

We’re really trying to optimize for our public website’s performance for SEO reasons…

…was the core theme of our meetings with one of our customers a few weeks ago. They run a Rails application with several different ‘sectors’ — a public website, two different user portals, and an admin ‘backend’ with several internal tools. It’s not an extremely complex application, but it is diverse in its traffic. After chatting with them for a few hours, we had a great solution ready for them — one we use ourselves but feel isn’t talked about enough! Running a second prod app.

We’re going to dive into that story and our clever suggestions for scaling sideways, but before we do, let’s clarify some terms so this doesn’t all become terribly confusing! We’ll use “main app” to describe the existing, single production application instance. We’ll then use “second app” to describe the new, separate clone of the main app — an instance still running all of the production app code (with all the same environment configs, etc.) but which is separate (more on that in a moment). Alright, let’s dive in!

What We’re Solving For Here

This particular customer has a very SEO-driven business. That means that their public website, which is served by their core Rails application, needs to be excellent: fast, steady, predictable, burst-ready. But the app houses several other sectors which are older, slower, and less performance-friendly — we all have ’em!

Unfortunately, in a multi-threaded world (hello, Puma), those slower endpoints don’t just take longer for the people who hit them; they raise the waterline for everyone by occupying threads that subsequent would-be-faster requests must wait on. The result is a p50 (average) request time that looks pretty reasonable… but a p95 that’s much worse and erratic. Oh, and a support channel that pings for performance issues when there seemingly aren’t any.

From a telemetry and metrics standpoint, we’ve seen this issue plenty of times: CPU saturation is nonexistent and database resources look boring, but request queue time (the metric that matters) spikes randomly and p95s are all over. In the case of our customer, it’s not that their public website got slower, per se; it’s that the requests for those public site pages had to wait. Thus we’ve met an old truth: multi-threading increases throughput but amplifies latency (something we dissected in “Why Did Rails’ Puma Config Change?!”). Boil it way down and it’s hosting costs vs. p95s.

But the reality for this customer is that they needed to tame and stabilize their p95 response times for their public website. Appeasing the finicky beast that is Google Search Ranking is a broadly unknown game, but stable performance does seem to be a factor.

The good news here is that we’ve got a creative solution. We call it “scaling sideways” — slightly different than ‘horizontal scaling’, yet still horizontal in concept: running a second, but subdomain-separated, instance of your production application.

Scaling Sideways

Let’s expand on the specifics of this strategy, since “scaling” can be a bit of an overloaded term. What we’re describing here isn’t “scaling” in the sense we’re likely all used to these days: changing the number of webserver or worker instances your production application is running at any given time (the core premise of Judoscale itself). Instead we’re talking about “scaling” in a much slower and more methodical approach: running a second production application, which is essentially a clone of the main app, on a separate subdomain with separate infrastructure. It’s still the same code-base, same deployment branch, and really should have all of the same environment and configuration variables… just a different place to request the same data and/or pages.

The key to this strategy is offloading traffic to slower and less consistent endpoints to the second app (via its subdomain) so that your main app can handle its own traffic more consistently and quickly. The main app becomes the home for predictable, latency-sensitive endpoints; the second app absorbs the messier stuff without letting it bleed into the public experience.

Luckily, we don’t need a microservices migration plan to do this. We’re not decomposing the domain model; we’re just decomposing our runtime. One deliberate split is enough: the fast path (main) and the heavy/volatile path (second). The payoff is that your main app’s thread pool stops babysitting slow requests and blocking higher-priority endpoints. Queue time stabilizes. Tails compress. (…Engineers stop arguing about whether going single-threaded everywhere is “worth it.”)

When Is It The Right Move?

We should recognize first that this strategy isn’t perfect for every application. It shines when at least one of a few conditions are true:

Your traffic has distinct “shapes.” If one slice of your app is bursty, slow, or just unpredictable (admin pages, CSV exports, report builders, portals, ‘real time’ (polling) dashboards), while another slice must feel instant and boring (marketing site, signup flow, product pages), you’re a great candidate. Sideways scaling lets you build a fast-lane for the steady stuff and a truck/carpool-lane (or two) for everything else.

You have different SLAs for different routes. Some requests just matter more. If a public route missing its p95 target is business-critical (SEO, ad landing pages, checkout, conversions), prioritize it on the main app and give it a calmer thread pool. If an authenticated portal can tolerate higher p95s without harming KPAs or other business targets, move it to the second app.

You can influence where traffic goes. This sounds obvious, but you need a lever. Many teams already have it: front-end fetch() calls, Turbo Frames/Streams, HTMX targets, or API clients you control. If you can change hostnames in those calls, you can steer traffic to the second app with minimal risk and no user-visible disruption. Especially if these calls are transparent to a browser’s address-bar.

SEO is part of the story. If Google’s crawlers matter a great deal to your business, you might consider splitting your public site from your other application chunks. Instead of the classic “let’s just rewrite the marketing site to static”, you get a lot of the benefits of a dedicated marketing site system (the main app) while retaining all of the comforts of a unified code base and singular mental/domain model.

Judoscale Does It, Too!

As it turns out, Judoscale itself satisfies three of those bolded conditions above. The Judoscale architecture is built around customers installing the Judoscale package, which is essentially just a light-weight monitor for request and job queues within the app. Those metrics ultimately get POST’ed back to Judoscale servers for processing and aggregation. Nice! But those POST’s happen every ten seconds for every process over thousands of applications. We have a ton of API traffic. As in, 3000-3500 requests per second 24/7.

Then, of course, there’s the Judoscale dashboard and user UI where you can see your metric charts, tune your scaling configuration, and do standard SaaS things. While those charts do have automatic 10-second update polling built-in, the traffic for that entire sector of the app trends much closer to about 50 RPS.

So… we (1) definitely have different ‘shapes’ — our API traffic is tiny payload and ultra-fast response whereas our dashboard traffic is small-to-medium payload and variable response. Additionally, we (2) definitely have different SLA’s for these two shapes. Our API needs to be available, but response times can fluctuate (there’s no human waiting)… whereas our dashboard needs to be as fast as possible since it’s customer-facing. Finally, we (3) can control where the majority of our traffic goes by tweaking the client packages to POST somewhere else (and/or some smart routing with Cloudflare).

We’ll get to the implementation specifics below, but hopefully this gives you an idea of the versatility of scaling sideways: applications completely non-SEO focused can still benefit greatly from segmenting traffic in this style.

How You Actually Do It

Spin up a clone of your main prod app. Same repo, same deploy pipeline, same environment variables (with a couple exceptions we’ll note). Point it at a sibling subdomain — ww2.example.com, api2.example.com, or simply 2.example.com all work. The goal is sameness: both apps should boot the same code and talk to the same primary dependencies (database, cache, storage, queue, file storage [S3 et. al]). Differences should be intentional and minimal: web process counts, thread counts, and possibly instance sizes.

From there:

1. DNS & routing. Create the new subdomain and point it to the second app’s router/load balancer/DNS target.
2. Environment parity. Duplicate secrets and env vars (including SECRET_KEY_BASE/equivalents so session cookies work across hosts if necessary — more on this below). Consider different Puma thread counts between apps (more on this below too!).
3. Traffic split. Start by moving non-navigational traffic: API calls from your front-end, background polling, Turbo Frames/Streams targets. These won’t change the URL in the address bar, so the move is low-risk.
4. Progressively offload. Next, migrate heavier, authenticated pages and long-running endpoints to the second app. Be deliberate around what addresses users might see in their browser’s address bar!
5. SEO guardrails. Add canonicals on anything public your second app might serve, ensure robots blocking is in place for that host, and keep sitemaps + social meta rooted on the main app.
6. Observability. Watch queue time and p95s on both apps. You should see the main app flatten out quickly.

Most importantly, treat this like a runtime composition change, not an architecture rewrite. You can ship it safely in small patches and keep rolling forward.

What Actually Moves

A practical rule of thumb:

• Stays on the main app: canonical public pages, sitemaps/robots, OpenGraph/Twitter cards, landing pages, docs/blog, marketing flows, and any route that shapes your public narrative or crawlability.
• Moves to the second app: authenticated portals, JSON APIs, front-end-driven fragments (Turbo/HTMX/Stimulus/etc.), polling endpoints, file uploads/exports, batchy or I/O-heavy controllers, and admin tooling.

For navigations, you have options but need to be intentional. Keep in mind that browser address bars remain highly useful for users copying or pasting URL’s in/out and potentially sharing those URLs with others. For intra-portal / authenticated endpoints it may not matter than a user sends a colleague https://2.example.com/portal/book/5 (especially if the colleague would’ve ended up forced over to the second app to log in to the portal anyway!).

But for resources and endpoints where the goal is speed and public accessibility, we’ll want to keep those endpoints pointing against the main app.

The good news is that we can be clever. For instance, if an endpoint is slow and synchronous (not recommended but we get it, it happens) yet must result in a public URL, we can still POST to the second app and do the work synchronously in that controller. We just need to make sure the response from the second app redirects back to the first. And since they share the same database, you can fluidly (for example) do an expensive create operation in the second app then immediately redirect to the now-existing record on the main app with confidence. There’s no delay in data propagation between the two applications!

In the case of our customer, this meant offloading most of their user portals and internal admin tools to the second app. Their public marketing site stayed put and immediately got calmer metrics. Problem solved!

Judoscale’s Setup

We mentioned earlier that Judoscale also runs a dual-prod-app setup, but we arrived at our split for different reasons — and with a different emphasis. We’re sharing that to underscore there isn’t one “right” pattern. For us, it was more about cost and UX than isolating slow paths… most of our endpoints are already fast!

Rather than sending volatile endpoints to a second app, we split by human interface. Our main app (app.judoscale.com) is the customer dashboard, so we tune it for UX: snappy, steady, predictable. Our API app (api.judoscale.com) serves the bulk of our traffic, but it’s non-human-facing and can tolerate small, occasional latency blips. The machines don’t mind! But people do. It’s not the fast-vs-volatile split we describe above (which is still the right path for this customer), but it delivers similar benefits: each runtime is optimized for what matters most to it.

Practically, this lets us fine-tune the API runtime for throughput and cost (concurrency, process counts, aggressive autoscaling) while keeping the main app conservative for a consistently great feel. The net effect: a calmer UX and lower hosting spend (more on cost below..). For many, the canonical split paradigm might be “fast vs. volatile” but for us it was “UX vs Cost”. It’s a different motive but the same playbook: split out a second prod app.

A Caveat on Cookies, Auth, and Subdomains

If you’re going to use a second app for a disparate, separate API or fully segmented authentication mechanism (like Judoscale did), feel free to skip this section. If instead you’ll be cleverly (and carefully) shuttling users between the two apps, we need to discuss shared authentication across subdomains.

The simplest way to accomplish this is to setup both applications with the exact same secret key base (or equivalent) so that cookie and session cryptographic signing validates to/from both. That is, if you log in on the main app, a subsequent request to the second app will see that you’re logged in. This strategy upholds the “keep both apps the exact same” principle by keeping sessions transparent between them. Both apps will read and write to the same session/cookie.

Once both applications are running the same keys, you’ll need to ensure that the actual cookie policies are setup correctly for both apps. Essentially we need to make sure that both apps are emitting cookies with the same sharing configurations setup so that browsers will send the same cookie to both apps. In Rails that might look something like this (for session storage via cookie):

Rails.application.config.session_store(
  :cookie_store,
  key: "_my_app_shared_session_key",
  domain: ".example.com",      # explicit eTLD+1; covers example.com + subdomains
  expire_after: 1.year,
  secure: true,                # if this fails in specs/tests, switch to `!Rails.env.test?`
  same_site: :lax,             # mitigates CSRF while allowing subdomains
  httponly: true
)

But, as with all things security-related, make sure you understand every config component here and are confident in your security strategy amidst sharing cookies between the two apps. YMMV.

Magic, P95’s, and Threads

It’s worth taking a little detour here to assess the magic of what we’re presenting: it isn’t. There’s no real magic at play here — this is just simple queueing theory with friendlier furniture. We’ve talked about queueing theory broadly in “Queue Time: The Metric that Matters” but the mechanism at play in scaling sideways isn’t radical. When slow requests leave the main thread pool, fast requests stop waiting behind them. That means lower overall variance in request speeds (e.g. lower p95’s) and an app that users will probably describe as feeling “snappier”.

In fact, we can use our “keep the fast app fast” and “keep the slow app slow” mindset in tweaking our thread counts in each app. For a main app we recommend three Puma threads. That’s Rails’ new standard and proves to be an excellent tradeoff: increased throughput with a reasonable, low tail-latency increase (especially after you move all of the slow requests to the second app!). That said, we recommend deliberately choosing a higher thread count on the second app. Maybe five, maybe six. Your mileage will vary on specifics, but when we design and spin up an application specifically to handle our slower (likely I/O-bound) requests, especially when we aren’t as worried about response times, we can really leverage the power of a large thread pool. This should allow us to keep our instance-count low — a single server instance running five or six threads should be able to handle quite a bit of stuff!

Autoscaling Two Applications

Finally, the last major topic to cover for scaling sideways is indeed autoscaling. First, you should use Judoscale (👋). Okay, obvious plug aside, there’s a little nuance here: you’re going to want both apps to autoscale. But they’re going to do so with different parameters and goals.

Main app: now that variance is down and your endpoints are consistently performant, we’ll want to clamp our queue-time thresholds a bit tighter. The target is a flat, boring queue-time line very near zero. In Judoscale, you should see low enough metrics that an upscale threshold between 5-10ms feels very stable and scales nicely with your actual traffic curves (not erratically)!

Second app: still scale on queue time but expect volatility and small spikes that self-resolve. We’d recommend a moderately high upscale threshold like 80ms as well as reducing upscale sensitivity to 20 seconds so brief jitters don’t cause thrashing (AKA ping-pong scaling, which we discussed here). We want to upscale when necessary, but wait a moment to be sure that upscaling is, in fact, necessary.

So, all of that to say, queue time is still absolutely the metric to watch for scaling on both applications. And Judoscale is still absolutely the tool to use. But refining our scaling parameters for each app in their own context is the real path to success here! We want tight bounds and strict expectations on the main app with looser, workload-aware settings on the second.

A Note on Cost

To address the potential elephant in the room: scaling sideways this way may cost a little more in your overall hosting bill. That’s true. But keep in mind that our first goal here was to optimize and speed up a sector of an application without refactoring the whole application. This is a “Can we throw money at the problem?” solution.

But there’s actually better news: it’s likely that this strategy won’t actually cost much more than your base hosting level now. Remember that the main app is likely going to run fewer instances the more surface area you move away from it. That’s savings. And the second app should make broader use of multi-threading, so it too may need fewer instances than you expect. That’s cheap!

At the end of the day, snappier user experiences and conversions tends to yield more sales, and more sales means you probably have a little more space in your hosting budget. We’re not advocating for going wild here — you should still autoscale both applications to keep things efficient — but this strategy is a reasonable cost-path forward for powerful performance gains.

Scale Sideways

We started with a simple ask: “optimize the public site for SEO”, and a familiar constraint: one app serving very different kinds of traffic. That’s why we reached for the often-overlooked move of running a second production app. It squarely addressed this customer’s need: keep the public face fast and predictable while letting portals and internal tools be as spiky and complex as they need to be. We should know, we do the same thing (though not for SEO purposes)!

The path there doesn’t require a big‑bang migration. Stand up the second app, put guardrails in place, and move traffic in slices. Begin with front-end calls, shift over some API action, then gradually migrate entire user-portals when you’re confident in your URL sharing… all while feature-flagging shifts to build confidence.

What you get for that incremental effort is real performance gain with little added domain complexity or cost. The main app’s thread pool narrows to the fast paths, queue time flattens, and p95s stop lurching. The second app absorbs the messy variance without leaking it into the public experience. Same codebase, two runtimes, each excellent at a different job. If your intro sounds like our customer’s (“we’re optimizing public performance for SEO”), or ours (“we really ned to optimize our API for throughput and reliability”), this is the strategy that keeps the promise without rewriting the product or doubling your spend.

I added one background job and now I’m priced out of Heroku.

I’ve heard some variation of this too many times to count. Your app hums along fine on Standard dynos…until you add video encoding, giant imports, or some other memory‑hungry job. Suddenly your worker needs a bigger box, and upgrading every worker to Performance dynos feels like buying a school bus because you might carpool once.

There’s a simple pattern that keeps your bill sane and your architecture boring (the good kind): Put the heavy job on its own queue, give it a dedicated worker process, and autoscale that process to zero when it’s idle. The rest of your app stays on Standard dynos.

This post focuses on a real example from Justin Searls on his podcast, Breaking Change—and exactly how I’d set this up on Heroku.

Justin’s story: 4K video meets 1 GB dynos

Justin’s adding support for Instagram Stories (and soon Facebook) to POSSE Party, his tool for syndicating your own content to social media. The shape of the problem:

• A minute of 4K HDR video is 700–800 MB.
• Instagram only accepts 1080p within strict codec limits.
• Custom server-side re‑encoding produces a compliant, compressed file.

Everything worked fine, until a real 4K file hit the server. On Heroku Standard dynos (1 GB RAM), FFmpeg spikes over 1.2 GB during 4K→1080p. Heroku starts swinging the OOM hammer. Sometimes the encode finishes just in time before Heroku shuts it down. Sometimes not. Either way, it’s “no way to live.” (Justin’s words)

2025-10-29:29:13.606617+00:00 heroku[worker.1]: Process running mem=844M(165.0%)
2025-10-29:29:13.608658+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)
2025-10-29:29:38.911996+00:00 heroku[worker.1]: Process running mem=810M(158.3%)
2025-10-29:29:38.913195+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)
2025-10-29:30:03.936646+00:00 heroku[worker.1]: Process running mem=1197M(233.8%)
2025-10-29:30:03.942490+00:00 heroku[worker.1]: Error R15 (Memory quota vastly exceeded
2025-10-29:30:03.944352+00:00 heroku[worker.1]: Stopping process with SIGKILL
2025-10-29:30:04.133323+00:00 heroku[worker.1]: Process exited with status 137
2025-10-29:30:04.179846+00:00 heroku[worker.1]: State changed from up to crashed

The apparent choices: 1. Upgrade to Performance dynos (ouch, $$$, especially if you upgrade all workers), or 2. Break out encoding to a separate service you run elsewhere (more moving parts, Active Storage integration gets annoying), or 3. Do it client‑side with WebCodecs (promising, but HDR tone‑mapping and codec constraints are tricky).

There’s a fourth option that’s dead simple and keeps everything on Heroku:

Isolate the heavy job on its own queue, back it with a dedicated worker that uses a Performance dyno, and autoscale that worker from 0→1 only when needed.

The perf dyno runs for a few minutes a month, all automated, costing almost nothing.

The “dedicated worker” pattern (at a glance)

1. Create a dedicated queue for heavy jobs, e.g. “memory_hog”.
2. Run a dedicated worker process that only monitors memory_hog with concurrency = 1.
3. Set that process’s dyno type to a Performance size. Leave the quantity at 0.
4. Autoscale that worker based on queue latency (queue time).
5. Enqueue jobs; let autoscaling do the rest.

If you’ve read our post on planning your Sidekiq queues, you know I’m a huge advocate for latency‑based queue names. This is the exception. When memory is the constraint, name the queue accordingly so its purpose is obvious and you can add similar jobs later.

Step by step setup

Let’s walk through the actual implementation of this pattern. I’m focusing on Sidekiq and Heroku here, but you can apply the same concepts to any job/task queue and cloud hosting platform.

1. Point the heavy job at a dedicated queue

# app/jobs/encode_video_job.rb

class EncodeVideoJob
  include Sidekiq::Job
  sidekiq_options queue: :memory_hog

  ...
end

2. Add a dedicated worker process

# Procfile

worker: bundle exec sidekiq -c 5 -q within_5_seconds -q within_5_minutes
memory_hog_worker: bundle exec sidekiq -c 1 -q memory_hog

Keep it single-threaded (-c 1) to avoid multiplying memory usage. If you truly need parallel encodes later, raise carefully.

3. Set the dyno type, not the dyno count

In the Heroku dashboard, open the memory_hog process and choose a Performance dyno size (whatever meets your memory needs). Leave the quantity at 0. This only tells Heroku what kind of dyno to use when you scale up later.

4. Wire up autoscaling in Judoscale

You can use any autoscaler that can scale your workers based on job queues. This is the Judoscale blog, so we’re using Judoscale.

• Process: memory_hog
• Scale range 0—1 dynos (maybe more?)
• Scale up when queue latency >= 1 second (anything in the queue)
• Scale down when queue latency drops below 1 second (essentially idle)

Now when you enqueue a video, queue latency rises, Judoscale starts one Performance dyno for memory_hog, FFmpeg runs, and the worker scales back to 0 when the queue drains.

5. Test it end‑to‑end

1. Enqueue a big video.
2. Watch queue latency bump.
3. See Judoscale scale memory_hog 0→1.
4. Encode finishes; dyno scales back to 0.
5. Celebrate not paying for a big box 24/7.

Gracefully handling long-running jobs

Jobs that consume a lot of memory also tend to be long-running jobs. We consider a “long-running job” to be any job that takes longer than the shutdown timeout for the given job processor, which is usually 25 seconds.

Out-of-the-box, Judoscale will downscale a worker service as soon as the queue is empty. As long as your jobs complete within the shutdown timeout, this is fine—The worker will receive the shutdown signal, finish the job, then shut down. If the job takes longer, it’ll be killed before it can finish 👎.

Judoscale handles this scenario with an opt-in configuration to prevent downscaling when jobs are busy.

To see this option in the UI, you must enable “busy job” tracking in your code. Check out the docs for details.

Alternative approaches

Hopefully it’s obvious why this beats the “just upgrade everything” approach. It’s completely unnecessary! If you only need a perf dyno for one occasional job, there’s no reason to pay for it 24/7.

The other alternative I often hear (and Justin mentioned in the podcast) is extracting this work into a separate service outside of Heroku. It’s cliche at this point to talk about how cheap hardware is if you get closer to the metal. Yes, you pay a substantial tax to use a PaaS like Heroku, but that tax buys your time. There’s simply no reason to waste time spinning up new infra when simple (and cheap!) solutions exist on your current platform.

Take action

Judoscale was built for exactly this: autoscaling workers by queue latency, including scaling to zero. Turn it on, ship your feature, and leave the school bus at the dealership.

If you want help wiring it up, email us or call us.

The popularity of JavaScript and Node.js means we get a handful of really incredible frameworks to choose from and plenty of great hosting options. In this article, I’ll explain what makes each framework unique and how you can choose between them. Ready to jump in with me?

Using Express.js

Express is the common standard for Node.js web frameworks. It’s the Node framework that nearly every Node developer has used or at least heard of. It’s a small framework that doesn’t impose a strict structure. This makes it pretty flexible. You can use Express to build everything from simple JSON APIs to full-blown web applications. Even if you don’t choose to build on Express, the framework you do choose may include it under the hood.

Getting started is very easy, as Express intentionally has a very shallow learning curve. If you’re new to Node, Express is often the first recommendation from the JS community. I put a HUGE emphasis on developer experience. It’s why I love Rails. The Express developer experience is straightforward, but it’s not an all-inclusive framework like Rails or even Laravel.

One of Express’s biggest strengths is its huge ecosystem of middleware and plugins. Because it’s been the most popular Node framework for years, almost any feature you need has an Express middleware available. This points out an issue. Express does not have all that much without this middleware, so you’ll be forced to make lots of choices for your app!

One important thing to keep in mind is that Express is not the fastest framework out there. It’s known to have a bit of overhead in its request handling. Other frameworks like Fastify or Koa can beat Express on throughput. Express is ideal when you want maximum flexibility and a quick start to a project.

Express has stood the test of time. It is mature, stable, and proven in production. Many high-traffic websites have been built with Express, an Express app can certainly scale to handle very large loads. Since Express leaves structure to you, it can scale organizationally if you apply good conventions. But you won’t get as much built-in support for scaling as you would from a more structured framework like Nest.

Using Fastify

Fastify is a newer player than to Express, but it has gained a lot of traction due to its performance. As the name implies, Fastify was designed for fast Node apps. This raw speed comes from an event-driven, highly optimized internal architecture with minimal overhead per request.

Fastify’s API and concepts will feel familiar to you if you already know Express. You still register routes and handlers, and you can also use middleware (though in Fastify they’re called hooks or plugins). The learning curve is moderate, certainly more than express. This is because it’s a bit more structured than Express, pushing you to define JSON schemas for your requests/replies that can be used for validation and serialization. This schema-based approach is optional but strongly recommended for the framework as it helps catch errors and improve performance.

Fastify does have an official plugin system with many plugins available to add features. The documentation is solid (I read tons of documentation), and the community is growing fast.

Performance is Fastify’s headline claim to fame. It consistently ranks among the top Node frameworks in terms of throughput and low latency. In a real-world context, this means a Fastify server can handle more concurrent requests on the same hardware than an Express server, which gives you room to grow.

If you’re autoscaling your Node app (you should be), Fastify’s low overhead is even more beneficial. Each instance can do more work, potentially reducing the number of instances you need and certainly making each scale event more granular. There’s no special magic beyond this. Because each process is more efficient, your scaling is more cost-effective.

A great use case is building APIs where performance is the most important requirement. Fastify stands out in use cases where every millisecond counts and in resource-constrained environments. The trade-offs are relatively minimal. Fastify isn’t as universally known as Express, and its community, while enthusiastic, is smaller than Express’s. That said, it’s not obscure – plenty of big companies use Fastify.

Using Koa

Koa is often described as the successor to Express. It was actually created by the same team behind Express, with the goal of making a smaller, more modern framework. Koa is even more minimalist than Express, shipping without any middleware (not even a router!). This sounds daunting if you’re used to something batteries-included like Ruby on Rails, but it’s by design.

Koa aims to strip the core down to just the essentials, leaving it up to you to add only what you decide you need. Koa’s core library is tiny, only spanning around 600 lines of code. It leverages modern JavaScript features like async/await, which allows Koa to avoid the legacy of callback-based middleware that Express 4 uses.

If you already know Express, picking up Koa is straightforward (conceptually, at least). You still create a server, define routes, and use your chosen middleware. Many developers find Koa’s usage of async functions makes middleware composition and error handling simpler than Express. The learning curve is moderate, more of a challenge than the other Node frameworks we’ve discussed so far. You will need to assemble your own stack of middleware, which means reading documentation and doing a proper implementation for those pieces. Koa intentionally does not maintain compatibility with Express-style middleware, so you can’t just drop in an Express middleware and expect it to work.

Koa’s lean design does have performance benefits. With less overhead, Koa can handle more requests per second than Express in comparable scenarios. In practice, Koa’s performance is pretty good for a Node framework, but not as good as Fastify.

Koa is a great choice if you want a modern, minimal framework and are comfortable assembling all the pieces you need. It’s well-suited for building APIs or web services where you might otherwise use Express, but you’d prefer a cleaner async/await-based flow. It’s like Express but faster and a bit more refined, at the cost of doing a little more setup on your own.

Using Nest.js

NestJS is a different paradigm from Express, Fastify, or even Koa. Nest is a full-featured, batteries-included framework. The idea behind Nest is to give Node applications a structured architecture out of the box. It adopts a modular approach, with support for TypeScript and features that will remind you of fuller frameworks like Rails or Laravel.

Working with Nest can be a mix of excitement at what’s included and dread all there is to learn. The developer experience is driven by the fact that NestJS forces a structure. Personally, I love this, but I’m already a huge fan of convention over configuration. It also makes Nest great for large teams since everyone knows where to put their code and how things should be structured.

The framework comes with a CLI that can generate boilerplate, just like Rails scaffolds. However, the learning curve is steep. You need to get comfortable with lots of concepts before becoming productive, since they affect the overall flow of Nest apps.

A NestJS application uses the Express framework under the hood. This means that Nest’s performance will be somewhere in the ballpark of Express, maybe even a bit slower due to the overhead. Still, you can swap out the underlying HTTP adapter to use Fastify instead of Express. With a one-line change, your Nest app can run on Fastify and instantly get a big boost in throughput.

That said, pure performance is usually not the deciding factor in choosing to use Nest. Most people who choose Nest do so for its organization and convention. It shines in large, long-term projects where maintainability is key.

NestJS does not include a frontend framework. It is purely a backend framework, but it can serve a frontend if you want it to. Since it’s just a Node app (built on Express or Fastify), you can add a view engine like Handlebars to render server-side HTML templates.

NestJS does not ship with a built-in ORM, which makes me sad! It does, however, cleanly integrate with whatever data access layer you prefer. The community has produced official and semi-official integrations including TypeORM and Primsa.

If your project will be worked on by many developers, or you foresee it growing significantly in scope, Nest provides the architecture to manage that complexity. It’s also great if you prefer strongly typed code and are a fan of Angular or Java-style frameworks. Many companies choose Nest for mission-critical apps because it imposes a consistent structure and has many reliability features built in.

Using Next.js

Node.js is also used in full-stack frameworks that integrate the frontend and backend together. Next.js is a framework for building web applications with React (frontend) and Node.js (backend) together, but it’s best described as a frontend-first option. With Next, you can render React pages on the server (SSR), generate static sites, and define serverless API routes all in the same project.

I have a love/hate relationship with Next. It has a generally good developer experience, especially for frontend engineers who want to expand into backend without leaving the React ecosystem. Next handles all the build tooling, code splitting, etc. The local development experience is genuinely great, boasting hot reloading, actually useful errors, and so on. Still, the learning curve Next is moderate. You need to know React, and then learn Next’s conventions. That sounds simple, except those conventions change in breaking ways relatively often. It’s not uncommon for me to visit an old Next project and find it needs lots of work to bring up to date.

Next.js is best for scenarios where you need to build a frontend interface along with your backend logic. It allows for server-side rendering (SSR), which means pages are pre-rendered on the server for fast initial load and SEO, as well as static site generation (SSG) where pages can be precomputed at build time and served as static files.

Since Next is a frontend-first framework, it does not provide an ORM either. If you need an ORM, you add it yourself just like you would in a plain Node.js app.

It blends the lines between a frontend framework and a backend framework, which may be a selling point or a problem depending on your perspective. That said, for many teams, the productivity gain outweighs these concerns. Next.js is often recommended for full-stack apps because it offers great performance to the end user and fairly easy scalability (if you deploy to a platform like Vercel, and it scales automatically).

How to make the right decision

Choosing the best Node framework for your project is hard. Of course, it depends on what you’re building and what your priorities are. It’s not one-size-fits-all. Still, I’ll do my best to give you a straightforward decision-making framework.

For quick prototypes of simple APIs, just use Express. Its simplicity is unbeatable for getting something off the ground fast. You won’t waste time fighting the framework or learning new conventions.

For high-performance APIs, use Fastify. If you expect heavy load or just want to maximize efficiency, Fastify will give you more headroom. It’s a great choice for building JSON APIs or microservices where throughput is critical. The trade-off is a slightly smaller community and middleware options, but the core features are pretty good by themselves.

Koa is an option if you like to keep things lightweight and are interested in using something super modern. It’s a nice upgrade from Express when you want to embrace async/await and have more control. Choose Koa only when you are comfortable assembling your own toolkit and you want a framework that stays out of your way.

For large-scale, enterprise APIs, NestJS is often the best choice. If you’re building something that might grow to dozens of modules, with several teams working on it, Nest provides the architecture to keep it maintainable. It’s the go-to if you prefer strong conventions, TypeScript, and a full-featured backend framework that comes with everything included. It scales just like any other backend option and has a great structure that’s easy to lean on.

For applications that need a strong frontend component or you simply like having both parts of a project in one framework, Next.js is hard to beat. Naturally, the full-stack nature of the platform gives you plenty more options for your work than Nest or one of the backend options. If your project is essentially a React app that needs server-side rendering or will benefit from static site generation, Next.js will make your life much easier.

I couldn’t help but add Rails and Laravel to this chart to make the comparison even more clear. I’m quite attached to those frameworks’ batteries-included approach to building web apps. With either framework, you can build fully-featured CRUD apps that support user authentication and a dozen other common features, all without adding a single library. Next is the only Node framework that comes close to this amount of support.

That being said, all of these frameworks are perfectly capable of building useful, scalable applications. Just keep in mind that they offer different balance of convenience, performance, and structure. Good luck choosing!

We’ll organize this like a listicle, but we’ll connect the dots as we go so it reads like one story: what queue time really tells you, why shared hardware is noisy, how scheduling fits in, and how to keep your scaler from thrashing.

1) Shared hardware is noisy

Multi-tenant machines are cheaper because they’re shared, but providers overprovision their shared machines. It’s not “well we have 10 cores so we can host 10 apps, each getting 1 core of CPU”. Depending on how much the hosting provider wants to profit cram applications into a single host, it could be a lot more like, “well we have 10 cores and apps tend to underutilize what’s available so let’s stuff 30 apps onto those 10 cores” 😬

When your app shares CPU with other tenants, one neighbor’s burst is another neighbor’s tail‑latency blip. We’ve talked about this several times, but the “noisy neighbor” effect is not groundbreaking or new. It’s also not a single-provider issue: every hosting platform that operates on shared hardware is subject to some degree of neighbor noise!

The takeaway: get familiar with what noisy neighbors look like in your app. Learn to read your charts so you can tell app‑wide issues from single‑dyno outliers (the noisy‑neighbor tell).

The action item: enable Judoscale’s Dyno Sniper (by-request only as of September 2025) feature to automatically detect and restart services that fall prey to a noisy neighbor’s delay. It’s free. It’s magic. It works everywhere. There’s really no downside.

2) Autoscaling “for headroom” is hard (most teams miss it)

Many teams use autoscaling specifically to keep a certain level of headroom available for unknown bursts of traffic ahead. It’s not the most efficient way to run an app, but the premise can prevent downtime for highly burst-prone traffic loads. That said, actually configuring your autoscaler to do that correctly is extremely difficult. Most teams end up very over-provisioned constantly (e.g. wasting money! 💰) or without the headroom they desire, and alerts when bursts arrive 🚨.

Most autoscalers operate on what we’ve come to call “reactive metrics”. These reactive metrics are excellent. They’ve always been excellent. When you’re using an autoscaler that’s watching them and reacts quickly (like Judoscale!), reactive metrics are absolutely the right answer for 90+% of applications. That said, if you’re in the other 10% (specifically looking for autoscaling-with-headroom), reactive metrics aren’t the right tool. If you need to maintain a proportional overhead of capacity relative to your scale as it changes, you’ll need Judoscale’s custom Utilization-based autoscaling. It allows you to say “Keep me at about 70% capacity utilization so that I always have 30% overhead”.

The takeaway: to our knowledge, Judoscale is the only autoscaler out there that offers true proactive, headroom-prioritized autoscaling via custom “Utilization” measurement 24/7. If you need that kind of behavior, get Judoscale installed and activated. You’ll be surprised at how useful extra headroom can be in high-burst loads!

3) Queue time ranges for healthy apps are lower than we expected

After several years of watching both customer queue time data as well as our own, we decided to lower the default queue time threshold for new apps on Judoscale. While that change could be worth an article of its own, suffice it to say that our realization was based around the stability of shared and dedicated hardware. Queue time thresholds for dedicated hardware (think Heroku’s Perf- series and/or Fir platform) can be very low. As in, “scale up if queue time hits 5ms”. And that’s ultimately a reflection of a very stable and operational stack — a request hitting Heroku, getting routed, and hitting your dyno consistently and predictably. The moment that dyno begins queuing requests for more than just a few milliseconds, we can be sure there’s a capacity problem.

Our realization came regarding shared hardware (Heroku’s Std- series). We’d long considered a higher default queue time for those tiers since shared hardware can experience blips of queueing even though you don’t have a capacity issue (yet) — those are the moments your neighbor’s code is running instead of yours.

What we found is that shared hardware can actually operate on the same degree of stability and low queue time threshold as long as you’re carefully and deliberately staying within the bounds of your own “Dyno Load” or overall “shared slice” lane. We’ll write about this more in the coming months, but here’s the takeaway: keep an eye on, and be careful about, how much of your slice of the shared hardware you’re really using. On Heroku, this means being careful about that “Dyno Load” metric. If you consistently push for more Dyno Load than you’re actually allocated, you’re going to have a bad time.

4) Scaling to zero is a super‑power (for workers)

Event‑driven workers don’t need to idle. If there’s no work to do, you shouldn’t be paying for workers. Sure, keep at least one of your low-latency queue workers around all the time — we do too. But if you’re following our “Opinionated Guide to Planning Your Sidekiq Queues”, you should allow both your less_than_five_minutes and especially your less_than_five_hours queues to scale to zero. Cold-start times once jobs actually hit those queues are around 30-45 seconds (YMMV) so you’ll be well-within your queue time SLA… while saving free money 💰.

The takeaway: scaling background job workers to zero when there’s no work in the queue is free cash in your pocket. Set up your Judoscale schedule and scaling range to allow for zero-scale and enjoy the free money you save 😎. Then, while you kick back, give our Sidekiq Queues Guide a read!

5) Y’all have too many job queues

We’ve seen it all: the “one queue per job class” approach, the “every feature gets its own queue” approach, and, of course, the “we’re just using this queue one time and we’ll clean it up right after” approach. All roads lead to more queues. The thing is, lots of queues create real issues:

• More queues means more queues to watch, and that means lots of polling and overhead from your job system. Sidekiq’s own docs stress that they “don’t recommend having more than a handful of queues per Sidekiq process” and

Lots of queues makes for a more complex system and Sidekiq Pro cannot reliably handle multiple queues without polling… slamming Redis.

• Long term maintenance of jobs, orchestration, and priorities is terrible when you have that many queues to think about. It’s more than you can comfortably hold in your head.

• Setting up autoscaling policies across 10, 15, or even 20+ queues is a massive headache. Trying to keep them all reasonably in-sync is worse. It’s not worth it.

You end up here:

Trust us, you don’t want to end up there.

Simply put, if you’ve got more than about five queues, you’re probably headed in the wrong direction. We urge and recommend only having three! And we recommend naming them based on an expected queue-time SLA, then setting your autoscaling up for each queue to reflect that. It’s a beautiful, job-agnostic way of handling queues!

The takeaway: read this guide and audit your background job queues. If you’ve got more than five, determine why and what should be merged. KISS!

6) On the web side, downscale by one (almost always)

Several of our larger-app customers have run into interesting headaches and unforeseen issues by downscaling by more than one at-a-time. Judoscale’s highly configurable nature does allow you to do this, but we’ve learned over time that you probably shouldn’t.

The reasoning is simple: cost vs. benefit. The benefit, in short, is that you’ll shave a little off your bill. If you’re going to downscale off some dyno load anyway, doing it in bigger steps means slightly larger savings accumulated at the end of the month! ‘Slightly’ is the key word there. The cost, however, can be less pleasant. You can end up downscaling too far! At that point your users could experience slowness, you could spark alerts, and you’ll inevitably upscale again soon. There’s just no need for all the thrashing.

The takeaway: downscaling by more than one at-a-time, for web services, isn’t worth the marginal gains. The risk of downscaling too far is real! Stick to downscaling by just one-at-a-time.

Our next step: we actually feel pretty compelled to remove this option altogether in the future. It’s exceedingly rare that downscaling by more than one service at a time is the right move. TBD, but this lever may disappear!

7) Understand intra-dyno concurrency

The short version of this story is based on understanding the interplay between request routing and how requests are handled within a single service. Many PaaS’s use simple random-based request routing: a new request can go to any active service/instance in the cluster. It’s not intelligent or load-based. So a single service could receive multiple requests in a row, even while processing its prior, while another service may get none!

It’s important, then, that each single service is able to handle multiple requests concurrently. Otherwise those randomly-routed new requests will be queued and you’ll have a consistently higher, but sporadic, average queue time for your app. 😬

In each of the runtime languages we support autoscaling for (Ruby, Python, and Node.js), the web frameworks cannot process requests in true-parallel (concurrently) per process — only asynchronously in alternating execution. That’s a mouthful, but we recently wrote a post that walks through that idea with great diagrams; give it a read here! The key is that a single process isn’t capable of true concurrency, but running multiple processes within your service does yield intra-service concurrency (the ability for a single service to handle more than one request in true parallel).

The takeaway: wherever possible, run more than one process per web service instance (“dyno”, “service”, etc.). This is a particular challenge on Std-1x-style, single-CPU-core, service tiers. But, if all other variables are held constant, it’s better to run a single service with two processes than two services that each have a single process!

8) Scheduling: boring, obvious, and wildly effective

Scheduled scaling is the easiest money‑saver with the least risk. If your traffic has a weekly rhythm, tell Judoscale about it. Keep your minimum scale higher on weekday business hours, then drop it during nights and weekends (for example). You can even leverage a tighter schedule to pre-scale up before large releases, big events, and other known spikes!

You can even get the best of both worlds by running a schedule and autoscaling together. Instead of scheduling hard-locked service counts, you can schedule the range of scale that you want to tweak at a given time. That gives you the flexibility to respond to unknown load changes while still controlling baselines in accordance with known load changes! 🎉

Of course, if you choose to not build a schedule, autoscaling itself will ensure that your capacity grows and shrinks to meet need. But depending on how sharp your traffic changes can be at known times, you might either end up with: 1) a few minutes of slow service as autoscaling ramps up your capacity amidst a large traffic burst, or 2) wasted dollars as your baseline (minimum) service/dyno count stays higher than it needs to be on off hours!

The takeaway: take a couple of minutes to think about your app’s weekly traffic patterns by day (even by hours!). Also consider any weekly rhythms your app has in terms of events, releases, or other in-domain things which drive influxes of users! Then bake all of those into a dynamic autoscaling schedule with Judoscale ✨

Wrapping up

If there’s a single throughline in all of this, it’s that autoscaling is a feedback loop living in a noisy world. The mechanics aren’t mystical: measure what users feel (queue time), add capacity when you’re near the edge, and give your system enough time to absorb changes before you decide again. Do that consistently and the chaos of shared hardware, random routing, and spiky traffic stops feeling like chaos.

Choose boring on purpose. Keep your queues few and meaningful. Let workers scale to zero when there’s nothing to do. Downscale web by one so you don’t saw off the branch you’re sitting on. Run real in‑dyno concurrency so random routing doesn’t turn little bursts into instant queue time. And if a single dyno is having a uniquely bad day, assume a noisy neighbor before you assume a rewrite (and let Dyno Sniper handle the whack‑a‑mole so your team doesn’t have to).

That’s it. No silver bullets—just a handful of defaults that make your platform feel calmer and your costs feel saner. After nearly a decade of watching this stuff in the wild, the “secret” is that stability isn’t a hero move; it’s a series of small, boring, repeatable decisions. Make those decisions once, encode them in Judoscale, and let the loop hum 🔁