How Propshaft Works: A Rails Asset-Pipeline (Visual) Breakdown

With Propshaft just recently reaching v1.0 (🎉) we thought it would be fun to demystify the magic — to explore what Propshaft actually does under the hood! After all, it’s now been around a couple of years, it’s the new default for Rails 8, and it’s the backbone for Rails’ simple asset delivery. There’s plenty to cover so let’s spin this thing right up!

MVC: A Review

As we all know, Ruby on Rails is a framework based on the Model-View-Controller pattern of web application software. And we’ve probably all seen diagrams of this at some point before:

That’s a great high-level diagram for visualizing how an MVC architecture handles a request! If we dig just a little bit deeper and put some real example paths and concepts into place, the diagram grows a bit (the example being a blogging application with many Posts):

But we still always have the MVC triangle: controllers that pull data from models and render views with that data. This is just how fundamentally MVC works! Even the largest apps are essentially following this model. Controllers, models, and views all working in tandem to answer a request.

But this model explains how requests to application resources work. That is, if you request an endpoint declared in the routes — an endpoint that requires some dynamic code to run and determine what to output. What about when we need things that aren’t dynamic or don’t require any code to run to be servable?

The simplest example of this is an image. Say that our show.html.erb example view now includes an img tag:

<h1><%= @post.title %></h1>
<p><%= @post.body %></p>

<img src="/beautiful-butterfly.jpg">

We tend to see that path and automatically associate it with a file because it’s named like a file, but let’s not jump ahead here — that’s still a request path! When the client’s browser receives the initial response to /post/hello-world and it starts to render that img tag, it’s going to issue another request to the path /beautiful-butterfly.jpg. Our application is going to receive an incoming request for that path:

Of course, we probably do have a real file called beautiful-butterfly.jpg that we intend to serve up for that img source. But how do we get it there? We’re not going to create a route and a controller that serves up just that file! There’s no need — we just mentioned that this is an example of when we don’t need dynamic code to run; this is a static resource.

In fact there are a few questions to be answered here!

• How do we actually serve static files (like images) in response to requests coming in for those file paths?
• How do we know which incoming requests are for static resources and which are for dynamic routes in the first place?
• What all counts as a ‘static resource’ that should be served this way, anyway?

Serve it Like it’s Hot

Let’s start with the first question above — how do these static files actually get served to incoming requests for them? The answer actually relies on a Rails convention: where we put our beautiful-butterfly.jpg file. Rails’ directory structure has a lot of opinions and this one is no different — we’re going to put our image into the ~/public directory. Technically what happens under-the-hood is slightly different when running locally vs. running in a production environment, but the net result is the same: any file (which can be nested under subdirectories) in Rails’ public folder will be made directly-accessible as a URL path.

That is, if we put beautiful-butterfly.jpg (the file) into the public directory, then requesting /beautiful-butterfly.jpg (the path) will automatically resolve to that file!

When we’re running the application locally in development mode, this slight-of-hand is handled by the ActionDispatch::Static middleware. Every time a request comes in to your local server, before it hits the Rails app (!), the Static middleware checks to see if you have a file in your public folder that matches the requested path. If you do, it’ll serve that static file and the request will never even hit the Rails application! (Excerpt from this code, comments mine):

# ...
# Static asset delivery only applies to GETs and HEADs
if request.get? || request.head?
  # `find_file` checks for a matching filename in `public`
  if found = find_file(request.path_info, accept_encoding: request.accept_encoding)
    serve request, *found
  end
end
# ...

resources :posts

example-app/
└── public/
    └── posts.html

Alternatively, when running your application in production mode, there are a couple of ways that static files can be delivered. As Heroku still tends to be the go-to option for new Rails applications, it’s worth calling out specifically that, on Heroku, Rails serves its own static assets just as noted above (via the Static middleware).

On other platforms and setups, a dedicated web-server can operate ‘in front’ of a Rails (Puma) server as a reverse-proxy and serve contents from Rails public directory directly. That’s a much more complicated setup so we won’t cover it here, but the end result is the same: the web-server serves the static contents directly and requests for static things never hit the Rails application.

So…

How do we actually serve static files (like images) in response to requests coming in for those file paths?

We don’t! We simply ensure that we’re putting our static files into the right directory then let Rails’ middleware automatically intercept requests to file-name-matching paths. Everything should just work™️. And, while the goal here is just figuring out how to serve static files at all, one of the nice side-benefits of serving static content with a Rack middleware hook is that it keeps our Rails app un-busied with requests for static content! The Rails application never sees them. ✨

What is Static, Really?

Let’s take a look at our second question above:

How do we know which incoming requests are for static resources and which are for dynamic routes in the first place?

We need not discuss this too much as we’ve mostly answered the question in our answer to the first question a couple paragraphs ago. The short answer is that the Static middleware checks every request’s path against all the files in your public directory for matches. For example, let’s assume we have a public directory like so:

example-app/
└── public/
    ├── beautiful-butterfly.jpg
    ├── jons-family-recipes.html
    ├── secret-page/
    │   └── index.html
    └── jons-favorite-books.pdf

With that setup, requests to any of the following paths will be served as static content:

• example.com/beautiful-butterfly.jpg
• example.com/jons-family-recipes
• example.com/jons-family-recipes.html
• example.com/jons-favorite-books.pdf
• example.com/secret-page
• example.com/secret-page/index.html

Now, given those six matchable paths, we can answer our question, “which incoming requests are for static resources and which are for dynamic routes in the first place?”

Any incoming request which matches one of those six paths will be served the static file content via the Static middleware. Any other path will be considered dynamic and get passed along to the Rails app.

Static Everything

Let’s tackle our third question:

What all counts as a ‘static resource’ that should be served this way, anyway?

There are really two answers to this question. The first is the snarky, unhelpful approach: anything in public is a static resource and will be served as such. The second answer is more around the “should” in the question. What kinds of things should we put in the public directory? What is a static file?

Well, the correct answer, unhelpful as it is, is that anything which doesn’t require dynamic processing and/or doesn’t need to be generated by Ruby code is a static thing. But more concretely, let’s look at some examples of files that are typically static in a Rails application.

First, media files. That includes the imagery (jpgs, pngs, etc.) and/or iconography that goes into your site’s design and pages — just like the beautiful-butterfly.jpg example above. These files don’t need to be processed by Ruby and aren’t generated by Ruby. They’re fully produced when exported from Photoshop and ready to go. We just need to serve them.

Second, non-interactive documents. This would be things like PDFs or even Excel spreadsheets that users might download from your site (or view in their browser, in the case of PDFs). They’re prefabricated files, so Ruby’s not interacting with them or generating them, thus, they’re static files.

Third, site-maps and robots! You might not think about sitemap.xml or robots.txt as being static files off the top of your head, but they are! Ruby isn’t generating them on an as-requested basis — they’re generated ahead of time (at some point) and stored, ready to be delivered to whoever asks for them. That’s a static file!

Fourth, font files. Lots of sites are using fonts from various web-based providers (Google Fonts is everywhere), but many teams still opt for serving their font files directly from their own project. Again, being a file that Ruby has nothing to do with, it’s a static file perfect for the public directory!

Finally, your Javascript and CSS. These are static files! It may not feel that way as you refine them and iterate on them during development, but think about what happens when your Rails application serves requests in production. Is Ruby regenerating any of your Javascript or CSS files in real time? Is it injecting anything dynamically into those files? Unless you’re doing some wild stuff, the answer is no. In perspective to Rails, all of your Javascript and CSS files are static files!

But, your Javascript and CSS files should not be in the public directory. Super contradictory to what we just said, right? Hold that thought. Just remember that they are static files.

An Updated Model

This article is called a visual breakdown, so let’s get back to the visuals! Now that we understand the role the Static middleware is playing and how it determines what to serve, we can add it to our traditional MVC diagram:

Look at all those files ✨

Now if someone requests a path which matches one of the file names in public, they’ll get the file contents!

And if they request any other path, the middleware will be ignored and the request will go along to Rails (whether or not Rails can actually answer it!).

But How Static?

You might’ve noticed that we haven’t actually talked about Propshaft once yet despite this article being titled “How Propshaft Works”. There’s good reason for that! We need to understand what static files really are before we introduce the asset paradigm.

We previously broke down the concept of static vs. dynamic by thinking through whether or not a back-end runtime (Ruby) needs to generate or modify a particular piece of content in order to serve a request. And we determined that files are considered ‘static’ when they’re fully ready to serve without Ruby having to do anything with them. In this way, we’re using the word “static” as “unmodified”.

But let’s talk about “static” in terms of time — as in, “unchanging over a long period of time”. The key question here is, just how static are these files? Again, examples will help us through!

If we’re talking about a static file like a favicon.ico, there’s a good chance that it may never change. Once you add it to your app, it’s there forever. Maybe you go through a rebranding once a decade. Regardless, a favicon tends to be extremely static.

Going down a level, maybe you have imagery on your homepage (stock images, team photos, etc) like our screenshots on the Judoscale home page:

You’re likely not changing these all that often, but it could be once or twice a year as you develop fresh content and design new marketing strategies. They’re static images, but they’re less static (in terms of time) than your favicon.

Finally, let’s talk about your Javascript and your CSS. There’s a good chance those are changing with every application deploy! These are still static files from a Ruby standpoint (in the “unmodified” sense), but they’re much less static than other files from a time sense (they change often)!

Why does any of that matter?

Caching.

Let me explain it with a metaphor. Let’s pretend that we’re both big rockstars 🤩. You know: the lights, the glamor, the #1 singles on Spotify — we’re basically both Taylor Swift. And we both have agents, of course. Let’s also say that we’re working on a secret demo together; the world has no idea what’s coming! Our agents, essentially:

Now, I’ve been working on this demo quite a bit the last few weeks, and I setup a plan with my agent to ensure that your agent gets a copy of my latest work as much as possible. I’m trying to keep you in the loop!

So every day my agent makes a photo-copy of my paper music sheets, puts them in an envelope, writes our super-secret song name on the front of the envelope (“Judoscale Bop”, what a jam), and takes the envelope to go meet with your agent.

Now, our agents are great at being agents, but they’re not musicians. They can’t read music! And, unfortunately, when my agent gives your agent the envelope, your agent simply says “Oh, I already have this!” and doesn’t take the new envelope.

But my agent persists, “but this is the new one!”

Alas, your agent simply states, “looks the same as the last one to me.” And moves on with their day. You might need to fire your agent.

End result? You don’t get the new music. Our rockstar relationship is on the rocks. The world crumbles without our new music. Pandemonium. Sadness. 😭

Caching Sadness

Okay, okay, metaphor aside, this is actually a decent representation of what happens with static assets when caching doesn’t go well. Back to the real world and our Rails application’s public folder, let’s think again about our javascript. If we have a file in our public folder simply called application.js, that’s great! It’ll get served when example.com/application.js is requested and should work fine.

The issue comes when we update that file. Servers, networking hardware across the internet, and even users’ own browsers really like to cache static files as much as possible. And that’s fair! The only thing better than a static file that loads ultra-fast over the internet is one that you already have stored in memory on-device! But what happens if we release a new version of our application.js while users’ browsers think they already have the most up-to-date version?

It’s the agent problem from our metaphor: “application.js? I already have a copy of that, I don’t need a new one!” — not realizing that the ‘new one’ is actually totally different in content from the old one. Just because they’re named the same doesn’t mean they are the same!

That’s exactly why we said above, “your Javascript and CSS files should not be in the public directory.” Because they change! Frequently!

In fact, anything which changes with any real frequency (let’s say more than once a year) should not live in the public folder!

So where should they live? And… what should they be named?

Hello, Assets! What’s Up, Propshaft 👋

The good news for us is that the brainy Rails developers of yore already solved this problem. It’s actually quite old news! Propshaft is certainly newer, but the general premise of fixing our agent-problem is not.

The idea is simply this: we’ll put all of our often-updated static files into a special directory, inside the Rails application, called “Assets” (you can think of it as “Static Assets” for more clarity). Then, instead of serving them with their plain-old file name, we’ll add a random value (a hash) to the end of the filename based on the file contents. That way when the file contents change, the hash changes, and thus the file name changes!

# Just like...
application-8fb79e02.js

It’s just like having a secret code on every Judoscale Bop envelope! A little indicator that can immediately tell a potential recipient that something inside is different without having to open it up and find out. Check it out!

Where 9A1 and 2CC indicate that the two envelopes’ contents are different, so too do asset filename hashes. So much so, that we call this a ‘fingerprint’. As in, a totally unique representation of the file. As in, if one little thing were to change inside the file, the fingerprint wouldn’t be the same.

Seeing fingerprint differences is just as easy as seeing the two different codes on those Judoscale Bop envelopes. Here’s two different versions of an example application.js file:

# Before we released new changes
fingerprinted_js = "application-3f97d6e2b8f7a34c56ab.js"

# After we released new changes
fingerprinted_js = "application-bc12a4d7e9fda123a0c5.js"

So… what is Propshaft?

Propshaft is essentially the mechanism responsible for adding the hash to each asset’s filename and preparing the hashed file to be served! Unlike tools of the past (more on that later), that’s all Propshaft really does. It’s a simple job: convert asset files to their hashed-filename-equivalent and copy them to where they’ll be served. And where is that? Well, public of course!

That’s right! We now put all of our frequently-changing static files into the assets directory, but ultimately they end up back in public thanks to Propshaft— just with their hash-name instead of the plain file-name we give them. That shouldn’t be too much of a surprise, given that, again, these are still just static files and static files should live in public, but it really comes full-circle here. Everything static ends up in public, we just needed a tool to fingerprint our frequently-changing assets before moving them to public so that they can be cached by consumers according to their (fingerprinted) filename.

What a mouthful! Let’s put it into our visual.

curl example.com/application.js
#=> 404 Not Found!
curl example.com/application-8fb79e02.js
#=> 200 OK!

One last detail worth talking through here — most Rails developers actually put all of their static assets (even the never-going-to-be-updated ones) in the assets folder and nearly nothing in public/ directly. And that’s fine! Since it all ends up in public/ post-Propshaft anyway, it’s really neither here nor there. There’s definitely no prescription or convention around this particular topic in the Rails universe.

The only trade-off between the two is that static files in assets/ must be referenced using the helper methods (hold that thought), whereas static files in public/ must be referenced without any helpers — they’re hard coded:

<%# Example of referencing a file in the public directory: hard-coding a root-relative path  %>
<%= image_tag "/my-evergreen-images/orly.jpg" %>

Which… isn’t a favorite of many Rails developers. Using helpers just feels more correct and connected. But wait, what helpers?

On Actually Using Assets

It’d be rude to talk through all of the how of Propshaft without, at least briefly, covering the how to use side of things. In short, since Propshaft is going to fingerprint all assets/ files, we can’t write our HTML markup expecting to load the non-fingerprinted filenames (see the Note just above):

<!--This will 404! -->
<script src="/application.js"/>

But at the same time, we don’t manually handle the fingerprinting process ourselves, so we also wouldn’t know what hash value to put in the filename either. Even if we wanted to hardcode it, we really can’t!

<!--Ugh, my brain -->
<script src="/application-????????.js"/>

Well good news. Propshaft automatically enhances the javascript_include_tag helper with fingerprint injection! So we actually can simply write:

<%= javascript_include_tag 'application' %>

And what will get printed to the DOM is the fingerprinted file name that matches what Propshaft is generating from the current file contents!

<script src="/application-8ca5d7333bbb.js"/>

And it’s not just Javascript! Propshaft natively includes a bunch of asset-finding helper methods for other types of things:

• stylesheet_link_tag
• audio_path
• font_path
• image_path
• video_path
• Many more…

Some tag helpers are even enhanced to automatically convert a file name to an asset-path and inject it into a tag all in one step. Something like <%= image_tag "some_image.png" %> does exactly that!

Now we have both sides of the coin. We just need to throw assets into the assets/ directory and use our built-in Rails path helpers to reference them correctly. We get to promptly forget about fingerprinting and pretend it doesn’t exist, and Propshaft gets to fingerprint freely without worrying about us breaking stuff! Sweet development harmony.

Propshaft is the Future

Finally, let’s close this way-too-long article with a few details about Propshaft from an ecosystem and curiosity standpoint.

What’s the deal with v1.0? Propshaft has technically been around a year or two now, and even in some production apps since the beginning. Given that it’s such a simple project and essentially only does one job, its reliability since day one has been solid. As with many projects, we believe the transition to v1.0 simply implies that the API has been settled and should now be considered stable. There was no large feature or functional change between the prior version (v0.9.1) and v1.0.

This also likely has something to do with Propshaft having been declared the future of Rails’ assets handling. Not that Propshaft’s API was unstable in its previous versions, but knowing that a particular library is soon to be the back-bone of Rails certainly brings some pressure to settle your pre-v1.0 discussions.

What was asset handling before? We didn’t want to get into it much in this article, but prior to Propshaft, handling assets in Rails was significantly more complicated. Sprockets is the tool that Propshaft replaces and Sprockets was made to do so much more than just fingerprint-and-copy. Too much, in most folks’ 2024 opinions. Thus, Propshaft was built to be much, much less.

Why is it called ‘Propshaft’? DHH has yet to actually mention specifics of why it’s named Propshaft, but we think that, like most software project names, it’s a metaphor. Of course, the name itself is “propeller shaft” — the linkage axle between a propeller airplane’s engine and the propeller itself. In those systems the propeller shaft is what transmits all of the power from the engine to the physical propeller, so perhaps it’s something similar here. (Starts hand-wavy gestures) Propshaft transmits the power of our own written Javascript out to the open internet ✨ or… something like that. Ultimately it’s surprising that DHH didn’t go with “Driveshaft”, the same concept but for automobiles!

‼️ Don’t Miss Part 2

There’s one more major piece of the static-files / static-assets puzzle that we can’t miss. We mentioned it very briefly above, but we need to have a fuller discussion about the giant, lovable, confusing, scary, but-also-friendly, Content Delivery Networks.

CDN’s play a huge role in how assets get to our end-users, the performance of our web-application back-ends, and how Propshaft fits into the stack! This will be a part-2 follow-up topic to this post, coming soon to a Judoscale Blog near you. You won’t want to miss it!

Sign up to the newsletter below and we’ll make sure to let you know when it comes out.

P.S. For further reading, feel free to check out the Rails Guide on The Asset Pipeline. The guide is still based on Sprockets and includes a lot of notes on asset preprocessing that most newer or well-updated apps are no longer doing, but it’s a great read, has some key tips, and gives a good perspective of where we’ve come from.

P.P.S. Do you know how few gifs of propeller planes there are?! I surely didn’t, but it turns out there just aren’t that many prop planes out there having videos-turned-to-gifs taken of them! What a shame.

P.P.P.S Okay, if you really want to hear the Judoscale Bop… here you go 😏

We’re, um.. much better autoscaling engineers than musicians 😅