Noel Rappin Writes Here

Or: Rails and JavaScript, Part 5

A quick program note: If you like this newsletter, you might like my recent books: “Modern Front-End Development for Rails” (Ebook) (Amazon) and “Modern CSS With Tailwind” (Ebook) (Amazon). If you’ve already read and enjoyed either book, I would greatly appreciate your help by giving a rating on Amazon. Thanks!

If you are really interested in how we got here, I wrote about the history of Rails and JavaScript, read Part 1, Part 2, Part 3, and Part 4.

Over the last few weeks, DHH and the Rails core team have announced a number of different JavaScript build tools for use with Rails 7. As I was writing this very post, DHH wrote his own post outlining the strategy for Rails 7 and JavaScript.

I’m going to try and sort out what it all means for you as as a potential Rails developer.

A couple of warnings:

• This is all unreleased and new, it is guaranteed to change, though I think that now that DHH has announced the whole strategy, it might have stabilized some. All the relevant tools are still being updated, though and the integrations and implications are still not clear.
• I’ve been testing these by upgrading Elmer, my project tracking tool. I like Elmer, but it’s kind of small and it uses Hotwire, so it doesn’t have many JavaScript dependencies. Larger projects with more complex JS dependencies will probably have other issues.

Here goes.

Rails 7 offers about five different of ways to interact client-side code from a Rails app, depending on how you count. Omakase, baby!

• The classic asset pipeline route via Sprockets and manifest files still will work as far as I can tell, but I think you might want to look at a newer tool.
• Webpacker is still under active development, and should release a new version more or less simultaneously with Rails 7. I definitely take from DHH’s post, though, that Webpacker is soft-deprecated in favor of the JS Bundling approach.
• Rails 7 will support “JavaScript Bundling” as of literally 10 minutes ago as I started this post. The JavaScript bundling tool uses the existing Yarn and package.json tooling, but places the bundle into the asset pipeline. For you your bundling took you can use Webpack, esbuild, a webpack replacement that bills itself as “An extremely fast JavaScript bundler”, which I suppose is better than a merely fast one, or a slow one, or Rollup. Rollup doesn’t have a fancy marketing slogan, but it’s also a JavaScript module bundler.
• The default Rails 7 tooling is called “import maps", which is a browser tool that lets you map a logical name to a downloaded module directly in the browser without needing to do further bundling on the server for the browser, and a Rails wrapper to manage that mapping from your codes.
• Finally, you can just use Rails as an API, and manage your client side code as a separate project using whatever tooling you want.

As an aside, this is a case where it’d be real nice if Rails had a more formalized RFP or road map process, so we mere mortals had some sense of what the Rails core team is really planning and recommending, rather than having to scramble through DHH’s Twitter feed or wait for an announcement on Hey World. The strategy here is interesting, it would have been useful to see it laid out in advance.

Just Tell Me Which One I Should Use!

Before I write a gazillion words on this topic, let me summarize the options, Wirecutter style.

Best Option for Most Projects: I think it’ll be the jsbundling-rails gem, or whatever they wind up naming it. The combination of the flexibility of a JavaScript bundler and the Rails convenience of using the asset pipeline – it’s early days, but this seems like a combination that will work for a lot of different projects.

If I was redoing the book right now, I think I’d build the project using jsbundling-rails because I’d be able to support the Stimulus and React code and not have to spend two chapters explaining webpack, which is less fun than it sounds.

Also Great if You are Using Hotwire: The Importmap-rails solution avoids any build step at all which seems like it’d easier to develop with. From my brief experience, import maps are super fast in development.

Importmap as a Rails tool seems a little limited (it was pointed out to me that it’s not clear how to deal with JS libraries that also bundle CSS). I think, but I’m not sure, it’d have transitive dependency issues where NPM allows multiple versions of dependent projects. Also, by design, no transpiling, which means no TypeScript, and potential JS versioning issues. But import map seems like a great approach if you can stay inside its lines.

I will probably try to keep my Elmer project on import maps, because it was the easiest to get working and I’m curious to work with them.

Also Still There If You Need That Level Of Complexity: Webpacker, which I would now mostly recommend if you really need the webpack ecosystem and plugins for performance or if you were doing something that you just can’t do in esbuild or Rollup.

I’m not at all sure I would recommend a new project jump on the Webpacker train, and I’d suggest that existing Webpacker 5 projects consider the jsbundling-rails route instead of upgrading to Webpacker 6 – they seem like similar levels of effort.

Also there: Using Rails as an API only. I’m glad the option is there, but all I really have to say about the API-only route is that I probably wouldn’t pick it on my projects without a very, very strong reason, but that I bet a lot of people will try it.

What’s the problem Rails is trying to solve?

Someone suggested that the problem is that DHH hates JavaScript, but I don’t think that’s quite it. I think that DHH doesn’t like being tied to the JavaScript ecosystem, but hey, I don’t like being tied to it either. Like many Ruby developers, I find the Ruby ecosystem to be easier to manage.

Specifically, webpack feels big, complicated, and fragile. It feels, to me, like it breaks every time I look at it cross-eyed, and I have used Webpacker since it came out and I literally wrote a book on it. I think that Webpacker has not been as successful as hoped at making webpack feel easier and more Rails-like.

I made an incorrect prediction here – when Webpacker came out, I thought the Rails community would write a lot of Ruby extensions that would make Webpacker even easier to deal with, but that didn’t happen at all.

Now, I think browsers have caught up with developers to a point where it’s possible for projects to have much smaller build overheads than before, and Rails is trying to allow projects to take advantage of that if they can.

Rails 7

To get these new features working fully, you need to update to Rails 7, which currently only lives in the main branch on GitHub. I don’t recommend running your production app off of Rails main, though Basecamp does, so if you have that level of control over your dependencies, I suppose go for it.

Just quickly, here’s how I do a Rails upgrade like this:

• Update the Rails version in the Gemfile to gem "rails", github: "rails/rails".
• Run the rails app:upgrade task. I accept all the file changes when it asks for them.
• Then I go through all the changes in my git browser, and more granularly check them. Usually what I’m looking for here is custom changes on my part that have been removed and which I need to put back.
• The Rails Guide on updates often has specific extra changes you need to make, at this point I don’t think there was anything new. (I even wound up deleting the defaults file and just changing the version in the configuration)
• Rails 7 no longer updates the Gemfile as part of app:upgrade, so I went to the actual Rails code for the template and checked for changes. This resulted in me removing listen and spring neither of which are part of Rails 7 default.

I did have a couple of hiccups. The Devise gem has a couple of compatibility PR’s that haven’t been merged yet, gem "devise", github: "strobilomyces/devise", branch: "patch-1" is a temporary branch that has the changes incorporated. I had to work around another gem temporarily.

CSS

One of the specific goals of getting off Webpacker is not doing CSS through webpack, which I think a lot of people found particularly confusing. So one goal here is to move people to asset pipeline CSS if you were not already doing so.

Sass is no longer a default. Instead, Rails 7 will have a new cssbundling-rails gem that allows you to choose between Tailwind (using the tailwind-css gem to install Tailwind into the asset pipeline), PostCSS, or Dart Sass as your CSS processor of choice.

For Elmer, all I needed to do was follow the cssbundling-rails installation instructions, choose Tailwind, delete Sass, and move my very small number of CSS selectors to the asset pipeline root at app/assets/stylesheets/application.css and everything continued to work.

I did, however, move image files out of the webpacker director and back to public/images, where the regular Rails image_tag could be used to display them. Probably, though I should put SVG files into Rails helpers so that they can be better styled.

JavaScript

A common feature of all the Rails 7 JavaScript build tools is a new default directory structure. (Webpacker hasn’t caught up yet, but will likely by the time you read this).

The new system puts all JavaScript code in app/javascript (singular – I know that sprockets used to pluralize it). JavaScript files at the top level are considered to be entry points, and I think the idea is that optimally there should be exactly one entrypoint at app/javascript/application.js.

For Elmer, this involved moving my file in app/packs/entrypoint (the Webpacker RC4 directory) to app/javascript/application.js, then moving my other directories (mostly Stimulus controllers) underneath app/javascript.

Using jsbundling

I’ll only go through jsbundling with one of the supported bundlers – esbuild, because it was released first.

The idea behind jsbundling is that you still use Yarn and the package.json file, but the built bundle goes into the asset pipeline download rather than being controlled by webpack. (And, I think, that you only bundle JavaScript, not CSS, and not static files).

Here’s what I did:

• Started with a Rails 7 upgrade, and the CSS changes above.
• Moved all my JavaScript code to app/javascript.
• Add jsbundling-rails to the Gemfile, removing webpacker.
• Cleared Tailwind and css stuff from the package.json file. Also cleared Webpack from it, making the package.json file a lot simpler.
• Ran bundle install and ./bin/rails javascript:install:esbuild (or replace esbuild with rollup or webpack . The installer
  • Creates an app/assets/build directory, appends that directory to the asset pipeline manifest at app/assets/config/manifest.js and .gitignores it.
  • Adds a javascript_include_tag "application" to the application.html.erb layout.
  • Offers to override package.json, but the only thing it adds is a script "build": "esbuild app/javascript/*.* --bundle --outdir=app/assets/builds" – even if you keep your package.json, you still need to add the script.
  • Runs yarn add esbuild.
• Re-ran hotwire-rails install, because the Stimulus index.js needs to be aware of this. This actually gave me the Stimulus 3.0 beta.

The default Stimulus installation uses require and a glob to autoload Stimulus controllers. You can’t do that in esbuild, so I needed to manually import all my Stimulus controllers with a for each controller line like: import("./css_controller").then(c => application.register("css", c.default)) – I’m quite positive this will be addressed soon.

At this point the recommended course of action is to run the development Rails server in one terminal and esbuild in another with yarn build --watch.

And this worked. JS file changes triggered a reasonably fast rebuild and the bundle was distributed to the page.

Import Maps

Import maps are a different way of thinking about sending JavaScript to the browser.

If I can oversimplify, the webpack family of products allow you to reference other files or external modules within your code because they resolve all the references at packaging time, and send a single file to the browser with all the references, so the browser just finds code in that file.

Once you are already bundling your JavaScript code, all kinds of additional feature become desirable, such as minifying code. And once you’ve got a big bundle, all kinds of other things attach to it and down that road, you get to webpack.

What import maps ask, is what if you didn’t do any of that?

Instead, what you do is send down all the individual files separately, and also send a text map relating the logical names you use in the code with the physical download URLs.

From the importmap-rails readme:

This frees you from needing Webpack, Yarn, npm, or any other part of the JavaScript toolchain. All you need is the asset pipeline that’s already included in Rails.

With this approach you’ll ship many small JavaScript files instead of one big JavaScript file. Thanks to HTTP2 that no longer carries a material performance penalty during the initial transport, and in fact offers substantial benefits over the long run due to better caching dynamics. Whereas before any change to any JavaScript file included in your big bundle would invalidate the cache for the the whole bundle, now only the cache for that single file is invalidated.

The immediate problem is how to manage sending an accurate map of files down to the browser, which is where the importmap-rails gem comes in. The gem provides a view helper to add the import map to your header, and a command line tool to manage the dependencies that go into the import map.

Here’s what I did to make this work, starting back from the Rails 6 version of Elmer:

• Upgraded to Rails 7. The Rails standard JavaScript libraries are updated in Rails 7 to work with import maps
• Moved all my JavaScript code to app/javascript.
• Add importmap-rails to the Gemfile, removing webpacker.
• Run bundle install and ./bin/rails importmap:install.

The installer adds the new javascript_import_tags helper into the application layout, adds app/javascript/application.js to the classic asset pipeline manifest in app/assets/config/manifest.js, sets up an initial importmap.rb file and adds an importmap command line tool.

• I updated the hotwire-rails install by rerunning that install command – it changes the Stimulus installation to use importmaps to identify Stimulus controllers. The name for the actual Stimulus distribution changed to @hotwired/stimulus, so all my references needed to update.

I changed my application.js to remove the Webpacker image path and also ActionCable and ActiveSupport references that I’m not currently using, leaving me with just this:

import "@hotwired/turbo-rails";
import "controllers";

Rails UJS is now soft-deprecated, so I’m not including it. Elmer didn’t use its behavior much, so I think I’m ok. (I had to change some link_to helpers to button_to.)

The head of my layout file now looks like this:

<head>
 <title>Elmer</title>
 <meta name="viewport" content="width=device-width,initial-scale=1">
 <%= csrf_meta_tags %>
 <%= csp_meta_tag %>

 <%= stylesheet_link_tag "inter-font", "data-turbo-track": "reload" %>
 <%= stylesheet_link_tag "tailwind", "data-turbo-track": "reload" %>
 <%= stylesheet_link_tag "application", media: "all" %>
 <%= javascript_importmap_tags %>
</head>

The top two stylesheet tags were added by tailwind-rails and the bottom one is the classic asset pipeline that’s going to grab app/assets/stylesheets/application.css.

The javascript_importmap_tags came from importmap-rails.

We’re so close to this working, but I do have some small external JavaScript dependencies, I have a form polyfill and a tool that I’m using for drag and drop lists.

The importmap command line tool lets us manage these external dependencies. The website JSPM provides CDN hosting for NPM modules and an API for generating a list of dependencies for NPM modules. The command line tool lets us “pin” dependencies into the importmap using this API.

So, from the command line:

$ bin/importmap pin form-request-submit-polyfill
$ bin/importmap pin sortablejs

If I wanted to remove them I’d use importmap unpin <the thing>.

Which results in an importmap.rb file like this:

pin "application"
pin "@hotwired/stimulus", to: "stimulus.js"
pin "@hotwired/stimulus-importmap-autoloader", to: "stimulus-importmap-autoloader.js"
pin_all_from "app/javascript/controllers", under: "controllers"
pin "@hotwired/turbo-rails", to: "turbo.js"
pin "form-request-submit-polyfill", to: "https://ga.jspm.io/npm:form-request-submit-polyfill@2.0.0/form-request-submit-polyfill.js"
pin "sortablejs", to: "https://ga.jspm.io/npm:sortablejs@1.14.0/modular/sortable.esm.js"

The first line is pinning my entry point at application.js, the next three pin lines are mapping Stimulus and Turbo locations to actual modules that are downloaded as part of the hotwire-rails gem. The pin_all_from is mapping everything in the app/javascript/controllers directory to the logical name controllers – the browser will still load the index.js in that directory when it’s referenced. And the final two lines are mapping my external dependencies to a CDN that can provide them to the browser (ga.jspm.io is a CDN that exists precisely for this purposed, maintained by the same people that manage the import map specification)

This works. Turbo loads, Stimulus loads, Tailwind loads. My Cypress tests work (I’ll need to keep a minimal package.json file around for them), and in fact are quite a bit faster to first run because they aren’t getting the overhead of a webpack build before the test run. (They might also be more stable, but I’m not sure about that yet).

The HTML page source gives me this

So that’s a big long list of imports mapping logical names to locations, and then a big long list of link tags downloading those locations – notice that the external dependencies are coming from the CDN (there is a way to download and vendor them if you’d rather not have that dependency).

That’s an admirable amount of Just Working, I got this going with relatively little frustration, and it does feel stable and it’s nice to not have a build tool – development feels very responsive. I definitely had the feeling of being able to write and display JS code quickly that I hadn’t really had in several years.

Wrapping Up

As the dust settles, there’s pretty good story here:

• Use importmap if you are not using much JS, you’ll get a great developer experience at the cost of some limited interaction with the JS world.
• Use one of the jsbundler tools if you want more interaction with the JS world.
• Use Rails as an API if you really have a Single Page App that is very big and very complex.

I’m excited about it, I think it’s going to be a better developer experience.