Rails Engines Using Webpacker (1/2).

creating a Rails engine using Webpacker

An Experience Report

Webpacker is the main line replacement for Sprockets and the Asset Pipeline, as of Rails 6. Webpacker wraps the webpack JavaScript module bundler such that it can be simply used by a Rails application, consistent with the current common practices of JavaScript developers.

Core to this process is the creation of a manifest.json file, and a recommended new structure for JS files within your Rails application.

While this works for Rails 6 applications, it doesn’t work as well for Rails Engines. Let’s give it a go.

Introduction

Webpacker isn’t new, but as of Rails 6[^1], it is now installed by default upon creating a new Rails application. That is, when you run rails new whatever, Webpacker is included, and bin/rails webpacker:install is run by default.

This isn’t true, however, for Rails Engines[^2]. An Engine is a Rails application wrapped up in a gem, and nested inside a host or parent application. Engines encapsulate reusable systems that require routes, controllers, and views in addition to other library features.

This becomes useful when you’re writing an add-on or common tooling element. For example, if you’ve implemented an authentication, you’ve probably worked with Devise[^3]. At Flagrant, we’re building a simple, but commonly scratch-built element of many web applications and encapsulating it as an engine.

At present, Rails Engines don’t have a clear way to include Webpacker out of the gate, and we must jump through several hoops to get things to work. I’d like to clarify this process and explore how to connect an engine’s Webpacker configuration to the configuration of the application using the engine.

We’re not going to make any assumptions of your familiarity with webpack or the JavaScript dependency management system, and will explain things (badly) where a little more context might be helpful.

Goals and Rationale

We’re going to build an engine that does a very simple thing (runs a JavaScript timer) on a page in our engine content. This won’t work out of the box, so we’ll need to shoe-horn in the Webpacker considerations, and try to get the implementing app to play nicely.

We’re taking this on to more fully understand the moving parts within Webpacker, what the boundaries are for using Webpacker within an engine, and (eventually) explore the possible of including a gem’s JavaScript content when compiling using webpack from a parent application.

Why? Because engines + Webpacker received short shrift, and we’d like to a) ensure that the stopgap documentation works as expected, and b) improve the experience for engine developers. If I achieve a certain level of understanding, I might go ahead and submit a PR to Webpacker, but that’s getting way ahead of myself.

Getting Started

Let’s building a foundation for understanding how engines work with a host application.

Creating a Rails Engine

We start by creating a minimal mountable Rails engine, using information from the Rails Guide above[^2], and reviewing the available options using --help:

rails plugin new webpacker-engine -S -M -P -O -T -C --mountable

This will serve as the stepping off point for the exercise. The flags: no sprockets, no mailer, no puma, no ActiveRecord, no gasp tests, no ActionCable, and the plugin should be mountable.

By glancing into the resulting directory, we realize that because we called it ‘Webpacker-engine’, we’ve unintentionally created a subdirectory structure. Ouch.

So let’s try this again (this time without the oops) with a clever name:

rails plugin new saddlebag -S -M -P -O -T -C --mountable

Why So Skinny?

By excluding all unnecessary systems, it’ll be easier to isolate any problems. Fundamentally, engine work concerns itself with what many Rails developers think of as ‘magic’. We want to avoid summoning stray code into our problem space, so we work as minimally as possible.

It’s also necessary to clean up the FIXME and TODO cruft in the gemspec.

Creating a `dummy` App

This app will host the Saddlebag Rails engine, giving us a platform from which to run the engine ‘in the wild’. We create it with the same constraints:

rails new saddlebag-dummy -S -M -P -O -T -C

Since the dummy and engine are in peer directories, we add this line to our dummy app’s Gemfile:

gem 'saddlebag', path: '../saddlebag'

Then bundle and we’re off to the races.

This is a great time to git init both places, and push initial content to some repo somewhere. In our case: saddlebag and saddlebag-dummy.

Adding a (JavaScript) Feature

To have something to show (and mount), we need some content. Let’s add a feature on a single page, and to ensure that the dummy app is picking up what we’re putting down.

Elapsed Time Counter

The feature we’ll use is elapsed time since page load. We don’t need anything fancy.

We create a controller in our engine, CounterController to pass through to a view:

# app/controllers/saddlebag/counter_controller.rb
require_dependency "saddlebag/application_controller"

module Saddlebag
  class CounterController < ApplicationController
    def index; end
  end
end

We build a simple JavaScript counter directly into the view like a common criminal:

<!--  app/views/saddlebag/counter/index.html.erb -->
<h1>HELLO!</h1>

<p>elapsed since page load: <span id="counter">0</span></p>

<script>
    let loadedAt = new Date().getTime();
    let updateFn = setInterval(function() {
        let elapsed = (new Date().getTime() - loadedAt)/1000;
        document.getElementById("counter").innerHTML = elapsed;
    }, 200); // update 5x second for maximum fan usage
</script>

We add a route so we can get to it:

# config/routes.rb (engine)
Saddlebag::Engine.routes.draw do
    get '/counter', to: 'counter#index'
end

See this commit for changes to the saddlebag repo.

Finally, mount ‘saddlebag’ in the dummy app’s routes.rb file.

# config/routes.rb (host)
Rails.application.routes.draw do
  mount Saddlebag::Engine => '/saddlebag'
end

See this commit for changes to the dummy repo.

Start the server, visit http://localhost:3000/saddlebag/counter and there’s our counter.

voila! the timer works

Adding Webpacker

We haven’t touched Webpacker yet. While it’s installed in our dummy app by default (at version 4.0), it’s not installed in our engine. The current version of Webpacker is 5.2.1, so let’s add this to our saddlebag.gemspec, and update the version in our dummy app too.

When adding Webpacker to an existing Rails project, there are a few steps that need to be completed. With a new Rails 6 app, these steps happen when the app gets created. You can walk through the steps listed in the Webpacker github repo[^4], but those install steps don’t work for a Rails Engine.

Instead, we look to a separate set of instructions[^5] that far more involved. They replicate manually the steps that happen automatically when installing Webpacker into a top-level Rails application. Let’s do it.

Copying Files

Step 1 is to create the engine, which we’ve already done. Step 2 is to import the following files from a newly-Webpackered app (such as our dummy) into the engine:

config/Webpacker.yml configures the Webpacker gem
config/webpack/*.js configures the webpack application, used by the gem
bin/webpack* are webpack-related binstubs
package.json is a list of JavaScript dependencies.

The package.json file we copied looked like this:

{
  "name": "saddlebag",
  "private": true,
  "dependencies": {
    "@Rails/ujs": "^6.0.0",
    "@Rails/Webpacker": "5.2.1",
    "turbolinks": "^5.2.0"
  },
  "version": "0.1.0",
  "devDependencies": {
    "webpack-dev-server": "^3.11.0"
  }
}

Add Webpacker to the Engine’s Module

To keep the Webpacker functionality for the engine isolated, we create an instance of Webpacker right inside the engine. There is some provided code in the instructions, but it defines a ROOT_PATH constant, so we used this instead:

# lib/saddlebag.rb
require "saddlebag/engine"

module Saddlebag
  # ...
  class << self
    def Webpacker
      @Webpacker ||= ::Webpacker::Instance.new(
        root_path: Saddlebag::Engine.root,
        config_path: Saddlebag::Engine.root.join('config', 'Webpacker.yml')
      )
    end
  end
  # ...
end

Configure Helper and Rake Tasks

We’ve skipped Step 4, and moved on to Steps 5 and 6, since Steps 4 and 7 concern modifying the same file. We’ll just copy and paste these sections and change the appropriate names.

Configuring the Webpack Dev Server

Steps 4 and 7 from the documentation, and operate on your engine configuration in lib/saddlebag/engine.rb.

In Step 4, you’re adding webpacker-specific changes to communicate with the webpack dev server via an included proxy, we simply copy that over.

Serving Packs

In Step 7, we’re adding a Rack::Static middleware to serve engine-local files via defined endpoints in the host application. I spent far too much time fiddling with this single step because I didn’t really understand why things were happening. On the off chance that you might benefit from a little context, let’s talk about how webpack and Webpacker work together via the config/webpacker.yml file.

Webpacker wraps webpack, which is among other things a JavaScript runtime for running scripts bundled with it. It determines what lives where using a manifest file, manifest.json. Webpacker’s configuration file (config/webpacker.yml) tells webpack where it will be looking for the manifest file after it’s compiled and bundled, embedding that information in the bundle itself.

In a default Rails 6 application, the beginning of webpacker.yml looks like this:

default: &default
  source_path: app/javascript
  source_entry_path: packs
  public_root_path: public
  public_output_path: packs

The source_path tells Webpacker where the root of its asset tree will be to begin compilation. (This can include images, CSS files, etc, but that’s another show). The source_entry_path tells Webpacker what directory contains files that are entry files–that is, what input files will make it into independent output files. The public_root_path is the place in the codebase from which static public files will be served when the application is running. Finally, the public_output_path is the name of the directory inside the public_root_path where the compiled and bundled artifacts will go.

So, using the host app’s webpacker.yml file above, the host application’s Webpacker source and output trees look like this:

app/ – source tree
- javascript/
  - packs/ – entry files
    - whatsit.js
    - entryfile2.js
    - index.js
public/ – public tree
- packs/ – compiled artifacts
  - manifest.json – manifest file
  - js/ – compiled JavaScript files
    - index-#####.js
    - entryfile2-#####.js
    - whatsit-#####.js

Most importantly, the webpack runtime will, after compilation, want to look for its manifest file at exactly servername:port/packs/manifest.json, because when the bundle is compiled, the contents of webpacker.yml get locked inside.

(NOTE: this is a simplificaion. This can be very configurable and fiddly.)

In an engine, the engine isn’t serving its own files, it’s depending on a host application to serve those files. The engine won’t be able to put the output of a something:webpack:compile into the host’s /public directory, nor will it be able to merge its own source tree into that of the host. To get around this, we make a couple of changes to the engine’s webpacker.yml file, and add the

default: &default                       # engine's webpacker.yml
  source_path: app/javascript           # same
  source_entry_path: packs              # same
  public_root_path: public              # same
  public_output_path: saddlebag-packs   # what?

Remember that even if the host application is using Webpacker, when it runs bin/rails webpacker:compile or bin/rails assets:compile it’s using a different instance of Webpacker, and generating a separate webpack runtime.

We want to be able to isolate the Saddlebag runtime, manifest, and compiled scripts, but still have them accessible while the host application is running.

Like before, only using the engine’s webpacker.yml, we have the following source and output trees:

app/ – source tree
- javascript/
  - packs/ – entry files
    - counter.js
public/ – public tree
- saddlebag-packs/ – compiled artifacts
  - manifest.json – manifest file
  - js/ – compiled javascript files
    - counter-#####.js

In this configuration, the webpack runtime will expect its manifest file at exactly servername:port/saddlebag-packs/manifest.json. Even with this change, when the app is running it won’t by default be serving files from the engine’s public directory. We can provide that capability using a middleware in the engine’s lib/saddlebag/engine.rb definition:

config.app_middleware.use(
    Rack::Static,
    # note! this varies from the Webpacker/engine documentation
    urls: ["/saddlebag-packs"], root: Saddlebag::Engine.root.join("public")
    # instead of -> urls: ["/saddlebag-packs"], root: "saddlebag/public"
)

By adding this middleware to the app (config.app_middleware.use(...)), we’re directing Rack::Static to serve files living in the saddlebag-packs subdirectory directory of the engine-local public directory on the /saddlebag-packs/ path of the host application.

With this in place, the webpack runtimes and bundles are served from two place while the host app runs:

servername:port – server root
- packs/ – app artifacts
  - manifest.json
  - js/
    - index-#####.js
    - entryfile2-#####.js
    - whatsit-#####.js
- saddlebag-packs/ – engine artifacts
  - manifest.json
  - js/
    - counter-#####.js

We made one change from the documentation here. Rather than trying to make assumptions about where the engine source resides relative to the main application, we just ask, using Saddlebag::Engine.root.

Final piece of context–when we’re using the Webpacker view helpers, it is once again the webpacker.yml which tells the view helper where to go. For example, in the engine, this erb:

<%= javascript_pack_tag 'counter' %>

translates to this html:

<script src="/saddlebag-packs/js/counter-69d8560626505c71825d.js"></script>

In the host app, the same erb would look for counter in the manifest, fail to find it, and error out.

Converting to Webpacker

With all the infrastructure in place (and all possible mistakes made), we now need to move our code into the Webpacker-appropriate places. For us, this is simply moving the counter code into its new home and file at app/javascripts/packs/counter.js:

// app/javascripts/packs/counter.js
let loadedAt = new Date().getTime();
let updateFn = setInterval(function() {
    let elapsed = (new Date().getTime() - loadedAt)/1000;
    document.getElementById("counter").innerHTML = elapsed;
}, 200); // update 5x second for maximum fan usage

We then replace it with a javascript_pack_tag.

<!--  app/views/saddlebag/counter/index.html.erb -->
<h1>HELLO!</h1>

<p>elapsed since page load: <span id="counter">0</span></p>
<%= javascript_pack_tag 'counter' %>

Finally, we need to compile the webpack. From the dummy application, we run the rake task we created above:

bin/rails saddlebag:webpacker:compile

This has populated our engine’s public/saddlebag-packs/ directory with our bundles and manifests. We can now boot our server back up and visit http://localhost:3000/saddlebag/counter and see the counter advance.

Easy, right?

Next Steps

In the next entry, I’d like to explore making engine JavaScript available to the host app, using Webpacker configuration.

Conclusion

It feels like Rails 6 Engines live in an in-between state right now. New engines can’t include Webpacker easily, even though it’s the default when creating a new Rails 6 application. Still, there are ways to make it work, even if it’s somewhat more sweat and tears than we’d like.

If you’d like to look at the complete project code, you can check out the saddlebag and saddlebag-dummy on GitHub.