File Access (and a Pattern)

I was working on an import process, basically slurping in a bunch of data and shoving it into a database. Well, a little more nicely than just “shoving”. There’s massaging the data a bit from the input format, validating it, and then going through the normal creation flow (at least the behind-the-scenes part). At heart, this import process is a way of allowing someone to not have to go through a form hundreds of times to get those hundreds of records created.

I’m not here to talk about the import itself, though that could be a nice topic for later. What I’m here to talk about is providing the data. The data is meant to come in a CSV format, and this was a first-pass effort on the importer. That meant it was easy enough to create a CSV file and give the importer the filename. Locally, this worked perfectly.

Then things got a little tricky when I wanted to verify this in the QA environment, and was bluntly reminded of something very important when it comes to Heroku dynos: they’re meant to be stateless. There’s no way to put files there. Even if you shell in, there are no editors installed. I spent a little bit of time seeing how people have tried to get around this, but really, that’s not what you’re meant to do. (Also, and maybe more to the point, none of the things I tried worked.)

This, by the way, is a good reminder that your code will need to work in all environments, not just on your machine. Decades in software development, and I’m still occasionally rudely caught up by this — especially when I’m trying to do something “quick and dirty” or “just as a first pass”.

So, Heroku dynos are meant to be stateless. You’re not meant to add or change files on the dyno itself, but get information and data from other places; a database, environment variables, S3.

Hey, S3. That reminded me — this app already had something for dealing with attachments and knowing what to do with the underlying files. In local dev it just used the local filesystem, but in deployment environments it used S3. Maybe I can use that?

In fact, I couldn’t easily use it as-is, because it was meant to back attachments and deal with “file fields” in forms and interface with the DB nicely. I looked around for other packages, but they all seemed to follow the same pattern of backing attachments, acting as some sort of database model. What I could do was make a simplified copy of the existing attachment-backing code. And in the end, all I had to do was provide the correct configuration for local dev vs. other environments. Oh, and go through the importer code to find calls like File.stream and File.write and change the File to FileAccess. Simple as that.

That’s one story, but there are many like it. There’s a general pattern here with this common interface implementing different behaviors depending on some configuration — welcome to the Adapter pattern.

You remember Design Patterns, right? When you look around, you can see these patterns all around you. That’s the point. As mentioned before, these are the “building blocks” of creation. But not all building blocks are the same size — some are bricks or boards, some are walls or rooms.

Go forth and look around. And enjoy the patterns.

If you’re looking for a team to help you discover the right thing to build and help you build it, get in touch.