Potions in a Cauldron: Streamline Elixir App Development with Code Generation
At SmartLogic, our team of developers works feverishly to make our clients' dreams a reality. We partner with our designers and project managers to chart the course of a project, whether we are creating a new application from scratch or evolving an existing application. In both scenarios, we often need to spend time performing the tedious work of installing and configuring dependencies before feature development can begin.
One of the responsibilities of an engineer is to find opportunities and methods for improving efficiency. Our team specializes in wielding the Elixir backend language for custom web applications, and the language offers several tools for streamlining the process of project setup and dependency installation. For the past year, I have worked to build internal tooling for our team with the goal of reducing setup time from hours or days into minutes.
Looking for an engineering team dedicated to delivering robust custom applications with an obsession for detail and efficiency? Reach out to learn more about what we can do for you.
Let's take a look at how Elixir empowers us to achieve this goal.
Mix Tasks
In Elixir, Mix is the build tool included with the language.
Mix is a build tool that provides tasks for creating, compiling, and testing Elixir projects, managing its dependencies, and more.
When we need to create a new bare-bones Elixir project, mix new is the Mix task for generating project code (docs). For demonstration purposes, let's imagine we need to establish a new Elixir application called Wzrds, a social platform for crafty Elixirists.
$ mix new wzrds --sup
This Mix task accepts an argument for the project name, wzrds and some optional flags like --sup. Project names are snake_case in Elixir-land, so another example might be elixir_wizards. With the project name wzrds, modules will have the namespace Wzrds, so an example database module might be named Wzrds.Repo.
When we passed the optional flag --sup, we told the Mix task to generate an Application module with a Supervisor pre-configured. The supervisor is a core feature of Elixir applications, inherited from Elixir's predecessor, Erlang. That's a topic for another article, but suffice it to say that the application supervisor is a differentiator for Elixir and Erlang applications - a luxurious piece of code powering performant and resilient systems.
After running this task, we have a simple Elixir application. From here, we could start locating packages in the ecosystem and installing them as dependencies. After installing each dependency, we will also need to add and modify code so it can be configured for development, testing, and production environments. Documentation for our favorite dependencies is quite good. Instructions are often provided for adding and configuring the dependency.
Following those instructions can take a few minutes for a developer who is familiar with the dependency and knows the requirements for the application. However, less-experienced developers sometimes struggle to "connect the dots" or fill in gaps in the documentation where there may be assumed knowledge. Fortunately, there are a few dependencies that include a Mix task for doing this setup work, often following the mix some_dependency.install naming convention. However, the number of packages that include this convenience is small.
What if we could build our own Mix tasks to configure dependencies based on the preferences of our team?
A Note on Magic
As wizards, we enjoy crafting and casting spells. Creating Mix tasks that generate SmartLogic-style code does feel like magic. It allows us to encode standards and best practices as we find them while leaving room for customization on a per-project basis. This power is important to our team and sometimes brings new developers to Elixir.
In the software engineering world, "magic" is often a dirty word. When a piece of code works in a way that is hard to understand, we often say it "feels like magic," or worse "feels like too much magic." With Mix tasks, we fortunately get to use tools that are familiar and relatively easy to understand.
Building a Cauldron for Custom Software Development
If we want to create a collection of reusable Mix tasks, we need a place to put them. If we're wizards casting spells in the form of Mix tasks, what better name is there than Cauldron? This is the name we chose for our internal Elixir repository. So far, it is simply a set of Mix tasks that we can use to install dependencies, add configuration, and generate boilerplate code - the tedious stuff that can slow down otherwise efficient developers.
While it's closed-source and proprietary to SmartLogic, we can share a bit of the strategy behind Cauldron and some of the ways we're crafting spells to speed up our custom software development process.
Code Generation & Templates
Why do we want to generate code in the first place? Couldn't we write code in one repo and use it as a dependency across our projects? Sure, that is a reasonable approach, particularly within a product company where all projects need to conform to expected behaviors. At a software development agency, however, the requirements of one project are often very different from other projects. By generating code, we provide developers a set of sensible defaults with the opportunity to make adjustments based on the project's needs.
Cauldron currently includes the following tasks for code generation:
cauldron.install.components: Installs common UI components and CSS styles for Phoenix applicationscauldron.install.fun_with_flags: Installs the FunWithFlags package for controlling access to features via feature flags, including starter UI for managing flags at runtimecauldron.install.commanded: Installs the Commanded framework for Event Sourcing & CQRS, as well as some custom Elixir behaviours for ensuring consistency across domain modulescauldron.gen.domain: Generates directories and boilerplate code for a provided Event Sourcing domain, including example code
Each of the cauldron.install.* tasks installs one or more dependencies and injects configuration code.
The cauldron.gen.domain is special. It's our approach to ensuring consistency across domains in an Event Sourcing & CQRS application, particularly when adding code for a new domain. This task generates files for a given domain.
For example, imagine we're adding a posts domain in the wzrds app:
▶ cd wzrds
▶ mix cauldron.gen.domain posts
▶ tree ./lib
./lib
├── wzrds
│ └── posts
│ ├── aggregates
│ │ └── some_aggregate.ex
│ ├── commands
│ │ └── some_command.ex
│ ├── events
│ │ └── some_event.ex
│ ├── fields.ex
│ ├── projections
│ └── router.ex
The cauldron.gen.domain task uses the Mix.Task behaviour, a contract requiring implementation of a single callback aka function in order to work. Under the broom, it takes a collection of EEx templates for modules that we want to include. These modules establish or follow the pattern we want to use in our projects so that a developer can get acquainted with a new application quickly. Without patterns like these, every application can feel like a thicket of bespoke naming and organization.
EEx templates are nice. For developers who have worked with Phoenix's HEEx templates, an EEx template will be familiar territory. HEEx, after all, is an extension of EEx templates, adding HTML validations and introspection via the ~H sigil. Once a developer understands assigns and the EEx syntax for interpolating code, our module templates will be easy to understand.
The router.ex template, for example, looks like this:
# cauldron/templates/create/domain/router.ex.eex
defmodule <%= inspect @domain_pascal_case %>.Router do
@moduledoc """
Routes commands to aggregates within the `<%= inspect @domain_pascal_case %>` domain.
"""
use Commanded.Commands.Router
alias <%= inspect @domain_pascal_case %>.Aggregates.SomeAggregate
alias <%= inspect @domain_pascal_case %>.Commands.SomeCommand
identify SomeAggregate, by: :id
dispatch SomeCommand, to: SomeAggregate
end
When we run mix cauldron.gen.domain posts, the generated router code looks like this:
# wzrds/lib/posts/router.ex
defmodule Wzrds.Posts.Router do
@moduledoc """
Routes commands to aggregates within the `Wzrds` domain.
"""
use Wzrds.EventSystem.Router
alias Wzrds.Posts.Aggregates.SomeAggregate
alias Wzrds.Posts.Commands.SomeCommand
identify SomeAggregate, by: :id
dispatch SomeCommand, to: SomeAggregate
end
See how <%= inspect @domain_pascal_case %> is replaced with Wzrds.Posts? Inside Cauldron's Mix task code, we create assigns (a key-value map) for each piece of data we need to generate code. These assigns are passed to each template, which may use some, all, or none of the assigns to interpolate values into the newly produced code.
In a mature application using Cauldron to generate domains, developers can expect to see strong naming and organization patterns. This consistency will be especially valuable to developers new to Event Sourcing concepts, where inconsistent code would add an unnecessary mental burden. As we discover pain points with these conventions, we can improve our Cauldron tasks to make the developer experience more efficient and enjoyable.
Sourcery: Modifying Code
New code generation is only part of the spell. For many tasks, we need to alter existing code. When modifying Elixir code, we have an abstract syntax tree (AST) and some built-in tools for manipulating the AST. While Elixir's Code, Macro, and Module provide powers for altering the AST, using them can be daunting. There is also a bit of complexity around maintaining inline code comments, which is why we reached for a package name Sourceror.
Sourceror still requires an understanding of the AST, but it makes navigating the AST a bit more approachable. It also allows inline comments to be retained in the correct place, which is important for providing guidance to developers.
With Sourceror, we can write a string representing some Elixir code, and parse it into an AST before it's injected into a target file. For example, this code is injected into dev.exs using Sourceror:
app_name = :wzrds
...
Sourceror.parse_string!("""
[
username: "postgres",
password: "postgres",
database: "#{app_name}_eventstore_dev",
hostname: "localhost",
pool_size: 10
]
""")
...
This is what happens if we run the code in IEx:
iex(1)> app_name = :wzrds
:wzrds
iex(2)> Sourceror.parse_string!("""
...(2)> [
...(2)> username: "postgres",
...(2)> password: "postgres",
...(2)> database: "#{app_name}_eventstore_dev",
...(2)> hostname: "localhost",
...(2)> pool_size: 10
...(2)> ]
...(2)> """)
{:__block__,
[
trailing_comments: [],
leading_comments: [],
end_of_expression: [newlines: 1, line: 7, column: 2],
newlines: 1,
closing: [line: 7, column: 1],
line: 1,
column: 1
],
[
[
{{:__block__,
[
trailing_comments: [],
leading_comments: [],
format: :keyword,
line: 2,
column: 3
], [:username]},
{:__block__,
[
trailing_comments: [],
leading_comments: [],
delimiter: "\"",
line: 2,
column: 13
], ["postgres"]}},
{{:__block__,
[
trailing_comments: [],
leading_comments: [],
format: :keyword,
line: 3,
column: 3
], [:password]},
{:__block__,
[
trailing_comments: [],
leading_comments: [],
delimiter: "\"",
line: 3,
column: 13
], ["postgres"]}},
{{:__block__,
[
trailing_comments: [],
leading_comments: [],
format: :keyword,
line: 4,
column: 3
], [:database]},
{:__block__,
[
trailing_comments: [],
leading_comments: [],
delimiter: "\"",
line: 4,
column: 13
], ["wzrds_eventstore_dev"]}},
{{:__block__,
[
trailing_comments: [],
leading_comments: [],
format: :keyword,
line: 5,
column: 3
], [:hostname]},
{:__block__,
[
trailing_comments: [],
leading_comments: [],
delimiter: "\"",
line: 5,
column: 13
], ["localhost"]}},
{{:__block__,
[
trailing_comments: [],
leading_comments: [],
format: :keyword,
line: 6,
column: 3
], [:pool_size]},
{:__block__,
[
trailing_comments: [],
leading_comments: [],
token: "10",
line: 6,
column: 14
], ~c"\n"}}
]
]}
Sourceror has parsed the string we provided and returned an Elixir AST, which can be programmatically injected into an Elixir file.
Modifying Non-Elixir Code
There are times when we need to modify code in languages other than Elixir, and Sourceror cannot help in those situations. The cauldron.install.components task, for example, copies some common starter CSS styles and injects some @import statements into a Phoenix application's app.css file. Here, we use Elixir's String module to find and replace code the old-fashioned way using regular expressions. The same approach is used to modify Javascript code.
Because changes based on regular expressions can be error-prone, we attempt to keep this style of modification to a minimum.
Igniter: Simplifying Setup
So far, we've been writing our custom Mix tasks using Elixir's built-in tools, and incrementally adding dependencies to make it easier to do so. Until ElixirConf 2024, this was the way I expected to write custom tasks. That changed when I spoke with Zach Daniel, known for creating the Ash framework, who had introduced another tool called Igniter.
While writing Mix tasks was becoming more familiar, it did feel like a special skill that would not be common across the team. Navigating the Elixir AST in particular would be an intimidating and difficult skill to learn for developers. With broader plans for adding Cauldron tasks, I was keen to learn more about how Igniter would make tasks easier to create and maintain.
After the conference, we discussed Zach's work on Igniter in the first episode of Elixir Wizard's 13th season (YouTube). After this conversation, I spent some time writing new tasks with Igniter, and found it useful and enjoyable. In fact, all of the Cauldron tasks listed earlier were written in a week using Igniter, with the exception of cauldron.gen.domain which already existed.
What I found particularly useful with Igniter was its ability to add and install dependencies, compose other tasks, and navigate code with a more readable API. Instead of pattern matching on the nested list structure of the AST, Igniter allows you to write instructions that read like plain-English:
def ignite(%Igniter{} = igniter) do
igniter
|> Igniter.Project.Deps.add_dep({:commanded, "~> 1.4"})
|> Igniter.Project.Deps.add_dep({:commanded_ecto_projections, "~> 1.4"})
|> Igniter.Project.Deps.add_dep({:commanded_eventstore_adapter, "~> 1.4"})
|> Igniter.apply_and_fetch_dependencies()
|> Igniter.Project.TaskAliases.add_alias(
"setup",
[
"deps.get",
"event_store.setup",
"ecto.setup",
"assets.setup",
"assets.build"
],
if_exists: :append
)
...
end
It's not only junior developers who will benefit from the readability of this code. Our senior and staff engineers can easily follow along, and I expect that even after months on other projects, the code will be easy to understand when we need to make changes.
There's much more going on in Cauldron, and we have plans to expand our list of tasks to include new ones for adding config for CircleCI, GitHub Actions, Ansible, Terraform, as well as additional Elixir code generators.
Peering into Igniter's source code, I have a tremendous amount of respect for its contributors. As a pre-1.0 package, there is some opportunity to improve documentation through more complete typespecs, prose, and example code. Still, the documentation is good and the modules are named with clarity. When I needed help or found issues, Zach was very responsive.
Discarded Spells
In the past, our team used a template repo in GitHub and a separate repo as an Elixir package for common functionality used across projects. Initially, Cauldron started as a new take on this approach, where we copied templates from the Phoenix source code and wrapped the mix phx.new task. The idea was to modify the templates to match our internal patterns, and include dependencies we tend to add to new projects.
However, this approach would prove to be brittle and hard to maintain over time. As Phoenix and other dependencies change, our generators need to keep up or risk falling behind. This isn't manageable for our team who are primarily focused on building client applications, not maintaining internal tools.
With the introduction of Igniter, we can take a more granular approach. Now, Cauldron is a collection of tasks for generating and modifying code instead of generating new applications. This way, developers are empowered to add just the code they need at the time they need it. There is still much more work to be done internally and in the community.
What's Next
During our podcast conversation with Zach, he raised the idea that package maintainers should include an installation task to automate configuration and setup. This could be a sea change in how we create brand new Phoenix applications in particular.
If packages like Ecto, Swoosh, Tailwind, and others included mix [package_name].install tasks, the mix phx.new task could generate a much smaller collection of code. Currently, it's tempting - maybe common practice - to simply accept all the defaults from the project generator because there's some chance they'll be necessary eventually and adding dependencies later could take precious time. If we as a community establish a pattern of including installation tasks for our packages, we can make it so that our dependencies are added only when they provide real value.
So, this is a call to action for package maintainers! Whether you create an install task using Igniter or Mix, automating setup with sensible defaults and CLI options can dramatically improve the developer experience and potentially lead to increased adoption. In fact, the WebauthnComponents package I maintain includes a mix wac.install task (code), which I intend to update and streamline with Igniter.
Happy coding!
If you are intrigued by our approach to solving problems, reach out to learn more about what we can do for you.