Elixir Trends, Fall 2021

Rose Hammer Burt -

This October we were delighted to attend ElixirConf® in Austin, our first in-person conference since the beginning of the pandemic. Both offline and online, in conversations and in presentations, we observed a few high-level themes. This document is a summary of recent trends we’re currently observing in the Elixir landscape.

Elixir, Edge Computing, and the Future of the Internet

Brooklyn Zelenka presented a keynote on a future-looking view of the physical infrastructure of the internet as well as a compelling case for how Elixir is set up to work well with the coming edge computing landscape. Current cloud computing data centers like AWS are centralized in locations that limit speed of access for vast swaths of the globe including massive emerging markets in India, Africa, and Asia. According to Gartner, by 2025, 75% of data will be processed outside the traditional data center or cloud, in part to overcome the physical limitations of the current centralized cloud server setup. Any centralized server system necessarily inherits and cannot escape the latency of the round trip from local device, out to server, and back to the local device. Brooklyn makes the point that infrastructure is already shifting away from this centralized model to a more distributed one where localized compute resources will be tapped to execute tasks, and next-generation ledger technologies will be used for distributed authentication to manage the various distributed computing systems executing on jobs to be done. Some early indicators of this move include Walmart’s move to install edge computing centers across the globe throughout their retail locations, current applications of extreme low-latency systems like remote surgery, competitive cloud gaming, and location transparency.

Elixir is well positioned with this shift to distributed computing and data-centric programming, Brooklyn quotes Chris McCord from What makes Phoenix Presence Special:

Instead of “which database would be best to hold presences?” we could ask, “How can we best replicate data in a distributed system without the user having to worry about it?”

Elixir’s strengths with distributed systems and data handling make it uniquely suited to handle the challenges coming with the shift to edge computing.

Elixir and the Future of Embedded Hardware

Justin Schneck’s keynote, as per usual, focused on embedded hardware. He made the case that the Internet of Things is both the same and not the same as “the cloud” (aka “somebody else’s computer”), and suggests a better name for the world of embedded devices might be The Fog.

Embedded devices are subjected to a wide variety of environmental hazards: wide-ranging temperatures, moisture, physical movement. But even devices in traditionally “safe” environments are subject to disruption; as our chipsets become more efficient and require less power, they also become increasingly vulnerable to cosmic rays and random bit-flipping.

Justin makes the case that in the world of The Fog, we increasingly need to design systems that are prepared to handle unexpected input of many types. In particular, as embedded devices are used to collect and generate the data that we use to feed massive difficult-to-introspect neural networks, our ability to ensure the accuracy of the data fed in by building devices that are robust and resilient to unexpected input is increasingly important.

Nimble design, then, is presented as a strategic approach. Flexible build platforms like Nerves, Embedded Linux, and RTOS allow developers to work at a level of abstraction that allows, for example, a manufacturer to change compute hardware altogether in response to the current global chipset shortage. Together with faster feedback loops and more exposure to the unknown unknowns, Justin calls for abstracting state from the hardware interface. Very is working on a platform to support these nimble design practices called Peridio, a heterogeneous management tool for modern hardware platforms announced at ElixirConf and scheduled for beta release in early 2022.

Elixir Community Growth

A recurring theme we’ve heard in interviews for our podcast, at the Elixir Wizards |> Conference this past summer, and again here at ElixirConf 2021 is an increasing understanding that tech does not solve problems endemic to a company’s culture; indeed, a healthy culture is necessary for high-performing teams and quality software development.

Miki Rezentes presented a talk on this topic titled “Greasing the Wheel of Adoption” which included tips specific to Elixir, but led with an overview of types of problems that tech cannot solve. In particular, Miki focused on identifying high-leverage problems, which in many organizations are people problems that cannot be fixed with technical solutions. Miki posited that many people, when presented with an ambiguous or uncomfortable situation, will try to treat the issue like problems they are more familiar with. Team members with more experience triaging technical problems want to be helpful but often end up looking for technical solutions and avoid solving the higher-stakes people problems in their organizations.

Another perennial topic in the Elixir community is hiring, capacity building, and training. We’ve seen wave after wave of hiring at fast-moving startups, often followed 12-18 months later by an equally large wave of firings. In fact just a week after ElixirConf, BlockFi laid off 60 Elixir developers. Typically these hiring waves are focused on rapidly hiring mid and senior level talent, with many organizations opting to cross-train mid or senior-level developers coming from Ruby, Erlang, or other backgrounds.

More recently we are encouraged to see the movement toward hiring Junior Elixir engineers. At SmartLogic, we’re once again running our Apprentice program, aimed at bridging the gap between formal training programs—from bootcamps to credentialed degree programs—and professional experience creating custom software applications.

Ben Marx also made the case that in the Elixir community, we need to value the perspective of beginners. Holding on to more of the beginner’s mindset even as we work in spaces where we are experts will enable us to continue to grow as individuals, teams, and broadly as a community.

This year we once again got overviews from José Valim, Chris McCord, and Frank Hunleth on the latest and greatest technical updates and improvements to Elixir, Phoenix, and Nerves. Here are a few trends we’re seeing, along the lines of technical developments within the languages and how the community is engaging.

Usability improvements

José gave us an overview of a variety of quality of life and usability improvements from Elixir v1.13, including semantic recompilation, mix xref, code fragments, and code formatting. The usability improvements here appear to be aimed at making it easier to manage large mature codebases.

Our engineering team continues to see the changes to Phoenix’s JavaScript (JS) dependencies that were announced this summer as positive indicators. As of Phoenix 1.6, new Phoenix applications will use the esbuild JS build tool instead of the previously required npm, which suffers from lack of reproducible builds over time. Assuming your project does not require any external packages, esbuild will generate the same JavaScript years from now, meaning better stability and maintainability for Phoenix apps going forward.

Phoenix is Phoenix LiveView

Chris McCord shared a run-down of changes and updates in Phoenix v1.6 and LiveView v0.16. While in its infancy Phoenix existed without LiveView, all contemporary framing around Phoenix is within the context of LiveView. At this point it is a challenge to find reference material on Phoenix sans LiveView. LiveView is now even bundled in the Phoenix package. We saw a number of talks and heard anecdotally in conversation that teams are building and deploying large scale production apps with Phoenix LiveView. For example: Desmond Bowe presented on the Single-Page App (SPA) that the Payitoff team built in LiveView.

In some situations we would make the case that keeping “dead views” alive and maintaining non-LiveView routes is worthwhile. Our Engineering Manager Eric Oestrich has built a new framework entitled Aino for this purpose; it is fundamentally a very fast server-side renderer and API framework, built on top of elli and loosely based on the Clojure library Ring. You can read more about Aino and it’s intended uses in Eric’s announcement post from September 2021.

Early days of Machine Learning as the new frontier for Elixir

As José originally announced in early 2021, Nx (Numerical Elixir) positions machine learning as the new frontier for Elixir. Machine learning emerged as a theme in a number of presentations and conversations over the course of ElixirConf, though not often taking center stage.

One presentation that focused on an application of machine learning was Vanessa Lee’s talk on her new project, Akin. Akin uses a collection of string comparison algorithms and machine learning to facilitate the disambiguation of names. People’s names are notoriously difficult to manage in any given dataset because of the high occurrence and variety of edge cases and forms of representation, making them a good candidate for a dynamic analysis project using machine learning.

Indicators so far give us the impression that while the Elixir community is curious about machine learning, developers continue to be primarily concerned with building web applications. We have not yet started to see much in the way of applications leveraging Nx. We are curious to see where Nx will start to take off, whether in the data science community, among Elixir developers, or perhaps somewhere new altogether.

Nerves and LiveBook

In his keynote, Frank Hunleth provided an overview and demonstration of new features of Nerves and Livebook, a collaborative code notebook. Our first impressions of Livebook are that it’s a very accessible learning and teaching tool; our podcast team got an early look last spring during a livestream with Frank. Owen Bickford, who joined our company one week after ElixirConf and who also attended NervesConf this year, reports that the participants had a mesh network running by end of day. Livebook appears to be having a positive impact on Nerves adoption by reducing onboarding friction.

We’re also curious about the new project the author of Phoenix, Chris McCord, is working on. Fly.io is a deployment service focused on geographic distribution with automated secure clustering and data management between nodes. This platform could have positive implications for ease of deploying Elixir applications. You can launch a Livebook instance with a free account if you’re curious to check it out.

Elixir and the Future

The Elixir community is continuing to grow and mature. The language and tooling continues to improve while the core of the language itself remains as stable, scalable, and fault-tolerant as ever. As we begin to see broader adoption of edge-deployed compute resources and a greater proliferation of networked embedded hardware, Elixir is well positioned for broader adoption and integration into these new massive distributed systems.

This article is also available as a downloadable whitepaper.

Header photo by Alexander Ant on Unsplash