Justus Eapen:       Welcome to Smart Software with SmartLogic, a podcast where we talk about best practices in web and mobile software development with a focus on new and emerging technology. My name is Justus Eapen and I'll be your host today. I'm a developer at SmartLogic, a Baltimore-based consulting company that has been building web and mobile software applications since 2005. I'm joined by my co-host and colleague, Eric Oestrich. Say "Hi," Eric.

Eric Oestrich:      Hello.

Justus Eapen:       And we're also joined by a very special guest, Mr. Jeffrey Matthias from Community.com. Say "Hi," Jeffrey.

Jeffrey M.:         Hello.

Justus Eapen:       So, our first series covers Phoenix and Elixir in production so, Jeffrey, that's what we'll be talking about today. Do you want to just give us a quick overview of Community.com and your experience using Elixir and Phoenix in production across your career?

Jeffrey M.:         Yeah. So, I'll try to explain Community. It's a product that's kind of cooler the more you dig into it, and I've discovered I really suck at selling it so, I'll do my best and then we'll go from there. We are giving people the opportunity ... and when I say people, we're talking about people with fan bases, the opportunity to interact with their community, with their following, via SMS, via text message, instead of having to be stuck on social media platforms. It gives them a lot better control. It also means that you could actually — we provide a phone number. You are texting with those people. So, we've soft launched at this point. If you do a little digging on the Internet, you can find places where you potentially can come up with phone numbers for Ashton Kutcher, Metallica, and Gary V — Gary Vaynerchuk — and we're slowly rolling out more clients.

Jeffrey M.:         Then, one of the big things we're doing is building tooling so that they actually really can manage that relationship between themselves and millions of people. Again, it's kind of interesting because it's through the interface of that text messaging app that you already have on your phone. That's what I'm working on here at Community.

Jeffrey M.:         In terms of working in production Elixir, I got my start in either late 2015 or early 2016, working at a company called Parkify. We were putting sensors in parking spaces so that we could get live occupancy data. That system actually originally started as Java, and we started to introduce more and more Elixir as time went on, initially as background workers, and then it started to do customer-facing applications as well, with Phoenix.

Jeffrey M.:         Then, after that, I was at a company called Enbala, which is load balancing of the power grid. That was a pretty exciting and very different use case where we actually had hundreds of thousands of assets out in the real world that were represented internally by GenServers. So, we were collecting telemetry and making decisions on what to switch based off of what the current situation was and what the current telemetry was for those individual assets. So, that was pretty cool.

Jeffrey M.:         Spent a while at Weedmaps, working on there, and worked on building out the online ordering and driver tracking — In California, you can have marijuana delivered to your home. So, that was pretty exciting, getting to do live map stuff. That was, both Enbala and Weedmaps, we got to leverage the channels pretty well for being able to actually drive live dashboards.

Jeffrey M.:         With Community, our biggest challenge has been scaling, so that's where Elixir has come into play is there are periodic things that happen that just spike our traffic really heavily, and being able to manage those heavy traffic spikes has been big.

Eric Oestrich:      All right. So, what made you, I guess, interested in using Elixir in production?

Jeffrey M.:         So, back when he first really announced it, José was making the rounds, José Valim, the creator of Elixir, was making the rounds, going city to city actually introducing Elixir to people. I sat in ... there was a meetup, the Denver full stack meetup, and he did a presentation on it. I watched it and it just ... right over my head. I was like, "Why do I care?"

Jeffrey M.:         But, for some reason, 2014, I was sitting at a conference talk with [Ben Tan] ... I'm trying to think ... I think it's [Ben Wei-Tan-Ho (Ben Tan Wei Hao) ... I apologize if I got his name wrong, but he's the writer of the "Little Elixir and OTP Guidebook," but I don't think that was out, at that point. It was a RubyConf talk in 2014, and he was talking about Elixir. Something about that conversation just clicked for me.

Jeffrey M.:         Shortly thereafter, I left SendGrid and was at Parkify, which is the first place I'd mentioned that we had that, and turns out that one of the other engineers there was super excited about it, too, so Brad and I kind of put together a plot on how to get our CCO to let us start trying out Elixir in production, and that started with background orders, where we were just reading from a queue, processing data, and then sticking it somewhere else.

Jeffrey M.:         They just didn't cause us problems. The biggest challenge around it then, at the time, especially, was figuring out how to get deploys out but, as far as once they were up and running, we had these simple little apps that were just bullet-proof, and it was a pretty cool way to get into it, especially because the first apps I ever wrote were actually just straight Elixir OTP apps, and not Phoenix, which ... I feel like Phoenix isn't as heavy-handed as ... for those of you who are coming from Rails, there's a lot of people who really ... if they learn Rails and Ruby at the same time, they actually don't really know where the division is between the framework and the language. I don't feel like Phoenix has that problem in anywhere near the same stuff, especially because it can't open up and actually change the behavior of existing classes, in the way that rails happily does.

Jeffrey M.:         With that said, it was still a really great way to get into it, by having to learn OTP and having to figure out how to do a lot of my own background processing and stuff like that before I ever hit Phoenix, which does a lot of that concurrency and parallelism for you.

Jeffrey M.:         So, then, after we got those little apps running, we started expanding, and it started becoming more and more of a platform. At that point, I was pretty hooked. I have actually worked pretty hard to make sure that the jobs I've had ... I've had one job since then that ... and that was just very briefly ... that wasn't dealing with Elixir, but I really enjoy the language.

Eric Oestrich:      So, what are some of the high-level advantages and disadvantages of Elixir?

Jeffrey M.:         In my experience, a lot of people — one of the biggest advantages — I mentioned the parallel work and the concurrency already, and that's really nice, it does make those concepts really easy but, in my experience, one of the biggest things is that Elixir and functional programming in general maps to the way that most people think about programming or think that programming should work before they ever actually learn to program. Then, we beat them down and teach them about object-oriented designs and programming. I'm not against object-oriented stuff, at all, that's not what I'm saying here, but it actually really is a pretty big mental shift to get to everybody's original, basic concept of just kind of going through things in a pretty detached way to, instead, there's this little ball of information and only certain functions that are attached to it.

Jeffrey M.:         I think, mentally, it actually ... for me, it maps to ... it's easier to think about and reason than object-oriented programming, so I got that functional aspect of it. The syntax is really nice. Coming from Ruby ... you know, I did work in Java, but the majority of my career was Ruby before this. Having similar syntax is great, but a lot of it is actually ... I don't read very well, and there are certain syntaxes that are easier for me to read and parse visually than other syntaxes, so it's really nice from that standpoint.

Jeffrey M.:         The other thing I've noticed is that, one of the biggest things I've had to do in this career is, because there's never enough Elixir developers is teach existing engineers who've got long, wonderful careers, and teach them Elixir. The adoption rate, the speed at which people can pick up the language is actually surprisingly fast, and I think that is probably its biggest asset.

Jeffrey M.:         I just named a bunch of things I really like about it. From a disadvantage side, though, I think the biggest challenge is deploys, and we're in way better shape than we were back in 2015 and 2016, in terms of deploys and how to get them out the door, especially with the concept that most people think of as how we do modern-day deploys, which is you have some sort of artifact that you're dropping into an environment based off of the environment variables. That's the only thing that's going to switch its behavior, so you're actually sticking the same compiled code all over the place, and that is not what Erlang was geared towards when it was originally created. To this day, a lot of Erlang's original design still shows up in Elixir, so one of the biggest challenges we had for a long time was dealing with the deploys.

Jeffrey M.:         It's getting better, but everything we're doing ... it still has that feel of a hack to it, in terms of getting releases out, and I definitely recommend releases over mix run. We can touch back on that if you want, but the biggest thing is releases are still kind of a big pain in the butt. Bitwalker — Paul Schoenfelder came out with Distillery 2.0 last summer, I believe, and that has helped us, in terms of giving us flexibility to get releases out, but the actual system and process itself is still pretty counterintuitive. I think that the biggest challenge is getting releases out.

Jeffrey M.:         The other one that I come across regularly is that the level of parallelism can make testing hard, sometimes. People get really excited — the language is great, in that it supports testing as part of — we barely step outside of the core standard library's testing — ExUnit — for getting our testing done, so it's really great in the sense that that support was there from the get-go, it was designed with testing in mind, but actually trying to figure out how to deal with a lot of the parallelism can be a challenge.

Justus Eapen:       First of all, I just want to say that, you mentioned the accessibility of the syntax, and that has definitely been a trend that we have seen with all of our guests ... maybe not all of our guests, but some significant portion of our guests have mentioned that.

Justus Eapen:       I want to move on to some of the deployment, system-level questions that we have. So, I'm wondering: right now, and throughout your career, how have you hosted Elixir applications?

Jeffrey M.:         Back when we didn't really have things that were very good for switching up reading from the environment, our very first pass on it was to actually have our CI compiling different versions for the different environments. So, staging got its own version compiled, built out as a Docker container, and then we just deployed that Docker container. Production had that.

Jeffrey M.:         To be honest, it worked, and it was ... it didn't involve anything that felt super hacky. We just had to compile it and we were using system_get_inf ... and that particular compile environment had those variables available. Worst-case scenario, that's actually a decent fallback if people are struggling to deal with other stuff.

Jeffrey M.:         Distillery 1.0 was a big shift for us, where we were building things out in containers. Distillery 1.0 was great, in that it gave us the ability to have a single artifact that could run in multiple environments, but everything that came in had to be a string. There are times where you're actually pulling things from the environment, especially dealing with library code ... like, older libraries especially weren't taking their configuration at start-up, they were actually taking their configuration in via the application environment, and so, you didn't have the ability to actually manipulate or convert a string to an integer. That's been a challenge.

Jeffrey M.:         Distillery 2.0 has given us the ability to specify that. It's still feeling kind of tough, but right now, the way that our deploys look is that we've got CI building out our artifacts, our binaries, for the system, and then we build out Docker containers for that, we store those in a repository — Docker repository — we use Key.IO — and then our actual deploy system is using Mesos, and we've got a bunch of extra additional tooling running around that.

Jeffrey M.:         Mesos is probably on its way out, in terms of — Kubernetes seems to be taking over the world right now. One of the things that it really brings to the table that Kubernetes doesn't have is it's significantly simpler, and there's no such thing as a small Kubernetes cluster. Once you've got Kubernetes running, you're in that ecosystem, and we're in a position where we initially got our stuff set up with Mesos, and we're using a couple other tools on top of that. Some are custom internal, and some are from ... a company called Nitro put out an actual deploy tool built on Mesos called NMesos, so we're using that stuff and, right now, our deploys are ... I don't actually personally like the concept of continuous delivery, but we're basically a push-button deploy. I've got a single line to run on my box to control the remote stuff to actually deploy from those repos, and so, we're pulling down Docker containers, and then the environment variables I was talking about that are so precious to us are done specifically by environment. That's all done through configuration, through the Mesos stuff.

Eric Oestrich:      Cool.

Eric Oestrich:      So, the Docker container ... do you do the multi-stage build, and then the final entry point is the release binary? Is that ... ?

Jeffrey M.:         Yeah. I'm trying to think of it now. I haven't thought about this stuff in about two months. Trying to see if I can cheat and just go back and look at some of my Docker files ... I'm trying to think ... I apologize, I honestly am not as awesome at Docker as I should be, especially because my brother wrote one of the books for O'Reilly, but-

Jeffrey M.:         ... Yeah, "Docker: Up & Running," by Karl Matthias and Sean Kane ... I honestly ... when you say multi-stage versus not, I mean ... they're pretty shallow containers. We're not pulling a lot of dependencies in front of them, so I'm not sure if that actually directly answers your question or not.

Eric Oestrich:      I think that's the new, recommended way, if you look at the Distillery docs, so I was ... yeah, cool.

Eric Oestrich:      Anyways, with Mesos ... this is the first time I've heard Mesos in the wild ... Are you able to get zero downtime deploys with this setup?

Jeffrey M.:         Yeah, and it's the boring answer, where it's rolling deploys and not hot code. I'm going to apologize in advance to anybody who was hoping those were the words that were going to come out of my mouth, but we're in a position where we don't have any long-running processes that are so precious that we have to maintain that state. We do have processes that are potentially — when I say long-running, we've got stuff that can take five to ten minutes, and so, if we're trying to deploy when that's going, we actually are — we've got shutdown handlers that will basically start weeding off the connections and start refusing new traffic on those, but let them run until they're done with their work. In the meantime, we're spinning up new nodes and then rolling off the old ones.

Jeffrey M.:         Our zero downtime does exist and, inherently, the nature of our domain, there's a lot of queuing involved between our stages of processing, and that gives us a natural way that when we send out the signal, we basically tell each of the servers "Hey, stop processing the messages. Go through the rest of your shutdown." During that, we've got up to five to ten seconds where they need to stop processing, and then we move on. So, we've got, the apps themselves have shutdown handlers, so we're just doing the boring old rolling deploys, but we do have zero downtime.

Eric Oestrich:      Cool. It's boring, but it works, and it's a lot more stable than the hot upgrades.

Jeffrey M.:         Our biggest challenge around that, actually, is that we have certain rate-limiting systems that we have to pay attention to and recognize, in terms of our stuff, and so, when we actually are spinning up new servers, we have to adjust the rates across the servers so that they're all recognizing ... because, collectively, they have to aggregate up to a maximum rate. So, we've had to build out — we basically query Mesos and say, "Hey" — well, actually, sidecar, at that point, but we're querying the system and saying, "Hey, how many nodes are there actually running," and our nodes themselves are self-correcting and doing the math to figure out what their individual max rate should be.

Jeffrey M.:         I think that's the biggest challenge we've had, and it wasn't ... this is one of those things where Elixir actually makes that relatively easy because we just have a process that, it's a long-living process that just sits there, and just queries on a pretty regular cadence to see how many there are in the system, and each of the nodes do self-adjust pretty quickly.

Eric Oestrich:      Awesome.

Eric Oestrich:      So, that kind of leads us to our next question ... so, you have a bunch of nodes. Do you cluster them?

Jeffrey M.:         Yeah. So, it's funny ... right now, we don't, but we will be shortly. We actually have, I think our system is, right now, five separate Elixir applications with more on the way, and each of those have multiple nodes. The nature of most of them, we don't actually need that, although that potentially would present a different solution for the rate-limiting I was just talking about, where if we actually could just do a pub/sub broadcast across all of the nodes, then ... what the new rate should be, instead of letting them individually correct.

Jeffrey M.:         So, that would be one of the few advantages, in terms of further back in our system, but our client-facing stuff, right now, our dashboard is built around pulling, and we are going to be switching over to being able to push stuff via sockets. That's one of the easiest places. We could use Redis for the pub/sub, but connecting our nodes ... Our system, not only does it report how many nodes there are of each service, it actually reports the IP addresses, and so, with that, you can connect relatively easily. So, in our relatively near timeline, we're going to be doing that.

Jeffrey M.:         Then, when I was working at Weedmaps, that was the other place where we actually were running connected nodes, and that was largely for the channels, being able to use Phoenix channels and drive a light dashboard. Honestly, it's nice. The node stuff, for the most part, just works. You do have to pick parts of your domain where the consequences of any sort of separation of your nodes ... my brain just totally blanked on the right word for that, but when you have that-

Eric Oestrich:      Netsplit? Is that-

Jeffrey M.:         ... Yeah. It's not what I was looking for, but that absolutely works. You definitely want to choose in your domain, when you're choosing to rely on that as part of your mechanism of communication between your servers, you have to choose areas where it's okay if things are slightly out of date for a short period of time, or ... so the data that you want to rely on passing via those routes has to be stuff that has low consequence if there are failures, but ...

Eric Oestrich:      ... I think it's ... CAP theorem might be the ...

Jeffrey M.:         Yeah.

Eric Oestrich:      Another thing I'll toss forward, I guess.

Jeffrey M.:         Partition, in fact.

Eric Oestrich:      There we go.

Jeffrey M.:         That was the word I was looking for! The "P" in CAP. That's exactly it. We got to the point where we introduced that, and our front-end developers were actually incredibly pleased with the Phoenix library, the JavaScript library for dealing with the channels. They really liked it. It got to the point where our front-end started suggesting "Well, could we use channels here? Could we use channels here?" We're like, "Hey, dial it back. These are not actually ideal places. We need to figure out what does it look like ... if there is that partition, what are the consequences for it?"

Jeffrey M.:         That said, from a user experience standpoint, when they're working, man ... channels just really drive home a really clean experience.

Eric Oestrich:      I think there was a ... the 2017 ElixirConf, I want to say ... someone did a GraphQL API over channels, so ... just do everything over channels.

Jeffrey M.:         It's really tempting, and it's funny: GraphQL actually would probably solve one of the biggest challenges we found, which is ... we're like, "Okay, we're here to write channels," and this was ... the beginning of the project I'm thinking of is the first time we'd really tackled channels. It's like, "Okay, what are the actual conventions for message naming?" We're moving away from REST, we don't have these REST concepts of Post, and these REST-y URLs ... what do we actually call this stuff? We had to work with our front-end team and actually design an API spec. To be honest, I think that ... fortunately, it wasn't huge, but there was enough inconsistency there that going back and auditing — it's because we were kind of dealing with things as they came up. Some of our original decisions were just awful.

Jeffrey M.:         They may exist, I'm sure somebody will be listening to this as I say these things and screaming at me that "Why aren't you just looking at X spec that already exists," but I couldn't find anything as far as recommended web socket specs, in terms of actual action naming and payload stuff. We were using JSON API for our actual payload structure, but the channels, the web sockets themselves ... I'm still waiting to see strong conventions arise around what that stuff should look like, but GraphQL itself could be one of those answers.

Eric Oestrich:      Real quick, before we move on ... there's a thing called AsyncAPI.com, it's open ... it's, what, Swagger OpenAPI 3.0 for web socket stuff.

Jeffrey M.:         Nice!

Eric Oestrich:      That's something to at least document what you've got. I don't think they push anything.

Jeffrey M.:         All right.

Jeffrey M.:         Well, that ties in with, as much as Elixir is a magical, in so many various ways, let's be clear: when I say magical, I mean fun to work in, and does lots of cool things. It's not magical in the way that you write a line of code and there's 50 things that happen under the hood, like certain other frameworks and languages, but as awesome as it is from that standpoint, there are still a lot of things that are common ... like, it hasn't changed how I work as a developer.

Jeffrey M.:         One of the main things that I drive whenever I come to a company and the process doesn't already exist is, whenever you're starting an API, one of the things you do is you work with your clients and you just agree, you document what that API looks like. It's painful, because nobody likes to write documentation, nobody likes to do anything, but what it does is help you drive a consistent API, so that would be ... a tool like that for documenting that stuff is great, and without even having specific convention recommendations, that should be your first stop.

Eric Oestrich:      All right.

Eric Oestrich:      So, how does your Elixir applications compare to ... anything else you might've had? I guess this might just only go back to Parkify, with the Java app.

Jeffrey M.:         It's nice. We live predominantly in a world where performance considerations just aren't a thing anymore. The scaling that we can hit is fantastic. Phoenix itself, under the hood, handles every incoming request as a separate process. Before that, I came from a world where we were running ... oh God, what was it ... Unicorn or something like that ... then, eventually, Puma, with the Rails stuff, to try and just run, literally, entire versions of the application just to be able to serve multiple requests. That kind of horizontal scaling was challenging.

Jeffrey M.:         Phoenix, just out of the gate, leverages the concurrency model and the parallelism of the Erlang/Elixir ecosystem in a way that ... I don't focus a whole lot on that, like I used to. One of the biggest challenges for us — and this isn't specific to Elixir, this is specific to any application you write — is just you've got to make sure you're instrumenting your stuff, and actually have full-on metrics for how your application itself is performing.

Jeffrey M.:         That's the biggest thing is that, for us, we spend a lot less time worrying ahead of time about what is going to scale and what isn't, and it's more about instrumenting it and figuring out what we have to scale. A lot of times that just means, instead of having to build out more nodes, we literally could find bottlenecks in a specific thing, identify if ... Maybe we ended up with a single GenServer that ... and the GenServer has to handle its own messages serially, so it can potentially be at a bottleneck. Sometimes, you take advantage of that but, other times, it turns out "Oh, hey, we weren't thinking when we made this decision, and there are ways to actually remove the state aspect of it or pull it away from being a GenServer and actually just a flat functional file instead.

Jeffrey M.:         I think the biggest thing is that, in general, it performs as well as we design it and, every once in a while, we get a little bit process-happy and we're using processes when we actually probably could've gotten away with pure function.

Justus Eapen:       So, our next line of questioning is around background task processing, libraries, et cetera. I'm curious: how are you solving background task processing? You mentioned earlier, a little bit, that you're writing them on your own. I'm curious: what does that look like, in practice?

Jeffrey M.:         I've tried a host of different ones. The one I think that's pretty well-known that I've never actually used is Verk, which is the Sidekick-style one. My general, go-to default is literally spinning up ... the very first pass I'll have on something, and it depends on what it is, but if it's a scheduled task or if it's something that doesn't have to be exactly on the dot when it happens, but it's something that needs to be recurring regularly in the background is literally just start a GenServer that then sends itself a message when it's done, or when it starts, it uses Process.send_after.

Jeffrey M.:         That actually could cover probably 80% of the use cases out there, and it is simple. Depending on how many jobs or anything like that you have, then that would absolutely be my first place to start, and then you could start getting into other things that I've had success with. I believe Quantum is one of the things where you can get a Crontab-style time schedule or whatever, and it'll kick things off.

Jeffrey M.:         At the moment, we're actually leveraging ... Mesos has scheduling built in with cron-style syntax, and it's just hitting an end point to kick off work for us. I've used all three of those with happy success, and wouldn't tell anybody to shy away from anything. Again, I think the first step is to actually look at what the system itself can do, or what the BEAM itself can do, and the background ... long-running processes, especially something like that GenServer I'm talking about ... that's often a scenario where you don't even necessarily need it to keep state. If you do, then you need to be potentially thinking about how to back up that state and stuff in the event of crashes, but if it's not, then having a supervised GenServer that just sends itself that send_after actually works pretty well. The supervision tree ... the supervisors, which is one of the main features of the BEAM is that supervision tree ... that supervision tree can all but guarantee that process is going to be up and running.

Jeffrey M.:         Then, like anything else, you should have some sort of monitoring or some sort of learning in case your job doesn't run so that you do become aware of it, but ... at first pass, look what you could just do with Elixir and its own background processing.

Eric Oestrich:      We started with Verk mostly as a ... we were incredibly familiar with Sidekick, and it's been interesting to see how it's un-winded to just "Oh, we just did a new project," and it's like ... this is the first real production server where I feel very comfortable with just GenServers and I know it's going to work.

Jeffrey M.:         That's great. I love hearing that because it's ... I think one of the very fun things about this whole ecosystem is, first off, the more I learn ... you see all these cool things you can do, and then you go build these really cool things that you could never do in another language. You're like, "Wow, look at the way I did this," and you show it to somebody. They're like, "Oh, you didn't know about ... " and they name X that was an existing feature in the language. It's like ... Erlang has been around for 30 years or whatever, and it turns out that they, too ... the developers behind Erlang ... were also aware of how awesome their system was, and they have been building tooling around it ever since.

Jeffrey M.:         The downside tends to be that it's awful syntax, or it's really not intuitive, by today's standards. I feel like development culture, as a whole, and Elixir definitely, is pushing that readable, clean APIs is a thing, and admittedly, a lot of the existing APIs in Erlang can be tough to grok, at first, but that said, it's really cool to get into these ... realize that a lot of the stuff we pull in is actually often overkill ... that, really, the core feature set of the language and the ecosystem and the BEAM actually will deliver that as values.

Eric Oestrich:      All right. So, we've mentioned a few libraries like Phoenix, Quantum, Verk, and a few others. Is there any other big ones ... maybe outside of Ecto ... that jump out as stuff you use?

Jeffrey M.:         Yeah. So, probably the biggest ... There's a couple that are just good for maintaining code quality. If you're greenfielding the application especially ... they're really hard to bring in after the fact ... but Credo and Dialyzer ... or Dialyxir ... are really nice if you can build that into ... from the very get-go, I like having that. Again, I'm sure somebody listening to me right now hates me because I just said Dialyzer ... and, again, if you've got an existing project, don't hate yourself enough to try and pull it in, it can be a real mess trying to clean it up, but if you're going from the get-go, it really helps drive specs ... it's a way to encourage people to actually be spec'ing out their public functions, stuff like that. Credo is great, in terms of its static code analysis.

Jeffrey M.:         Honestly, the biggest checks that we guarantee to have across all of the tools are actually part of the existing library. Our CI doesn't pass unless you run a check for already ... to use the formatter to check that the code is ... whatever changes are already formatted, and then the second one is mix compile --force-warnings-as-errors. They're not extra libraries, they are features that I wish every project had from the get-go, and they tend to be a little easier to add into the projects after the fact, as well.

Jeffrey M.:         As far as ... the libraries that show up in almost every single project I've ever worked on ... I'm not in love with the API of it, per se, it's a little too Rails-y, but we use ExMachina pretty heavily, as a factory library. Every time ... I know that Ecto ... I think it was Ecto 2.0 ... they actually tried to drive it so that it was really easy to make your own factories, your own factory functions, in there.

Jeffrey M.:         The thing that I've discovered is, every time I go that route, because I'm like, "Ah, this thing is totally overkill," or even better is, "Ah, we're not actually using Ecto," I just keep ending up replicating that same feature set and, at some point, I'm like, "All right, this is just getting silly." It's to the point where, even when I'm not using Ecto, I just pass in a dummy name because you have to, when you use ExMachina, pass in a repo name, but if you're never actually interacting with the database, you just give it a dummy repo name like MyApp.DummyRepo, which hopefully, somebody sees that, they recognize it's not a real repo, and you've still got the ability to use those factory functions. Then, we just build additional data-type functions in that same factory file ... like, if we need a phone number, we construct our own helper functions to do that.

Jeffrey M.:         I'd say that's probably the one that I end up pulling in, and it's surely out of laziness. Now that I'm used to that particular style of working with the factories, just kind of keep doing it. I'm trying to think of any ... Oh. I don't know ... I can't believe I just forgot this one. The other thing ... and you're going to notice that both of these dependencies are pretty much test-specific ... testing is probably the place where I feel the most comfortable ... I use Mox ... M-O-X ... the mocking library built by José in response to his own article about mocking ... I'm sorry ... about mocks. I need to correct myself ... mocking is a noun, and mock is a noun, and not a verb, but ... we use that pretty heavily. It's to the point where I don't know anything I can't really do with Mox, at this point, that I was able to do in other languages, but it has a nice barrier to entry that it forces you to have behaviors defined for the things that you're mocking. Behaviors are not heavily enforced in Elixir, but that actually does push us to have those standardized, built-in interfaces for the stuff.

Jeffrey M.:         It also ... there's enough of an overhead using it that you really do have to choose when that is the route to go versus finding some other way of dependency injection or just, honestly, centering your code around being functional, as opposed to actually needing to have those heavy reliances on outside stuff. I think, if I had to do this over again, I would actually have named Mox as the number one thing, and then yeah, ExMachina's helpful or whatever, but the reason I like ExMachina is because it helps me build out randomized data for my testing, and you should avoid hard-coded values in your tests as often as possible. Mox, to me, is definitely a bigger thing.

Jeffrey M.:         Then, yeah, Phoenix.

Justus Eapen:       Do you have any third-party integrations that you've worked on that either posed any difficulty for you, or were interesting?

Jeffrey M.:         That's a great question. So, I've used Rabbit at a couple different jobs, and that's a ... you host it yourself, but I think that sufficiently counts as third party because it comes with its own API. We've pretty much exclusively used the AMQP library for that, and it works well, but it is just-

Eric Oestrich:      That's RabbitMQ, right? Is that the-

Jeffrey M.:         ... That's ... RabbitMQ is the actual queue system, and then AMQP is the Elixir wrapper around an Erlang library, if I recall correctly. It's about as far from intuitive as it gets, as far as working with it. We actually had to marry a Rabbit consumer with a GenStage producer recently. That was hard, but it still is probably the biggest dependency, in terms of consistently across different companies, that I've used. I think this is the third company that I've been using Rabbit with, and this is ... for anybody who's, again, yelling at the podcast, Broadway just came out last week, and it actually ... they haven't officially started supporting Rabbit yet, anyway, I don't think, so that's why I had to do this stuff on my own, although I'm lying. I say on my own ... José Valim actually helped us a little bit with that.

Jeffrey M.:         That's probably the biggest one. I'm still looking for a metrics third party system that I'm in love with. We're using AppSignal right now, and it's the closest I've gotten to being happy with something, but it ... given how much of Elixir is background processes, they really only provide immediate first class plug-and-play kind of support for the Phoenix endpoints in Ecto, but we're using Cassandra right now for some of our stuff, and we're reading off of queues, which have nothing to do with endpoints, with the Phoenix plug endpoints, so we're definitely still having to roll out a lot of our stuff there, which is not the end of the world at all, it's just ... the only challenge is that it shows up in AppSignal as a different screen, the metrics, and that's a ... I think, in general, every running application out there should have some sort of metrics system that it's interacting with.

Jeffrey M.:         I'm trying to think of other stuff that we've really kind of struggled with ... I mentioned Cassandra. We're still in search of a driver that we absolutely love ... or a library, I should say. The one we're currently using is erlcass, which is a Erlang library, and thank God we can just jump back and forth and just grab Erlang libraries willy-nilly. Then, there's Xandra. They both have ... Xandra's problem is it didn't have SSL encryption supported out of the box, and it was kind of hard getting communication to the maintainer about getting that added, whether it was us doing the work or somebody else. Then, erlcass actually just brings with it a whole pile of awkward syntax because it is an Erlang library and then it uses NIFs ... Native Implemented Functions ... with a C++ driver, and that actually has been kind of a pain from a build standpoint because the C++ driver occasionally changes on us, and we've had to figure out ways to lock in the version and all that.

Justus Eapen:       Well, we're going to move on to our last set of questions here, the first of which is, do you have a story where Elixir saved the day in production?

Jeffrey M.:         Yeah. I think that the biggest thing there is that, when it comes to debugging your runtime application, Elixir is awesome. One of the big reasons that I like releases as much as I do is the ability to jump into remote console and actually interact with your running system. There is nothing like hunting down memory leaks. I think ... the majority of the people who are listening to this have had the experience where something was going wrong, their box is running up its memory, and then falling over, restarting from whatever outside services have that, and they're trying to figure out what on Earth is going on. Being able to do something as simple as just asking the system "Hey, here's an anonymous function I wrote that just asks for the top five processes, in terms of memory usage, and what are they?" Just being able to jump into your running application, where you actually have the issue happening and you're not trying to simulate it or do something else somewhere else, has been absolutely huge.

Jeffrey M.:         Now, there are some companies that don't want their developers touching production servers, and that's a philosophical choice of that company itself. Maybe at that point, you have to work with ops or whoever it is that's your gateway there but, the majority of the companies I've worked at, I've been able to do that. Being able to actually jump in and actually do something like write that stuff or query the VM on it ... This is actually the first company I've worked at where we actually have Observer running live against our remote servers, but that would also be another way where you're not potentially at risk of running arbitrary code, and whatever issues can come with that, but you still can get a lot of insight into the BEAM.

Jeffrey M.:         A lot of it is we don't actually just see a bunch of the issues that we have, just from the design philosophy of "Let it crash." There's a lot of stuff that just doesn't come up as issues, but when we do have them, being able to jump into the live running servers and debug them directly is priceless. At the end of the day, it's probably ... I mean, it ranks in my top three about just what are the reasons that I like Erlang and Elixir, and I just totally didn't mention it at the beginning of the episode.

Jeffrey M.:         Being able to connect remotely ... and, from that standpoint, too, if people are trying to wonder what their start is, or how to get into that, I'd mentioned remote console, but there's a book from Fred ... I'm going to butcher his last name, even though it's really ... I think it's just Hebert, but ... H-E-B-E-R-T ... it's called "Erlang in Anger." I think it was co-produced with Heroku, and that is a go-to guide for anybody who wants to know — all the concepts transfer straight over to Elixir because it's all about debugging the BEAM and dealing with that, and, again, I'll repeat the name: it's "Erlang in Anger," and if you just search Erlang in Anger, you should be able to find it. It's a free PDF, and everybody should read it.

Eric Oestrich:      We'll make sure to link that, as well.

Jeffrey M.:         Great.

Eric Oestrich:      So, our final question: if you could give one tip to developers out there who are or may soon be running Elixir in production, what would it be?

Jeffrey M.:         So, I have spent a long time ... and this isn't even necessarily about production versus just kind of learning on your own. One of the most valuable — when people start feeling like they really know what they're doing with Elixir, one of the things that they should do is go and implement the GenServer module themselves, without looking at the existing code, just set it up. Give yourself some tests where you build a GenServer, you have some testing around your GenServer, and then you go and, at the very top of the file, where you see a "Use GenServer," you should switch out GenServer for a module that you write, and it should do two things: it should be a drop-in replacement for GenServer, and it also should handle sending the messages, GenServer.cast, or GenServer.call.

Jeffrey M.:         I think that going through that exercise is the best way to become familiar with what's happening under the hood with a majority of OTP is just an expansion of what you're going to learn there, and one of the catches there is you've got to make sure that you're actually making darn sure that the message coming back matches the request you made ... use make_ref, just a little hint there.

Jeffrey M.:         Doing that is one of the things that really — I've watched people — it just clicks in their head and, suddenly, so much of Erlang and Elixir becomes demystified, and so, I would really recommend that for people. It doesn't necessarily, again, tie directly into getting your apps into production or running it there, but it will help you understand what's happening under the hood in a way that nothing else would.

Justus Eapen:       Okay, well, that is great advice, and this has been a jam-packed episode. It went a little bit longer than usual, but I think that is a good thing. Jeffrey, we're really glad that we were able to have you on the show. Do you have any final plugs or asks for the audience before we let you go?

Jeffrey M.:         I'll just throw out there, I am IdleHands ... @IdleHands ... on Twitter, and sometime in the next ... I don't know how long it's going to be ... sometime in the next ... probably four months or so, a guy named Andrea Leopardi and I are going to have a book out, or at least the beta version of a book out, on testing Elixir from Pragmatic Programmer. We're working on it right now, and ... again, I don't know the exact timeframe, but just keep an eye out for it. I mentioned that testing can be ... there's some challenges in there, and we're trying to give people a reliable guide, or at least a start, on how to start tackling some of the bigger challenges in testing with Elixir.

Justus Eapen:       Super. Jeffrey, thank you so much for joining us today. Once again, this has been Smart Software with SmartLogic, talking about Elixir in production. Join us next time for another conversation on Elixir. Jeffrey, thanks so much.

Jeffrey M.:         Cheers! Thanks for having me.