S10E02 Cory OĠDaniel
Elixir Wizards S10E02: Cory O'Daniel and Future of Elixir DevOps
Sundi: Welcome to another episode of Elixir Wizards, a podcast brought to you by SmartLogic, a custom web 
and mobile development shop.
Bilal: This is Season 10, where we are looking to the next ten years of Elixir!
Owen: WeĠll be talking with our guests about what the first ten years might tell us about the future of Elixir.
Sundi: Hey everyone. I'm Sundi Myint, Engineering Manager at SmartLogic.
Owen: And I'm Owen Bickford, Senior Developer at SmartLogic.
Sundi: and we're your host for today's episode. For episode two, we're joined by Cory O'Daniel, CEO and co-
founder of Massdriver. Welcome, Cory. Thank you for joining us. How are you doing today?
Cory: I'm doing pretty good. Thanks for having me.
Sundi: I'm so excited. I do have to say, before we dive in, I really do need to know if you ever mastered the hot-
to-honey ratio, the spicy honey combo that you were looking for. Did you find the perfect duo?
Cory: Yeah, so, I am a fan of very hot food and I'm, I'm generally not happy unless I'm crying a little while I'm 
eating, so, I go with a lot of hot, so I did a, like, where, where I'm at is about a tablespoon of clover honey to 
about a tablespoon and a half of crystal hot sauce. And that is, that is perfect on some fried chicken.
Sundi: Owen's making this face right now. I don't know what this face is for!
Owen: My mouth is already on fire.
Sundi: Yeah, for our listeners, this was a Twitter conversation. I think it was around the Super Bowl. I had a full 
bottle of Mike's hot honey in my pantry and then got a bottle of Mike's Hot Honey. It was part of a charcuterie kit.
And so I was like, well, today's the day. I guess I'm gonna find out what this tastes like. And it was great. Fully 
wish I had opened up my bottle before.
Owen: I think the hottest thing I've had lately is there's a food truck nearby and Tika chicken tacos. Maybe not the 
hottest but they were ... kinda ... were spicy!
Sundi: Well, Cory, you might appreciate this then. I had a Hot Ones party in September. I fully plan on doing this 
as an annual event. We had wings and then we had the full current season lineup. I think it was season 18 at the 
time. We just went through it and I felt kind of bad cuz everyone who came over to my house left crying and I 
don't normally invite people over for them to leave crying. But that was, that was the event. It was great.
Cory: That's awesome. I, uh, a long time ago there's this restaurant where I grew up that had hot wings that. If 
you finished them, they put your photo on the wall and you didn't have to pay for the wings. It was really nice. 
And they had the, I guess the scoville units of being maced. And so my dad and I went there and we're like, well, 
we gotta try this.
And we ate them all and the photo of us on the wall, like our faces are puffy and we're crying. Um, and then we 
went back and they had one that was twice as hot and I'm like, we gotta try this. I finished one and my dad 
blacked out at the table and I thought I'd killed him., so, uh, that was, that was a freaky one. He just faced first 
right into his plate, and I was like, "uh oh."
Owen: Wow.
Sundi: Okay... change everything. We're making the name of the episode "how Cory almost killed his dad." And 
then just moving on. So, I mean, speaking of fun life stories, can you go ahead and tell us, you're the co-founder 
of Massdriver and you're the CEO. Can you tell us what Massdriver is and how that kind of concept came about?
Cory: Yeah. So Massdriver's a tool for visually managing cloud infrastructure and applications. Our idea is to try 
to land somewhere in between a pass and the full power of the cloud. And so what the platform allows you to do 
is connect together infrastructure, provision that infrastructure, and get back to writing software.
But on the underside, we're not like a code generator, like some of these other sites out there. It's actually a 
package manager for infrastructure as code and applications. So, when you're connecting stuff together, you're 
connecting actual software written by somebody who's an expert in whatever that piece of cloud infrastructure 
software is.
So the idea came from, I was actually doing consulting for Google a few years ago, and I've worked for a number 
of companies over the years, in operations and I got invited to work at Google. I have this weird niche where it's 
like, I know Elixir and Erlang and I know Kubernetes. And I guess like there was a point in time where if you 
Googled those terms, I was the first thing that came up, and Google was migrating a really big customer to 
Google Cloud that was running Erlang and they didn't know what they were doing and so they, somebody just 
Googled it and at Google and they were like, oh, this guy. And so somebody reached out to me and was like, you 
want to help migrate a giant customer to Google Cloud? And so I got the opportunity to do this. And, while I was 
working with this company, I was seeing a lot of the same pains that I had as an operations engineer.
Small companies I'd worked for, had large companies I'd worked for, had. And what I really wanted to do is make 
a way to easily take some of this commodity knowledge that operations engineers have and be able to give that 
to developers so they could move quickly, focusing on their use cases rather than like the ins and outs of a cloud 
API and Terraform and all that jazz.
Owen: So is this, are we talking Docker, Kubernetes, or is it an alternative or an abstraction on top of that? Like 
what kind of infrastructure is this tying together? 
Cory: Yeah, so it depends. So like I said, it's a package manager under the hood. So like the, if like get into the 
nitty gritty details, it's a package manager and it's a cloud type system. So, what we develop is the cloud type 
system. Anybody can add these packages and the packages essentially conform to one of these types.
And so what that allows you to do is anybody can extend the platform. So if you go look at the platform today and 
you're like, oh, I wanna run Databricks, [00:05:00] it doesn't have Databricks, you can publish a Databricks 
package and now it supports it. Similarly, our application run times are extendable as well. So today we support 
Kubernetes on all three clouds and bare metal, um, VMs.
Lambdas, app functions on Azure. Somebody is actually open source adding Databricks support to the platform, 
so you can just push your notebooks up there. So, we pretty much can run anything that you can put in an ISE 
tool.
Sundi: That's super cool. The reason that we wanna talk to you today, other than the fact that, how many times 
did his name come up, Owen? Like three or four. Was it Dave? Maybe Lucia? We've had many requests for 
many guests, and we're gonna fulfill these requests this season. Welcome, guys. But, yeah, so we wanna talk to 
you, Cory, because you were requested, but also because this is the future, the next 10 years of Elixir, and we 
really wanted to get your DevOps expertise on what the future of Elixir devOps might look like. Learning about 
what your company does and the background about that is a really great jumping off point into some of these 
more nitty gritty concepts we're gonna get into. Owen, it looked like you had a question there.
Owen: Yeah. So, before we get to the future, I think it's good to start with the history of deployment with Elixir and 
Erlang. Going back, let's say, Elixir v1, or maybe even pre- v.1, how would you deploy an app then? How has it 
evolved over time? And I think maybe in a few minutes we'll kind of get to like what, drives you to think of Mass 
Driver as a new solution or like a new company to solve this problem?
Cory: Yeah, so I mean, how have I deployed Elixir over the past, over the years? I am a big time Kubernetes 
proponent, like even though like our product is designed to kind of meet people where they are and run it how 
they want to, like I run most of my stuff on Kubernetes all the time. I even develop locally on Kubernetes.
And the reason I do this is like, not because I'm a fanatic, I am, but honestly like I don't like learning. I'm dumb, 
like I'm a dumb ol' OPs person. Like I don't like learning things. And it's like, I know the Kubernetes way and it's 
like, it's the same on AWS,. It's the same on GCP, it's the same on Azure, it's the same locally.
So, um, I've been working at Kubernetes now since, geez, I wanna say like seven or so years. So, the first time I 
deployed an Elixir application that wasn't, that Basho product, that database, I can't remember what it's called, 
was on Kubernetes. So I've just been running Kubernetes and Elixir since I started developing in it.
What's changed a bit is like it becoming container aware. So like when you originally put it on Kubernetes like 
seven years ago, it was not a nice neighbor with other applications and so you'd have to like taint them and give 
them their own nodes and your node pools to run, so it wasn't taking RAM and CPU from other applications, but, 
that's my go-to way of deploying it.
Owen: Are you thinking of Riak? Is that the database?
Cory: Yes! That one, gosh, that's, that's a trip down memory lane.
Owen: This is one of many databases I've not personally interacted with. I've just heard it, so I listened to enough 
podcasts that I, it comes up, you know, with like Erlang folks especially, so.
Cory: Wow, man.
Owen: So, okay. Interesting. So like, and I kind of come at it from a, from the other angle of like, kind of just 
recently getting into Kubernetes.
I wouldn't say I'm like an enthusiast. I'm interacting whenever I need to. But, I definitely enjoy Docker for like, 
local development and kind of, you know, using Docker composed to, to simplify that thing for like, getting 
onboarded on a project. I see the value in Kubernetes of having a reproduce, or configurable, kind of deployment 
system through code that's kind of configuring all of that. I hear some debate within the community about Elixir. Is 
it a good fit for Kubernetes? Is it a bad fit? I don't think there's maybe a clear answer. I think it all depends, like 
everything else, but yeah. What's your thoughts there?
Cory: So, in all conversations Kubernetes, do you need Kubernetes? The answer is like a resounding no. You 
can run whatever you want on a Raspberry Pi. Sure. Right? But like at some point in time there becomes like 
operational complexity of things, right?
There are bus factors in your business. I have one ops person, I have one person who knows how to deploy this, 
right? This is why I like Kubernetes. Again, like I, I don't like learning new things. It's like I got something that 
works. I'm, I'm fantastic. I'm just swinging this hammer everywhere. Right? So when I think about deploying it, it's 
like, okay, could I put it on a bare vm? Sure, I can, I can absolutely run it on a Bare VM. What's the benefit of 
that? Like, what's the benefit of running things on like Bare VMs nowadays? You don't get bin packings, so you're 
actually, you're renting something by the minute and you're not utilizing it to its full extent, right? Resource 
utilization isn't great on VM versus running something on Kubernetes where you can bin pack it together, right? I 
have to solve logging, I have to solve metrics, I have to solve secrets, I have to solve all these things. So I start 
doing it in bespoke ways, right?
So my team's figured out, oh, we've dialed in this perfect ops way of configuring all this stuff to run on VMs and 
autoscaling groups. It's great. Person that knows about that leaves. Now, the next person that comes in has to 
figure out how you did it versus, hey, last company I worked at, we were using Kubernetes.
This is all the same. It's just some [00:10:00] configuration that may change from, you know, one cloud to the 
other, but the API remains the same and like that is the truly appealing thing of Kubernetes is that there is this 
standardized API that is easy to extend, that you can just take that knowledge with you. I can take it from 
company to company.
I can take it from cloud to cloud. That's the thing that I really see as the valuable part. When you think about it 
that way, is Elixir a good fit for Kubernetes? Yeah, because you have this portable knowledge that you can take 
everywhere. You're not reinventing stuff, right? Like we wanna spend more time writing software as an industry 
and as a planet, we need to spend more time writing software and less time operating it, right?
And we spend a lot of time operating it today, and not a lot of software developers have that deep operational 
expertise. And that's what I think we really need to be focused on is like getting it to, like writing more software, 
less operating the software. And so I think you can do that with Kubernetes and now there's gonna be people that 
are gonna hate that answer.
And it's like, okay, yeah, Kubernetes is hard. It's like, yeah, if you're running kOps or you're doing Kubernetes the 
hard way, sure, but like the cloud's taken care of most of the control plane for you. Auto-scaling groups are the 
same auto-scaling groups in Kubernetes as they are in ECS or with bare VMs in AWS, right? So it's like, yeah, 
there's some operational burden there, but is it any more than the operational burden you have of figuring all this 
stuff out yourself and reinventing the wheel? I'd say no.
Sundi: So speaking of the wheel, how has the idea of DevOps changed over the last 10 years? Like what was it 
10 years ago versus, what is it today?
Cory: Yeah, so it's funny, it's like, what is it today? I'll tell you what, if you ask Reddit, am I allowed to cuss on this 
podcast?
Owen: We
could
Sundi: we've done it 
Cory: Sorry, you can cut that part out. If you ask Reddit, I have no idea what the fuck DevOps is, which I think is 
funny cuz nobody on the planet does like, and I feel like there's some serious revisionist history going on with 
what DevOps is, right?
So it's like, when you think about it like early 2010 ish range, right? Like it actually made sense I think for a brief 
point in time, right? So it's like, okay, there's data centers and then there's operations people and there's software 
engineers that wanna run stuff in these data centers. And it's all hard.
You've got gray beards that are mad and they don't wanna change anything. Cloud comes around, it's like, oh, 
it's easy to go, make an easy two instance. It's easy to make a queue, like let's tear down these barriers, like get 
rid of these silos and allow people to own and operate the code that they write.
And that made sense when our applications were simple and our scale was simple and our compliance was 
simple and our security was nothing. But like in today's world with compliance and security and how complicated 
our apps have gotten. That is for two reasons. Way, one global audience, right? And like the demand of your 
apps quickly, and two, the clouds just made it complicated, right? So with all of this stuff, we're shifting a lot of 
accountability left nowadays, but we're not shifting the expertise with it. We're saying, oh, you're, you're the 
engineer. You like keyboards, well go do some operation stuff, even though that's not your expertise, right?
My lawyer doesn't do my taxes. My taxes do, you know, isn't my lawyer, they both work in law and legal stuff. 
Right? And so I think it made sense then like where you get to today, it's like there's just too much stuff for an 
engineer to be effective at operations and security and writing software when their job is to ship features to make 
the company money. And so, where you see DevOps today is. There's these multiple camps now of what 
DevOps is. Some people are like, oh, it's engineers. They do everything. And some are, oh, well this, when we 
said tear down the silos, we didn't mean you did both jobs. We just mean we were better at communicating. And 
it's like, that's revisionist history.
And then there's people that say like, there's a DevOps team, and it's like there's not a DevOps team. There's an 
operations team and there's an engineering team. And that's fine because what you have there is expertise. And 
I think calling an operations team, a DevOps team does discount the expertise that they have in operations 
because it sounds like this thing that everybody can do when there is actual deep expertise in in operating 
systems.
Sundi: Well, what you said just there has me jumping around in the timeline a little bit because, if you could 
envision a future, in 10 years, and there's an Elixir engineer somewhere, just kind of fiddlin' around, IEx-ing it up, 
and they say, what do I need to know about DevOps to be an effective Elixir engineer?
What would your answer be? What do you think the answer is in 10 years?
Cory: Yeah. I mean, in 10 years, the same answer it should be today. You shouldn't have to know much. Right? I 
was actually having a conversation with somebody earlier today about this. When you look at, let's just take, put 
Azure and GCP aside. Just talk about AWS. How do you just run a Docker container on AWS?
 I just wanna host and a port, run this thing. There isn't an answer. It's, it's like you go set up Kubernetes or you 
go set up ECS, or you go build a VM and an auto-scaling group and a load balancer and you run Docker in it. 
Like, there's not a way to just like run a container, right?
And it's like, we've got to this point where, we feel like we have Dev/prod parity, right? It's like, oh, I got this 
container locally. It's the [00:15:00] same-ish container that I'm gonna run in the cloud. But like, that's not actually 
how we operate our software. And I think for engineers, like a good platform, whether it's a cloud or a pass, 
exposes this minimalist layer where it's like your engineers, if they want to just think about code, they should 
have that Heroku Git push model.
It's like I push my code and now it's running now, right? That's where I think everyone needs to start moving and 
you see a lot of passes there and that's great. There's a lot of opinion around the way they do things, but the 
cloud's not quite there, and that's kind of the gap that we're trying to bridge is like let you get any of these run 
times that you want in the cloud with a very simple push model.
That's what DevOps fully realized is, is like I don't have to think about this stuff as much. Like why do we care 
about this? As long as my application can run, I got the services that I need from the cloud and nothing's waking 
me up at 2:00 AM cuz it's broken.
Sundi: Ugh. The worst!
Cory: It's a simple wishlist.
Owen: How many people need to click a button to get the thing into production, right? 
Cory: Yeah, I guess that's up to your compliance auditor!
Owen: Right? Yeah. What kind of business are you, right? So... part of this, I think, so I think I'm trying to like get 
in the mind of our listeners as well. In the Elixir space, I think it's more common than some other languages to 
build a monolithic application, as opposed to either services or microservices, that seems like microservices are 
kind of going out of style.
Sundi: You're willing them so, Owen. You're willing them so.
Owen: I keep my ear to the ground. I'm happy to not live in a super microservices world, when I can avoid it. But. 
Monolith versus service or microservice, does that impact the feasibility of Kubernetes? Does it alter like the tech 
you wanna use for deploying your apps?
Cory: I don't think so. Again, if you have that, if you have somebody that has today, if you have somebody that 
has that Kubernetes operational knowledge to get you there, I, I don't think it matters if you're a monolith or 
microservice. Microservices are obviously gonna get more out of it cause you're making more changes, there's 
more stuff connected. But, I think the key thing there is, is like, what expertise do you have on hand, right? Like, if 
you have a team of five people, you got a principal engineer, you got four people out of a bootcamp and nobody's 
run in the cloud before and you've used Heroku, you should probably deploy to Heroku again, right?
I don't think that anybody should be going out there and jumping into a, a pool full of things they don't have 
expertise with if they're trying to just ship software for functionality sake. Now, if you have some Kubernetes 
expertise, or if you have experience with how to use it or you have the time to learn how to use it and deploy and 
manage stuff in there, it's like a monolith runs great there.
And, and again, the nice thing and the. Always lean towards this is, it's the knowledge that you have that's 
portable great. I run my monolith on here. I pushed it up in a deployment or a staple set, whatever it is, my 
application's running. That's great. When I'm ready for microservices, nothing's changed. I just make a new 
deployment or a new Helm chart to ship this thing, right?
I don't have to go, oh yeah, you know, that model is on a VM, and we had an agent installed on it that we 
installed with... I think Ansible... and then like somebody SSHed into it and they added, you know, the Logging 
agent, and then we added a Datadog Agent right? Being able to have like a reproducible way where you say like, 
this is how my cluster ships logs someplace. This is how my metrics get shipped someplace. You put an 
application in it, and the rest of the operational stuff just starts happening for you, like that's where you really start 
to reap the benefits of it.
Owen: Kind of going back to, I wanna push some code and I just wanna see it in production, right? So, 
Massdriver. Do you have extensions or plugins or maybe webhooks for hooking into CI? So that like... I push a 
commit, let's say my PR gets merged and, let's say I'm using GitHub Action CircleCI, you know, the list goes on, 
does Massdriver hook into that CI pipeline so it can automatically deploy?
As long as everything is successful?
Cory: Yeah, so we have like one like tenant that we've leaned into since creation. There's a couple, there's like a 
couple of like real strong things that we believe in. And I'm gonna say this one, and everyone's probably going to 
think that it is dumb, but I challenge you to think about it a little harder. So, everything that we do in Massdriver, 
we do it via our CLI, so we're API first, GraphQL API in Absinthe, and then we have our UI where you can draw 
stuff, but everything's API first. So, we have a CLI tool if you wanna do the GitOps model, or you have a team 
that's doing GitOps model, you can do that with Massdriver. You generate your configurations and push them 
using our CLI, and the cool thing is, if you're looking at the diagram, all of a sudden your diagram, your lucid chart 
of the world gets updated with the truth of what's happening.
And you can also make a change in the chart and that gets synchronized back, right? So you can definitely tie it 
into your Git pipeline. Now we, what we don't do is, we don't do Webhooks, and the reason why is everything that 
we do in Massdriver, we do it without ever having access to your source code.
So even though like we do a lot of wild stuff like auto-generating IAM roles and policies, injecting database 
credentials, like directly into the runtime of your application, we never get access to your source code. And like, it 
just blows my mind that we just like freely give any company access to our source code.
Nowadays, I feel like it's a thing that like doesn't even phase people anymore, but every once in [00:20:00] a 
while you'll see one of these vendors that, you've owe off-ed and given access to your GitHub, and then 
something happens and all of a sudden there's a source leak for like 12,000 companies.
And so, we've made the decision to make it as painful as possible for us to make sure that we don't ever actually 
get access to source codes. The way that our builds work is, in our CLI we have this mass image command, and 
it is a cloud agnostic way of pushing and provisioning container registries across all three clouds.
So if you've never set up ECR or ACR and Azure or GCP or Azure, if you do mass image push, and you have 
essentially a credential reference which ties it to your cloud account. We'll set up all the stuff behind the scenes 
and put your image in there. And so what we have the ability to do is we have the ability to schedule images.
So like we never actually get access to your source code. We can just simply say like, okay, there's this image 
and they want to run it in a lambda or this image and they want to run it in a, Kubernetes. And we just do the 
coordination for placing that in the cloud, but we never actually get the access to your source code.
Owen: I think part of what you said there kind of triggered in my mind, the thought of being either cloud agnostic 
or multi-cloud. This is not something I've had to contend with yet.
Sundi: You're making up words now, Owen. Multi-Cloud?
Owen: Well, I'm wielding some words that I have some understanding of cuz I listened to, you know, a couple 
podcasts. But, so let's say, well, I guess to back up a little bit, so multi-cloud is a real thing, right? So it's like 
there's AWS, there's Google Cloud, there's Azure, Digital Ocean. Those are the ones that I'm most familiar with.
So, if I deploy my app today and I'm on AWS, and let's say there's an AWS outage, the idea of multi-cloud is 
maybe you deploy to AWS and Google Cloud or you know, N number of, platforms that way, one outage does 
not take your entire site down. So, a platform like Massdriver, is part of what it's doing, kind of giving you like a 
more kind of cloud agnostic configuration, where you can either switch platforms or deploy to multiple platforms 
at once?
Cory: Yes and no. So, when we actually originally thought of the idea, we were like, you know what we're gonna 
do? We're gonna make, what Massdrive originally was, way back in the day, was a cloud agnostic API for 
managing infrastructure. And what we found out really quickly was, If you look at the cloud, a network or a 
Kubernetes cluster, and you look at it in an agnostic way, you get the most watered down, boring, featureless 
version of that thing possible.
So I see a few companies doing this today, and it's like, it's always silly. It's like, okay, I want a network. But then 
you'll see there's an attribute and it's like AWS and it's like, oh, here's how I need it if it's on AWS. And so it's like 
they immediately step out of like cloud agnosticism, right? So like we, we pivoted away from that, but at the same 
time, like the answer's still a bit "yeah," because there's a few other ways that you go multi-cloud, right? So what 
we actually like to push people towards, and this is something that's hard without something that acts as like a 
metacloud that handles a lot of like the boring, tedious IAM, and all the other not exciting service parts of the 
cloud is hybrid cloud.
Our industry, I feel like we're a bit bullshitty. We love to say. We use the right tool for the job. We don't do that. 
We use the last thing we saw on a media article, right? We are hobbyists like we say that to like give like product 
managers like good vibes, but we're like... "shit man, I just saw this on Hacker News. I can't wait to put it in prod." 
Right? When you get to the cloud, like that falls apart cuz there's just so much stuff to deal with. So, you get to 
the cloud and you're like, " we're serving models." And it's like, what are you using SageMaker? You're like, oh, 
why? Because we use AWS, right?
 When you get to cloud services, you don't see many people using the right tool for the job. They use whatever 
tool is available on the cloud that they're on, or an open source tool, right? What we have is a lot of customers 
that are doing hybrid cloud. And the really cool thing is on your mass driver canvas, the cloud, the canvas itself is 
cloud agnostic.
You can put multiple clouds on one diagram and provision, but they'll each be in their own cloud and they're 
unique things. You see, this is an S3 bucket. I see that this is a GCS store, but I can do things like have, you 
know, my view of the world as an application developer. Maybe I'm running on Kubernetes and I'm using RDS 
PostgreSQL.
But, I'm calling some Pub/Sub thing in GCP for some reason. Like I can have that in one canvas that I see, but 
it's on, it's on different clouds. Getting people towards like hybrid and using the right service or a service that 
makes more sense. True multi-cloud. I mean, where come from a language where we like to talk about nines, like 
every extra nine costs you money, right?
And it costs you a lot of money. Costs you a lot of time. The true source of like idea of like multi-cloud. It's really 
expensive, right? Like it's, it's hard to do. We can deploy our applications in multiple clouds. What about your 
data, right? Are we like, are we gonna do everything as a stateful Elixir app and just like distribute it around their 
cluster?
Or do we need to use PostgreSQL? Uh oh! We're using PostgreSQL, what do we do? It's like, okay, well you can 
have RDS, which is great. It's super highly available over in AWS, and I'm also gonna deploy in GCP, and if AWS 
[00:25:00] completely goes down, I'll switch over to GCP. It's like, okay, like that's a lot of engineering effort. Like 
are you the 9 1 1 system or are you selling t-shirts? Nobody needs to buy your shirt if AWS is down. Right? Like I 
feel like we don't take a lot of like reality into things when we build stuff. So it's like, okay, let's say your 9 1 1, 
AWS is down, people still need ambulances.
It's like, okay, well it's like, how are we now? It's like every single data store becomes. Conversation and a 
problem to solve for multi-cloud. It's like, okay, how do we do Postgres across AWS and GCP replicating this stuff 
and being able to fail over a master, right? It's like that's, that's a lot of engineering effort.
Can you do it? But who is doing that with data stores at scale, that isn't super critical, right?
 Where it gets to be interesting though, is again, not to ride the pirate ship of Kubernetes, but, when you start to 
look at your applications, like if they are running on Kubernetes, and this is one of the things we design, 
particularly Kubernetes applications around in Massdriver. When you put an application in Massdriver, what you 
actually do is you send us a bit of metadata about your app, and one of those things is a list of your cloud 
dependencies.
So you say like, I'm dependent on Kubernetes, I'm dependent on PostgreSQL, I'm dependent on Redis. Our 
cloud type system goes, ah, I know that ElastiCache is a Redis. I know that RDS is a PostgreSQL. I know that 
Aurora is a PostgreSQL. And, so it lets you design your applications the way we think of 'em as software 
developers, ah, I need PostgreSQL. Docker installed it locally and it works, right?
That's what we try to do with Kubernetes applications in particular. Let you focus on what your application needs. 
So, we have customers that do have, you know, their application. It's configured to run on Kubernetes, and what 
they can do is they can deploy it into their customer's accounts. And so now their customers can actually get data 
locality, they get their compliance security, it's inside their VPC, but our customers have to write their application 
once, they have one way of describing the IAC, and they can provision it into other people's accounts and 
actually get that like operational data and observability data back.
So that is like the one way that we still kind of do cloud agnosticism is like, okay. Yeah. If you want to be able to 
package your application, sell it to multiple people, if we package it in Kubernetes, that makes it really easy to do.
Sundi: So would you say that that would be something you hope that Kubernetes will change in the future? So 
that future Elixir developers or our listeners might be able to package their stuff up faster, deploy faster, kind of all 
of that. You've definitely touched on things that you hope to see in the future. But, is that the one big thing?
Cory: Yeah. I think so, and you know, it's actually, it's really funny. All of our apps are different, right? There's a 
ton of stuff inside of them that you can, nitpick how to configure, but you know, I feel like when you look at like a 
lot of what we do, it's like we, we do copy and paste a lot from Stack Overflow, right?
And we do copy and paste a lot from like other things that we find on GitHub, right? And so it's funny, the view of 
Massdriver was built by operations people. Originally, you started your network and you build all your 
infrastructure up and put your apps on top.
And so we've kind of shifted our approach where now you publish your application and we actually have this 
recommendation tool that can recommend and generate infrastructure based on your dependencies and it can 
kind of generate your entire tree of infrastructure. Switching this idea, focusing more on the application, we're 
trying to go and build a bunch of, essentially I said earlier, our run times are open-sourced and extendable. 
Anybody can write 'em. And so we're writing very framework specific run times right now. And a lot of those are in 
Helm. When you get into Node, like, you know, we'll do like a Lambda-specific one for express or land specific, 
one for Nuxt or next or whatever, right? But it's funny, I'm going through all of my old Phoenix and Elixir apps that 
I've deployed at five or six different companies, and I shit you not like I got 'em all. I got all the charts and I Diff-ed 
across all of them and they were almost identical over like seven years, not much has actually changed in how I 
package an Elixir application to ship it to Kubernetes.
Like there's some secret stuff, there's like a weird cloud config thing here, some environment variables, but like 
the gist of it is, is like, ah, there's a service, there's an ingress there, deployment. There's some secrets, right? 
Maybe if I have a stateful application, I got a stateful set. But like the configuration is really funny.
We're reaching a point where it's like there's a real similar way of running a particular framework, especially on a 
Kubernetes cluster. And so that's what we're trying to do is build these Helm templates that are generalized 
enough that it's like this can run a Phoenix app.
Pretty much whatever your Phoenix app is, you put it in there, it's going to run it. If you get to the point where it's 
like, ah, you've left that 80 22s case, you can fork it and modify it, but like that's the power of the platform. You 
don't think about it. And when you need to, you fork it and you can extend it, publish it back into our package 
manager and fine tune it. So, I think a lot of applications can get there though, especially, you know, ones that 
aren't super, super niche.
Sundi: Yeah. And when you say, I mean, it's interesting that you looked at your projects over the last seven years 
and it hasn't changed, although at the same time, I'm not that surprised cuz you are the same person. Like yeah, 
you might look at your code and six months be like, who the F wrote that? Oh it was me.
You know, like, that's everybody. Right? But it's not terribly surprising to me that the way that you [00:30:00] 
package up your projects doesn't change if there is not much to change about it. But it, it does beg the question. 
Does everyone do it the same way that you do? If you compared your seven years to, to Dave, seven years to 
Owen, seven years to Dan's seven years, you know, I'm just throwing out names. Would it look the same?
Cory: I think it probably look pretty similar, right? So, when you think about packaging things up in Kubernetes, 
right? It's like it all comes back to that API. There's a standard API in Kubernetes, you can extend it and add 
other things to it, but, the core of what an application is, it's a deployment or it's a staple set.
It's either, you know, stateless or not. Right? There's a service, there's a way of routing traffic to it in ingress, so 
it's like you have a lot of these high level objects and then those configurations change and that's fine, right? Like 
that's what we kind of exposed to the platform is like what is your configuration?
What are your environment variables? Which ones are secrets versus configuration? Where's your Docker 
container come from? The structure of that object doesn't change. That's the nice part about Kubernetes. It's not 
some bespoke bash script. It is a thousand different organizations saying this is the structure of like how you lay 
out a deployment.
Your values might change, but the, you know, there's a strongly. Or there's this type system and this, this schema 
that describes like what it is, right? So, you don't have a lot of structural change unless somebody's added one of 
these extensions, right? So maybe somebody's added Prometheus or not, right?
But this is where it starts to get funny into like the bike shed-iness of like software development. We like recreate 
a lot of stuff a lot of times and it's just like, ah, somebody's like, ah, I went and worked on this for two years cuz I 
didn't like this one feature of Prometheus.
It's like, why didn't you just fucking add it? It's one of these things where, if you go and look at the cloud Native 
Compute Foundation, like if you look at the landscape, there's like 20 different tools for, for doing metrics and it's 
like, but like the one that's one that everybody uses as Prometheus.
So, if we looked at the way I collect metrics for my applications and somebody else for Phoenix might use 
something else, why does that matter? Why does it matter? Like are your metrics getting to where you want 
them? Yeah. Is your application running? Yeah. Who cares if it's Prometheus or this other thing?
That's kind of where our approach is. Give people the best one, the one that's winning, the one that everybody 
uses. And you're gonna have people the want this niche other thing. But it's like, okay, if you're like the tinkerer 
and you like to play with things, fork it and go play with it, right?
Go add that functionality. But, we're just gonna focus on the stuff that gets people back to writing software and 
not worrying about the operational portion.
Owen: So, you, you mentioned a t-shirt company as an example earlier, and my brain since then has been 
thinking about a new startup where Sundi and I just sell Kirby versus Jigglypuff shirts.
Sundi: Oh my God. Stop it.
Owen: So, when we launch this highly successful, multi-region application, it's gonna run on Elixir Erlang and I 
think I've heard people, not to siderail this too much, but like, there are questions some people have who are kind 
of newer to Kubernetes.
Are we losing anything with the Beam VM or Erlang specifically? Does that make it harder or does that kind of 
make some features kind of just not work? Does it change the tools that we have available to us when we're on 
Kubernetes?
Cory: I don't think it does, but at the same time, I think this is one of the things that people are kind of, maybe 
weirded out by when they first see our code bases. We do use Elixir. And, I don't remember who said this a long 
time, actually, I'm not gonna say what I'm gonna say cause I know the person who said it last time got absolutely 
fucking eaten alive.
I got too much going on right now to get Twitter hate. But, I agree with that opinion that somebody else said about 
six years ago. I believe the strong, I believe in it strongly. If you know what I'm talking about, wink, wink, nudge, 
nudge. We are deploying all day long. We deploy a bunch of times a day.
We're autoscaling, right? But, we got demands. Scale the thing up. None? Oh, we'll scale it down. We use 
serverless PostgreSQL, we scale our, our PostgreSQL hits like zero in the middle of the night, no CPU, right? We 
are full on auto-scaling the things, controlling our costs, right? When you think about that, and then you think 
about Elixir, stuff starts to get weird, right?
Like we're talking about DevOps, right? And operations, and there's a lot of Elixir that is operations and DevOps 
like in the code itself, right? It's like, okay, we're clustering our nodes. It's like, okay, so now you're talking about 
like finding each other on a network. You're talking about service discovery.
Okay. And we're auto-scaling? Great. What happens when we go from one to 30 back down to two to 10, and 
like there's data distributed across this cluster? Now we're starting to write software inside of our software to 
move our data around. And so we don't use a ton of OTP. If it's in a library, we use it, but probably the extent of 
the most Elixir-y thing we do is we connect nodes so that when a notification comes out across our GraphQL 
subscriptions that it hits people and honestly, we're considering just switching to using Redis just so we don't 
have a bunch of noise when the nodes connect and disconnect in the logs, cuz it, we're also log-free.
The only thing we ever see in logs is nodes connecting and disconnecting. We're big open, open Telemetry fans, 
so we just, we open Otel all the things.
Owen: And you're saying that's not [00:35:00] useful? That's not useful logs? 
Cory: That's the gist of it, is like our, our application's stateless and so it doesn't matter if something connects or 
disconnects and it's really just noise.
It's like, oh, something showed up in logs. We thought maybe somebody put in a frivolous logger or there was an 
exception. It's like, oh, no, a node just disconnected. Who cares? It disconnected because it's shutting. It should 
be disconnecting, right?
Or, oh, it connected. Why is it connecting? Oh, we auto- scaled up. Right? So, you know, you know, I think that it 
doesn't necessarily impact the tools that are available, but I think that what it does give you is it gives you a 
different way of thinking about like your compute being commodity. Why do we have three instances running all 
the time?
Are they running at a hundred percent? Cuz if they're not, you're wasting money, right? You're paying for a 
hundred percent of that compute, right? We just try to bin pack everything and get it down to as, as minimal 
amount of CPU time running as possible. And so, that's one of the things that I think kind of goes counter to like a 
lot of things you'll typically do in an Elixir application where you do want to have state, you do want to have 
processes that you're pushing around. Like we, we don't use it at all. So, boring app. 
Sundi: We kind of touched on this, but would you say that in 10 years an Elixir developer wouldn't need to know 
how to deploy their app and they should just use that service or that button or that git-push kind of thing? Are you 
standing by that one or is there kind of a little bit more nuance to that? 
Cory: I think that's where we need to get to and here's why. I actually just wrote an article. Microsoft, let me write 
a thought piece on the future of the cloud, which was an insane thing for them to do. You know, Bill's gone. So 
they're just doing whatever now. But, there is like this existential dread that I have when I think about our 
industry.
We joke all the time about like, oh, we don't know what we're doing. We're out here guessing, no, we're just 
copying, pasting shit off of Stack Overflow, right? Like, if you go on Twitter, there's a million developers a day 
joking about how they don't know what they're doing. They're making $200,000 a year.
My dad would've loved to have made six figures to not know what he was doing, right? And a lot of what we do is 
based on hobby, right? Like we like writing software. It's what we do after work. It's what we do at work, right? 
Like we're, we're, we're people that like to have fun. We're always learning. And that's why it's fine for us not to 
always know what's going on.
But it's really funny when you think about what we are. We are extensions of a business. What we do is a tool 
that the people that know the business don't know how to use, right? Like that's like a woodworker coming in and 
saying, " hey, I wanna build a table, but I forgot how to use saws," And is telling somebody else how to cut some 
wood to design this table.
Like that's what modern businesses are today. We have product managers, business side, like thinking through 
how the thing works and then they come over. It's like that scene in office space. They come over, they tell the 
engineers like what they need, right? And we go and build it for 'em.
Where this starts to get worrisome is, the average tenure for a software developer is like 18 to 20 months. Like 
we constantly hop, why do we hop? Because nobody gives us raises. And if you hop, your salary goes up 10 to 
20%, nobody wants a 3% raise, right? And so we're hopping around and a lot of people hop from industry to 
industry, not getting like deep business knowledge, right? So, where this starts to get freaky and like why I think 
we need to get to the point where things can easily be deployed, US Department of Labor Statistics says that the 
software industry, or the software role in the US is growing at five times the rate of any other job. 70 years ago, 
there was one software developer on the planet. Today there's more software developers than the population of 
Australia. The average software developer has less than three to five years of experience, and they're operating 
software. They're writing software, which is something they have to do because not enough people know how to 
write software that the experts and professionals in our industries don't know how to write software. So, they go 
get expert software developers to write the software for them, right? What's kind of worrisome about this is, you 
only get operational knowledge from operating in prod. And that's why I was saying earlier, like, I think DevOps 
kind of discounts the operational expertise. It's not something everybody can do. I mean, everybody can do it, but 
we're guessing our way through it. 
And so... What are we doing? So we're running, we're writing software, we're selling it to people, we're telling 
them we're professionals, and then we're like, I just copied and pasted some shit for how to secure this off of 
Stack Overflow.
And guess what? My product manager just wants us to convert. They're not concerned with the amount of time 
I'm spending, securing and operating the thing, as long as people can click the button and buy more stuff, right? 
So, it's like, the incentives of the business and software engineer do not align with operating and securing 
software well.
In the meantime, 8% of software developers in Stack Overflow's last survey have cloud expertise, but 
everybody's trying to move to the cloud, so less than one in 10 people know how to run this stuff effectively. In a 
world where software developers are growing at five times a rate of any other job we're making more and more 
software developers. We're making less and less people, less doctors, less lawyers, less pilots, less everything 
else, cuz everyone's becoming a software developer, right? And so, 10, 20 years from now, we need to know 
how, we need to know these systems that are being created by people that aren't going to medical school for 
eight years, but they're writing AI to cut somebody open,[00:40:00] at least can operate the software securely, 
right?
Like you might have a bug that cuts too far, that's fine, but like at least their, like, their healthcare data is secured, 
right? That's the thing that really worries me. We have too much stuff to worry about day in, day out as a 
developer. Like there's the business rules that I'm trying to process and figure out how to turn it into code so a 
computer can understand it.
They're securing the thing, there's operating the thing. Like there's a lot of stuff. And at the end of the day, what 
I'm supposed to be doing is delivering value. And if I'm doing the operational part, I'm not doing the part of the job 
that I'm paid for, I'm not doing the part of the job that my non-engineering coworkers see.
That's why we need to get there. The role of operations person is still very important, but they start to be experts 
in these different passes and cloud systems that know how to operate these things. And that's fine. Like that's 
where they should be. That's how I think Dev and Ops really should ... the silos are fine... they delineate 
expertise and that is okay. So that's my soapbox, Reddit!
Sundi: Every time you said somebody is operating software. You said it in a way that made me think of like a, a 
toddler operating a forklift. That's the image that came to mind. 
Cory: It's wild, right? We effectively have root access to every single one of our customers cloud accounts, right? 
So like we, we do a, a painstaking amount of work in security, right? And it's just like, there are people on our 
teams that haven't worked in the different security.
It's, it's really funny. Thinking about Azure development, ownership and like letting people like move around their 
stack. It's actually hard to do that at Massdriver because there's some stuff that's like, this requires so much 
security knowledge to secure this thing.
This is one of the reasons we lean so much into Otel, we have an entire part of our system that no human can 
access, and that's where like all of our credentials are actually running. It's equivalent of like the most air gap 
thing you can have in the cloud. And we get some telemetry data out of it of what's going on and that's how we 
debug it, right? That's what's important to us. Like that's what we had to do to like get this system secure and it's 
like it takes a lot of effort. I don't know that. If this was a company built by somebody who had a, uh, an MBA that 
didn't care so much about security, that they would put as much on their engineering team to secure that system, 
right? But it's important to us, it's a part of our brand. It's very important that we don't have any sort of exposure. 
So like we go through all this pain to do it. Our banks, our healthcare companies, right? Like you see healthcare 
leaks. I just had a bank recently just leak a bunch of my information. I was like, yay. Right? And so like this 
happens all the time. And sure it could be a mistake if somebody who's an expert, but we know that one in 10 
people are experts. So, it's more likely that it was a mistake of somebody that wasn't an expert. 
Owen: Yeah, there are a lot of admin password credentials in the cloud. Hopefully fewer every year, but I think 
you're, you're right. More engineers, more problems, baby. Maybe that's the way to sum up this entire episode. 
We could spend an hour on, I think the, the credentials stuff sounds really interesting.
We could spend an hour on just Kubernetes deep diving even further. But, we've somehow already reached 
almost an hour. So before we wrap up, do you have any final plugs or , any asks, anything you want to say to the 
audience before we go? 
Cory: Yeah, I mean, check out Massdriver. We'd love to have people give us some feedback on it. Like I said, 
we're starting to lean more into the application side to make it easier for application developers, so we'd love 
some feedback there. We do have a deal for Elixir developers where you get $5,000 of Massdriver credit.
Then, we also have just like a base level free plan that's free for life. 
So, your first $500 of cloud spends never build. So, we definitely appreciate any feedback there. We're trying to 
make it easier and easier for people to just focus on their apps, tell us what their dependencies are and get back 
to writing software. But yeah, besides that, put hot sauce in honey and put it on your chicken. I don't have a lot of 
asks. That's it. that's two things. I asked two things of you.
Sundi: And that's full circle folks.
Owen: Full circle, right? Hot takes... literally. So, awesome. Well, we, we will have show links. We'll have links in 
the show notes for that $5,000 Massdriver account. That'll be great. And yeah, thank you so much for joining us, 
Cory. And thank you to the audience for joining us. Thank you for sticking around and we will catch you on next 
week's episode of Elixir Wizards. 
Owen: Elixir Wizards is a production of SmartLogic.

Sundi: You can find us online at SmartLogic.io and weĠre @SmartLogic on Twitter

Bilal: DonĠt forget to like, subscribe, and leave a review!

Owen: This episode was produced and edited by Paloma Pechenik for SmartLogic

Sundi: WeĠll see you next week for more on the next ten years of Elixir

Yair: Hey, this is Yair Flicker, president of SmartLogic, the company that brings you this podcast. SmartLogic is a 
consulting company that helps our clients accelerate the pace of their product development. We build custom 
software applications for our clients, typically using Phoenix and Elixir, Rails, React, and Flutter for mobile app 
development.

We're always happy to get acquainted even if there isn't an immediate need or opportunity. And, of course, 
referrals are always greatly appreciated. Please email contact@smartlogic.io to chat. Thanks, and have a great 
day!