S11E05 Embedded Systems in Elixir vs. C With Connor Rigby and Taylor Barto
Intro: Welcome to another episode of Elixir Wizards, a podcast brought to you by SmartLogic, a custom web and mobile development shop. This is Season 11, where we're branching out from Elixir to compare notes with experts from other communities.
Sundi: Hey everyone, I'm Sundi Myint, Engineering Manager at cars.Com, and I'm your host for today's episode. For today's episode, we are joined by Connor Rigby, Software Engineer at SmartRent, and Taylor Barto, Lead Embedded Software Engineer at Eaton. In this episode, we're comparing notes on embedded systems with Elixir, C, C++, and Java.
So this is a whole nice grab bag of languages today. Thank you both for being here.
Connor: Great to be here.
Sundi: It's been a long time since we had you, Connor, and Taylor, your first time guests, so,
newest first, Taylor, do you want to tell us a little bit about yourself?
Taylor: Sure. Yeah, I'm a embedded firmware engineer at Eaton. Recently just switched from working at GE to Eaton this year. So just coming up to one year here. been doing embedded systems since I was in high school. In high school, I started off doing some desktop applications and things like C# and VB.
Eventually I kind of got bored with that and, started discovering electronics and found that I could kind of mix doing development that you would do on desktop with some electronics and firmware was kind of like that perfect sweet spot for me. so I started doing that and haven't looked back since.
Sundi: Cool, and Connor, what about your background, for those of us who haven't heard from you in a while?
Connor: Yeah, I think a bit of the same. Somewhere around high school, I found electronics were pretty cool. , stuff like Arduino and all this was popular at the time. I sort of just got into that along with getting into Android phones, which is kind of just embedded Linux on itself. So, you know, it naturally progressed that way.
I've done a lot of other non. Embedded stuff, but I always seem to come back here.
Sundi: Yeah, so, I guess what's interesting about today's conversation, just as a primer for our audience Elixir is A small niche language, as we all know. Except for maybe Taylor, you didn't know that. but maybe you did? I don't know. Do I talk about it ever? 
Taylor: Heard it from you a couple of times, 
Sundi: Okay.
Just, just once or twice. Yeah. Fair. 
Taylor: once or twice. 
Sundi: so it is not necessarily the traditional pick for anything embedded. Connor, why are you going the route of Elixir for your embedded systems work?
Connor: I will say maybe that we get to go really early into the discussion of what is and is not embedded. You might get a more concise answer here from Frank, but a lot of people will take up in arms over Linux being or not being embedded depending on the context here. I think that's an important distinction to make early on because, I'm not sure what Taylor does on the day to day, but I'm blessed with being able to do things from a Linux level, which some embedded developers may not agree with being true embedded, right?
I'm not writing anything bare metal. I have a whole entire operating system to keep the guardrails on for a lot of those things. and so why I like to use Elixir, I guess, for it is, You get all of this stuff that comes with a nice operating system, right?
there might be real time operating systems and things in this real embedded space But you know that just the stuff that you get from Linux you get good debugging you get reloadable executables without having to burn an entire new firmware image, etc, you know and Elixir is just sort of a nice way to take that hardened Beam VM into a sort of more modern developer experience for applications that can support it well.
You'll find that you simply cannot run Linux on all devices. So, it's one of those things that you get to use when you do, or if you don't, you still have to kind of fall back, if you will, to some of those other platforms or frameworks.
Sundi: Yeah, it, it occurs to me that maybe we need to define embedded and I'm really trying to refrain from this desire to phone a friend and call Frank and get our definition.
Connor: Frank has got the most accurate definition, unfortunately. It's the one that favors me the most, but it is the most accurate, in my opinion.
Sundi: Yeah, I'm going to refrain because that is the equivalent of live coding in front of people. Taylor for, for context, Frank Hunleth is, one of the, primary contributors to nerves in the community and is almost the name synonymous with that. And Connor, your, your boss,
Connor: I think John Carsons, who is another character you'll see in the area, is technically my boss. I guess Frank is my boss's boss, though, so he's still my boss. 
Sundi: that's fair. I can't hear the word John Carstens without seeing that, medieval, instrument that he was playing at, EMPEX Utah, Salt Lake
City. 
Connor: Oh, like a ukelele, maybe? Or, like a sitar? 
Sundi: It was sitar shaped, yes. 
Connor: Yeah. 
Sundi: yeAh, we have, we have a fun time in the Elixir world. Taylor, since we're not gonna call Frank, maybe you can give us your definition of embedded systems.
Taylor: I think Connor hit the nail on the head right there. You could argue embedded includes or does not include a full operating system. you could definitely work both ways in embedded world. I've historically stayed closer to the bare metal and working either in like straight C or maybe using an RTOS, dabbled here and there using systems that have full blown Linux on it. And I'm interested in learning to see how Elixir could be solved in those situations. I'm also interested from this conversation to learn, how small of a micro has Elixir been on before. Because, you know, even within the world of having an OS, wide range of what, what that could mean.
Sundi: Yeah, Connor, do you have a answer to that?
Connor: maybe I'll partition two answers. The realistic answer is you need, like, in MMU, a memory management unit is the bare minimum required to get Linux running on something, so your ARM 9 I think is like the oldest it goes, ARM 7 doesn't have any of the extensions required for it, so I think ARM 9 is as old as it goes now, and you can get mainline Linux running on pretty much any of those cores, and they're usually marketed as like applications processors or something like this.
And so that's about as low as it goes realistically. There are some sort of like meme answers in that there's this project called AtomVM, which they re implemented Beam in its entirety in I think it's just a C, a native C. It's just a re implementation of Beam in the same way that like OpenJDK is a re implementation of the Java VM.
So, they can run without an operating system, you know that, that joke of Erlang is your operating system? They kind of took that and ran with it literally, where it actually is the operating system. Another example of this is Grisp, which is like a sister project to Nerves in a way, where they are doing bare metal Erlang.
, and that one actually is, is a, fairly hardened project at this point. It's just one of those things where if you have an application that needs an operating systems the scale of Erlang, you probably also want Linux to come with it, for the most part. You know, there are exceptions to that rule, and, you start getting into the pedantic discussion of what is and is not hard real time.
 It usually just comes with the territory, right? Like, when you want one, you usually want the other. And so as small as it gets is as small as you can find a..., microcontroller with an MMU that can run ARM9. Like, there's a lot of RISC V stuff now as well. I'm not as versed in RISC V stuff, but I know that they've got some really low, cost, low power MCUs that can run Linux.
Taylor: yeah, so it seems like to me, the benefits of using nerves would be like where I have used, full, OSes on microcontrollers would be in applications where we have Java using JNI to work with C, C programs through dynamic linking. It seems like maybe that would be the ideal place that this nerves would fit in as an alternative to Java. 
Connor: Yeah, and I think it works really well in an analogy as to Java, because I know that there are a lot of... Embedded applications that are running Java. I'm not sure what the runtime is for those I don't think I've ever worked with them, but I do know that it's like a pretty I don't know about popular But it's a well known setup and I would say that any place that you could do that You could probably put nerves in place of it, I don't want to say that like globally, but it could potentially be an option the reality is the only real thing you need is Linux, which when you say it, you can say it with a connotation one way of like, all you need is Linux, like, okay, that's an entire operating system.
Of course, all you need is that right. But then on the other hand, it's like, it's a pretty portable piece of software that can run on a lot of different hardware. And all you need is that really kind of does sum it up, I think.
Taylor: Yeah, sure. Like I could see, there's two realms to embedded one where you're making a million devices. So if you have to add expensive processor, extra memory and all that times thousands and thousands of units, you're going to probably try to gear yourself towards something that maybe doesn't include a VM and an OS and just run bare metal code.
Connor: Yeah, definitely. And I think the, sweet spot is where you'll find these internet connected devices. You know, we've got some things out there, like the ESP32, and I think Nordic's got a Wi Fi chip, and all of these, I think the ESP32 uses a FreeRTOS type environment, if I remember correctly, it's been a while, but, you know, it's a big task to start asking these little chips to, have a constant WebSocket connection open, you know, now you have security holes that you gotta worry about, and you're just Hoping that you're able to do all this stuff and see correctly, and, you know, a proficiency programmer is going to not have a ton of problems with it, but in my opinion, if you're connecting things to the internet,, it can sometimes be easier just to leverage all of that existing hardened work, you know, open SSL, which is baked into Erlang and, having Linux as your protection for a lot of things that you might focus on yourself otherwise. And, yeah, it definitely is, uh, if a budget allows it. Because, you know, if your internet connected thing needs to cost five cents, unfortunately, there's not really anything you can do about it.
You're not going to be able to run a Linux computer that cheaply.
Taylor: Yeah, I think that's where a lot of these devices, you know, you'd have the 5 cents device that's running something real small. And then having a low bandwidth network that would then talk to some central device that would then have all these fancy features on it. It's typically the implementation. You'd see. 
Connor: Yeah, and that's what we do you know, we have the, Z Wave devices, which is, like, a whole suite of devices can support this. And they all are really cheap and... Easily manufacturable, very, simple devices that all talk to the smart rent hubs that have the same radio on them that is, running a full Linux and nerves and all that business on it.
Sundi: Yeah, it might actually be helpful for Taylor, but also for the audience who hasn't ever listened to a season of Elixir Wizards before, because I think we've had a SmartRent person on every season since the beginning. yeah, do you want to give like your quick primer on what SmartRent is?
Connor: Yeah, so it's, internet enabled home automation targeted toward rentals. So there will be big installations Any apartment complex might have one of these devices in each of the apartments. And each of the apartments might have a small array of actuators or sensors or whatever.
 For example, leak detection is one of the ones that are really common. It's, it's a little square device about, I don't know if this is a primary listened or, do people watch the podcast? I was going to make a shade. It's about an inch wide. That's a really simple sensor that just detects leaks in, pipes such that it can be reported to the apartment manager or owner or whatever.
And so, SmartRent specifically makes the hubs for those types of devices that gets mounted in a closet or on a wall or whatever, and all of the devices will talk to that hub. And then the hub is used to report and do all of the fancy internet stuff.
Sundi: Digit still works with you, yeah? 
Connor: Correct. Yep. 
Sundi: another smart renter, and I were talking, I think we, we bought our homes around the same time, and Taylor, I know I called you right when I bought my house, because I was like, eh, we have things to fix. I had similar conversations with Digit, and I think Digit also talked about doing, the leak monitoring, and I was like, I need to do that, except for I think I actually do know where the leaks are, because we just got the inspection done.
Connor: Yeah, they told us where it is. Don't have to go detecting
Sundi: Yeah, true. but I did think it was pretty cool. And I did also really appreciate this, concept of, when you get smart house stuff, it all kind of goes out to the internet and it's being sold and you don't know where it's going and I just, I really appreciated Digit's approach to that kind of thing. But I think that might have been more of a hobby thing at the time. I don't know, Connor, are you guys doing that at SmartRent with the leak protection stuff?
Connor: it. It's sold. That's one of the packages. I don't actually know a ton about the business as a whole, if I'm being completely honest. I know that they install a lot of hardware in a lot of places across the country. 
Sundi: Yeah. 
Connor: on the device side we've got, you know, Elixir running , Nerves firmware itself, and then... The website where all this stuff connects is another Elixir, service. It's a server up in the cloud, That these devices are all reporting to as well. So that's a nice little bonus, I guess. If you're, happen to be making both, you can do a lot of sharing in that case.
Sundi: Yeah. And, and you're correct. we are not a video podcast. However, I think everyone could hear your air quotes just now on the cloud. We could feel it in our soul. 
Connor: I agree. I, I, emphasize them enough.
Sundi: yeah. So with that primer on SmartRent, I guess, Taylor, what are you doing at work, that kind of pertains to this conversation?
Taylor: Honestly, it's very similar. I work on wireless meters, for electricity, gas, water, flow type, things like that. So we have a network that communicates wirelessly with these devices. And then every so many devices, we have a central device that communicates with these, and then it has a back end internet connection.
 But unlike using Elixir, all of our device firmware is written in C. And then we have this whole JNI thing where we're running some Java on the gateway. Seems like... Potential application for using something like elixir fit well there. 
Connor: Yeah, it sounds like more or less the same ish setup. The same topology, if you will, I guess. Cool.
Taylor: So what made you guys land on using Elixir for that? Because I feel like if you were trying to, hire people, you're going to limit the number of people you're going to find just based upon who's familiar with embedded systems and Elixir and nerves. So it seems like that could potentially be, you know, a risky choice. So I'm curious how you guys landed on that. 
Connor: So for SmartRent in particular, I think it was just luck of the draw. I'm sure Frank will be on at some point to tell this story, so I don't want to spoil it too bad for him. But Frank created NERVS, Frank Hunleth, who is now, senior hardware engineer at SmartRent or something. He was an early developer there, and so it was sort of a case of he invented this thing, and then...
Found a company that would let him work on and use the thing that he invented and it paid off, paid dividends there because now, I think the the hidden benefits of Nerves may be less on the tech side and more on the hiring front is, you know, while Elixir isn't a super well hired language compared to Java, you know, you're going to find 15 Java developers for every Elixir one, right?
But there is a pretty large, pool of Elixir web developers just due to its, roots in Erlang and whatever. You know, it's a pretty popular web language. And the cool thing about Nerves is... You know, you can close one eye, squint, and look at it real closely and it feels kind of like web development in a way.
And so we get those developer niceties like hot code reloading and, unit testing and all of this stuff baked in for free. In a lot of other embedded frameworks, you either have to rely on like kind of a lazily created framework by, you know, the vendor or, rolling it yourself.
 A lot of the benefit comes from those sort of things in hiring... You're not specifically looking for embedded developers as much as you are looking for Elixir developers. Which, you know, is a chore in itself, but it's a little bit more narrow than... Just embedded developers who know Elixir and NERVs and all of this all at once, right?
Taylor: Yeah, that's kind of a good point because I've always viewed Embedded as like the center of development because you kind of have to be a jack of all trades. You have to be able to talk with iOS or Android people if you have a Bluetooth app, so you need to have some idea of like Objective C, Java, Swift, you need to be able to talk to back end people because most likely it's talking to a database.
You need to talk to hardware engineers, so you're doing all the scripts for testing and stuff. So if you have someone on the Embedded team that has strong web knowledge, that helps to bridge the gap for sure. I could see that being a plus. Absolutely. 
Connor: , it ends up working kind of well for, if, for example, you hire for the embedded side or the firmware side or whatever you want to call it, and then end up needing help on another part of it. It's like one of those kind of, like, small lean team things where you can... We swap people around at Smart Rent all the time John Carson's works on firmware.
He works on Web, he works on, B2B stuff, a lot of Elixir developers end up being jack of all trades developers by themselves because it's the only way you can get hired? You're just going to end up there at some point anyways.
Taylor: Yeah. Web development and firmware are like completely different how you think about things, you know, and firmware, you're thinking about timing and bits and all that, but then in web development, you're doing all this asynchronous stuff. It's kind of like two different skill sets. Almost. 
Sundi: Yeah, it's, it is kind of rare, I've told the story on the season before, but, Frank talking to, my friend Joe, a few seasons ago about, similar questions, Taylor around like nerves or why, and just like, why are you doing this with this language and all of that?
Frank was so excited because he was getting to talk to somebody from the Embedded side. First, usually he's interacting with folks from the Elixir side who are getting into Embedded. And so that's usually the way that people are entering the nerves in the Embedded Systems space from the Elixir side.
It is a small niche language, like we've mentioned, but, we've got a pretty strong community. The conferences run 400 at the smaller ends and like 800 ish to the, the bigger end. So, I wouldn't exactly say there's a shortage. You put out a job posting, you're gonna get a couple hundred applicants
Connor: agreed. And I will say, this is pretty anecdotal, but I mean, I guess anecdotes count in a relatively small community. But at my last job, I hired another, you know, I was the first and only developer for the first year or whatever, and come year two, we buttered it for a second embedded developer while I worked on the website of it after I had sort of brought the hardware up and running with Nerves, right? And I didn't hire a specifically Elixir developer. I didn't hire a specifically Nerves developer or anything. I was able to just hire a embedded Linux developer, of which there are quite plenty.
It's a pretty common job. And he, he was able to pick up, programming in Elixir and Nerves pretty quickly. I mean, like you said, embedded developers are kind of a jack of all trades. So, like, picking up a language, I don't think is going to be, the end all of the world. It's a weird language, it's a little funny and quirky, but...
At the end of the day, , if you can learn a new language, you can learn a new framework or whatever, and I, I don't think it's too big of an ask, I guess, if you're looking for the right person, right? There's going to be a reality where there's people that just want to use the things they know, they don't want to learn any new stuff, and that might not be the right hire, right?
But, if you get someone who is... Into it or wanting to learn or branch out or whatever. I feel like it's a, pretty easy position to fill there.
Taylor: This sounds almost like it's a, you know, big slash older company versus small slash newer company thing, where you're willing to try these new frameworks and tools versus a company that's been around for a while. It maybe has devices already in the field that they need to continue to support. It'd be much more likely to use tested, tried and true methods.
I think typically embedded people tend to fit in that. I mean, I think nowadays you see things like Rust that are new, that people get excited about and go into, but I don't know, outside of a few people I talk to, I don't know a whole lot of people developing on a day to day basis and things like this. 
Sundi: Yeah, Taylor, I was actually wondering, outside of just like you have a bigger hiring pool of folks, are there advantages to using Java C, C++ in your kind of daily life? 
Taylor: Yeah, I think it's kind of just the same point that... It's tried, true and tested, you know, that there's a lot of support from chip vendors that might have stacks you're using, that people are going to know how to just work in it. You don't need to have some fancy development tools. Pretty much just get dumped on the computer, able to do things in C you're close to the hardware.
If you want to, you don't have to always rely on these types of stacks and tools to interact with hardware. 
Sundi: that's fair. And I guess, on the flip side of that, Connor, I know you're a tinkerer, so I was wondering actually if you've ever done any Rust with nerves or Rust with, uh, at all. Yeah.
Connor: I know Rust. I'll save my opinions on it for a different podcast. But I'm not a super huge fan just on the surface value. but I do use Zig a lot. Which is a relatively newish language, you know, getting even more new and more niche than Elixir., I don't want to call it a competitor to Rust because that doesn't, that seems disingenuous, but it almost is a competitor to C in a way.
It's got interrupt with C, it, it can emit C code, as well as, it is a C compiler itself. Which is a pretty nifty little trick, for embedded in particular. So, and when I say embedded in that particular context, I mean like, I'm gonna say real embedded. Very, very close to the hardware, not running an OS or anything.
And as an extension of that, it can also compile the OS, cause, you know, an OS is... It's more or less written in C. You might have some assembly for specifics or whatever, but you know, the majority of it is written in C, targeting a particular libc. So, if you can bring up libc on a device, you can compile something like FreeRTOS or any of the other RTOSs out there, I guess.
Zig in particular is one of those that can do that, where instead of needing to just write it all in C, it can compile all of those C projects and you can write your project in that language as well. It sort of behaves in the same way, just with more of the developer niceties that we're coming to, love with things like Rust or Go or other statically compiled languages.
Sundi: yeah, cool. We talked about on the Elixir side, you get to do like in unit testing and all of that. It kind of led, led me towards the questions around project workflow. And so I guess Taylor was wondering what your development testing and deployment processes look like.
Taylor: Yeah, I'll say just in general embedded, it's probably a disaster compared to what you're used to on the website. Just things like updating devices. If they're internet connected devices, it sounds like you could easily update them, but the reality is if we have this environment where we have a couple devices connected to the internet and then a low bandwidth network that communicates to the vast majority of devices, it takes a long time to update devices.
And you want to be very careful because if you're spamming a bunch of firmware updates to thousands of devices, you could bring down networks. So anytime you have a firmware update, you need to be very careful. Full start small, do lab testing and all that. But even depending on the lab testing, find some smaller, , real sites that you could keep a very close eye on as updates are happening and, raise flags early, , versus in a web development environment where you have a lot more automatic tools and, you push firmware and it's there, it's done.
In terms of actual development, working way up to releases though, that's. Where I think C is nice. It's simple, it's basic. You just use any sort of ID you want. Embedded gets a little weird with the tool chains and versions of everything. I've recently been using Docker a lot to encapsulate build environments to have exact versions of tools across the board.
I've been loving using that. And then, using tools, debugger type things. I guess a lot of embedded people would probably use command line GDB type things. I personally like all the GUIs available. Different things like IAR has a pretty nice debugger and embedded workbench.
VS Code has a whole bunch of cool extensions where you could, you know, poke at memory, see at memory, set breakpoints when memory changes. those tools are all great. I think when you get kind of more into the space our Java side stuff that would compete with Elixir, you know, you have the Java debuggers.
 I don't know that I'm necessarily making full use of those. I'm sure, modern languages like Elixir probably have a lot of cool tools that kind of simplify things and make things easier to do. It'd be something I'd be interested in hearing from Connor's side.. 
Connor: Yeah, a lot of Elixir tools are sort of rooted in Erlang. Maybe we glazed over that in the beginning, but Erlang's this really old esoteric programming language that is, running the backbone of the internet. It's sort of under the hood of all of those things, you'll find, Erlang nodes are often what's holding them up, so because of that, we get a lot of their tools for free, so there's a pretty good debugger built into Erlang and, aside from that, just the language itself being functional is, it depends on who you ask, but some people say it's easier to debug because, you can only see the functions you've called, therefore, it's got to be one of those functions.
None of that's object oriented, you know, like, overloading or anything like this, so. I'll say it's, it's largely opinion, and I, having done Elixir now for five, six, seven years now, I still prefer a C debugger 100 percent of the time, and it, maybe not 100 percent of the time, that's an exaggeration, but there's just something so simple to being able to have like a register by its address and you can just put a single breakpoint on it to see what was being written into that register and um I think it's because the problems you're solving in those, very embedded or those C type things, you know, it's usually if this, then that.
There's not a ton of, huge complicated logic that needs to happen when you're debugging these things. Or if there is, you know, it's maybe time to start thinking about grabbing something a little bit more complicated that hides that, logic behind the scenes or whatever, you know. But I will say that the debugging experience Can sometimes be better on those smaller devices just due to the problem space that they're in there It's just a little bit easier to conceptualize to me.
Taylor: Yeah, I think in embedded though you get the extra layer of complexity of say you shut off your optimizer so you could run the debugger or, you add breakpoints. Now you're affecting timing. So in embedded systems, that becomes a huge thing. So you have this uncertainty principle of, , is this an artifact of debugging or is this real?
But there's a lot of nice tools in the embedded space for that, especially with RTOSs, where you could keep track of all your different tasks and, what's pending on different objects, RTOS objects and what kind of CPU usage you have just in real time. Those tools are really nice. And I don't know that in the Elixir space, you would be looking at that level. Cause you're looking more at the higher application level.
Connor: Yeah, a lot of the optimizations that we'll be doing on , any nerves project is more Similar optimizations as you might find on a web project I think that the Sort of interesting part is that like You can say that it looks kind of like web development, and if you squint it kind of does, but when it comes down to it, like every edge case that you find in web development is just like a normal case that has to be handled in, on the nerve side.
 on a Phoenix Elixir app, your database might go down or might get slow or whatever due to bad queries or whatever, and it'll just kind of pick itself back up once the traffic stops or, you can add in all of your tracing and metrics and all of this business to, see what's going on.
But at the end of the day, it's going to kind of solve itself or you just reboot the server and it solves itself. And you have a device out in the field somewhere,. Services going down aren't really edge cases anymore. You have to handle that. And, you know, your device will stop working if this doesn't work, right?
So you have to handle those things. So, in Elixir and Erlang, we've got this concept of, let it crash, right? So, like, if it, if something isn't working right, just start it over again. You know, take that particular process, throw it away, it'll crash, it'll burn, whatever. And then we start it over again, which works really well in Nerves.
It's, it's a good thing to have. But, you know, At a certain point, you have to stop letting it crash, right? Like, if your device is, keeping a human being alive, letting it crash all of a sudden sounds much less interesting, you know? And so I think there's a lot of that, , web crossover where we think, just let it crash or whatever.
It'll pick itself back up. , it will, but, like, at what cost? You know, you have to start thinking of those edge cases that we don't usually have to think about when you're doing Erlang and web development or Elixir in web development areas,
Sundi: I suddenly have this memory that's like a dream almost of Frank sending out these Raspberry Pi kits and the Elixir Wizards hosts were like playing with it and pressing the button to the numerical value where like it could contain 100 and we knew that if we got to 101 it would crash. We were just like pressing the button to get to the crash state.
We're like, oh we don't know anything but we know how to 
Connor: right? Yeah, I think we, I'll say abuse it a little bit, maybe not abuse, that's not correct, but you know, it is a thing that in the web world, you know, let it crash, whatever, it's not that big of a deal, maybe it's a me thing, but I've been bitten by letting this happen on, edge devices, you think, Oh, it'll just crash and pick itself back up again.
But like, why did it crash in the first place? Maybe we should think about that a little bit. And then all of a sudden, now you have this like huge refactor that has to be done because you just let it crash. And it was a big deal, you know,
Sundi: Yeah, hardware bugs are things that I feel like we don't talk about that often. I don't know, Taylor, have I ever told you about the big bug in my Subaru? So the head unit, if it's connected to Apple Play, Apple Play? CarPlay? 
Connor: CarPlay? 
Sundi: CarPlay, or Android Auto. And we test it on both. We'll crash the head unit and freeze and scream and it doesn't fix itself unless you turn the car off, which is great if you're on the highway.
 People ask for more software in cars and I'm like, please God, no. So medical devices too, when I think about that and it's like inside humans and people are relying on it.
A lot of type one diabetics have the pumps and then, various folks in my life definitely have heart monitors and things. And I just think about the let it crash mentality there. And I'm like, Oh, please, no, don't do that.
Connor: That's another case Where I, I would prefer to see, just a pure embedded, the language not particularly important there, I guess, you know, C or Rust or Zig or whatever you want to write there. But on those type of devices, just conceptually, I prefer the simplicity of just having your code and your hardware and that being it, right?
Like, the, the attack surface there, very small. It's just that one thing, right? It might be easy to physically break into the device, for example, but you know, you don't have to worry too much about it being like compromised over a network or whatever, which, it's just something that will happen eventually.
 Right. I feel like these are the sweet spots for traditional embedded, where I wouldn't, I would never. Claim nerves would be perfect for your bluetooth enabled heart monitor, you know Like maybe you just don't need a bluetooth enabled heart monitor actually.
Sundi: Taylor, I feel like you probably haven't done medical devices, right?
Taylor: I've tried to avoid that. I have a friend that works in healthcare, on CT machines 
doing Windows desktop application programming and 
the amount of efforts they need to go to and, , 
the speed at which projects goes is just completely foreign concept to me.
Connor: Yeah, that was what my last project was it was not medical directly, but medical adjacent and Every single thing that ever had to get done outside of our product itself was just an absolute nightmare you know SOC 2 compliance is a huge issue and that affects not only the firmware but The website as well if you're trying to host a web app and all of this stuff.
So, yeah, I, I feel your sentiment there. I'm not really trying to be in medical either, regardless of what framework or software you give me.
Sundi: Yeah, 
Taylor: I give myself enough of a heart attack releasing firmware and potentially getting into like a reboot loop or breaking a device. For something that controls a light switch or something or a gas meter. I couldn't imagine that same feeling with someone whose life depends on it.
Connor: Yeah, and that's another problem that, you know, adjacent to Nerves we have this Nerves hub, which is like a firmware update server. Which we use to, deploy firmware to all of our ENG devices. Be it Wi Fi or LTE 
connected or whatever, you know, Ethernet, . and even having this, rather large piece of software that is, pretty hardened, we use it in production. It's still scary to push out a firmware update. I don't think there's any getting around to that.
Sundi: yeah, one of the biggest jokes for web developers, I feel like is yeah, pushing the prod is you're putting information on the internet, whatever. You're not launching a rocket was like the first thing I ever heard at my first internships. So definitely feel that pressure there.
Props to anyone out there who, who does medical device work. And we all have somebody in our life who depends on something. And so thank you for your work. Sorry we can't help. 
Connor: Sorry. I know Frank used to do, medical stuff before he started nerves. I don't know if those two things are related, but that might be an interesting conversation from him. I, I think he did imaging devices like x rays or something, if I remember correctly.
Sundi: The, the most I know about those devices is that I can't wear my magnetic nail polish inside them. 
So. 
Connor: And you could, you might just not like it though.
Sundi: Yeah, yeah. I, yeah, that's true. I don't know what happens if I don't take my nail polish off, that's fair. 
Taylor: I don't think you want to be the beta tester for that. 
Sundi: no, definitely not. we've talked a lot about the Elixir community and Taylor, I'm actually kind of curious if you have any resources or communities that have played a pivotal role or are helpful at all in shaping your career with embedded systems.
Taylor: I'd say in general with embedded, unfortunately, the answer is kind of no. Depending on what parts you use, there are some good communities. When I was in high school, I started working with microchip just because the local electronics shop. That's all they sold, and I didn't have the ability to buy stuff on DigiKey and Mouser like you can easily do today.
And Microchip had fantastic forms, great community. My first job at GE, we previously used Microchip parts and all the FAE contacts were amazing there. , learned a lot from them. But then, you know, other vendors that I've used. There's absolutely nothing you're kind of on your own. I think for me in the embedded world is really the best way I've learned and grown was just from having mentors that I've worked with and developed friendships with them and constantly having conversations similar to this on various topics where we can learn and grow together.
I think really embedded that's the best I found. Unfortunately, there are some good resources like, There's a organization led by Michael Barr. He's pretty well known in the embedded world. He does a lot of trainings. I've taken some of those and they're fantastic But that's not something someone who's just getting into the field is probably gonna get into unless their company is supporting sending them to a training like that.
Connor: That's sort of been largely my experience as well with, sort of the non nerves or even I'll say the non embedded Linux, communities is, it's very, enterprise y, or enterprise focused, right? Like, these hardware manufacturers, they're making, billions of chips and,, they want to sell them, and, to do so, they're putting on these, conferences and, and all of this stuff, the same stuff that you might see in like the web world, but, they're really, really targeting the other businesses more than like developers. You know, the developers are just the ones sort of putting it all together, but there's so much other jobs involved here that the community itself or just the development seems pretty small in comparison to a lot of the other more business side y stuff, I guess.
Taylor: It makes sense because, , to get an embedded system going, sure, there's maybe some simulators here and there, but largely you're going to have to have hardware made. And so that's just another bar to entry versus if you want to spin up a web app, you could, you know, download some sample code. So you could spin up a server, maybe pull a Docker container, start it, and you're up and running. You could see your button and you make your change. But, you know, in an embedded world you either have to do hardware yourself or work with other people who do that kind of stuff.
Sundi: Something else you mentioned, Taylor, do people... Do they kind of ever land in the embedded systems world, kind of by accident, the way folks tend to in Elixir? Or do they sort of graduate college and that wasn't any question, that's what they were going to do. 
Taylor: I think most people I know were just interested in Embedded from the start. Maybe they all didn't start in Embedded, but they worked adjacent to it and then transitioned. But I don't know of really anyone making a huge switch from something like web development to Embedded. I can't think of anyone off the top of my head who would fit that bill. 
Sundi: Oh, that's funny, I was actually talking about like stage managers in like acting studios or,
Taylor: further away.
Sundi: like very far away. 
Taylor: No, I can't think of anyone that I've worked with or friends that fit that bill. 
Connor: I'll echo that as well, I don't know that I've ever, like, the closest I've ever found is someone that can go from like a Phoenix web app or whatever to, you know, Embedded Nerves. That's probably the closest I've seen of this sort of thing happening myself. 
Taylor: I think, on the internet wise, that you kind of see this, I know like some people who are into art get into like arduino stuff. Um, and I've seen some cool projects where people are focused, mainly focused on the art, but then they use Arduino to maybe like control lights or, you know, buzzers on their art projects and they get into that. So maybe that would be the one place I would see it, but beyond using like Arduino type projects, I haven't really seen a whole lot of 
Sundi: For the folks I know who do Arduino for art, I think they were on the embedded side first. And then they started, like, creatively showcasing their lights in jackets, and fashion, and boots, and stuff like that. 
Like, I've seen that. 
Connor: Yeah, the cosplay communities have a lot of this crossover as well, where, they're maybe learning it more as a means to an end. Which it does count if they're doing it, so it counts, you know, but, I don't think they're getting a job off of that because like, Taylor said there, there's sort of this jack of all trades sort of mentality.
You got to know a little bit of electronics, you know, you can't get into embedded without having at least like a 101 foundation knowledge of electronics, right? You need to know, the basics and how to read a schematic and how to read a data sheet, which are adjacent skills that are required, but, you're not getting paid to read data sheets as your career, you know, well, if you're lucky, maybe, I don't know, but there's so many other things that you need to know to be able to actually just have a job in this career outside of the, coding itself, you know, there's so many other things you need to know that I, I think even if you were to like, , pivot your career in art or whatever toward, right.
Embedded electronics, it would be like a pretty big hill to climb because not only are you gonna have to learn all of software development as a whole, but like all of these adjacent things, like technical reading and writing and testing is a way bigger deal than you might find in other industries.
Right. You know, if you're making a device, you gotta test that device. How do you test that device? You write down instructions. That's a pretty particular skill that not everyone is just born with, right. There are all these other little things that you've got to know to get sort of into the field, I guess.
Sundi: Yeah, very fair. This has been really fascinating and even being kind of adjacent to this world my whole life. I really still don't fully understand it. So thank you both for making it very easy. I know that sometimes we can be pretty heads down in our jobs, so it's very rare to have an opportunity to talk to somebody from another walk of life that's adjacent.
So just as an opportunity, do you have any questions for each other??
Taylor: I have a question for you, Connor. So now that I'm getting older, I know some people rolled their eyes at that, but I find myself not using all my free time to just do hobby projects and you know, projects for learning. If I do a hobby project, it's probably because I want to do the project first and then any sort of learning or new technologies I use are going to be second to that versus just doing a project to learn something 
new. 
Connor: Yeah, definitely same. I've been feeling that a lot lately.
Taylor: Where Do you think Elixir would be good for someone like me that might want to learn this, but, don't want to just do an Elixir project for the sake of Elixir? 
Connor: I'm sure you've heard of or seen or probably have Raspberry Pis. That's like our, go to hardware for just like testing things out. Anything that will work on embedded Linux will work with Elixir and NERVs. Some people might have a better time doing it first on like Raspbian or whatever their operating system is called now. And then, once you understand the problem you're solving, then move to Elixir. I know a lot of people do that, or it's sort of a... A common pipeline, if you will. So, I guess it depends on what kind of projects you've got going on. But for me personally, what got me originally into it was a hobby project. I do a little bit of road racing here and there, and I wanted to build a telemetry system of sorts for All of the data in the vehicle as road racing, so, you know, this is things like accelerometer and GPS and speed and blah blah blah, coolant temperature, whatever, right? My very first version of that was just a Raspberry Pi with the stock operating system that, would just kill itself all the time.
Because if you just like cut power to a Raspberry Pi on the stock operating system... It will just nuke the SD card and stop working entirely, which, you know, if you're trying to do something to either be competitive or, you know, like if you want high fidelity data, if you want your data to not have gaps or be broken or whatever, you need some additional solution.
And so that's sort of what led me down the path of learning all of this stuff in the first place, which was how do I harden this system such that it can be reliably used and work the same every single time without spending 18 minutes to boot up, right? And so there are a bunch of ways to tackle this problem.
How I happened to do it was, getting into Elixir and Nerves here. And at the time, I don't know, I think it was called Nerves at the time, just as a little history lesson, but it was real early. I just, like, stumbled onto a forum post or a talk from Frank just by sheer coincidence in, like, 2013. 
Taylor: how did you find the startup of that, like the learning curve, initial learning curves, because I know like in embedded typically learning curves could be pretty steep and, you know, documentation and startup guides aren't always complete. 
Connor: Right. 
Taylor: So I'm curious what your experience with nerves was with that. 
Connor: the experience with Nerves at the time was really good because... It was relatively new and it only did, two things and learning those two things were pretty easy, right? It boots up a stripped down version of Linux and then it boots into Erlang or Elixir, you know? the learning curve of learning Elixir I will say is not particularly forgiving if you've been writing OOP language for your entire life.
But if you've been writing C your entire life, I don't find it to be that. steep of a curve, right? Like, the syntax is a little funky, it's, Ruby meets Haskell. It's, it's a little, it's a little funny looking. But once you get past the syntax of it, you sort of know all of the patterns already by doing, like, if you've worked with FreeRTOS, the concept of processes and all of that stuff will seem, similar.
Our version of processes will look, very similar to FreeRTOS's tasks. And just the syntax on how you work about them is going to be a little different due to, , having do instead of curly brackets. 
Taylor: I'm excited now. I feel like I need to find a good project to test this out with. 
Connor: That's been my problem lately, is finding those, projects that are, just large enough scope to learn something, to do something new, but not so big in scope that they never get completed. 
Sundi: I haven't obviously done the project, but Frank did write a book with I think, is it Alexander Kudmos, 
Connor: Yeah, that's one of them, and, uh, Bruce Tate is another one that's written a couple good nerves books. I think they did a, weather station.
Sundi: Yeah, yeah, that's the one I was referring to, the Build a Weather Station with Nerves book. And for those listening, we can add that, that link to the show notes and Taylor, we can send that to you as well. fun fact, I bought that book right when it released and I have one of the misprints the spine of it says, 
Connor: Oh no. 
Sundi: help your boss help you.
Connor: Hmm. Oh, like as in it was a placeholder or something that they forgot to 
Sundi: Or something. Yeah. I, I felt like it was a very fun, rare thing and next time I see Frank, I have to have him sign it because I think that's hilarious. 
Connor: that'll be good 
Sundi: but yeah, that could be a good one and, we could totally compile a list of starter projects and if we do it in time, we could even add it to the show notes.
So that for sure is, is fun, fun to look into., well, cool. I mean, this is, again, been a really fun conversation. Thank you again, both of you for being here. Do you have any final plugs or ask for the audience, Connor,
Connor: cHeck out nerves That's pretty much it. It's fun. It's a cool project
Sundi: For the person who knows nothing and knows absolutely has no idea where, where to go, where would they find nerves?
Connor: nerves dash project org or github. com slash nerves dash project
Sundi: I love how you did not look that up. You had that in your brain. Perfect. 
Connor: Said it really confidently, but there's like a small chance that it's wrong.
Sundi: Okay, we'll put the right link in the show notes. Taylor, do you have any final plugs or ask for the audience? That could be social media, where to find info on you or your projects, if you have any 
Taylor: No, I guess I could commit the cardinal sin and software development, not posting in my personal projects 
to get hub. So nothing to plug there. In terms of asks for the audience, I guess for those of you who are listening to this that aren't embedded, I'd say give Embedded a go. Maybe we could get some more people that aren't natively coming from embedded into embedded and get those kinds of thoughts and opinions from different backgrounds. That'd be fantastic. 
Connor: I Definitely want to echo that as well if you have ever wondered how electronics work, you're already like most of the way there. You know, you just got to learn the technical crap behind it afterward now. 
Sundi: All right. Sounds good. Audience, you have your marching orders and we will see you on the next one. Thanks everyone.
Outro: Elixir Wizards is a production of SmartLogic. You can find us online at smartlogic. io, and we're at SmartLogic on Twitter. Don't forget to like, subscribe, and leave a review. This episode was produced and edited by Paloma Pechenek for SmartLogic. We'll see you next week for more as we branch out from Elixir.
Yair: Hey, this is Yair Flicker, president of SmartLogic, the company that brings you this podcast. SmartLogic is a consulting company that helps our clients accelerate the pace of their product development. We build custom software applications for our clients, typically using Phoenix and Elixir, Rails, React, and Flutter for mobile app development.
We're always happy to get acquainted even if there isn't an immediate need or opportunity. And, of course, referrals are always greatly appreciated. Please email contact@smartlogic.io to chat. Thanks, and have a great day!