EW S4E3 - Transcript EPISODE 3 Steve Bussey Justus Eapen: Welcome to Elixir Wizards, a podcast brought to you by SmartLogic, a custom web and mobile development shop based in Baltimore. My name is Justus Eapen, and I'll be your host today. I'm joined by my esteemed cohost, Eric Oestrich. Say hi, Eric. Eric Oestrich: Hello. Justus Eapen: And this season, season four of Elixir Wizards, we're talking about system and application architecture and we're joined by our special guest. Today's guest is Steve Bussey. Hi, Steve. Steve Bussey: Hey. Justus Eapen: How are you? Steve Bussey: I'm doing well. Thanks for having me. Justus Eapen: Oh, we're so glad to have you. So glad to have you. We normally open up with some personal questions, so we're going to do that, but this one comes from a fan of the show. Amos King asks, "What is your favorite pair of shoes?" Steve Bussey: My favorite pair of shoes. That's a great question, I love that. I have two favorite pairs. I got a more formal pair and I got my more casual pair. I have this pair, it's a Nike Dunks. The cool thing about this pair of shoes is that it's from a skateboarder and he used to paint his shoes, so you couldn't tell. And then as he skated it, the paint would wear off. So the shoes come in full black paint, and then they wear off into different patterns. And you can hit it with a little alcohol wipe and stuff if you want to accelerate it in places. So I love those ones, they're super cool. And then I have a pair of Margiela shoes that have paint splatters on them, which is just like... I love the look of those shoes. I didn't pay full retail price for anyone that looks those up, I bought those used. Same for the other pair too. Justus Eapen: I've only heard of Margiela in rap songs, and I don't even know which one. Steve Bussey: Yeah, it's in different rap songs. Justus Eapen: It'll come to me though that later and I'll shout out my favorite rap. For some reason, I was expecting dress shoes. I thought you were going to say something like, I don't know Carminas or something that. Steve Bussey: I do have dress shoes, but actually, I don't know at what point I stopped wearing them. I think maybe when I got more into sneakers, because I can find sneakers that are dressier and that work with different outfits. I've been more into sneakers. And if I ever go to a conference, you'll see, I always bring my favorite shoes to conferences and events. Justus Eapen: Just out of curiosity, how did you get into this as a hobby? Steve Bussey: I think there's a stigma around engineers maybe not having the best style taste. And I remember I would go to conferences and stuff, I work in a sales company, and you get comments of like, "And here's our engineer, Steve, you can tell," tongue in cheek. It's funny. But I started thinking more about like, "Well, what if I spent a bit more money and make sure I get stuff that I like, and that I'm a little bit more curating about what my outfit is." And I started getting into something like button downs, nice button downs. And then at some point I found a pair of sneakers that matches a crazy button down I had. And it got me into sneakers and I started finding sneakers that matched the button down. So actually you can wear a nice button down shirt, like a short sleeve or something that, but then have like a pair of shoes that goes with that shirt, even if it's like a crazy shirt. Justus Eapen: Yeah. That's a really cool hobby. I like shoes as well. Eric, do you want to start with some real questions? Eric Oestrich: Sure. To continue on the about you train, where are you from and how long have you been into doing programming? Steve Bussey: I guess now I would say I'm from Atlanta, but I originally grew up and went to school in Pennsylvania. I moved down to Atlanta six years ago, about. And I've been programming probably since like the sixth or seventh grade. I used to play a lot of video games as kids do, and back then it was the beginning of when people would like... Well, not the beginning, but as MMOs were coming in and people were hacking MMOs, because a lot of stuff was still client side. And I just remember really being interested in that stuff. And I wasn't good at any of it, but it was like an entryway into programming. And I'd say I actually started getting decent at programming, actually being able to create programs from scratch back in high school and then went to school for it. And Elixir in particular, it's only been about three and a half or so years for me. Coming in from a Ruby shop, Elixir made a lot of sense, and I've been enjoying it ever since I started. Justus Eapen: I am a little curious how you found the Elixir world. Talk a little about that. Steve Bussey: Yeah. I have a coworker, he's smarter than I am. And I don't know where he heard about it, honestly, maybe a Ruby conference or something like that. I know that a few years ago, you're looking at four years ago, people were actually evangelizing Elixir at Ruby conferences. I haven't been to one since, actually since I started using Elixir, I just go to Elixir conferences now. Maybe that's something that still happens. But I think he heard about it there and started looking at it. And it's just one of those things like whisper in your ear, again and again and again. So that's a good way to get a colleague on board, just bring it up and be like, "Oh, this would be really easy in Elixir because of X, Y, Z," or, "Oh, this would work really well here." And it was organic over time. It probably took like six to eight months to actually really get it to take hold and actually introduce it into the organization and serve like a pilot capacity before we vetted it out and saw that it was working. Justus Eapen: Can you say which organization you're referring to? Steve Bussey: Oh yeah. Sorry. I work at SalesLoft in Atlanta. Justus Eapen: Nice. Cool. And we'll make sure that you have time to plug them at the end of the show. Now we do have actual technical questions. And I didn't mean to say that it was more interesting, it was not an insult. People come here for the meat, they want to know about the tech. They want to hear about Elixir man. But yeah, can you just talk about architecture, because theme for this season is a system and application architecture. And I think that this word means different things to different people, so I just want to hear what it means to you personally. What is architecture? Steve Bussey: Yeah. To me, architecture is about identifying, planning and even possibly mitigating decisions that are going to be hard to change. I don't look at architecture as things like, "What is the class structure of your application or your module structure of your application?" That's a very low level thing that you could change that out, and probably you're going to have a not fun time doing it, but you could change that out fairly easily. But I looked at it as things like, if you're in a microservice world, how are your systems talking to each other? Who's the authoritative source of things? We say things like, "Well, databases, aren't really architecture because you can change them out, but at the same time, changing out of the database sucks." I've only done it in the capacity of a database to a very similar database, but I know that it's not something I'd want to switch out very frequently. So that's something that in the beginning of a project, maybe thinking about over time, what is this data storage going to look in this data model? And so it's really about planning those things. You want to solve it, but it's about, if you see that there's a possible avenue to take that could make it easier to change in the future, it's about finding and going down those avenues so that eventually if you do need to change, hopefully you don't, hopefully everything goes great and it just works, but if you do need to change it, you have that avenue that you planned for yourself in advance. Eric Oestrich: One of the things that I feel like, you can architect a system and you can design a system. What do you think the difference between the two is? Steve Bussey: Yeah. I just talked about architecture a lot, and then design I do think is a little bit different. I think design gets into the more nitty gritty of the code and the actual act and process of writing that code. So I would look at your module structure as a component of design. Obviously, we have the word design pattern, so hopefully it deals with design or else maybe it's just poorly named, but when you look at design patterns, you're looking at, how is this code going to interact with this other bit of code and make it so that it's easy to modify. Again, you're usually looking at making it easier for yourself, easier to change, easier to understand. So design really fits into that more tactical act of writing code. And then I think architecture's a little bit more, maybe the strategic side of code, if you will, a little bit more of the planning side and just trying to see different avenues. Which is one reason why, as an architect, I think a lot of people might turn up their nose a little bit at a term that because there's a connotation of the ivory tower architect that looks down on all the engineers and tells them what to do. But if you were all strategy, that would be the case, because you would not be doing anything tactically, you would just be telling other people what to do from a strategic perspective, but that's not- Justus Eapen: That's not what architects do. Steve Bussey: Yeah. Well, I guess it could be. Maybe at certain companies, people fall into that trap, but something that I do and I'm very careful of is, I write code more time than I'm doing anything else. I'm actually doing that tactical side of things, because one thing I've seen happen is if you're only focusing on strategic, you could be missing things like, "Well, I have this strategic decision I want to make, but if we go down this route, it's going to be five times harder to actually build the thing." And so that's something that you to actually consider in your trade offs or else everyone's probably going to hate you a little bit because you would be making decisions that are not taking their perspective into account and the things that they're going to have to do to realize that. Justus Eapen: I really liked this idea of architecture being about the things that are difficult to change. I that a lot. I'm going to think about it a little bit more, but I that a lot. I think that one thing that's been coming up a lot in the Elixir world and I think it's a product of Phoenix contexts is domain-driven design. Can you talk a little bit about what you think it is and what your opinions on it are? Steve Bussey: Yeah. I definitely do not do domain-driven design. I haven't been taught it or really spent time learning it in particular, because I know there's books about domain-driven design, and some people are very big proponents of it. So I don't have enough to be like, "I don't it," or anything that. I would say I do a light version of it and I probably use context in a more forgiving way than some people do. So for me, the important part of the domain-driven design is making sure that things that are part of a subsystem that are different from another subsystem are separate from each other. So that basically the more that your subsystems can be isolated, the more that they reach over and talk to each other directly. While it's increasing the coupling of your system, it could make changing a subsystem much more difficult. So let's just give an example like I work at a sales company, so I'll say the phone dialer and the emailer, these are two distinct subsystems. Placing phone calls is very different from making emails. But if somehow these two systems were at the code level linked to each other, making calls across the boundary or doing stuff, it would be really hard to change the dialer because of your emailing code. That wouldn't make much sense and would probably be a pain to make those changes. So one aspect of domain-driven design and what Phoenix contexts are doing is the encouragement or at least putting it forefront in your mind that it's something you should be doing. "Oh, hey, I should actually separate this at the code level, separate that at the code level." And then it raises the question of, "Should I allow things to reach deeply into another space?" So when I do Phoenix contexts, I like to put all of my public interface functions at the top level on a module. And if you want to interact with that subsystem, you do it through that module via its public API. And I try to avoid those things that reach in like a.b.c.d and then call a function because now you've increased the coupling of these two systems. And it's probably not as bad in Elixir because you have a compiler. Like if I was to change a.b.c.d.function, and I didn't change it somewhere else, I would at least get a warning about that or it wouldn't compile. But in other languages Ruby, you wouldn't get anything, you would get a runtime exception. So hopefully you have tests and stuff for it, but that can be a lot very difficult to figure those things out in a fully dynamic, non compiled world. Justus Eapen: Okay. I think I hear what you're saying. I'm trying to understand with, how is what you described with making all of your public functions at the top-level module in Elixir, how is that distinct from decoupling sub systems as a domain-driven design purpose? Steve Bussey: Yeah, sorry. That is how I do that. So that's how I realized that decoupling that domain-driven design once, I do that at the module level usually by making sure that things have a very well-defined public interface. I think the difference is when you get into true, people that are real strong DDB proponents, you're going to be looking at actually making those subsystems potentially impossible to talk to each other directly. There is no escape hatch, you must go through the public interface. Which can be a little bit inflexible if you need something that would be difficult to do otherwise, especially if you're dealing with a SQL database, which you may need to do joins and stuff like that, which is inherently going to be coupling two things together. So I don't reach the level of like in Elixir that could be umbrella apps where you actually have a domain, domain, domain, their umbrella apps. Actually, at the compiler level, they don't have a way to talk to each other, they'd have to be tied together through the umbrella orchestrations. Justus Eapen: This is a great segue because you bring it up umbrella apps. I know that Eric, you've got some opinions about umbrella apps, which I think amounts to like, don't do them. Maybe you could explain your position so I'm not misrepresenting it. And then Steve, could you maybe just respond to that? Eric Oestrich: Yeah. One of our first Elixir apps we reached for an umbrella app. Justus Eapen: Also, could you just tell us what it is in case somebody doesn't know? Eric Oestrich: Yeah. So an umbrella app is an Elixir app comprised of many other apps locally. You have your main project and then the apps folder. And then every folder inside of that is its own Elixir project. They all have the same mix.lock file and they can talk to each other if you say, the dependency is in the umbrella, it'll look next door in the folder structure. And then it also prevents circular dependencies, because I don't think you can have one thing say that it's going to use another, but it can't request it back, so it kind of forces you to split stuff up and make sure that you only talk to things that it should be able to, or whatever. The reason I don't them is because we reached for them way too quickly in that projects life cycle. Like we had three applications, I think. One was the app and then the other two were single files that just talked to a remote API and was like, "This is how we're going to talk to Stripe. Let's have a full Stripe app," or whatever. And so we just reached for it maybe incorrectly, just too early, whatever. Eventually, we just got to be such a pain to work with that we just collapsed it all down to a single thing. And so I feel they might make sense once you have a application and you break things apart into pieces that make sense. Like I've got my one big side project. I've got what I'm calling a ponch-brella because it's not an umbrella and it's not a poncho, because it's a single get repo, but they're just doing dot-dot-path, dot-dot-slash next to me to do it. I don't know if it's better, but it's working. Steve Bussey: For me, I saw this on Twitter and I wanted to work it into my weak ones. So I think it's applicable here, which is, I don't know enough to have an opinion about it because I haven't personally used umbrellas because I think the things that I've read have been to avoid them. And I think one of the big reasons, like something I do know about from reading a lot of documentation is that libraries, and in particular when you get to things around deployment and how maybe some of the major libraries expect you to work, they do usually have, like if you're using an umbrella to do this instead. But I feel you're working against the grain a little bit. And so I feel the documentation is always like, regular application first, and then there's a little aside about umbrellas. And then I've seen things in the Slack channel and on the forum where people will be like, "Hey, I'm using umbrella and I don't know how to set XYZ up." It's just a little bit more friction for them. And I haven't needed to use them because the applications that have... I guess maybe it's because we're using microservices at SalesLoft for our Elixir apps. They're not huge domains that we're coding into an application. So maybe if it was a more monolithic application, the umbrella app would be a good way to split that up. So you still have one thing that you're deploying, but you're able to code it a little bit independently. It'd be something that'd be willing to try out on a project, but I don't think it would be the first thing that I would reach for, if I was building a new complex application, like let's say one that I intended to be a monolithic type repo. I don't think I'd reach for that umbrella right away, just because I would want to reduce friction and I haven't seen the need for it over the past three years. Justus Eapen: Yeah. This is really interesting thing to think about. I would love it if anyone listening would reach out and share with us their experience using umbrella apps. Steve Bussey: Yeah. I'd be curious to hear, especially if people have success stories, because I think it's probably... Doing something well takes time and dedication, it's easy to bow out and say, "Eh, that didn't really work out." I'd be curious about people that put in the time and dedication to make it work and they're like, "Oh yeah, this was actually a really good thing in the end." Eric Oestrich: All right. So one of the reasons we brought you on was because you have a new book out within the last month, I think, called Realtime Phoenix. So in the vein of that, when you start working on a new real time feature, where do you start? Is it the front end, the back end, figuring out channel stuff or what? Steve Bussey: I know there's definitely different schools of thoughts here. What I've always been drawn to, I don't know, it just feels more natural for me is to start thinking about the front end. I guess one way I think about it as I'm thinking about, "What's the user experience going to be? How does the user want to interact with this feature? And what are the different needs from that perspective?" And then I usually work my way from there. Sometimes I might jump around, I might figure out how I want the front end to be and then go to the backend and implement and then go back to the front end, especially if I'm just going to go straight into building the channel application. Sometimes I've done stuff where I'll actually mock out a front end. Like there is no backend server, I just have a fully operating front end. That's actually something that I actually really like to do when I'm building a new, more complex feature especially if it's in something React or whatever, so I can work out exactly how everything is going to tie together. I'll do just a completely fake, everything's in JavaScript, there is no backend. And then I'll figure out how I want it all to work and then get the user experience right. And then I'll go and implement the backend from there. Usually, you don't have too many restrictions and I think channels don't really give you that many restrictions either. It's not you're going to have this front end design that you worked out and then go need to dramatically change it because you're using channels or whatever, or it's going to be web-request based. So for me, I like to figure out what I view as the important thing, which is, what's the user going to be experiencing? Justus Eapen: And do you have a pre-code design process? Like even prior to building out that front end, do you have anything that... do you whiteboard, do you sketch out on a piece of paper? What's your pre-code process look like? Steve Bussey: Yeah. It definitely depends here. Like if I'm doing a project for myself versus a project for work, because at work we do have a process that we follow for making sure that other people have seen what you're going to build, at least like your general direction that you want to take, which we call feature development plan, so FTP. So we would put something that together, especially if it was a bigger feature, mainly the big thing there is you're looking for pitfalls, someone has some experience or some perspective that you don't have. The goal would be that they can help tell you about that before you go and experience it in production. So that's something that I would do at work. And if I'm working with a team, there's going to be whiteboard discussions and breaking everything up. It is a bit different if I was doing something for myself, I generally, when I start a project for myself, I'm doing one right now and I have to do list of like, "All right, here's everything I want to do." It's my task management, I guess. It's just a markdown file or whatever. And then I will usually flesh out some of the general data structures, that type of thing. But I like to get right into it, especially when I'm doing stuff for myself because it may be stuff that I just haven't done it before, so I'll do a more spike type stuff, like write some code, make it not pretty, it doesn't need to really work that well, I just need to flesh out the idea. And then I'll throw it out and actually make something better and copy paste stuff around and whatnot. So it depends. I personally to just get right into. And what I'll normally do... like I remember the first iOS app, for example, I ever wrote, I shipped it to the Apple Store and it was for myself, so it wasn't work related, and the app ended up, I think the folder was still called hello world in the final thing that I shipped to the store. Because I was starting to get into it and I fleshed stuff out and I was like, "Oh, I can actually see what I want to do here." And then I was just like, "All right, let's just do it and go." And by the time I wanted to rename it, it was going to be so much hassle redo it just because of the iOS world, that I was like, "Yeah, I'm just going to go ahead and ship it." Justus Eapen: What was the app? What did it do? Steve Bussey: It was called Noisy Dog Log. I took it off the store because no one ever bought it and I didn't want to pay $100 a year. I had an extra iPhone or iPad, and my dog... We have a Greyhound and I think they're generally susceptible to getting separation anxiety during the day, and so rather than watching the webcam and having it being an active process and rather than spending $250 on a new piece of hardware, like a nest cam, I just repurpose my iPad. It would sit there and listen all day based on the ambient noise, if it got above a certain threshold of that within a five second period, it would start recording and register a push event and then I would be able to get it from my other device and be able to see that, "Oh, hey, yeah, my dog is barking right now." I couldn't listen to it until he was done. He had barked for a long time before, so I couldn't actually tell until the end, but it was a cool little app and it worked really great for me. I didn't really market it or anything or try to push it. In hindsight, if I wanted people to actually buy it, I should have done that better, but it was cool. Justus Eapen: Very cool. We once, at my last job, we built an app that did that except with human beings who snore. So there's definitely a market for it. Steve Bussey: Yeah. Yes sir. Eric Oestrich: So when you're developing new real time stuff, what are some gotchas? What's the toughest part of developing real time stuff? Steve Bussey: So this is the great thing about Elixir and Phoenix and what they've done, developing it is probably the easier part of the process. I feel the documentation for things channels are really good. The front end client is really easy to work with, and so is the backend. So from that development perspective, I do think there's not major gotchas there. When I saw issues, it was in the actual rollout process and shipping it into production. An example of that would be the first real time app that I ever did, the one that spurred my desire to do something about it, and that do something ended up being, writing a book about it and doing some upstream contributions as well, was it probably took three to four weeks to build this app, and then the rollout process took six months because I would ship it out... And there's a blog post about all the different things I ran into on my own StevenBussey.com, there's, I think it's called Memory Isn't Free. It was about the memory issues I ran into. I ship out this app, and we're going to have to ship it out to tens of thousands of users, and I ship it out to 400 users and see that, "All right, the servers taking up a gig of memory." So I do a bit of extrapolation from that and figure out that like, "All right, I'm looking at 30 to 50 gigs of memory just to get the connections for this real time app all set up. And that process of undoing that and figuring out where I went wrong took a good bit of time and a good bit of load testing and whatnot. The end result of that had to do a lot with, if you return big payloads in a Phoenix channel response, you have to basically garbage collect both the channel and the WebSocket process itself because it actually does the serialization from the server to the client and that takes memory, you're doing something, it's not free. And that's not in the typical garbage collection path that Phoenix provides you, and so you have to actually send it a message to garbage collect. So it seems like, "All right, I need to make sure I'm triggering garbage collection at the right time." In that case, also, one thing I did was, if I'm going to have this big response, go to the server and then get a big payload and send it back, actually just spawning a process to do that, let the process do the request and send the response, and then when it's done making the request, the process goes away. So you don't have any memory of garbage there because the process goes away and it takes all of its memory with it. Versus if you do a big request or it could be a big database request or a big API request in the channel itself, you're hanging onto that memory now because there is latent memory for just how Erlang's garbage collector works. If you miss the major garbage collection cycle, you could end up with a fair bit of latent memory sitting around. And that was a really frustrating process for me because I had this app that I wanted to ship and I couldn't ship it because it was going to take up more resources than any other app in our cluster. Eric Oestrich: Can you talk a little bit about testing real time features? Because I think that's something that probably confuses a lot of people when they're getting started. Steve Bussey: Yeah. As usual, I look at two types of tests, real time features, you're going to have your typical unit tests, then you're also going to have... I guess there's three types. You're going to have the unit tests, maybe you're going to have automated integration tests, and then you're going to have that human integration test as well. So for us, we have a QA team that is doing manual testing of the application when there's new features to make sure that things are working as expected. So the Elixir based unit testing, there's good libraries for it, but it can be a little bit weird. I think testing in Elixir can get a little bit complex because you're dealing with processes. So you're not in one process and you're just going through tests, you're spawning off a different process for the actual channel. Maybe you have a global process somewhere, you have to worry about how those things are going to interact with the database. Maybe if you want to mock something, you have to worry about how that mock is going to get into the inner process. So it can be a little bit tricky, but there's pass-forward for all that. You can write really good unit tests for all of your real time features. And Phoenix provides that out of the box with a channel tests that comes with Phoenix, and LiveView as well now that LiveView is out. And coming into maturity, there's good testing for that and to make sure that your LiveView is working properly. And it benefits from channels because it's built on top of channel so you're getting a lot of that strong foundation there. The other really important thing that could be missed especially if it's a small team or a new project, would be human integration tests, I've actually going through it and trying to put weird situations in place. One of the ones that we ran into for instance was around, everything's connected, your laptop's open and everything, and then you shut the lid and then you walk away for an hour and you come back and open the lid back up. And we ran into some issues where it wouldn't reconnect. So that was one where it's like, "All right, let's figure out what's going on there." And unit testing or integration testing that, automated, is going to be just really hard to do. So there is that human aspect that's so useful. And then there were other things as well that would come up in the testing world that our QA team probably found like three, four different things that we hadn't thought about in the actual development, and are things that would be probably not come up too often, but we were able to find them and get them caught. I don't know them all off the top of my head, because it's been a little while, but it would be stuff around... And usually particularly to our application, how reconnections would work. We also have a Chrome extension, how would the Chrome extension work throughout this life cycle? Those sorts of things, they were really helpful. Eric Oestrich: You mentioned LiveView in that, do you use LiveView? What are your thoughts on this? Does LiveView replace 90% of the book that you wrote? Steve Bussey: That's a great question. One thing I had mentioned previously is that LiveView is built on the foundations of channels. So I do think there's this important aspect of understanding how channels work and understanding what you can do with them, because you can do the same things with LiveView. For example, if you're building a data pipeline, you need to move data around your system and eventually get it to the user's LiveView, that's going to be exactly the same between LiveView and channels because it's based on Phoenix PubSub. So you're going to need to know those exact things. Deployment, you're going to run into the same issues there because you're running with Websockets and LiveView is based on channels, so it's going to have that same story of deployment and the different things you should do there, potentially, maybe more sensitive on deployment because if your servers are down, it's not offline compatible. So if your servers are down, you would have issues. So you need to be even more cautious with deployment than you would be for channels. If your channels goes down and maybe it's just a thing that augments your application, the thing that augments has gone for a little bit, but your app's still there. So that is something that you'd have to be really careful of. And then I think LiveView is generally though great for the community, people are really excited about it. People that building applications with it seem to really. It doesn't fit our use cases at SalesLoft. And so we have used it on some internal apps for developer enablement type tooling, and that seemed to work really well for the team that did that, and I think that they enjoyed using it. For our main platform, I think the constraints of it wouldn't work for us in particular. And then also we have to do a lot of things over JSON, because I mentioned, we have the Chrome extension, we have mobile app. It's a lot of our stuff is JSON as the primary mechanism for communication. So that that's really important to us. The fun thing about LiveView is that you can turn an existing Phoenix space, static rendered page into a LiveView really easily. So the example there from the book is in the last chapter or the second to last chapter, I think it is, I'm starting to forget now, we actually take the project that we had worked on earlier in the book, which is channel space, and I did the different ways that you might use channels to get HTML to the front end. Honestly, it's not the most fun thing to do, the channel side of things, you have to do your own HTML parsing on the JavaScript side, or you had to do HTML rendering and the channel and parse it down. But then you go through and rewrite that into live view, and it's awesome because you just copy-paste the template and you copy paste it to a file with a different extension. And then you add in your live view and you set up your render and you set up your binding to the Phoenix PubSub, and in about 20 lines of code, the thing that we had built earlier in the book was able to be turned into a LiveView without all the JavaScript that had been associated to it, so we removed the JavaScript so we had less code, and we didn't have to go and rewrite anything, we just added the LiveView stuff, because we were able to utilize the same templates other than a change file name. So I really liked that, I thought it was awesome. It shocked me, I knew that it was going to be important part of the chapter to show people that it's easy to turn something into a LiveView, but when I actually did it and I was like, "That was easier than I thought it was going to be," it actually did surprise me and take me back. And I think I even had sent something to Bruce Tate at that time to be like, "Man, you're definitely right here." Because he's been talking about LiveView and how it's a game changer. And I was like, "I just did XYZ with it on the book and it was painless and it was way easier than I thought it was going to be." I'm excited about it, I don't know if I have a use case for it right now, maybe in the future I'll have a use case for it. I think people that are using it seem to really like it and great for the community, obviously. You're looking at Phoenix 1.5 and what they're doing with Live dashboard and getting LiveView, I assume close to that 1.0 mark I don't know any plans around that, but the community is really excited about it. Justus Eapen: I have a note here that you have a funny story about writing that chapter of the book? Steve Bussey: Oh yeah. It was just how easy it was to do it. The fact that I had expected it to be easy and then blew my own mind with how easy it was, and with taking it back. Justus Eapen: You're really good at explaining these things. It's been interesting to listen to you. When you're designing, how are you thinking about security inside of your architecture? How are you thinking about authentication, authorization? Where are you putting this? Are you putting in the context at the controller level? Steve Bussey: This is really important for me and for SalesLoft. And if I was to write a new application, this would be really important for me there as well. A lot of software, at least in the web world these days is SaaS based, where you basically have potentially multiple teams in the same application. You have a user that signs in, they're going to be different from another user and you need to make sure that you have these contexts set up. Well, when I say context, I don't mean Phoenix context, I mean, authentication and authorization context. So if you do it wrong, you're going to end up leaking data to someone that shouldn't see it, and leaking data is the number one worst thing that can happen to a web application, it's 10 times worse than having downtimes. No one's going to sue you over downtime. You get yourself into lawsuit territory when you start sharing people's data, especially if it's private. So I definitely care a lot about this. Luckily, one thing I mentioned is that at SalesLoft, a lot of our Elixir code is microservice based. Well, all of it is microservice-based, we have a ruby monolith that is the legacy of the past and we still add stuff to that, but we don't add a whole lot of new stuff to it. So that's where our authentication is born and lives, and we pass around database tokens for microservices, and we make sure that everything we do in a microservice has that context on it. So we want to know the user that's requesting it and the tenant that's requesting it, so we can do things like put that in our database queries, and if we have any other things that happen, make sure they have that contextual information on them so that we remove the chance of running into issues. One of the things I had worked on in the past, I don't know, it's probably been four months or so now, or maybe even five or six, was a way to enforce tenancy being set on Ecto clues. So essentially, you have to make your Ecto query with a certain column present. So in the case, let's just call it, Tenant_ID. You have to say, where Tenant_ID=1, or something that. And if you do joins, you have to make sure that your joints have tendency on the joins as well. And it basically goes into a whole bunch of edge cases that you have to handle that tendency properly set up. You can't do things say where Tenant_ID=1 or Tenant_ID=2, because now you could have two tenants worth of data coming back from the query. Once you get into administrative parts of code, you may need to do stuff like that, but you should never need to do that as part of your normal code path. So that's something I've been putting in Elixir projects that I work on now, if I had a multi-tenant application, I would add that in from day one to prevent any oh, moments where you accidentally realize that you've been making a query with the wrong Tenant ID or something like that and you just showed someone else's data. Eric Oestrich: So I guess if you have multi-tenant applications that might have a lot of data and whatnot, that you're potentially loading, how do you deal with any performance issues that you might encounter? And I guess if you talk about it in the book, specifically with WebSocket real-time stuff. Steve Bussey: Performance is also top of mind for me often. Well, it was probably about eight months ago or so now, or maybe it's even been a full year, I actually was like, "Hey, we need to dedicate time to just spending time on getting performance up." So I actually had a colleague and I, that literally went through everything that we could possibly find to increase performance and we were able to increase it fairly significantly in the application. There's some very basic things that people talk about. You get into things like, a lot of people have performance issues when it comes to their database. So that's obviously something to look at and making sure that you don't have n+1 queries, which Ecto, I love the fact that it's really hard to make an n+1 query in Ecto versus in Rails, where it's very easy to make... It's like n+1 is one is the default in Rails and an error is the default in Ecto, which is absolutely the right thing to happen. So there's that basic type stuff. One of the big things for me, at least in our APIs, because we do have a public API that anyone can hit, it's like any customer or partner can hit it, just as you probably can't unless you go and sign up. So people are hitting this and they're often doing things like, "I want to slurp all of my data down," which usually you don't need to do, but people get in their head that that's what they want to do and they want to do that every hour to check things. And it's like, yeah, there's probably better ways to do this, but I can't stop you, you're allowed to use the API. So one of the things that we do there is, when we're doing things pagination, we recommend not using page ID offsets because in Postgres that's really non-performance. So we use what I call cursor-based polling, where you say, "You know, my cursor is X, give me everything greater than that." And then you keep moving your cursor so that you're always at page one in your database, but you're shifting through the data in a windowing-type fashion. So that's one that significantly increases our performance on the API by doing that versus the typical pagination. Justus Eapen: How did you come up with that? Steve Bussey: I think it was natural for us, honestly. So we had pagination originally, just normal standard, and we still have pagination, and we were seeing the performance issues from it. So we said, "Well, what are the alternatives?" So in Postgres you can do things actual cursors in the database, but that requires transactions, which are incredibly expensive in practice. Transactions are the thing that will bring our app down, so we have to be very careful about opening them. So it's like, "All right, well, if that's not on the table, what's the other thing?" And it's like, "Well, essentially it's a cursor." A cursor is an okay string that you give back to the request every time you make it, and every time you make the request, you're going to get a new cursor back. And so we just implemented that in application land rather than in the database. And we allow you to do cursors based on IDs or maybe timestamps. Timestamp is definitely the best one to do because you can provably prevent yourself from missing any data, and if you have data updates going on and whatnot. So that's how that one came about. And we will still get hit with people that are doing pagination, but we also made it expensive. So when it comes to our rate limits and stuff like that, we were like boost the cost of your API queries, if you're using typical pagination. We do everything we can do to get you to not use pagination and we use the cursor base. One of the things that we do with our real time apps is we try to prevent ourselves from having to go to the database, so one of the things that we have is a website tracking mechanism. And rather than going to the database constantly to figure out who's on that and to send those updates down to know who's on your website, that are your prospects, instead we keep that in memory so that anytime that the channel is asked or it needs to be updated, everything's in memory, so there is very fast performing queries, especially with the amount of throughput that happens on that service, that's an important one. And then the other big performance thing that has popped up on our real-time apps in our case, a lot of real-time apps are getting data from a source, like let's say another API is pushing data up in a post-API. There may be doing something with the data, maybe saving it to the database, maybe enriching it further because they're actually doing something as a service and then passing it down to the client. So one thing is that it hit us in the past is using things like task.async to do something like, "Let's just pass that async fetch something from the database or task.async, hit this API." And it quickly came crashing down on us because we essentially had this unbounded parallelism in our application where things were hitting other APIs that 4,000 requests per second, when it should have been 4,000 requests per minute. It's like everything came in at once and just overwhelmed downstream systems. So the takeaway from that was to stop using tasks.async, and we had switched over to gen stage because it gives you a pooling mechanism out of the box. For me, I copy-paste some lines of code, it takes me five minutes to have a very basic memory data pipeline set up with gen stage. So that's what I've been using on those types of projects. And the big reason for that is that controlled parallelism. So instead of having 4,000 things reach out to an API, maybe you have 30 things that are able to reach out to that API at once. And honestly, the things happen so quick anyway that we don't see any issues from that. We're able to have really performance data pipelines still. And that is a chapter in the book. That's probably the one that I was most excited to actually write, which is about how to write a data pipeline, and I think it's called the Performance Pitfalls. It's looking at things that can really drag your application performance down. So having an unintentional data pipeline is one of those things. So if you're calling task.async to do things, I would say that's usually unintentional, you haven't really planned out what's the ramifications of doing that. Not to say there's not use cases for tasks.async, but in my experience, when I did that, it was because I wasn't thinking ahead and planning it out. Or if you're in a channel, a channel is a process. A process can handle one message at a time, so if you're sending a lot of messages to a channel and expecting them to respond back quickly to you, you either need to make things async by using a spawned processes that can respond back to the client, that's something that Phoenix channel supports out of the box. And I just had even read a form post today where someone was asking about, "How do I do this in Phoenix Channels because I'm getting delays because things are backing themselves up?" So Phoenix provides that out of the box. So that's another thing is making sure that you're not overwhelming your channels with expensive operations. Justus Eapen: Could you talk about some of the deployment considerations one might want to be aware of when going into production with a real-time application? Steve Bussey: Yeah. So that's a really great point around deployment. I think that we have it fairly good now, but there's still a lot of gotchas with it. So the biggest gotcha with deployment of real-time apps is that when your application is deploying, unless you're using hot code reloads, which I think most people in the community are probably not using hot code reloads today, unless they have a very particular use case for it, I'd say most people are using a more classic, bring it down and bring another one back up type thing. That's what I use. So if you bring it down so that it can come back up with a new code, your clients are disconnecting and then reconnecting. And it might only be half seconds that that takes but that's a point of downtime in your application and you can even get things where you come down, you come back up, you connect to a server that's also about to come down, so you come back down again, you come back up. If you had 20 servers in a rolling deploy, the chance that you're going to get multiple failing servers is very high. So that's something to consider when you think about uptime availability, especially with something like LiveView where your application actually, you want it to be as up as possible. So I would personally be more careful if I were deploying a LiveView application just for that, I would be thinking about, what's the ramifications of the server going down and whatnot? The other thing that can happen is you can end up overwhelming one of your servers really easily. So with a website good connection, you are connecting to a server and then you're just basically going to stay there until you disconnect, which could be hours. I've seen website connections last days, people just leave their desktop app and they have a page open and the website connections open. So load balancers are typically a little bit more passive where it will be when a request comes in, let me load balance it. And maybe even you can do things around the number of connections that are present, but at least the ones I've worked with and the ones that are on Amazon for all our application don't have that option, they're just route it in a round-robin fashion. So we would do a deployment and ended up with one or two servers that have 10 people connected and one or two servers that have 10,000 people connected. So you get a big disparity. And let's say that you were then getting alarms going off saying you have load issues, so you say, "Oh, let's add some new servers to the system," well, those new servers are going to come online with zero people connected. So you didn't actually help your load problem. One of the things that we do there is our servers are constantly pinging each other, asking how many connections do you have? How many connections do you have? And basically they aggregate that information together and the one that has the most connections, will look at the difference between themselves and the average. And if they're with outside of a certain threshold, they'll actually terminate a certain percentage of connections, and those connections will reconnect. So if you do a rolling deployment within, let's say 10 minutes in our application, every server is balanced within a five to 10% margin, whereas when we started deploying real-time apps, we would do a rolling deployment and it would be 10,000% margins between the first and last server. Eric Oestrich: You should definitely package that up and push it out the Hex. Steve Bussey: I have a GitHub repo for it. I actually was looking at this the other day because I was thinking about it and I was like, "How close is it to releasing?" It's usable right now, it is the code that we're using at SalesLoft, it's been running in production for over a year and a half now without any issue. So it probably could be released pretty quickly. The only thing that I was hesitant on because I had one at the last ElixirConf, what is now Dashbit sort of consulting package for a month. And this is one of the projects I took to Jose to say, “Hey, what are your thoughts on this and how might you approach this?" And he had a few different suggestions in the route that I went. So it was a little bit hesitant to release it onto Hex, just because it may not fit other people's use cases. But like I said, we've been using it for a good while now and it's worked really well. So that is something I might look at actually releasing. If you can get some users for it, then I'll be more motivated. Maybe if there's like a need for it, if people hear this and they're like, "I wish I had that," then hit me up and then I'll be motivated to open sources. What is open source, but I'd be willing to package it into a one dollar release. Justus Eapen: You heard him, folks, tween data's, which brings us to the end of the episode. I want to give you a chance to sell us on your book, although I think that this conversation probably has sold us on your book, but I'll give you more time to sell us on your book and also any other final plugs or asks for the audience that you want to... Anything you want them to do, find you on social media, where to find info on SalesLoft, or get involved with any open source projects you want to support. Steve Bussey: The book is Real-Time Phoenix by Pragmatic Bookshelf, it's released now, you can get a print copy on Amazon or e-book on Pragmatic Bookshelf website. I think if you're hesitant about it, I would suggest looking at some of the different customer reviews and seeing what people have thought about it. I love Amos King's quote about, "That the knowledge in this book is hard-one knowledge, it's going to help you make informed decisions about your project." Which is definitely true. I mentioned this project goes very frustrating to work on and get released. And basically, this book is the distillation of that frustration into a physical form of like, "Here's the book I wish I had before I started this project so that I wouldn't have been so frustrated and I wouldn't have spent six months getting it sent into production that should have taken two weeks." So yeah, definitely check that out. And let's say maybe you haven't done real-time apps or you haven't used Phoenix Channels, or maybe you could be listening to this and maybe you haven't really done that much Elixir development. So one of the great things about Pragmatics, all of their books styles are this way, and it's a style that I definitely adopted for this book, is that all of the examples in the book, you can go through and basically, the number one, you're getting all the code, but then as you go through the examples, everything is broken into the code snippets that are explained really well. And then you can just copy-paste those essentially into your own project. So let's say that you don't know Elixir that well, and you're reading the book, you could just type the code. Maybe you don't fully understand all the Elixir-isms yet, but as you go through the book and as you see more Elixir, things will start to click and you're not going to get hung up on the small things of like, "This project is just not working." Because you can just grab the code from one line and use that or you can just enter the code exactly as it is in the book. Justus Eapen: Got it. Eric, anything else we should talk about before we sign off? Eric Oestrich: I think we're good. Justus Eapen: Steve, thank you so much for being on the show. Steve Bussey: Thank you for having me. I enjoyed it Justus Eapen: And we'd love to have you back sometime. That's it for this episode of Elixir Wizards. Thank you once again to our guests, Steve Bussey, and my co-host, Eric Oestrich. Once again, I am Justus Eapen. Elixir Wizards is a SmartLogic podcast here at SmartLogic, we're always looking to take on new projects, building web apps in Elixir, Rails, React, infrastructure projects using Kubernetes and mobile apps using React Native. We'd love to hear from you if you have a project we could help you with. Don't forget to like and subscribe in your favorite podcast player. You can also find us on Instagram and Twitter and Facebook. So add us on all of those. You can find me personally @justuseapen. And you can find Eric @ericoestrich. And join us again next week for Elixir Wizards for more conversation about application and system architecture.