S15E12 Owning Your AI Stack with Zach Daniel
===

Charles (00:33)
Hey everyone, I'm Charles Suggs, staff engineer at Smart Logic.

Emma Whamond (00:37)
And I'm Emma Whamond, a Software developer at SmartLogic, and we're your host for season 15, episode 12. We're joined by Zach Daniel, creator of the Ash Framework and Igniter, VP of Engineering at Remedy Meds, an upcoming ElixirConf keynote speaker. In this episode, we're talking about AI agents, context layers, harness engineering, open source LLMs, and why engineering teams should be careful about outsourcing systems that hold their knowledge and decision making. Hey Zach.

Zach (01:10)
Thanks for having me. I'm glad to be back on and after you guys have such an amazing track record. It's just it's an honor to be back.

Charles (01:17)
Thanks. Yeah, we're glad you can make it. for any of our new listeners, since you've been on the podcast before, can you tell us a little bit about yourself, your background, and what you've been working on recently?

Zach (01:28)
Yeah, absolutely. So I've been doing Elixir since kind of the Stone Ages when it comes to Elixir itself. There's I think I've been doing it like twelve years now or something or thirteen years, and it's like what, a fifteen, sixteen year old programming language, I don't know. and yeah, so I've been, you know, active in the open source scene there for a long time. About six years or so ago, maybe longer now, I started Ash Framework, which is it's not like a web Framework for those who aren't sort of aware, it's more like an application core framework, how you might model your business logic, how you might model APIs and all these sorts of other things. typically complements Phoenix. So most people who are using Ash are using them together. and at this point, Ash has like, I don't know, 50 packages and extensions. you know, you get s a lot of stuff out of the box, too much to go over in an elevator pitch. And yeah, so for the last I guess year and a half or so, I've been working at Remedy Meds as a VP of engineering. so I oversee an engineering team that's 70 some odd people. and you know, we're building a really ambitious telehealth stack. and it's all with some very few exceptions, it's all one big Elixir Ash Framework monolith. we've created a lot of Elixir engineers in the process of all of this and and also, you know, it's been great to employ a lot of really amazing talent from the Elixir community. and yeah, that's that's kind of what I've been up to these days. And and of course I guess more topically, a lot of what I do at this company is driving sort of our AI stack and AI contracts and decision making and and how we go about augmenting our engineering team and also the rest of the business with AI tooling. So that's been the theme of my work life for a while.

Charles (03:07)
that sounds pretty interesting. last last time you're on the podcast, this was October twenty twenty four, we were talking about Igniter and code generation, project patching, but a lot has changed since then. So maybe you could tell us about some of the biggest shifts that you've witnessed from from your vantage point.

Zach (03:27)
Yeah, it's I mean, I think big shifts for sure, kind of monumental, in in all honesty. And I think most people can can see that, you know, at this point. it's just any any time anybody's doing engineering at this point, at least something is different thanks to LLMs. and you know, Igniter was it's it's actually really interesting because, you know, Chris McCord gave a a talk at at an ElixirConf a a couple of years back. It was called Code Generation is Dead, Long Live Code Generation, right? And his the the concept of it was that, well, why do we need like installers anymore? Like if we have LLMs who can just like make whatever it is that you want, why do we need not so much installers, like Phoenix.new, but it's more like the you know, generating a a view layer for you or a controller or whatever. Like why would you why would you ever need that, right? and I think that there is some truth to that. Right. And I we've seen that a lot in terms of like I I pilot a large, you know, open source ecosystem. and I see a lot of a lot of kinds of questions that nobody is asking anymore. Right. Like n nobody needs to ask a certain type of like, well, my syntax i is wrong in this thing, or I'm getting this like error message. Like they just ask Claude or whatever they happen to have, and it tells them, yeah, you typed that wrong, or you, you know, you you constructed that particular thing wrong. And so now like most of the questions we see. are more maybe architectural or bigger picture questions about what's the like I want to keep a track of all of the changes that happen in my app and I also want to be able to roll back to some point in time. Right. Everybody skips these like menial, trivial questions now at this point and they're all asking the big questions. And that's clearly like if you walk backwards from that an indication that like LLMs are raising the the floor of what what people can do, even when they aren't necessarily like don't have the same level of training or don't have the same level of skill. It's just nobody has problems with syntax anymore. Or at least not that they're gonna go ask for help on the internet for. And so I think that's been one of the biggest shifts from like just how I've seen it in the open source world. I don't think it's obviated the need for generators and installers. And in a lot of cases it's actually like I I tell my LLMs in in instruction files to use those generators as a starting point, right? Because the the question is like what is mix phx.gen.live or phx.gen controller? sure it's code, but it's actually like a statement about what a good controller looks like. It's like so at at the end of the day, it's like start with Chris's opinion about what a controller looks like, or start with my opinion about what a resource looks like, and then go iterate and do whatever LLM, you know, coding, agentic stuff you want to do. So I think there's still very much a place for it. but the I had these lofty goals for Igniter as like a general project rewriter. you don't need that anymore. Like there's that that's actually like it's great for make a starting point, make a solid base, get an example of a thing. and so I think we'll always have those or in especially installing packages. but I just don't think that I need to make it this like crazy fancy project patch or LLMs are just too good now to to waste my time on that sort of thing. so you know, it's definitely been a a big change. And of course, you know, we maybe a separate topic to how it's changed like professional, you know, enterprise level engineering. But in terms of Igniter and open source, that that's been the biggest changes that I've seen.

Emma Whamond (06:45)
So speaking of asking big questions and diving right in, would you be able to give our audience a quick explanation as to the difference between deterministic code generation LLM generated code?

Zach (07:00)
Yeah. I mean it's a big it is a big question. I think I'll actually start with some of the nuance in deterministic code generation. because a lot of people have maybe a simplistic view of what code generation is, right? They think like I'm gonna build some text up and then I'm gonna like save it to a file. and in in general, especially if you when you know if you look into Igniter and how it works, you can do extremely complex and nuanced things with real, you know. deterministic code generation, right? So what I mean by that is the way Igniter works is it takes that code that exists and it breaks it down into an abstract syntax tree. So it it turns it all into data. And then you can do things say like like say rewrite this function name to this other function name. the things that you might have like an LSP do for you in your editor. You might right click and rename function. And we can do really, really interesting and complex things that know about your project's context, right? Like It knows what a module has been aliased as. So it knows like if you say rename this module to that module, it can go fix it everywhere that it that it exists, right? so I think the the beginning thing here is deterministic. It's all scripted. Like you're writing code that modifies or generates other code. but it can be really, really nuanced and smart. It's not just sort of dumb put a text together, save a file. With LLM code generation, I mean how Quantifying what exactly that is is like a really tall order. and I think everybody has opinions on this sort of topic, right? Like personally, I see it very much as like an LLM is fundamentally like a, you know, it's reductive, but a next token predictor, right? It's like if I had a bunch of texts that started with please write me a module that does X, the LLM predicts what the text following that would be. And it happens to be the module that does X, right? So this at the end of the day, that really is what LLM code generation sort of actually is. The the interesting thing about it is you can extrapolate all of this really complex, you know, behavior and sort of emergent something that feels like intelligence behavior as a result of that, right? So LLMs can call tools, but they aren't making choices, they're just writing text. That gets parsed by a harness to call a tool. it's all it's all sort of quite complex. But at the end of the day, what this means is you can put some text into a box, you can put some context into a LLM token window here. And if you know, as these algorithms get better, you see that you get better results, more like what you wanted to get on the other end of it. and as you learn to use them over time, you get better, you understand context engineering. I'm gonna put all of this here, some examples and some of this stuff. And my next token predictor, which is freakily good, right? Like it's crazy that Autocomplete is able to do this, but it is, right? It's just we kinda have to all live with that now, right? will produce some usable thing that I want after the fact. And my two cents is that's not intelligence, that's not anything beyond like a a very, very useful algorithm. and and something we should all learn how to use well, right? So I'm I'm not sort of dunking on on AI. I try to call them LLMs, it's not AI. but yeah, so at the end of the day, you can use these things to generate code and and you'll get far more nuanced results. The A the AI, you could you can have it loop and you know, run your tests and try like all this sort of stuff, you're never gonna build. You're never gonna get that level of dynamism or emergent interesting behavior from a deterministic script. You just never, it's not possible, right? so they're two very fundamentally different paths, the different mechanisms of action, different properties on the other end of it. and I would use them in entirely different scenarios as well.

Emma Whamond (10:42)
so in your opinion, would the most efficient workflow use both these like probabilistic LLMs and the deterministic code tools? Like where does each one shine and what does the ideal workflow look like then when you're using both?

Zach (10:58)
I think i I mean exactly how you described it, like the if you're talking about something simple like, hey, build me a new page that does X, having your agents start with generators that build like a you know, your opinion of what a good page looks like or what a good controller or LiveView or you know, resource or whatever it is looks like is an excellent starting point. and it just sort of cements the like if you know about if you know, as you learn about context engineering, you you see that like the more you can provide it that puts it on the rails of what you want to build, the better, right? And so deterministic code generation can do that. The other interesting thing that I don't think is super explored a lot, right? is and I I've used this multiple times in my own, you know, in my own programming, which is if you look at something like Igniter that does this sort of complex code transformation, writing the code that does that is actually like pretty heady. It's it's not super simple to say, I'm gonna go transform some arbitrary code into some other code. You know, you have to think about a lot of edge cases. it's actually, I would say, really fun programming. It's the kind of programming that you should try to do just that like it's like puzzle solving. It's it's and it enlightens you about. the the code that you work in in the first place. So I think you should try it just for fun, you know, like advent of code style. Just do some challenges, right? but with that said, if you want an agent to make some sweeping change, you could say, go replace all instances of X with Y, or go rewrite all of my modules that do this to look like this. Or you could tell the agent to write code that will change, like deterministically make that change. across all of these files. And now you have something reusable, you have something testable, you have something introspectable, right? And that's I've used that regularly. I'll tell my agent to write an igniter script that that will patch my project in a very specific way. And it's it's really, really useful. So you can mix and match these patterns to great effect.

Charles (12:49)
This this interestingly dovetails with a conversation I was having just last night with an academic who has been working with machine learning and LLMs and tools like that for several years in an academic setting. And he was saying that what he's been doing lately is setting up the LLMs and the agents that he's using to only source their code. Like they've been trained on like the world of code, but to only source code from deterministic code that he's already written. That that's the library it can pull from, in order to kind of constrain its tendency to sometimes go willy-nilly to use a technical term.

Zach (13:31)
Yeah. It's it's interesting because I the the the the shape of that problem is interesting and th this is one of the things and a lot of this stuff I'll probably talk about today. I don't wanna spoil everything I'm gonna talk about on my keynote, but it's there's gonna be some some very you know heavy crossover. But one of the thing like i it gets to this thing of like LLM's most dangerous thing is that they they multiply. Well, one of the most dangerous things, they multiply what you already have, right? Like th they also can introduce other new things, but but by and large especially as you already have a large application surface area, they're just gonna like keep doing and remixing the things that they already see. And the more like that what that means is like how good your code is when you start, or how good your code is at any point is like the that's the coefficient of that equation, right? That that it's multiplying you know as you go along. And we've had, you know, we've seen everybody probably has this, especially if we build a large enough app. Some of the code's good, some of the code's bad, right? Usually the code that's bad is the code that was written under time pressure and duress, or it's the stuff that maybe you thought you might throw away later. You know, it's like the front end code for the for the like whatever, and you're like, whatever, it's fine. Just and so what we've done, and this is this is one of the the cooler things that that we have in place, which is it's called Project Dripfeed. So the way Project Dripfeed works, and I've I've this is comes from like something I've been instilling in a lot of my engineers over time, which is that good architecture, yes, it comes from this big upfront architectural thinking, but it also comes from just drip feeding small but important architectural changes over time, right? So what that means is like you can't just rewrite your whole app. Like I mean you could you can if you want to like stop the show for a couple years, but like nobody that's not reasonable, right? So you have to just drip feed good architectural decisions. And so what we did is we wrote and we're still iterating on this But we wrote general, unrelated to our app, guidelines on how we practice our craft as software engineers. Right. So this is for like we have Next.js, because we have some Next.js front ends, we have Elixir, Ash Framework, we have, you know, testing and all these different things, right? And what we do is we have a job that runs multiple times a day, that looks to see, that just reads that guide and goes and basically takes a pot shot. At some part of our code base that doesn't line up with our craft, like with what we've stated we believe in is how things should be done. doesn't really matter w if it finds the best thing or the right thing, it doesn't matter. And we have it open up to a limit. I think it's like two PR two pull requests per like area of the code that it's sort of diced up to that'll keep open at any given time with a label called drip feed, which just fixes that problem. And it's designed to make a small so it might go. find some duplication over here and do this. It might go find some tests that are kind of useless and or some flaky tests or all these different things, right? and it just opens a tiny little PR that's easy to review. and you it it's not making it should make no feature changes, no behavior changes. So it's really just a nice, you know. And I think we've already had like a couple hundred drip feed PRs come through that have been approved that just make our code 1% better or 0.1% better, right? And so in that way, we're combating any badness introduced by the LLMs, but also any like starting point that we had, you know, that that was making LLMs potentially worse over time. and so that's like it's interesting because you think of software engineering, especially when you did a big like leadership of a software engineering team, there's a lot more about competing forces, right? We want quality to be pushed upward. We want velocity to be pushed upward, but we have to like apply the right amount of pressure that, you know, that these dials get sort of tuned correctly. And so I think with LLMs, we have a whole other set of dials to turn, like LLM code review, LLM code generation, Claude MD files, all this sort of stuff, right? and drip feed I think is one of my favorite of those that that we've got going. And it's it's going very well as a project.

Charles (17:37)
I think software engineering is kind of a collection of trade offs.

Zach (17:42)
Yeah. Yep. A hundred percent. I think the the hardest thing I've found is how do you know that your team is making the trade-offs? Like I you know, with seventy engineers, I I can't review all of their code. I I read a lot of it. I really do. And I review as much of it as I c as I can. but how do you know that they're making the trade offs that you want them to make, right? There's no like test suite for for like, you know, architectural trade offs, right? until now. So that's the thing.

Charles (18:06)
Yeah.

Zach (18:08)
With Claude Review. Right? Like I I our Claude will, if you try to add a caching layer into our application, Claude will yell at you. We like we we took we said we don't like that. Like make it fast, don't add caching, right? But there was no test suite or linter that said like they could have caught somebody adding a caching interface, right? so I think that's also very interesting when it comes to like leading an engineering team, is you actually now have this like soft fuzzy. matcher that can sort of detect patterns you don't like and and push back on them.

Emma Whamond (18:43)
So seguing off of that conversation into feeding LLM's context, on Twitter you made a pretty bold statement about engineering teams outsourcing their context layer around their AI tools. You argued that since building custom tools is easier than ever, teams should be careful not to sell their organization's spinal column. could you expand on what you've meant? for our audience, what is the spinal column there in the context? And how do you think engineering teams need to rethink their workflows around AI?

Zach (19:14)
It's a it's a really good question. and I have a lot of opinions. I think the root thing that engineering teams need to do is understand that they are a bigger part of the organization than like feature builders, than request takers and and thing makers, right? and I tell this a lot to the engineering team is which is that we have to be the adults in the room, right? We we have to think more. Then just, yeah, I'll do what you asked me to do, or we'll make what you wanted me to make or whatever. and a lot of that comes down to simple things. Like, for instance, we in our monitoring tools, we monitor for like top-line business metrics, not just how our is our app up, but we monitor for like how well is our marketing funnel performing, like how well, you know, how what's our patient experience like? We monitor for those things. Because we have all the tools and the skills and and and the the setup to actually monitor those things. We should not just rely on external parties to come to us and tell us like there's been a business level problem that has emerged here that would never have showed up in our in our metrics, right? but that also comes down when it comes like when it comes to things like knowledge, right? Managing the knowledge of your organization, right? Knowledge management is a technology problem. It's not a people problem. Right. At least I mean, well, it's I guess it's sort of both. but it has a technology solution. and a lot of it involves restraint, right? So there's a lot of tools out there that you can buy that will any t anytime I see like a context layer for an agent that even Claude's like built in memory layer, I'm very, very wary of any of those things where it's just sort of like capturing stuff, putting it into a context lake and structuring it and linking it all I don't know. Honestly, you know pardon my French, but it's a bunch of bullshit, right? Like that we're gonna put in all of this that that somehow I made a knowledge graph and now everything's better, right? It it's I just don't buy it. I think it's just massively overcomplicated this the the space of what knowledge management actually is. and so for us, for example, our knowledge base, the one that we can there's plenty of people who write. Docs. I can't stop people from using Notion, right? But I don't care about any of those things. And I'm not gonna hook them up to any of our like AI stack. I like I'm I'm not gonna that. so we have a knowledge base that is like business knowledge that we've built up that lives in our GitHub repository, like in our docs folder. We have things about like how our brands, how our telehealth brands operate, what the clinical protocols are, all this sort of stuff, right? and The the biggest thing there is we get all of the stuff you want out of a knowledge management system kind of for free, right? One, you get revision history. You get to know who changed it. You get to know how out of date this doc this knowledge potentially is. these are all features like offered for these massive enterprise contract knowledge management systems that you also just get from Git, right? But the other thing that we do though is we have a this is one of the biggest ones. We have a reviewal and acceptance process, which is called a pull request, right? Which is if you want to change how we state some particular thing works, which is gonna feed into decisions that humans make, decisions that that LLMs make, context that things provide, like everything now. Knowledge feeds into everything now in a way that is just very hard to manage these days. if you want to do that, you it needs to be worded correctly, it needs to be in the right place, structured correctly, and you know it needs to make sense. And so we actually have an agent whose job is to read all of the Slack messages that happened in the last day, in like public channels, of course, not like it's not creepy, right? and it reads all of the projects and linear issues that were completed in the last 24 hours. And it looks at all of the code that was committed into the monorepo in the last 24 hours. And it makes a pull request to the documentation, updating it with something that might have changed. Right. So it'll it'll go in and say, actually like, you know, that's not how refills work for patients on this medication anymore. Like you you changed that, right? And so part of this now, we we have a new layer of defense against stale and old documentation that's beyond that goes beyond just like somebody full-time trying to keep docs up to date, which never works. I've never seen it work, never in in a history of the world has it ever worked. and yeah, so I think you have to own your knowledge layer, like your the that stack and that and it needs to be co-located. And I would never farm that out because It it impacts now everything. So everybody's Claude at the company is now hooked up to a MCP server that we built that that can only read the knowledge base that has been committed to the code base, right? So it can only source documentation that engineering has agreed, has accepted is like canon knowledge. Yes, that is how that works. and we're doing a lot of other things like that that I could you know, I could go into, but but that to me I think is just look, and it it's easy. Like it's not hard. You know what I mean? Like at the end of the day, like I built y you know, you know that meme like Iron Man, you know, Tony Stark built this in three days in a cave or whatever. It's like that's how everything is now with Claude, right? So this it takes me all the ten minutes to create a new agentic workflow because I invested like what two or three weeks to build a new harness, to build our own custom agentic so runtime. and so now I just am like, yo we call it Edenbot. Yo, Edenbot, set up a new thing. I want you to check every ten minutes for this. And like that that's all it takes. Right. So I just can't imagine buying something, especially not at the prices they're gonna charge you for these crazy tools, right? It's just no way.

Charles (24:42)
So LLMs, we know about hallucination and and and that as a a a thing that they can do. And if they're writing your documentation, how are you ensuring that that documentation is indeed accurate to the current state of the code? And of course this is a problem with humans too. We talked about you know how challenging it is and has been for years to keep documentation up to date with changing software. but how do you do that in an LLM context?

Zach (25:07)
I mean, yeah, that the the key is just the human review step, right? So an LLM never changes documentation autonomously. It goes through the same code review process that any other thing goes through, which means at least two other engineers have to agree. Like they have to vet that that like, is that true? I don't know. And like so they read it. And half the time it's me. Like I I, you know, I get notified about these, I go look through them and I'm like, Yes, that is true. I know that that has changed about our business. and so at like at that point, like if if two human beings, you know, two checkers, I say, like, yep. That's that's true. Then I think we're we're pretty much good to go, right? and I think that's kind of the that's always going to be the layer of defense. Maybe it's not always humans checking. I don't really know. but for me like I can't imagine letting the agent like letting the agent modify its own context fully autonomously. I don't think I would ever do that. There would always be human review because it they just aren't that good, right? Like they're amazing. Next generation miracle tools, but they're not actually that good at that type of thing. At managing their own context, they don't really know how to do that. They don't know the whole surface area of everything and every place that this is used and what it feeds into and what all of their options were for where they put it. It just they don't know those things. But yeah, human review, that's that's the answer.

Charles (26:18)
So you definitely don't ascribe to the philosophy that we should just stop reading the code.

Zach (26:23)
hell no. No, no, no, no. Not at all. It's okay, it is complicated, right? So let me here's I'll tell you about another project that we that we built to help to help with this because engineering is going way faster. We have like every project is a one person project now. It's like one person plus claude. It doesn't matter how ambitious it is. Everybody is like doing crazy things and we're moving so fast, you know. I think we have like fifty pull requests merged a day on like a light day, you know what I and so we do have this problem. This problem that we are generating, you know, code faster than we can reasonably review it. now of course good.

Charles (26:57)
Sorry, how many people so h yeah, how many staff, how many engineers are generating like fifty PRs a day? Just to give us some kinda context to the

Zach (27:04)
seven so seventy people ish. Fifty PRs a day I would say is excluding like fifty big PRs. We actually probably have like a hundred and fifty PRs a day that are like tiny it's hard to like a bunch of like little things, but fifty PRs that were like people like really need to like dig in and you know, part of reviews. so yeah, like fifty a day, maybe each person op I you know, actually I'm kind of I'm kind of doubting this now that I say this. I wonder how many That might be an old metric. I'll I'll have to I'll have to check. But last I checked a few months ago was 50 a day. So yeah, and the what we did to help solve some of this problem is we have our own code reviewer that's basically it's all it's all just Claude under the hood, you know. I I could talk about the implementation of our agent. It's this one agent that I built that can do a lot of different things. but the code reviewer agent has a final task after reviewing the code and making comments. It also it's all its job is also to rate three things on the PR. so the numbers it gives you are complexity, one out of ten, deployment risk, one out of ten, and alignment with the original linear, like we use linear, it's like Jira, right? So with the lit original linear issue slash project, right? And if your pull request, according to the agent, it's a ten on alignment, like yes, this does exactly what Project said it was supposed to do a one on complexity and a one on deployment risk. Then we add a label to the pull request called Fast Track. And what Fast Track does is it lowers the amount of human reviewers. We still have two human reviewers required, which most people would call like pathological in the age of LLMs, but we we still do. So it goes down to one person as a required reviewer and it enables the QA step. We have a product QA that that does PR view and it it triggers an agent to do the actual QA step. So it spins up a browser, it clicks through, it goes through all the stuff and validates that you didn't break any other thing. And so all of that together means a lot of our simple PRs are just like you just need one sanity check from a human. You don't need like this whole sort of hullabaloo and you don't need QA and you don't need all these other things. So I think there's a world where you can safely identify the code that is low risk, not dangerous. I in any high volume work, you've always had to learn what code matters and what doesn't matter and apply more scrutiny to the code that matters more. And less scrutiny to the code that doesn't matter, right? so we're just trying to formalize that and create kind of different lanes for things to go through when they meet that criteria. And there is a world, honestly, where that code requires no review at all, where fast track is like, now the original engineer needs to read that code, right? We are not vibe coders, but but at least no other human necessarily needs to read that code.

Emma Whamond (29:43)
So is what you're describing harness engineering? Is that is that the correct term?

Zach (29:50)
And I think the the difference I I don't really know what the you know, what harness engineering is in the zeitgeist to today. I think they're mostly talking about like building your own little Claude type thing or like Pi or you know, codex like like that type of harness. For me, I'm much more interested in like our harness is actually like runs in GitHub actions, you know what I mean? Like and everybody has Claude code locally. Just use that. If you're if I don't engineers who are writing features, just use Claude. I don't you know, we we ha pay a lot of money for our Claude subscription, You know, so and so but the interesting bit about our harness is it's like a general purpose multi connection point agent that can be triggered in a lot of different ways. including like you can assign a linear issue to the agent and it will just do it. It'll like open a pull request to do the thing. And you can assign a project to the agent and it will scope out the project into linear issues, it will sequence them in the order that they need to be done. And then when you move it to in development, it will start on all of the issues that have no dependencies. So it will like i it can end to end an entire project. and so that's the harness that I'm more interested in, right? Is that is not the I'm never gonna win against Claude code and I don't care to. It works great for what it does. But for the our business's general purpose AI and agentic stack, that is the harness that I'm like focused on primarily and like People add it from Slack. Everybody's using this thing everywhere, all the time. and so it needs to be good.

Emma Whamond (31:16)
So is owning your AI stack, we keep hearing this term, is it mostly about cost, like being able to use the agents more efficiently, or is it about control or a combination of both? What what's your opinion on that?

Zach (31:32)
good question. So for cost, I mean cost matters, right? Like our company, we we do care about cost, but in my very humble opinion, the LLM tokens are still worth their weight in gold, you know, relative to their cost, right? So like somebody we were talking about when you know Fable came out and one of our one of my team members was asking like is it it it costs a hundred percent more money to use Fable. Is it one hundred percent more better than you know, Opus four point eight? My answer is no, I think it's twenty percent better. And he said, Well then why would you ever upgrade? And I'm like, Well that that's a really good twenty percent. You know what I mean? Like like and and we we came to the realization like what that meant is that Opus four point eight is underpriced for the value that it gives, right? Relative to if you would take a like a nonlinear gain over a or you know, like a like a lower gain for an equivalent cost and price, then that means that It was worth more than you were paying, right? and I still think that that's that's true. So cost for me is I'm not super worried about it, really, honestly. and like we found that we don't have to hire engineers to repl like we've you know, we've had engineers who have moved on. We haven't had to hire to replace them, right? We have we're not we do like an AI layoff type thing, but we are not like having to aggressively grow our team, right? And what that means is like I mean, engineers are really expensive, right? So like i it just in general, I think it it it pays for itself. and for me, the two things are control and security. Like the the the original question, you know, about why you wouldn't farm this out, right? It takes a lot of effort to craft a good experience for people who are using your AI stack. And I'm not gonna buy 12 different tools. Like I hate this. Every tool has its own little chat with your, you know, and now like in those, you can also add your other products as tools in that. I I just no, I'm not gonna do that. It's you know for a lot of different reasons. But it's just not usable. It's not really usable for as like a person at our company to say, like which of my tools has connections to which of my other tools and at what privilege level? And how do they sign in? And does it know it's me? And does it have our context and all this sort of crap? It's just not reasonable. but then there's the security aspect, right? Like making a like an an AI based mesh network of all services that your company subscribes to, how are you gonna lock that down realistically, right? Like like it's just can't. You c you cannot. And a lot of people here ask me all the time, like, there's a lot of things that Edenbot, our agent, cannot do. And I have to explain to them that any agent has to conform to the lowest security level of any interface that it ever goes through or has access to, right? So because it can like talk in Slack in our general channel, it can never get access to anything that is not available to everybody in the general channel. Because there's nothing that stops it from t daisy chaining information and putting it somewhere you didn't want it to put it, right? and I have a lot of ideas about how I'm gonna solve that problem in general. but right for now, like we're s I'm not as focused on it. There's more value in just building out our core, like every person accessible agent. But but yeah, I would say that for me, control, crafting a good experience, something that people can really use, and then security is is huge.

Charles (34:48)
I keep feeling echoes of Battlestar Galactica in these conversations and these agents that as they're networked, the what they're able to do and the damage they can cause when allowed to to run amok. how should teams be thinking about, you know, we've kind of d dug into this some, but how should teams be thinking about privacy? Portability, maintainability and vendor lock in, as These LLMs become a much more significant part of the engineering flow. if a if a team's workflow lives inside Cursor or Claude, what happens when they want to change vendors or they decide to sunset the tool? Or you how do you guard against the the risks of that?

Zach (35:31)
It's a it's a good question. I mean, I think owning the stack as much as you can is one of the biggest you know, protections against that risk. but I also think it's sort of like a it's just inherent to the business model of LLMs in general, that they are like self-commoditizing, right? w you can hook up so let's say for example, Claude raises their prices, Anthropic raises their model prices by 10x, and we're like, well That's good, you we we can't afford that anymore. there are LLM routers out there that will let you keep running Claude's like front-end infrastructure, Claude code and all this other stuff, but just route to some cheaper model, right? and and be because like at the end of the day, like what they're selling is something that you meter, the way you meter like electricity or water, right? And it's like at some point, yeah, maybe some water tastes better than other water, but like you you can't like own the ability to. give me water in the first place, right? Like so I think it it's just a I I'm not too worried about that in all honesty, like happening. and I think like even for Claude, like e all of our stuff, if I wanted to switch to OpenAI, for example, I would rename a bunch of files to agents.md and then I would write like rewrite a couple like some of our agent harness stuff that uses the Claude the Claude TypeScript SDK to use like a there's d a different SDK. And like funny enough, I have an AI that can do that really quickly. And like I don't really care how well it's implemented because it's like not on a critical customer path, right? It just needs to work. and so I think like I just not really afraid of of the the the lock-in from a model perspective, from an LLM perspective. but from a data storage pipelining perspective, very much I don't want it to touch any other thing. I don't want other stacks having any of this stuff. I want to go as close to directly to the model as I possibly can. then I'm insulated. Now I can just change models, change providers and and not really worry about it too much. but on this topic, you know, I think open source models are actually so I don't like ever use open source models. Not not because I like have anything against them, but because they are materially behind the s the state-of-the-art models. And like I mentioned before, I would pay a premium for. even a marginal improvement in how in my entire engineering team's output or our entire code review process or all of these different things that But open source LLMs are the reason why LLMs are never going to do a 10x price jump. Right? Like not realistic, they're never gonna be not going to able to, right? Because there exists a distance between open source models and SOTA models. And if that if this distance gets too high from a price perspective and you know, but is narrow enough from a quality perspective, then at some point I'm gonna say, well, this is too expensive. I'm gonna go invest my time and money in setting up my own stack to run these open source LLMs, and I'm gonna accept that they're ten percent worse because the the now the distance is greater, right? And they continuously are are like right on the heels, you know what I mean, of of these state of the art model providers. So at the end of the day, like I think like while I appreciate very much what open source models are doing, I'm not using them because they're not quite as good as the state of the art, but they are absolutely like necessary as part of our ecosystem to continue drive like to keep prices reasonable for our state of the art model providers.

Emma Whamond (38:46)
So changing gears a little bit, I just wanted to bring up something that the audience might be interested that you wrote. You also wrote that AI has erased the illusion that junior engineers were ever just a cheaper version of senior engineers. Their real job was always to learn, grow, and eventually become an asset to the team. How are you thinking about the role of junior engineers right now?

Zach (39:08)
Yeah, it's a so really good question. And I think, you know, I've I've had this general opinion for ever now, for years, you know, that that junior engineers are an investment, not a asset. Right? They're like they're they're different. There's something you intend to grow and build that will eventually be worth more to your company. But the the sad flip side of that is that most companies never really wanted to invest in junior engineers. Like that wasn't what they were attempting to do when they hired junior engineers. They were attempting to have a sort of cost spread in in and skill level spread, whereby like, well, if I have three juniors, maybe they're kind of worth as much as like one senior, and we have like, then we only need one senior instead of like two seniors. It's it's that sort of like mental gymnastics that people were doing the whole time. but now that it doesn't really eliminate the need or the desire for your company to have and invest in junior talent. But now it's clearly just an investment. Because if all you needed was basic grunt work to be done for a year or two while they level up, you just use Claude, right? Like it does, it does that stuff just fine. and I think the problem is that like I think it's going to force companies to build things like apprenticeship programs and to build and to actually try to like if as the engineering talent sort of ages out of the system and you don't have any like human beings that actually understand how these things work anymore. This is a bit hyperbolic, but let's just say that happens and that goes that way. you're gonna have to start training them the way that like trades train individuals, right? Where we we intentionally invest in apprentices and in you know interns and all this sort of stuff. the problem is then you're gonna have to figure out how do you incentivize them to stay, right? Because Companies have the hardest time ever paying people what they're worth when their value rises astronomically as you train them. And that's what happens with engineers, right? A a junior engineer is worth twice as much after one year of being a junior engineer as they were at the at that's my opinion. In general, a good junior engineer doubles their value, their market value, what somebody else would pay for them after just one year on the job. Now take that to any finance person and say, hey, I want to double this junior engineer's salary, because if we don't, somebody's gonna come scoop them up. And like, no, that th that the conversation almost never goes the way that you want it to go. And so your junior engineers leave, right? so it's a bit of a catch twenty-two. And I don't know what the solution is personally, but I do think that that AI makes the problem much bigger. Like it it it it kind of lays bare that we do in fact have this problem. and I I've been thinking about how we can solve it at our company. Like we're a very senior heavy company. We don't really do junior engineers. We never have but I've been considering like you know Maybe we need like a like a proper apprenticeship program that's like scoped out for a multi-year growth trajectory, right? Like a lot of companies have done this in the past and still do this now. and I think that there's a world where we need to do something like that ourselves as well. Like that this is a three or four year program to bring in apprentices and and in you know, start them small and build them up and that sort of thing.

Emma Whamond (41:58)
like that. it's not something that we hear too often, giving back in some way. we were talking with Bruce Tate in an earlier episode and he was talking about how AI productivity gains is a dividend that teams can then reinvest in people. and that's kind of how I feel at at this level. It's okay.

Zach (42:17)
It sounds it sounds wonderful and you know, I think we all the the subtext there is that most dividends are not returned back to the workforce. Like that's not how it's typically works in at least, you know, most businesses I've ever worked at. But I do think, you know, it's an interesting question, like where where it all sort of where it all sort of leads to. And I I would like to champion programs like that as much as I can. especially when, you know, you can articulate a long term gain to your company, as as a sort of, you know, tit for tat type thing. but yeah, I don't I don't know if enough people are gonna do that for it to have the i you know, the impact that we want on the the programming programming world and ecosystem. I guess it may be a little doom and gloomy, but you know, we'll see.

Charles (42:57)
hi history in in in capitalist economies does tend to show that the gains from those efficiency improvements tend to go up as opposed to out or down or all directions.

Zach (43:11)
Yeah, I you know, okay, while we're talking about things I've tweeted, I just recently was talking about this on on Twitter where it's like there's a mentality that a lot of people have, and I think it dovetails into this conversation where they'll say things like, you gotta catch up. Like you're you if you're not using and getting good with AI now, then you're gonna get left behind because the models are gonna get way better next year or the year after. They're on an exponential curve of intelligence. And I'm like, to me that I think that's one of honestly, like it's a very silly Or it's it's a very surface level analysis of what's actually gonna happen, right? and so here's my two cents of this is you can't leave me behind if two years from now we're gonna have a genie in a box that can summon up whatever thing I could possibly want or tell me how to do anything I could possibly wanna do. Like we all have the I can pay twenty dollars a month for Claude just the same way as you can. You know what mean? Like it's not like it's like exclusive access to you. And so the the reason I say this is like what There is an equalizing factor here. Not necessarily equalizing everybody on the playing field, right? Like the people who own NVIDIA and Anthropic and OpenAI, obviously gonna be, you know, pretty far ahead versus the the everyday person, right? But for, you know, I don't know, middle class, lower class, whatever, right? we all just sign up to these services the same way we all sign up to Netflix and get access to intelligence on demand, right? I'm not saying this equalizes again all of society, but to me and some random person on Twitter, i if you're like using AI better now than I am, you're not gonna be ahead of the game because I'm gonna get the same thing. I'll just use the same magic box to get whatever it is that I don't know yet and I'll know it then. So I don't see what the problem.

Charles (44:45)
It's not like you missed the train when it left the station 'cause there's multiple trains, it doesn't matter. You're gonna get on a train and you kinda get on where the train is at as opposed to the train being where you're at in a way.

Zach (44:59)
Yeah, it's a it's a magical train that picks you up from your house. Like, so it's just I think yeah, and I I guess my my point to that is when we talk about what's gonna happen to the industry, to people's livelihoods, right?

Charles (45:01)
Yeah.

Zach (45:11)
The the other thing that we see as a tangent to all of this, or or or as a complement to all of this, is how many people are building businesses and building projects, right? And and building things that actually, even if they're not gonna make them money, are meaningfully making their lives better, right? So people are building apps that they use to track, you know, their health. They're building tools that they use to stay in touch with their family. They're building websites for their small businesses that are much better than they ever could have had before, right? With booking interfaces and all this crazy stuff that they never were gonna spend 20 grand on to get like contractors in to do. Right. And so all of these like that's also happening, right? And and I'm not I don't know if it like really sort of offsets things, but I do think there's a world where like a lot of people who are programmers actually end up being Technical Technical versions of X other thing. You know what I mean? Like so you start an arcade, but your arcade is like super cool and awesome because you have AI and you're really f you you're really good at using it and you know how to build like a great website and a great, you know, tech thing that hooks up all your games together so you can track high scores. I don't know, just making stuff up, right? so the it's just I'm not so sure that that yeah, maybe a programming as a profession shrinks, but programming as a thing you do to make things happen. is increasing and more accessible in the in a wider audience, right? So I think we may be in a world where that has a meaningful like uplift as opposed to you know, there's less dedicated programmers, but there are more ways you can use programming to make your life better or to make money or to build businesses, et cetera.

Charles (46:41)
so we've covered a bit of kind of the some of the the benefits that we're we're seeing from these tools. but if we could take a moment to talk about some of the cost, you know, so using using LLMs that it it could be the cost can be tokens at the individual or the organization level, or it could be energy, water, land use, or societal impact at the aggregate level. you know, one one prompt isn't gonna use up all the water in Utah, but maybe a lot of people hitting a large data center in Utah will have a significant negative impact on water in a dry state. So how do you balance these costs, both the, you know, kind of the token costs you've talked a bit about and these larger costs when you when you're thinking about what to spend these resources on?

Zach (47:29)
It's a really good question. You know, it it it it gets into I think what I don't necessarily consider this a political conversation, but I think there are a lot of people who might consider like the economic impacts of AI like a a political or or partisan conversation. I'm not sure. But w what I will say is I respect everybody's opinions, but but my personal opinion is that for lack of a better word, like it's fine. Right. And and I'll explain. New demands and new technology. It says like it's we'd we've never reached the pinnacle where we stop thinking about solving whatever problems are brought up or created by new technology. We like keep going. And in general, I consider myself a technologist and I've seen that technology is a mostly one-way street, right? Like we bec because of the tragedy of the commons, like pe people in gener somebody's gonna keep using this thing. You can't stop them. Like not realistically, right? So instead You go the only way out is through. You go through the you you you build, you make safer, you you learn, you expand, you add guardrails, whatever it is, right? and in the same way that like what's a good example here? you know, TVs, we maxed out the resolution that a TV that you could ever possibly want on a TV. It was like 12k or something, 8k, right? At some point it's like, okay, who cares? Like right, Then they came out with the Apple Vision Pro or the MetaQuest VR or whatever, right? And now that you put the screen, you know, one and a half inches away from your face, all of a sudden, new demand for higher resolution screens came into play, right? And I think that it's not the best example, but we have the same sort of effect with LLMs, which is like at the end of the day, like, okay, we're getting this other benefit. And now we have new demands on power grids, we have new demands on how we can cool a data center or make them quieter or make them whatever, right? and I just don't think that there's a world like if we just shut down all the LLMs, they're gonna want data centers for something else. You know what I mean? Like like it's it's not like we get to s to not solve the problem of large scale compute. We're always gonna get to that point. And so let's use all the cool intelligence that we've built, all the let's use LLMs to like help us solve those problems, you know? I also like don't I'm not the scientist that would look at those things, but as far like I don't think that they're really gonna use up all of our water. Like I don't think that we're all just gonna like r run out of w I don't I don't know. So I I definitely feel like sympathy for communities that are affected by that. And I don't think you should be allowed to like negatively impact somebody's like home and the place that they live. You should have to figure it something else out. And that's something that our government should help people do. but just in terms of the resource usage, it's like I think it's an inevitability to some degree, that we need to solve those problems. And a forcing function is always like part of how those problems get solved.

Emma Whamond (50:23)
Transitioning away from that, I just wanted to touch on something else. Since we are Elixir wizards, there's conversation in our ecosystem that Elixir has a real opportunity in this AI moment, specifically because of what the beam offers: supervision, fault tolerance, concurrent processes, hot code reloading, all the fun stuff. Where do you think the Elixir ecosystem has an opportunity to lead in AI tooling? What's your opinion about that?

Zach (50:54)
I mean, yeah, I a hundred percent agree. Elixir is for a lot of different reasons. Ignoring you know, we've had like there were like studies and stuff that that showed that like Elixir the you know produced better Elixir code. To me that that's like a surface actual evaluation about like how accurately it wrote the code we wanted it to write, for example. It's not really the same thing. That's not the part that I care about. for me, very simply, like there's one major thing. LLMs, when they work on your code base, have a narrow view. They have some predefined context that they were given. And then they have like they they don't just load up your whole code base. That's never how it works. And the bigger your code base gets, the l less likely that is. And so the number one thing in my experience that dictates how well, how good results you're going to get is what is the blast radius of any one given change, right? How likely is it to break some other totally unrelated thing very far away? and In a lot of cases you can architecturally solve that problem by like doing this like crazy, like a huge amount of like decoupling and you know creating microservices. I don't like any of that. Like that to me like just increases then the cost of understanding and so but Elixir has this natural property, right? Which is that everything runs inside of a ring fenced process. And in general, if you change something in one place, you break one place, right? You you don't typically break your whole world. of an a you know you you could but you know your basic test suite covers that. It's it's it you know there's just so much less s I call it like spooky action at a distance, right? So you think about like building like a Rails and a Ruby app. You go and you like you access a property three deep and you don't realize that like one of those was like a like a mutating method that you accidentally called that changed the state of this object in some I don't know thread over there. You just don't have those same problems. And I think that that's the biggest win that we get. with the ex I mean cost also. I mean Eli Elixir and Erlang are so cheap to run, even at like h huge scale. and they're just yeah, they're safe. The agent can't really break things the way that it could in other places. so yeah, I think by like a wide margin, I trust LLM generated Elixir and Erlang much, much, much, much more than and we have we have other languages that we generate and it's categorically worse.

Charles (53:04)
Is there a way that we should be positioning ourselves as a community within this kind of larger shift to know I keep hearing talk about how, you know, we just need another big company to adopt Elixir and to publish some open source libraries or whatever, and then you know, the uptake of Elixir use across the industry will really take off. but maybe maybe the world of LLMs is that. inflection point for for the community.

Zach (53:31)
It it could be I you know, I I've spent a lot of my career thinking about Elixir adoption, right? From a but just for in terms of like my open source life, you know, building a framework for Elixir. but also even like from my like financial success. Like I I want to have good job o good job opportunities moving forward. a lot of people I remember when I was first talk started talking about Elixir adoption openly, you know, people would often chime in with like a I don't really want To get big adoption. Like I I like our little community. And I like, that's cool, but I have a lot of people I care about and I want them to be able to feed their families, right? We have a lot of people who are using Elixir. I want them to have a lot of options. I want them to be highly paid. I want them to be highly sought after, right? So I think that it's a better, more responsible thing for us to do to try to get more adoption, to keep our community well fed and flexible and you know empowered, all those sorts of things. I don't know if that's reasonable anymore. Not And what I mean by that is I don't know if I'm not saying that Elixir adoption isn't gonna go up, but the it's such a saturated thing. It's not we're not gonna have another Rails moment, I think, ever. Like where where Rails sort of ate the world and and and because the thing is like the difference between Rails and what you used to have was this wide chasm, right? The difference between Elixir and all of the other options is frankly not as big. Like it is still better, still much better, but you're still gonna be able to build your enterprise app with Next.js or Rails or Laravel or whatever. It's not you're not really, especially with AI, like you're not really gonna fail. It's just gonna be worse, right? And worse doesn't kill businesses, right? Like worse technology is not on its own enough to kill a business. So what I think now is that it's more of a slow game. It's more about, you know, what we can bring to any business. And and it's it's much easier to sell a new language than it ever was before because you don't need to know a language to program in it anymore, And I think it's gonna be more of that, like a slow expansion. and I do think I I told people this. People ask me, what can I do to increase the adoption of of like Ash Framework, for example? This is a common thing. Like how do I get know, how can I help? Right. And I the same thing I tell everybody, get stupid, filthy rich and then just tell people how you did it. Like you don't have to you don't have to publish open source libraries or anything, but just just you know, make a a billion dollars and then just let people know that you did it by, you know, you built a really great Elixir application. And or you built your business off the top of Ash Framework, whatever it is, right? So I still think that's the thing that people need to do. They just need to be successful because they chose Elixir. and that is the only real thing I think that that really works nowadays. Cause you know, I'll expand on this one more thing, which is software engineering at this point is mostly identity politics. I don't most people do not make language choices purely on technical basis anymore. Because let's say you're at a you a you're at you've been ten years writing JavaScript, and then your company's like, we're thinking about going with Elixir, and you're like, well, do I wanna be the guy with 10 years experience in the programming language that we that we use, or do I wanna be the newbie again? Like, is it no, of course you don't, right? So you're you're trying to like protect like this isn't just a meritocratic technology or technocratic ecosystem. This is people's livelihoods, right? This is how they feed their families, this is how They live their lives, all of these sorts of things, right? So of course you're gonna have headwinds when you try to increase adoption. If it was just a technical conversation, Elixir would be like would have eaten the world at this point. but but it's not. And so we have to deal with that. It's like different problems, different solutions, at the end of the day.

Charles (56:48)
Yeah. startup founders hearing that, well you'll never find an Elixir developer, so you should use JavaScript. Or I I think it's also a little bit of that.

Zach (57:06)
There there is a little bit of that, yeah. I I hope that qualm at least gets erased because again, like we can teach an elixir we can teach somebody elixir in two weeks now. We can teach somebody elixir plus ash in three or four. Like it's just trivial to to learn a new language and and to to be effective even while you're learning, right? so I hope that LLM's at least squash that particular complaint.

Charles (57:15)
Yeah. Mm-hmm. Well, Zach, we are almost out of time, unfortunately. I think we could keep going for a while. you're I think a keynote at ElixirConf coming up, yes, in in September.

Zach (57:38)
Yep. Yeah, I get it's my my first keynote at an ElixirConf, so I'm I'm excited about that. I'll be doing the closing keynote and talking about a lot of the AI stuff that we sort of touched on today. More practical, a lot of things like project drip feed, stuff you can use to like really level up an engineering organization.

Charles (57:55)
Great. looking forward to that. We'll we'll also be there. I think we have a a discount code in the show notes, so check that out. is there anything else that you'd like to share with our audience or anywhere listeners can follow your work and learn more about what you're building?

Zach (58:10)
Yeah, so I mean the big thing is after Goatmire we're doing an AshConf we'll put a link I think maybe to the tickets in in the show notes if if we can. And then you can also see it on like my Twitter. I've I've posted about it, that sort of thing. and also on the Goatmire website. So it's kind of a in collaboration with with Lars doing Goatmire. and it's gonna be a full day. We'll have talks from the core team, from the community, from from business leaders who are using Ash. it's gonna be a lot of fun. So, you know, definitely join us if you can, especially if you're gonna be in Goatmire. Like you went all that way to to to Goatmire. You might as well do one more day and come to Ashconf.

Emma Whamond (58:47)
So, for our audience, that was Zach Daniel. Zach is going to be a keynote speaker at this year's ElixirConf US, and Charles and I are attending as well. So ElixirConf US will be held in Chicago, this September 10th - 11th Come hear more from Zach Daniel, Bruce Tate, and many other voices in the community, and come say hi to us. There's a 10% off virtual and in-person ticket promo code in the show notes, and we're excited to see you there.