Dev Interrupted

Iteration Velocity is the Key to Engineering Success | Vercel’s Malte Ubl

• Season 4 • Episode 37

Share episode

This week, our host Dan Lines chats with Malte Ubl, CTO of Vercel, about why iteration speed is a game-changer for teams trying to deliver more efficiently. Malte shares how accelerating the development cycle and embracing platform engineering can supercharge productivity, helping teams ship faster and innovate more effectively.

We explore what it really means to streamline workflows, reduce bottlenecks, and create an environment where developers can focus on what they do best—building great products. Malte also gives us a behind-the-scenes look at how engineering strategies are evolving to keep up with the ever-growing demands.

Topics:

  • 01:14 How do you see platform engineering changing the way modern software engineering functions?
  • 04:52 Decentralized vs. centralized teams?
  • 11:40 How Vercel is using AI
  • 13:30 All about AI SDKs
  • 20:41 How should platform engineering teams should be using AI?
  • 28:02 Power struggles between high level ICs and a high level managers?
  • 30:10 Key factors to focus on when optimizing AI applications?
  • 35:11 Advice for other CTOs or engineering leaders

Links:

Support the show:

Offers:

Malte Ubl:

Iteration velocity solves all known software problems. Yeah, yeah. Eric Schmidt at Google always said, Revenue solves all known problems. And I think that's actually bullshit. Revenue doesn't solve Google's antitrust problems. But like, you're a professional software engineer, you have to realize at some point that you're not gonna, you don't know the future. you're gonna make a wrong turn here and there. but if you are able to react quickly, then it's not as bad and you can, you can, you can adjust, right? Developer productivity can make or break whether companies deliver value to their customers. But are you tracking the right metrics that truly make a difference? Understanding the right productivity metrics can be the difference between hitting your goals and falling behind. To highlight how you can adjust your approach to both measure what matters and identify the right corrective strategies. LinearB just released the engineering leader's guide to accelerating developer productivity. Download the guide at the link of the show notes and take the steps you need to improve your organization's developer productivity today

Dan Lines:

Hey, what's up everyone. Welcome back to Dev Interrupted. I'm your host, Dan Lines, co founder and COO of LinearB. And today I'm joined by Malte Uebel, CTO of Vercel and former. Engineering Director for Google Search. Now Malta has extensive experience leading engineering teams at scale. And of course, today we'll dive into the changes happening with the rise of platform engineering, Vercel's approach to AI, and how engineering leadership is evolving. Malta, welcome to the show.

Malte Ubl:

Thanks for having me. This is super cool.

Dan Lines:

Awesome having you on the show today. And to our listeners, if you enjoy this episode. Just take a moment to rate and review DI on your podcast app. It helps us continue bringing you great conversations with leaders like Malta. All right, let's kick this off into our first segment. Let's talk about platform engineering and some AI workflows. Multa, platform engineering has emerged as a centralized movement for supportive, functions for software developers. And I know it's a topic you're passionate about. How do you see platform engineering changing the way modern software engineering functions?

Malte Ubl:

That's a great question. It actually reminds me. So I went to college. It's kind of embarrassing how long ago that was, but like over 20 years. And I technically have a software engineering and a business degree. And I hated every minute of the business part, but it probably was actually more useful to me in the long run than

Dan Lines:

Oh yeah. Where'd you go to school?

Malte Ubl:

had this like, it was in Germany, northern Germany. SchoolCodeNordAcademy. we literally had a class on IT organization. And the one thing I took away was that you have this oscillation between a centralized and decentralized form of, of doing broadly IT. And this holds true for, for almost every kind of non core function of engineering that I've seen across these years. So, Like there is also this word of shadow IT and like in every, every time you, you find like a way too inefficient, like we have all these people here that kind of do, let's say, platform stuff. Let's turn, let's put them into a team so that they can professionalize what they do, right?

Dan Lines:

highly

Malte Ubl:

And then you're like, And then they have like, oh my God, their roadmap is like two years planned out. They're not going to do what I want. So I'm going to like secretly like hire this one person on my team and I'm going to have them do platform work. Right. And so you have this oscillation and eventually you're back to like, okay, there's so many of them I need to centralize. And so, clearly, like, platform engineering as a discipline itself is like a centralization movement. Like, because you're now saying, okay, we're going to professionalize this work that's been doing for a while but just didn't have a name, probably didn't have a character. And, like, I actually have been, like, I've been part of this myself in my career at Google when I started there. Um, I was in a team that was part, actually, funnily enough, in the Google Plus organization. And we were like, well funded, we could do, like, build infrastructure. And so we built Google's microservices orchestration system. I built the framework that now every single, like, consumer front end is using on the client side. And But we were just this like small team and then now, nowadays, and I mean, this happened maybe like five, six, seven years ago, Google created the core product area, which is basically Google's platform team. And it's not like a thousand engineers, right? But it was, it was made from that and kind of this team emerged in it. And so that, that's kind of my career path. And, and what we're trying to do at Vercel in a way is that, if you don't have a platform team, you can kind of buy our software and it does a big part of that job for

Dan Lines:

Yeah.

Malte Ubl:

And we kind of are that for you or you are a platform team. And because we are particularly focused on the frontend space, it's very commonly true that people have like a very experienced backend platform engineering team, you know, they may be focused on running their one cloud instance in, in Virginia. Right. Or maybe it's like dual homed, but you're probably not doing global operations. Right. And so it makes sense to have, like, something that from a different discipline kind of come on the side. And so we have a very Happy kind of supporting platform engineering teams on the front end side of their, of their operations

Dan Lines:

Yeah. Do you have an opinion on, you know, I guess size of company when like decentralized is better or centralized? I've worked with like a ton of companies and I, I don't know if it's like a debate, but I see you said it was oscillating. Do you have like a recommendation or maybe for your customer base, what you see works best?

Malte Ubl:

Yeah, I think there is a size where the platform team is not even an option, like you have to do it, but the oscillation actually starts after, right, because you, you as you kind of get too inflexible because you have the centralized team. But I would definitely say that. As your engineering headcount, gets to about a hundred, the inertia of that organization is going to be so much going towards kind of messing things up slowly, but there's so many of them that like things are not going to, they're not staying on the right track unless that's someone's job. And so that's when you have to kind of make that investment.

Dan Lines:

Yeah, I think I'd agree with you. Like, honestly, the companies that we're working with, you know, over, definitely over a hundred developers, sometimes a thousand developers, the best platform engineering teams that I've seen, uh, in the best, I guess, in the best, You know, operating principles at scale. Usually I see a centralized team. That's why I was asking you. That's in my experience. Now you might have some of what you call oscillations. Like you have this centralized team and they have a mission, they have goals and they're servicing the other teams. And maybe in some of the other teams, you might have one engineer that's like interfacing with that team more. But yeah, it was good, good to get your opinion. Cause what I've seen, like from the most productive teams, usually I would recommend centralization.

Malte Ubl:

so, I mean, the idea of DevOps was the opposite one, right? Of decentralizing it. And it was a good idea, because now you have these, like, engineers, they're on the ground, and they feel the pain, and they kind of make their own thing better. It's just that that doesn't scale, and so you need both, right? Like, that's, that's how, where you, the, where, where on the extreme you are between them is kind of the oscillation. But, the professional thing, actually, is to recognize that these are, both of these, Extremes have trade offs and that you want to be somewhere in the middle that you want to have the DevOps like boots on the ground of like, I, I feel the pain. I'm going to make it better. And the like centralized team were like, it's my job. I'm going to, I'm getting paid for creating great available experience, highly productive teams, etc. Right? So that, so it's my main focus. And that's just creates better outcome than it's being someone's like, Uh, you know, 10% project at best.

Dan Lines:

And usually when I see these, uh, platform engineering teams, and this will be interesting to see what, you know, Vercel's up to, but typically it's a lot around, you know, I would call it like pipeline orchestration, like the flow of code, efficient system, CICD, getting things out to production, and then maybe built, building some of those more, I think you said like, Like backend services, but I think you mentioned Vercel a little bit more on the front end that could be used by platform teams, or if you don't have a platform engineering team yet, could kind of be that, um, happy to have you dive in to a little bit of what, what you, you all got cooking there for platform engineering teams.

Malte Ubl:

Yeah. I think the, I mean, we, at our core, what we do is do a very opinionated CICD workflow that deploys your application. Right. And, and we built the entire developer experience around it. And the other thing that we also do, and I think that is. A core part of platform engineering is we, first of all, built frameworks like the we are the makers of Next. js, which is on the on the front end side, the most popular React framework. and then we make these things kind of One wholesome package versus having these, um, like everything kind of be detached. Right. And so because we don't treat the apps that people deploy to us as black boxes, right, we're not, we're not running like an EC2 for you and we don't really care. Even a Kubernetes, like you could put anything on here, right? Like what do we know? Right. Um, we. Um, while we support like, I think, 35 framework or something crazy, kind of like that, we, we deeply understand the patterns, how people build applications inside of them. And that allows us to, to kind of automate the operations, uh, based on those primitives in a way that if you kind of treat applications as a black box, you just can't. And I think that's a, that's a great point. We call that approach Framework Defined Infrastructure, and it's been really, really successful. you know, it's actually not completely clear that this approach absolutely scales to every angle of your infrastructure, but certainly in the front end space where we are the experts, it has proven to be very, very, very successful.

Dan Lines:

You said it's got kind of like a very opinionated, right? In the, in the workflows or when you say opinionated, like, what are the values of that opinion? Or like, what's the philosophy behind it? If you can say in a few words.

Malte Ubl:

have this phrase that I say, which is that iteration velocity solves all known software problems. Yeah, yeah. it's a code from, well, Eric Schmidt at Google always said, Revenue solves all known problems. And I think that's actually bullshit. Revenue doesn't solve Google's antitrust problems. But like, you're a professional software engineer, you have to realize at some point that you're not gonna, you don't know the future. you're gonna make a wrong turn here and there. but if you are able to react quickly, then it's not as bad and you can, you can, you can adjust, right? And so everything we do is basically About literally physically reducing iteration cycles. Uh, so that starts with the frameworks we make support hot code module replacement, right? You type in VS Code. And your screen updates in absolute real time. um, we actually try to make this like literally for 60 frames per second. Um, we just might be a bit over optimized. And, um, then it was obviously build times. That's a, that's a very obvious one. we invest a lot of feature flags, right? Which is, I think, an important topic. We made that a first class, platform feature. we are big believers in like preview deployment. So like every time you make a branch, you get an associated application copy, right? and then even when you deploy. And you're in production, we have a feature called Instant Rollback, where we guarantee you to be back to the version of your choice in under 300 milliseconds, uh, globally.

Dan Lines:

Yeah, very

Malte Ubl:

And if you compare that to like a traditional kind of more, um, Kubernetes setup, where you, it can be anything, right? It can be definitely minutes. Um, but we, we have absolutely instant rollback and you can also do it to any version that you had in production before. Like technically it could even be a year old.

Dan Lines:

Yeah, totally. I mean, kind of had me at like improving iteration velocity and all of that, like decreasing wait times, decreasing build times, you know, again, all the best companies that I'm working with are measuring those things, working to improve those things. So that's kind of like right in line where we come from, the LinearB product. Now. AI obviously is super hot. Everyone's, talking about it, trying to talk about it. Want to talk about it. How does like AI fit into these workflows or like, what, what are you seeing in terms of either your company or other companies, like utilizing AI within, uh, workflows?

Malte Ubl:

Yeah. I guess it's the, the question of the hour. I would say like probably like a year ago, maybe a little bit more, obviously like lit that then literally every company would start saying like, Hey, there's clearly some revolution going on here. what's our space in it? Like, what's our role? Like, do we even need to exist? and I'm actually really happy that, that at Vercel, we, we have found two answers and I really love the answers because the answers are so, they feel so native to us. Like they feel really like, okay, this is actually. It took us a while to figure out, but now it's like, obviously this is what we should do, right? And so the two answers are that we make an AI SDK, so a software development kit for building AI applications. And that really fits with us because, you know, we're at heart framework makers, right? We make software frameworks, so we give you an SDK. to, to build AI based applications, right? Um, and the other thing that we have built is a software tool called V0. Which is, the initial idea was it builds the V0 of your app. So you type in the prompt, and out comes the React code that looks great, that implements your app. Right, and we've now kind of started iterating on that and turned it into, uh, Um, that plus an assistant that, like, teaches you how to use the frameworks you're making and so forth. And so we have this, like, thing that builds your app and as someone, like, that's, that's like our job anyway. And so being able to, to do that is, is, is I think really key and, and, and it has, like, resonated wildly with the community, um, which obviously makes us super

Dan Lines:

I think both of these are interesting, but maybe let's take it. One by one, you were talking about the AI SDK and you started with like building an AI application, like what is an AI application? And then how would I use like the AI SDK? And then we'll get to the other thing.

Malte Ubl:

alright, it's such a big topic, right? But like, big picture thing is that you mentioned I was at Google for the longest time. So everyone around me was building AI apps and that was whatever, five, six, seven years ago. But everyone isn't quite true. Those were all like ultra experts. and that's the revolution of AI engineering is that now everyone who can call an API can be an AI engineer. And so the workflow is completely different. It used to be that you had to like Go collect data, and then you train this highly specific model, and then go collect data,

Dan Lines:

like really specialized,

Malte Ubl:

probably can like do something quite reasonable, like, at your first shot, right? I mean, and you can literally prototype it by, say, Trying out ChatGBT and see what it would do for like your problem space. And so there are some other issues like, cause those are really expensive to operate and we can get into that, but like you start with something really. really good using the Frontier model and using that system is just calling an API. but it's not really just calling in and API because there's some other stuff. And so we built this SEK and it basically has two jobs. harmonizes the APIs between all the different model providers. And one of the learnings we had, that it's actually. Less work than you would anticipate to switch from OpenAI to Entropiq or to Google. Like that's actually something you could do. but obviously if you're, if you're a, if your software treats them as the same, you can think about it like, like JDBC in Java land, 20 years ago, harmonized how you talk to a database, AI SDK harmonizes how you talk to different AI models. that's really kind of, again, native to what we do on the front end is that taught AI. You can think about it like in your app. It can be internal, it can be external. You have domain UI. My favorite example is a seat map for an airplane. And you could totally imagine you have this chat assistant where you say, Hey, I would like to change my seat map. My flight, my seat, and this booking code. And then, like, ChatGPT, by default, would answer with, like, whatever, like, the, all the codes for the open seats, and you're like, what the fuck? Right? Like, but if it could answer with a seat map, and you just go pick one, and then the AI learns about the choice you made. that would be great, right? And so that's what this SDK does. It kind of, it teaches the AI about your UI component and your, really your business components, right? Like that, that belong to your domain. and then it makes it interactive, which kind of is really, really helpful for building a kind of sophisticated AI applications.

Dan Lines:

That's really cool. I like that. What's been the, uh, like, response to it so far?

Malte Ubl:

I know it's been, it's been really good. Like it's, it's growing like, like crazy. Um, there's other things like, for example, it supports schema full JSON. So one of the things, the first thing that you, you talk to an AI and it kind of, you really want it to respond with JSON because like you want to come process the output. You don't want text. And that's kind of nasty. And so the SDK helps that. And that's been super, super successful and popular. And I think what we, what we got right. And now that obviously wasn't, totally wasn't clear that we would get it right. But what we had tried was to, we didn't over abstract. early in the career of kind of modern AI, there were a few kind of frameworks, libraries out there that. It's kind of very complicated and one insight I had like many years ago as like a framework designer is that that's, you can make a thick abstraction if you have built this type of application many times. And I think that counts for more like five, like you, you really fell on your face or five times and you're like, now I really know how to do this, right? Like now I can give you the perfect abstraction to, I don't know, to build a web application. Like that's been around for 20 years. We absolutely know how to do it. Nobody has any idea how to make an AI application. so we give you, give people like the absolute minimum abstraction that took away, kind of figuring out how to get JSON out of something, but we don't have like some like crazy callback system, you know, whatever, like, uh, what you could imagine, because who knows what people are going to do, right? Like everyone's kind of still figuring it out. And so this is really kind of like very simple piece of software that kind of even. Like evens out the, the, the known kind of tricky things, but it doesn't, it's not an application framework of the sense that, that you, uh, you know, might

Dan Lines:

Yeah, that's really interesting. I mean, I'm sure everyone listening, like, definitely check that out. I have a, a note here, You have a note saying like latency is always a concern with large language models and other AI systems. maybe you can talk about that a little bit, but how does Vercel approach this challenge in user facing apps?

Malte Ubl:

Yeah, totally. Like, I think one, one interesting change that wasn't at all introduced by AI in engineering, but suddenly became kind of necessary. is that you build applications in a fully streaming fashion. kind of everyone knows in quotes, that's kind of stream processing of data is important. But for like a normal API call, You'd probably not use it because it's got X complexity. And, you know, if the whole thing takes a hundred milliseconds, you're like, okay, I'm just going to wait for the response. Right. I'm not going to do it. Like partials, JSON parsing, or like, you know, use the, the fully Bidi streaming version of gRPC, whatever you like. Like you, like you don't go there because it's. But if your API, as in the AI case, streams for 30 seconds, you have to do full on streaming processing, right? And what we found is that just through the entire serving stack, because of these biases to say, like, whatever, we'll just do it without streaming, nothing was actually ready for it. Um, so we deal out in the front end, React, the most popular framework out there. Until a couple of years ago, it just straight up didn't support streaming. You could only respond with a single response. Now, everyone doing PHP would know, okay, no, no, we've been doing this for 25 years. What are you, what are you talking about? And that's, that's right, right? But, but basically the biases were in the stack. serverless infrastructure like AWS Lambda, until recently, just straight up did not support response streaming. And so we really kind of had to work through this entire stack. To make everything, first of all, work on the infrastructure side, but also like get both the AISDK as the processing system, but the frameworks on top, like the Next. js that we make really ready for, for this kind of streaming processing and make sure that it works end to end. Because, again, if your API takes 30 seconds, you want it to show something to the user as soon as you can, and you don't want to wait the whole 30 seconds.

Dan Lines:

Yeah, definitely, like, thanks for sharing all that information. Definitely really interesting stuff. especially if I'm going back to platform engineering teams. do you have any other like advice of where platform engineering teams should be using AI to help developers or where not to, you know, can it overcomplicate, things? Like, do you have any tips for our audience generally there for platform teams?

Malte Ubl:

Yeah, I think, I mean, it's, they both, as a platform team, I interact on both of these angles of A, I probably have my internal customers who AI applications, right? Because that's what everyone's doing. So that's, and that's my job now as a platform team, like, enable these teams to do that. And the second part is Obviously, AI is an incredible tool that makes engineers more productive, if possible, right? And so I called, I talked about our own v0 tool, which is, uh, Like, specific for, for the front end use case, the most obvious one is, is GitHub Copilot. And so like, I think as a platform team, it's, it is like my job as the leader of the team to be advocating for either introducing something like this or, or evaluating how. How it makes sense in my, in my process.

Dan Lines:

Yeah, it makes sense to me. I mean, most of the platform teams are sometimes actually, uh, maybe a little different, but like developer experience teams, most of them have a mission to say like, how am I going to introduce AI to improve productivity for every developer in my organization? know, so, you know, definitely be dabbling there. Most of the, the, uh, Uh, Developer Experience Teams that I work with. Yeah, they're starting with CoPilot. Um, but just wanted to see if there was anything else that you saw that was interesting. Uh, to maybe start dabbling with.

Malte Ubl:

No, I think it's really, like, the the side that I think is key for platform teams is like, how do I enable my, like, my customer teams to be effective at building AI applications? Right. And, I mean, AI applications itself is like, You, everyone goes to like chatbots and like, those are actually kind of valuable things to build, but what we see a lot is kind of what I call invisible AI, where you use AI to do something that was before, used to be, it was possible, but kind of Maybe too much work, and so you wouldn't prioritize it, and suddenly it becomes really easy. Like, we actually have a blog post out over, Puny code domain formatting for Farsi domains, so the Iranian language. And there just wasn't a library on the internet They actually knew how to do that.

Dan Lines:

Interesting.

Malte Ubl:

there was a spec. And OpenAI, GPT 4 O model can like write those, these domains. It knows how to do it. just because the spec exists, right? There's just no software on the internet today That can actually do it. And so voila, right? Like you have this like, uh, problem solved that, that otherwise, you know, in, in, in hours basically. Right? And, and, but it's this tiniest use case, right? Like it's, it's, and there's no UI

Dan Lines:

to do a task or I need to get something. Yeah, that's interesting.

Malte Ubl:

but it's only an easy task if you've saw, if you've, if you have a contract with the eye provider. If you figure it out, all kinds of legal stuff, right? the first time you do it, this is actually not so easy, right? Like the, the second, third, fourth time, you now have this like super powerful weapon in your, in your toolbox that, that can be super effective. But you have to kind of break through that wall the first, um, doing it the first time.

Dan Lines:

If I'm switching topics here, because you've, you've had a, like a really interesting career. Let's talk a minute about engineering leadership. So you've led both large scale engineering teams at Google and now Vercel. And how have you seen the relationship between individual contributors and managers and engineering teams change over time?

Malte Ubl:

Yeah, this is actually my, my, my one of my pet peeves. like of all the things that Google has done, the one thing that it got right is that it has like this very trivial, job letter thing where there's an IC letter and an, and a manager job letter and they both have 11 levels, right? So you can be a senior vice president or a senior fellow and. And you're on the same level, right? And so no one's forced to go through the

Dan Lines:

Yeah. Usually it's the management track to like grow your career. Yeah. Before that. Yeah.

Malte Ubl:

Right. And, and like, I mean, you know, there, that there's like the Peter principle, right? Like everyone gets promoted to the level of their incompetence, which is like, that's just a straight up true in these like traditional organizations where you At some point, if you want to advance, you have to go into management, right? And so it's like one of my absolute most core values to have that IC track so that the people who are on the management side, self selected to be there. the other thing that I think is really important is that, and I, you know, I correct people here all the time when they send like, so and so got promoted to management. And I'm like, dude, That's not a promotion because it's a lateral transfer. Where if that turns out to be not the good idea because this person either hates it, or they're just not so good at it. But like they were probably like, we gave them this chance because they were like really good at their jobs. So what are the outcomes if it was a promotion? they feel like they were like, they're now in this role and they're kind of tied up to it. Because. I've seen it demotion and no one does that, right? But it was very clear it's lateral. Then you can say, yeah, you know, we did, we tried for six months, this is just, it isn't for me. I'm going to go back to my old job and it's, it's a lateral movement, right. And so if you, again, like the, the, the opportunity cost of flat blending, of having this person stay in this job that they hate, that they're not good at, because they feel that leaving that, that role is a demotion, it means just that you just have this like crappy manager who also used to be amazing at the IC

Dan Lines:

Yeah, it's almost like you lost two people. You lost the amazing IC, and you have a bad manager now.

Malte Ubl:

And so that's, that's why it's really important that it's a lateral move. And We try to practice this at Vercel. And it's actually, so my role is kind of specifically the, just the highest version of this. So basically actually in our

Dan Lines:

Like your role currently, you're saying, your role is like CTO?

Malte Ubl:

I'm a CTO. I am an IC. And so the way we, we are, we have it organized is that, we have a CPO, so Chief Product Officer, and, and they're, they have engineering, design, and product reporting to them, right? So they, and so we're basically this like, you know, Uh, Peer, IC, Manager, Pair. And so we're trying to have, at every level, like there's the VP of Engineering, but there's also the, the kind of, VP level, kind of, UberTL, whatever you want to call this person, who's responsible for the entire space, but from the engineering angle. And so, like that kind of trickling down

Dan Lines:

Yeah, just out of curiosity, because I haven't, You kind of got that experience at Google and now you're doing it at Vercel. And when you have those equivalent lathers of like an IC, like you're saying, Hey, I'm the highest level IC right now in a sense of CTO. And then you also have like the manager form of that. How does it work between like a high level IC and a high level manager? Is there any like power struggle? Is it really an equivalent? Is it non equivalent? Just understanding some of the human dynamics. Cause I, and where it's coming from my head, it's like, You know, hey man, I have like, 400 people on my team. I can do what I want with these 400 people, and you have just you. Like, are we really equivalent? Just putting it out there, like, how is it?

Malte Ubl:

No, it's a, these two people have to be best friends, right? And really kind of be able to, you know, they have to talk to each other every day. there is one, actually, by the way, really interesting question.'cause I've, I've definitely seen both and, and, and I am not particular opinionated, which is the right version, which is it could also be that not only are they. this one person has 400 reports and the other person zero. It can actually also work that, that, that person with zero reports does report to that manager. And, my formula is, that works if they're like the best team. And if they've worked together, for like the longest time. And so it's super natural, because then it's often better for the IC, reporting to the CEO, for example, it's like, you know, maybe not what you want to do for your career development, if you care about that, right? Because they have other things to do. So it can be, it can be really good to have them as your manager, but that only works if you have like a really, really deep trust relationship. If, on the other hand, you kind of, you make this team, um, it's better for them to be peers. So that you know, maybe someone could still say, well, you know, you don't tell me how to, what I do with my 400 people. I think that's something that you can kind of, you know, Manage towards getting right and I, I actually honestly haven't, haven't seen being too much of an issue, but I can understand that, you know, that you would have that concern, but like basically, yeah, I think it's, it's something that, that, that's absolutely manageable.

Dan Lines:

no, that's really cool. Thanks for sharing that insight. Cause I think, uh, you know, a lot of developers have that on my mind. Like what could be my career, progression if I want to go the IC route management's not for me. so that's really interesting. Yeah. Let's take it over to our. Last segment, optimizing AI applications. We might've touched on this a little bit, but building AI prototypes has become much faster, but scaling and optimizing these systems does remain a challenge. What are some key factors that engineering teams need to focus on when optimizing AI applications for performance and cost? I think we touched on

Malte Ubl:

Yeah. I mean, we talked about the streaming part, which is like kind of like this very, like, I'm just accepting it slow and I'll make the best out of it. but not, you know, really about, uh, about making it faster. So like one thing that actually is a huge transformation in the software engineering process overall, is that when I introduce AI to my system, I, I give up on the last kind of hopes of deterministic software. And you could kind of argue that I already ran out of the window with With like distributed systems in general, like they do weird things, but like the way you handle that, it's, I think, very different from an LLM. Which literally has like a temperature temper argument, right? You can't turn it off, that it's not deterministic. You give it the same input and something else comes out, right? And so all these like things, how we release software, how we, how we do testing, they just don't work. Because you can't, you can't make assertions, right? You can't say this thing in, that thing out. That just doesn't work. And so what we're definitely seeing, and this is like literally on my own team, like a transformation that we. That we're going through and I was afraid about it, but seems to be going better than I thought it would. Is that this notion of the eval comes into the software development process. So an eval is like a test, but more fuzzy, basically. And, like, this is actually also interesting, like, because I come from Google where that's like normal, right? Like the, like Google, you make a search ranking change. You can't run a test suit. because there's like the input space is like so big. And so what happens instead is you have maybe your hundred or thousand most important queries and another thousand random queries, and you literally Screenshot them the results and give them to people to rate, right? Before, after, and because something probably would have gotten worse and something gotten better, right? It's not a black and white thing. and so that's the process I was used to, but no one else because I also was really slow, would develop software like that because normal software you can automate tasks, you can have

Dan Lines:

Yeah, you, you have your suite, and it's not like

Malte Ubl:

Yeah. and now. So this process kind of comes to everyone and there, you know, we, we use a software called BrainTrust, to help us do these evals. And what's really interesting, and that's certainly different from how this used to work at Google, is that AI can help you with this thing, right? so what you do in practice, and again, this is not for like the small AI app, it's for the big AI app. Right? But there you change your prompt and you want to know, was it a good change or a bad change? So you have, let's say, a hundred inputs and then you run it through your new system and you now let a different AI rate whether it got better or worse. And this is particularly interesting if you probably like you are optimized, you optimized your, your, productionized your AI to be efficient and maybe not the hottest latest model that, cause that's too expensive, but it's probably okay to use a very smart reasoning model. For the eval side to just judge if the output is good. And so that kind of gets you kind of to that process. That's, that's fully automated, but it's very

Dan Lines:

probably do a whole podcast on just that topic. This is like AI

Malte Ubl:

exactly. So this is like, it's kind of a comfort zone thing, right? Cause you're really in a, in a different world now. But yeah, once you have that set up, you can think about, I was already mentioning. let's go from, like, we have this like working in the, you know, GPT 4 class Frontier model, but it's costing us 5 cents a pop. how about we lower the cost by a hundred X by going down to, and now, now, I mean, in a way OpenAI is the weakest here, but like all the major providers have like the stack of models. They're kind of similar, but they're, um, the different in size, the and they're certainly different and the smaller ones are faster and cheaper. and so you say like, how can I get something to evolve as well in the simpler model, right? And so that's why it's so important to have a good way of evaluating whether how good you are. Um, so that you can like be aggressive about making these changes where you say, okay, I optimized my prompt so much or I fine tuned in a way that the cheaper model can do what the bigger model did for like my domain, because it's not the whole world. It's just my domain. And again, like you need these eval flows in there so that you can actually make those decisions in a way that, you know, doesn't evolve, yourself going through a hundred use cases and being, human about it and making mistakes.

Dan Lines:

That's, that's really cool, man. As we wrap up here, kind of a final question, a little more of a general question, but, you know, being a CTO, highly successful company. what advice would you have for either other, you know, CTOs or engineering leaders? Like, what's top of mind for you today in today's world? We talked about AI, what, what are the things that are like Most top of mind for you as you think about, being in what other CTOs and leaders should be thinking about.

Malte Ubl:

I mentioned earlier how we went through this process of kind of finding us in the disrupted world, right? Like what's Vercel's role in this space? And again, like we found these solutions which are they're native to our company. And I think that This is a really good exercise for everyone to walk through. I feel that while we have these like universal assistants now that are really useful, it's also like become really clear that domain specific stuff is actually continues to be extraordinarily useful. and so there is this space for everyone in this future, but you have to find it. Right. And so I think it's like, it's a really good line of thinking of like, what is like my company's spot in this, in this world. and hopefully everyone can find an answer. Um, but if you do, it's really exciting.

Dan Lines:

Awesome, man. Well, Malta, thanks so much for coming on and sharing all of your insights. It's been great having you on the show today.

Malte Ubl:

Awesome. Thanks for having me. So, so much fun.

Dan Lines:

on these topics, be sure to check out Vercel's latest work, and I'm sure we'll attach it. to the show, notes. And as always, thanks tuning in to Dev Interrupted. Be sure to subscribe to our newsletter on Substack for deeper dives into the topics we discussed today and everyone. See you all next time. Thank you. All

People on this episode

Head Shot

Dan Lines

Host
Head Shot

Conor Bronsdon

Co-host