Episode 61: A Key Piece Missing From Coding with LLMs

Transcript

I don't think I've talked about AI in a while, so here we go.
I, like many people, have been spending increasing amounts of close personal time with various forms of AI, largely LLMs.
I've been on this show before talking about how it's not necessarily all it's cracked up to be.
It might be one day, and it will probably continue to improve, although there's some indicators that even the improvements are starting to hit sort of diminishing returns.
But regardless, it still is improving, and it does amazing things.
So, like, I want to say all that at the outset here.
One of the things that I use LLMs quite heavily for these days is in development workflows.
So, like, you know, programming and building out systems and stuff like that.
If you were a web developer coming up through, like, the mid and late 2000s, you would know that there were far fewer technologies that you had to learn to do that work.
And then as you got into the, you know, 2010s and the 2020s, by now it's a whole other world, there's a lot more that you have to know about to be proficient in that field.
So, yes, if you want to be just, I'm a React developer and that's all I do, that's fine and good, and I'm sure you'll find work just fine.
But if you are, you know, doing more kind of full stack stuff or bouncing between different clients or something that have varying systems and, you know, however you want to look at it, you need to know a broad swath of things these days to do that work.
And so, where LLMs were really, really helpful is helping you get up to speed in a lot of ways as a developer on something that you may not be familiar with.
You know, that initial learning curve, like, I remember, you know, if I wanted to learn a new language or I wanted to learn a new framework or something like that, you know, 10, 15 years ago, my journey typically started with watching super, super dry videos on YouTube, right, or some such platform where they start you at the very beginning and you start looking through things.
And, yeah, once you get a little more experienced, you can also, you know, be looking at code at the same time or starting with some boilerplate stuff.
But the ramp-up, regardless of what your methodology was, the ramp-up was much slower.
What's nice about the LLMs is you can kind of go in there and say, hey, here's what I'm seeing and, you know, copy and paste in, like, what's maybe in the code already or whatever.
You know, what's going wrong here or you could say, hey, I need to build a system and such and such.
It needs to do this.
Can you give me some starting points?
Can you give me a script that might be working, if you're lucky, 60% of the way there?
It just speeds up that on-ramp so that then as you're working with it, you're kind of learning on the go in a different way that I think is pretty productive.
With all that said, one of the groups that I think is rallying against some of this LLM stuff the hardest is developers in many cases.
And I don't even think that they're rallying against the idea of LLMs in general.
I think they're rallying against, myself included in some cases, the idea that the technology where it is now can just solve everything.
And that it can just, you know, you stand it up and it'll just write all your code for you and you really don't need developers anymore and whatever.
So some of the rallying is definitely because jobs are being displaced.
And that should be acknowledged, right?
I've spoken before about where I think this will go ultimately, where there'll be a downturn in the need for development teams briefly.
And then over the course of a few years, it's going to really swing back in the other direction.
I'll link to an episode where I think I talk about that.
But beyond that, there's another thing that I've noticed, having done this now for, you know, quite a while, quote unquote, in LLM parlance.
But this workflow of using LLMs and associated tools to help build things out professionally, there's another part of this flow, or maybe it's the whole flow in general, that I think it's very bifurcating.
It adds to frustration in a way that coding didn't used to.
So I'm going to try to explain this workflow as I see it, and we'll see how that goes.
As a developer, so let's talk pre-LLMs.
You would start with something small, right?
Maybe you're trying to learn a new language or a new framework or whatever.
A lot of times, the go-to first thing is just, hey, can I get something to show up on the screen?
So in different languages, that can look different ways.
Sometimes it's just one line of code that it takes to do that.
In other languages, it might be, you know, five or ten.
But it's a pretty simple thing.
So you kind of download, again, whatever this language, framework, runtime, maybe there's an IDE associated with it.
You kind of get your environment set up on your computer.
And then your first task is just to get something to show up on the thing.
Once you get there, you would then probably be, and I'm going to assume that there is some project that you are working towards.
So whatever the project is, now you start thinking about, okay, what am I trying to make this thing do?
And what are some good first steps?
Like, what might be the first thing I should dive into trying to make here?
So I don't know.
I'll use some silly example just because I don't want to overcomplicate this.
But I don't know.
Maybe you decided you're going to make a new tic-tac-toe game or something.
And maybe the first thing that you decide you're going to do is draw the grid and try to register when someone clicks in the thing.
So you do that.
And that might take you some amount of time.
You know, depending on the complexity of the language or the framework, it could be in your experience level.
That could be five minutes.
That could be five hours.
That could be five days.
Who the hell knows?
But after some period of time, you now have this working first step.
You have a grid.
It shows on the screen.
And maybe when someone clicks in a box, it sends a statement to the console or something or pops up a message and says, hey, you clicked in box number one.
Maybe your next feature you decide to tackle is the idea, the concept of turns.
So player one clicks, player two clicks, and it goes back to player one.
So that takes some time.
You've got to build that out in the back end.
And the idea of two players and blah, blah, blah.
But you get that to a point where it's working.
And now your little thing, first you click in the first box and it says player one clicked box one.
Then you click in another box and it says player two clicked box three or, you know, whatever.
Then maybe your next feature is disallowing players to, you know, to make some record of the X's and O's.
So now player one is X and player two is O and you're going to keep track of what they've clicked on and not allow them, the second player to click for the box from the first player.
And you get that working.
And then maybe you move on to the scoring and you move on to who wins and so on and so on and so forth.
The point of outlining all this is that it's incremental.
First, you take one step.
Then you take a second step.
Then you get a third step and a fourth step and a fifth step.
And every step of the way, you can feel yourself like, OK, I got through feature one.
Now I'm going to start feature two.
Now I'm 5% of the way through feature two, 10%, 50%, 75%.
And as you encounter bugs and glitches and problems, you feel like you're getting closer.
And that's the feeling that I'm trying to encapsulate here is this feeling of as you progress, you know that even though you're encountering errors and problems along the way, because you definitely are, you feel like, oh, once I solve this problem, I'm done with this feature.
I'm so close.
I know every step I've taken to get here.
And I know where I'm trying to go.
And I'm one bug away.
Right?
That kind of feeling.
So that was the sort of, at least for me, the traditional development workflow of incrementally getting closer, knowing where you've come from, knowing where you're going.
It takes a long time.
Right?
So I think where the LLM, now let's fast forward to the LLM world.
The difference is I can go into an LLM and I can say, make me a tic-tac-toe game in whatever, in this language or this framework or whatever.
And it's going to spit me out probably almost everything I need.
And then if I go and run it and it works, cool.
If it doesn't work, that's still okay.
At least in the back of my head.
I think it is.
Right?
Now, you know, now you start interrogating it and you start thinking, okay, well, it didn't work.
Maybe you go back to the LLM and you say, it didn't work.
This, you know, maybe if you're, if you've done a little bit of troubleshooting and say, well, this seems to be happening.
Then it spits you out with new code.
Put the new code in.
And if you're doing this with like agents or something, it's a very similar idea.
Maybe there's less back and forth, less copying and pasting.
But the difference is you don't feel yourself getting incrementally closer.
At least I don't.
What I feel is that it gets me 90% of the way there all in one shot faster than I ever could have by myself.
But to do that last 10%, I don't actually know if I have 10% left or if 60% of the original code is wrong and needs to be ripped out.
When it gives me new code, I don't know how much of it is going to overwrite or what new bugs might be introduced.
And you don't, you, you, I end up in this frustrating cycle when I use these heavily.
And for me, what I've identified is that it's the difference.
The bug could be the same, but it's the difference between feeling like I've been making progress all the way along, right?
Where I took step one, step two, step three, step four, step five, as opposed to being placed directly on step five.
And now just shooting in the dark, trying to figure out step six.
And I think, and I've talked to some people who have a similar experience, maybe they articulate a little bit differently, but that's the same kind of idea.
Where it's almost like you don't feel that feeling of accomplishment.
Because a piece of this, right, like you could make the argument, well, you don't understand steps one through five and that's what's frustrating.
And I agree, that's true, that is a piece of this, but there's another piece of it, which is more subtle, which is this idea that in, in the, in the previous workflows from yesteryear, you, you knew you were close.
You knew you were almost there and you said to figure out this thing.
Now, you're not really sure if you're almost there or not.
You don't really know if you have one more step or five more steps, or if the LLM is going to rip all your code out and start over again, or if it really has any understanding what you're even doing anymore.
Has it totally lost the context completely?
And there's a difference there, the difference between incrementally improving and feeling accomplished versus jumping to the end, but never being able to finish off that last, that 1%.
So, I don't know what the, what the solution is here.
Maybe there is, doesn't need a solution, but I have noticed this, this idea of the, the feeling of accomplishment is kind of gone, but it's not just that it's gone.
It's gone in a way that hinders, I think, the overall workflow and hinders the end, reaching the end goal.
I don't know.
Are you, are you a developer?
Have you done any of this stuff?
What do you think?
I know that there's stories out there of, you know, I built this entire thing and, you know, I never touched a piece of code.
I don't even know how to code and all that kind of stuff.
But, but I would put good money that for every story that's like, I never learned to code and I coded this, I did this whole thing with LLMs.
For every one of those, there's like thousands of screwed up text files filled with code somewhere that never quite worked, that were all 80% of the way there.
But because they were rocket ships to that 80%, that last 20% became impossible.
What are your thoughts?
Reach out, uh, lmc.fm contact page on there.
Always, always like hearing your thoughts.
Always like hearing your thoughts.