Let’s talk about the big three:

• Free will
• Self
• Identity

Free will

Endless flows of ink have been spilled debating the nature of free will, and given that I only hold half a PhD^[1] (not even one in philosophy) I don’t think I’m qualified to do justice to them all^[2]. Nevertheless, for the purposes of context, I’m obliged to at least delineate what I see as the three main camps:

• Libertarian free will: The belief that free will — the capacity to make choices — exists, is real, and is not an illusion. This is the natural view that we all start with, because as we grow up, it certainly feels like we’re making choices.
• Determinism: The position that all future events rely on chains of causation, and that this precludes the existence of actual free will (at best, the illusion of free will is permitted).
• Compatibilism: A series of gymnastic somersaults that philosophers perform to make the claim that true, non-illusory free will totally can exist within a deterministic universe. I won’t bother describing it further because it all strikes me as definitional sleight of hand and not much more than smug cleverness for cleverness’ sake.

In practice, if you’re religious, you probably subscribe to libertarian free will. Without this kind of free will, all this business about choosing to believe doesn’t really make sense (and the idea of burning in hell for not believing in God seems a wee bit unfair, if you don’t have any choice in the matter). Other beliefs adjacent to or overlapping with religion — things that might be labelled spiritual, or mystical, or supernatural — are also likely to lead you to choose this version of free will; if you believe in something like a soul with an existence that is independent of the physical substrate of our bodies, if you believe in reincarnation or out-of-body experiences, for example. Even if none of the above apply, it’s still possible that you might have a view of selfhood rooted in your subjective experience which leads you to feel like there is something inside you that is actually making decisions.

Atheists, scientific types, and such, are more likely to believe that determinism rules out real free will as a possibility. When two particles in a system collide, the laws of physics tell us what happens next. Your body and your brain are made of lots of these particles. Given enough knowledge about the configuration of all those particles at any given point in time, these laws should tell us what the configuration will do as it evolves over time. In this view, the appearance of free will is just an emergent phenomenon that arises from the low-level details of the system playing out. At the macroscopic, subjective level, we perceive our decisions as being products of "free" will, but the argument is that they’re all really determined.

The ironic thing is that an utterly pre-scientific Buddhist monk wandering over a thousand years ago through what is today India could have and would have arrived at the same conclusion without any discussion of particles or the laws of physics either, but rather through quiet observation and meditation. The same subjective awareness that some would use to claim that free will is real, can be turned inwards and lead to the complete opposite conclusion. People who meditate often recount how they become aware that things like decisions — and not just decisions but also ideas, words, feelings, and more — appear to arise as if from nowhere, with no sense of internal authorship or ownership.

I personally hadn’t thought much about all of this until about 15 years ago. When I did start to think about it, coincident with dabbling in meditation, I found through the combination of the simple logical argument for determinism with the support of observations I could make in my (inexpert!) mediation sessions, it was pretty darn obvious that free will was indeed an illusion. If you don’t already believe this, I’m not going to try and convince you here because I know there is an ample quantity of well-crafted and convincing argumentation online.

We humans are basically machines, complicated organic "computers" that run algorithms more or less deterministically^[3], and when we agonize over a choice and eventually arrive at a determination, the decision that pops into our mind is no different from a punch card popping out of a computer at the end of a computation. When I make the choice whether to eat junk food or a healthy snack, I might feel like I’m weighing it up in my mind, but really all that’s happening is my hardware is carrying out its programming and arriving at its inevitable result. And moment to moment, when I’m selecting what word to say next in this sentence, I can’t really claim any real agency in the process; the words are very much just arriving from "somewhere", the output of layers of calculation that I can only begin to fathom. There is no more a "me" inside my head wielding the quill and ink than there is a "self" inside an LLM deciding what the next token should be.

There are some smart fellows out there, like Roger Penrose^[4], who try to square all this in such a way as to make determinism compatible with free will. They do this not in the way that philosopher compatibilists do (eg. by doing things like redefining free will as nothing more than "the subjective experience of making decisions", and claiming that somehow means they can claim that free will is real — ie. not an illusion — after all, while remaining totally compatible with determinism). Instead, they make the argument that conciousness only exists because of quantum processes inside the brain — and not because of processes happening at the cellular level — and that within their framework, true free will somehow emerges due to (hand-waving) something about quantum state. All this stuff is beyond me, but I will say that it doesn’t appear to pass the "sniff" test; it may be that Penrose is right and everybody else is wrong, but the more likely case is that the widespread skepticism of this particular idea among his peers is probably well founded.

Believing that free will is an illusion doesn’t relieve you of your obligation to respect the laws of society (that is, you can’t go off on a rampage of theft and murder and then make the claim that none of it was your own doing). I think that one of the reasons society at large works, and also why local relations between individuals "work", is because we’re mostly all operating under the tacit agreement that free will does exist. Or put another way, that even though it is an illusion, it behooves us all to behave as though it weren’t^[5].

Self

I said that free will isn’t the only problem though. When you allow the domino of free will to fall, the next one to topple is usually that of "the self". The same kind of meditative practice that can lead you to conclude that free will is an illusion can in turn take you to a place where you realize that the self is an illusion as well.

It does feel like a bigger leap to make, because the sense of selfhood that we all subjectively experience accompanies us much more intensely, continuously, and pervasively than that of free will. Most of the decisions we make in our life are so low-key — like where to place our next foot-step when walking down the street —that they barely rise above the threshold of our awareness. But the experience of self-hood frames and constitutes almost everything we do.

There is an almost ineludible sense that there is a "you-ness" located somewhere inside you, most usually inside your head, looking out into the world through the windows of your eyes. This sensation is so normal and so continuous that we rarely speak of it, taking it for granted, and we instead name other states — like "flow states" — for those much rarer moments when you somehow lose your sense of self; like when you get absorbed in a good book, or lost in a task that captivates your attention and makes you lose track of time.

But if you think that a human being is a deterministic machine chugging along according to the rule of physics, it’s hard to explain how consciousness arises within that machine, and once it does arise, how and when it crosses the threshold of self-awareness^[6]. Clearly, it does cross that threshold, but doing doing so doesn’t necessarily give us a handle on what the self is.

If you could simulate your connectome and all the interconnected biological hardware on a computer, could such a thing be said to be conscious? If you believe in substrate independence — and doing so is a small step to take once you’ve accepted determinism — then it seems inevitable that a simulation must be said to be capable of consciousness. In this view, defining what consciousness is might boil down to something reminiscent of the Turing test: conciousness is that property that entities appearing to exhibit consciousness possess; a tautological and insufficient definition if ever there was one^[7], but fortunately, not one I have to defend to a thesis committee.

Once again, we don’t have to seek scientific explanations here if we want to arrive at this particular destination. Meditation alone is enough to convince us that what we know as the self is an illusion. I am far from an expert practitioner (and in fact, it’s been many years since I meditated regularly and in a structured, routine way), but even my inexpert dabbling was enough to provide me with access to a range of observations that revealed how contingent, arbitrary, and fragile the fabric of subjective experience is. Thoughts, emotions, sensations, arise as if from nowhere. The brain is very practiced at sense-making and arranging these into a semblance of order, a linear narrative^[8], but when you stop and observe what the experience of experiencing is really like, in detail, it becomes clear that what is going on is a lot more happening and not quite as much being as you perhaps thought originally. That’s a pretty vague thing to say, but here once again I’m going to make use of my grand excuse that there are many others out there who have described all of this far better than I can, and their work is easily accessible for those who are interested.

We’re so darn good at making narratives of things, that I think our very experience of the passage of time is mostly a construction that masks the reality of what’s really happening in our neural machinery. Our subjective experience feels linear and sequential; even if our minds appear to jump rapidly from one thought to another, we only ever seem to be thinking one thought at a time, just as we can only really look at or attend to one thing at a time (even though our eyes might be darting all over the place). In reality, I think our mental apparatus is massively parallel, and it’s only at some very high level where our consciousness resides that we impose a single-threaded integration of all of those parallel processes running underneath^[9].

So, the self is an illusion, yes, but a very useful one, as I will describe in the next section.

Identity

Maybe it’s a mistake to try and draw a line between "self" (which I’m using above to refer to the construct that we maintain in our moment-to-moment experience) and "identity", but I’m going to try.

The self may be an illusion, but the concept of a real stable identity is an obviously necessary and useful conceit. Your conciousness may be an emergent phenomenon relying on the operation of an incredibly complex biological machine, but as part of society it is evidently very handy that we assign ourselves names, addresses, driving license numbers, and so on.

The bank probably doesn’t care if you believe in free will, or even if you’ve mediated so much that you’ve started to question the boundary between your own organism and the broader world in which is it is embedded^[10], but it sure as hell wants to know where you live and a bunch of other "metadata" about you before it will loan you the money to buy a house. And in your relationship with your family, your friends, your employer, and the people you interact with in your neighborhood, it is anywhere from "handy" to "essential" that you present a stable identity to the world that is sustained over time.

If you’ve gotten this far, it will come as no surprise to you to learn that I think this kind of thing is an illusion of sorts too. I’m not even referring to the fact that most of the cells in our body gradually turn over^[11], turning us into living, breathing Ships of Theseus.

I’m also not referring to the fact that the "you" you see in the mirror slowly ages and changes, such that you can look at photos of yourself from 10 years before and marvel at how different you look.

Rather, I’m talking about how susceptible to change your internal model of yourself is. Part of that is related to memory and forgetting, which seems to affect us all as we age, even those of us who never fall prey to actual dementia. I think the Inside Out movies do a great job of fictionalizing aspects of something that we’ve all experienced: the role of "core memories" and "core beliefs" in sustaining and defining a personality (and how these can be eroded or replaced over time); how things get shunted into long term memory; how things get forgotten or lost forever.

Conclusion

I want to say that the big three topics that I’ve addressed in this blog post arrived one-by-one in my life, and in the order that I’ve discussed here. But, I don’t know if I can trust my memory about that.

You see, one thing I’ve had a growing awareness of is that my memory is changing as time goes by. Old friends sometimes recount episodes of things we did together that I have literally no recollection of although they do sound like things that I might have done. In general, my impression is that among my friends I am probably the one who remembers the least. And this was a bit surprising to me, because I had always prided myself on my nearly photographic memory^[12] back in my university days, which had seemed to me at the time like a totally unfair advantage.

I don’t think this is all about aging, although some of it might be. As an avid computer user, an early internet adopter, and assiduous archiver and note-taker, I have observed in myself a tendency to remember less and less as time goes by, instead developing skills and practices that enable me to find information again when I need it. That always struck me as a learned adaptation to my environment as opposed to a change in the functionality of my memory. But there has also been a sense of "overflowing", that my brain somehow filled up long ago and no longer makes much of an effort to retain new information. Nowadays when I see movies, it’s common for me to forget basic elements of the plot only weeks after seeing them. When I was younger I could give detailed recounts of entire scripts, but now I basically just remember whether I liked the movie or not.

It’s reflecting on memories that prompted me to finally finish this blog post. I created this file, blog/free-will.md over two years ago with the intent of saying something about it, but never made a start until a couple of days ago. The thing that pushed me to actually do it was an experience last week wherein I was trying to account to somebody why it is that it is so hard for me to focus on myself, and why I love "flow" states so much, because they allow me to completely lose myself in something "other"^[13].

The first order explanation is that I find those states subjectively pleasant, but the second order one is that I like losing myself because I don’t necessarily have a very strong sense of identity. And when I dig deeper trying to understand why that might be, I have to admit that I’m skeptical of the reliability of my own memories^[14], and I’ve spent a lifetime pursuing distractions that took over me and had me thinking about anything but myself.

There are contradictions and ambiguities to unknot here — like why I would spend so much time analyzing my own operation, why I would feel so "motivated" to do so even when I quite clearly don’t believe in free will to do anything at all — but the nice thing about a belief in determinism is that I don’t have to feel bad about shrugging my shoulders and ending abruptly. After all, it couldn’t have been any other way…

---

1. Quite possibly the most useless non-qualification of all. ↩︎

2. The Wikipedia page on Free will makes a good start though. ↩︎

3. When I say "more or less" here I’m not questioning that determinism is real at the level of the laws of physics. When you zoom in to the quantum level, there are certainly stochastic processes that we can’t predict the outcomes of (eg. we don’t know what the result of a measurement is going to be, although we can assign probabilities to different outcomes), but the laws themselves — like, evolving a wave function according to the Schrödinger equation — robustly define what happens to the system given a set of initial conditions. The "more or less" I was actually intending here was a reference to the unreliability of these human machines: cells degrade, DNA gets mutated, we age, and if our lives can be a thought of as a quadrillions of computational steps carried out by a machine comprised of trillions of cells, it’s to be expected that some of the operations are not performed exactly as expected. Nevertheless, such imperfections mean nothing for the status of our will, free or otherwise. ↩︎

4. Winner of the Nobel prize in Physics in 2020 for his work on black hole formation. ↩︎

5. An analogy I like to draw here is with the Efficient Market Hypothesis. I don’t think that markets really are efficient, but the most useful (and therefore rational) thing for most of us to do is to behave as though they were. ↩︎

6. So many fun rabbit holes to go down here, leading to interesting (but in my opinion implausible) places like panpsychism and such. ↩︎

7. Perhaps something only a compatibilist could accept. ↩︎

8. We do this at a very high level when we tell stories about the world, or create symbolic edifices that seek to explain and illustrate truths about reality, but we also do this automatically on a moment-to-moment basis as we integrate sensorial data and just go about our business of existing. ↩︎

9. This is how I explain one of my favorite surreal experiences, that of the "premonition". I routinely feel like I know what song is about to come on next in shuffle play, and when it does, I get the strong impression that I somehow started hearing it before I actually heard it. I don’t think this means I have the ability to see (hear) the future, though. Rather, I think the part of my brain that’s responsible for integrating sensory data and for presenting a linear thread to my subjective experience is having some kind of hiccup, and it ends up presenting events out of order (imagining that I had a premonition, when I really just heard the next song popping out of the shuffled playlist). ↩︎

10. Unless, perhaps, the mediation interferes with your ability to hold down a job. ↩︎

11. I’m no biologist, but my understanding is that some cells turn over very rapidly indeed — with the skin, hair, and liver tissue being among the fastest, on the order of weeks — while others are much slower or don’t turn over at all (neurons, for example). ↩︎

12. I say "nearly photographic" because I didn’t seem to have a universal ability to recall what I had seen, but rather I had a specialized version of this where I could rapidly find again something I had read in the past because I had a spatial recollection of where the words I was looking for had appeared (both in terms of how far into the book, but also where on the page I was to find them). ↩︎

13. And the reason I was having that conversation was because I’d had a weird experience a night or two before, when I’d been lying in bed around dawn and for no apparent reasons had suddenly remembered something I hadn’t thought about for over a decade and had in a sense "forgotten" (although evidently not completely), and which in turn made me question one of the "core beliefs" that I had come to hold about myself. ↩︎

14. I’m not talking here about confabulation or hallucination or anything like that, but simply a kind of gradual amnesia that slowly sets in over time, leaving larger and larger lakes of darkness. ↩︎

By the time you read this, if you read this, it’s because I’ve transferred the old domain, wincent.com, to a new owner^[1], and moved all of my stuff over to a new domain: wincent.dev. (Technically, I actually moved the stuff a month or so before the transfer, and set up redirects so that anybody who visited a page on wincent.com got redirected to the equivalent page on wincent.dev.) As I said on Twitter/X, I’m sorry for any hassle caused by the switch! In this post I’m going to give a bit of background on why I decided to make this change, but before I get into that, here’s a summary of what you should do:

1. Update your bookmarks and links.
2. If you cloned something from git.wincent.com, update the URL to git.wincent.dev instead.
3. If you have a wincent.com email address in your contacts, update it to investor.relations@hurrell.net instead^[2].

Details

Web links

Please update any bookmarks you may have for wincent.com to point at wincent.dev instead.

I hate broken links, and I’m routinely disappointed to look at articles I wrote 10 years ago — and sometimes much more recently than that — and find that almost every link to an external site has succumbed to bit-rot and now 404s^[3] or goes to the completely wrong place. As far as internal links go, however, I’ve gone to some effort to keep links working over the years, setting up redirects when necessary. Some things have slipped through the cracks, but overall, my internal links have been pretty robust over the years.

Well, this time it’s my turn to be the bad guy: now that the domain name transfer has taken place, all of a sudden there are a bunch of links out there on the web that no longer work, and there’s nothing I can do to redirect them because I’ll no longer control the domain. Some of these links I have the ability to edit and have already updated (eg. links from descriptions on videos on my YouTube channel, links from READMEs on GitHub projects etc), but there are many others that are controlled by third-parties and are going to stop working. Breaking all those is not something I take lightly, but the deal presented itself on favorable enough terms that I had to conclude that accepting it made sense^[4].

For the most part, I think search engines will be relatively quick to pick up pages at their new locations. As an example, soon after setting up the redirects I did a search for "understanding rebase conflict markers" at Kagi, and it immediately showed the new domain. The same search on Google^[5], however, was showing the page at the old domain. A few hours later, it caught up.

Git

Another issue is that many people have made clones of Git repositories hosted at git.wincent.com. These should all be updated to point at git.wincent.dev instead (or heck, just switch to GitHub URLs, which won’t be changing any time soon). Example commands to update the remote for a project (in this case, Command-T) are:

# To set the "origin" remote's URL to git.wincent.dev:
git remote set-url origin git.wincent.dev:public/command-t.git

# To set the "origin" remote's URL to GitHub:
git remote set-url origin git@github.com:wincent/command-t.git

Email

I mostly stopped using wincent.com email accounts years ago^[6], so there is unlikely to be any change needed here for most folks.

The future

I can’t promise that wincent.dev will be around forever, but my plan is to keep it around for as long as I can (that means, as long as I’m alive and of sound mind, and able to pay the bill!). I chose it as a replacement for wincent.com for three reasons:

1. I have a lot of muscle-memory for typing "wincent followed by a three-letter TLD^[7]". I figure I should be able to retrain myself to type wincent.dev pretty quickly^[8].
2. I’m not aware of any other development-focused "Wincent" in the world, either a person or corporation, that is likely to want the dot-dev domain in the future. As such, it is a reasonably safe bet that I won’t receive any offers to buy it and it should be easy enough to retain it effectively indefinitely.
3. Registration fees for dot-dev domain names are, for now at least, in the same ballpark as for dot-com domains.

That’s a wrap

And that brings us to the end. I don’t know how many folks will find this post, but I hope that at least some get to see it. Thank you all for your understanding!

---

1. Fun fact: I didn’t actually know who the new owner was until after the sale, because they acted through an intermediary. My guesses were (not necessarily in order):

   1. A Swedish drum stick manufacturer running their site at https://www.wincent.se/
   2. A German singer running his site at https://www.wincentweiss.de/
   3. A crypto start-up running their site at https://www.wincent.co/
   4. A gun safe manufacturer running their site at https://www.wincentpro.com/

   In the end, it was the crypto start-up. ↩︎

2. This last one is a joke. Or is it? 😉 ↩︎

3. "404s" — a neologism for returning a "404 Not Found" HTTP status code. ↩︎

4. There’s an emotional component to this as well, because some of my most important online activity for the last 20+ years has taken place on wincent.com (I originally acquired it back in May 2003). And the dot-org, wincent.org, was the first domain name I ever registered (on 18 September 1999, unless I’m mistaken). Since then, I’ve been using wincent as a handle in a bunch of places, some of them quite meaningful (eg. GitHub and Twitter, among others). I built up my blog to the point where I was getting 50,000 page views per day (in the lead-up to the first public release of Mac OS X). I made a living writing and selling software products under the Wincent brand, hosted on the dot-com domain. The path that eventually led to me getting hired at a Bay Area startup, getting hired later by Facebook, and later on by GitHub, started with Wincent. It was only after weeks of deliberation that I actually got to the point where I felt comfortable with proceeding with the sale, and the negotiations spanned months. ↩︎

5. Google is a search engine that aims to "organize and make accessible the world’s information". You may have heard of it, but in case you haven’t, I have included a "hypertext" link that you can click on and check it out. ↩︎

6. I did have one poor email user who had been using a wincent.com email account intensively, as their primary address, for many years, though, and for her the migration was painful. 😔 ↩︎

7. Top-Level Domain. ↩︎

8. Funnily enough, I still own wincent.org and wincent.net, and those are also three-letter TLDs, but given that so much of the content I post and host is development-related, the dot-dev seemed like the best fit. Take, for instance, git.wincent.dev, which seems somehow better than git.wincent.org or git.wincent.net for hosting Git repositories. 🤷 ↩︎

Once in a while I take a stab at a big, uncertain topic — like COVID or Bitcoin — as a way of recording a snapshot of my thinking. Now it’s time to do the same for AI (Artificial Intelligence), in a post that will surely be massively out-of-date almost as soon as I’ve published it. Be though, as it may, a doomed enterprise, I still want to do this, if nothing else because my own job as a software engineer is one of the ones that is most likely to be dramatically affected by the rise of AI. And while I could go on about image and video generation or any of a number of other applications of the new wave of AI products, I’m mostly going to focus on the area that is currently most relevant to the business of software engineering; that is, LLMs (Large Language Models) as applied to tasks in and around software development.

The current state of AI in software development

In the world of programming, LLMs are being crammed and wedged into every available gap. I say "crammed" because the textual, conversational model doesn’t necessarily always feel like a natural fit within our existing user interfaces. Products like GitHub Copilot seek to make the interaction as natural as possible — for example, proposing completions for you when you do something like type a comment describing what the code should do — but fundamentally the LLM paradigm imposes a turn-based, conversational interaction pattern. You ask for something by constructing a prompt, and the LLM provides a (hopefully) reasonable continuation. In various places you see products trying to make this interaction seem more seamless and less turn-like — sometimes the AI agent is hidden behind a button, a menu, or a keyboard shortcut — but I generally find these attempts to be clumsy and intrusive.

And how good is this state of affairs? At the time of writing, the answer is "it depends". There are times when it can produce appallingly buggy but reasonable-seeming code (note: humans can do this too), and others where it knocks out exactly what you would have written yourself, given enough time. Use cases that have felt anywhere from "good" to "great" for me have been things like:

1. Low-stakes stuff like Bash and Zsh scripts for local development. Shell scripts that run locally, using trusted input only, doing not-mission-critical things. Shells have all sorts of esoteric features and hard-to-remember syntax that an LLM can generally churn out quite rapidly; and even if it doesn’t work, the code it gives you is often close enough that it can give you an idea of what to do, or a hint about what part of the manual page you should be reading to find out about, say, a particular parameter expansion feature. The conversational model lends itself well to clarifying questions too. You might ask it to give you the incantation needed for your fancy shell prompt, and when it gives you something that looks indistinguishable from random noise, you can ask it to explain each part.
2. React components. Once again, for low-stakes things (side-projects, for example), the LLM is going to do just fine here. I remember using an LLM after a period of many months of not doing React, and it helped me rapidly flesh out things like Error Boundary components that I would otherwise have had to read up on in order to refresh my memory.
3. Dream interpretation. Ok, so I snuck in a non-programming use case. If you’ve ever had a weird dream and asked Google for help interpreting it, you’ll find yourself with nothing more than a bunch of links to low-quality "listicles" and SEO-motivated goop that you’ll have to wade into like a swamp, with little hope of actually coming out with useful answers; ask an LLM on the other hand, and you’ll obtain directed, on-point answers of a calibre equal to that of an experienced charlatan professional dream interpreter.
4. Writing tests. Tests are often tedious things filled with painful boilerplate, but you want them to be that way (ie. if they fail, you want to be able to jump straight to the failing test and be able to read it straightforwardly from top to bottom, as opposed to having to jump through hoops reverse-engineering layers of cleverness and indirection). An LLM is good for churning out these things, and the risk of it hallucinating and producing something that doesn’t actually verify the correct behavior is far more benign than a comparable flaw making it into the implementation code that’s going to run in production. The bar is lower here because humans are at least as capable of writing bad tests as LLMs are. This is probably because it’s harder to ship a flagrant but undetected implementation bug because if anybody actually uses the software then the bug will be flushed out in short order: on the other hand, all manner of disgusting tests can get shipped and live on for extended periods in a test suite as long was they remain green. We’ve all seen ostensibly green tests that ended up verifying the wrong behavior, not verifying anything meaningful at all, or being mere facsimiles of the form and structure of the thing they purport to test, but utterly failing to express, exercise, specify, or constrain the expected behavior.

But it’s not all roses. One of the problems with LLMs is they’re only as good as the data used to train them. So, given a huge corpus of code written by humans (code with bugs), it’s only to be expected that LLM code can be buggy too. The dark art of tuning models can only get you so far, and curating the training data is hard to scale-up without a kind of chicken-and-egg problem in which you rely on (untrustworthy) AI to select the best training material to feed into your AI model. In my first experiences with LLMs, I found they had two main failure modes: one was producing something that looks reasonable, appears to be what I asked for, and is indeed "correct", but was subtly ill-suited for the task; the other was producing code that again had the right shape that I’d expect to see in a solution, but which actually had some fatal bug or flaw (ie. is objectively "incorrect"). This means you have to be skeptical of everything that comes out of an LLM; just because the tool seemed "confident" about it is no guarantee of it actually being any good! And as anybody who has had an interaction with an LLM has seen, the apparent confidence with which they answer your questions is the flimsiest of veneers, rapidly blown away by the slightest puff of questioning air:

Programmer: Give me a function that sorts this list in descending order, lexicographically and case-insensitively.

Copilot: Sure thing, the function you ask for can be composed of the following elements… (shows and explains function in great detail).

Programmer: This function sorts the list in ascending order.

Copilot: Oh yes, that is correct. My apologies for farting out broken garbage like that. To correct the function, we must do the following…

In practice, the double-edge sword of current LLMs mean that I mostly don’t use tools like GitHub Copilot in my day-to-day work, but I do make light use of ChatGPT like I described in a recent YouTube video. As I’ve hinted at already, I’m more likely to use LLMs for low-stakes things (local scripts, tests), and only ever as a scaffolding that I then scrutinize as closely or more closely than I would code from a human colleague. Sadly, when I observe my own colleagues’ usage of Copilot I see that not everybody shares my cautious skepticism; some people are wary of the quality of LLM-generated code and vet it carefully, but others gushingly accept whatever reasonable-seeming hallucination it sharts out.

One thing I’m all too keenly aware of right now is that my approach to code review will need to change. When I look at a PR, I still look at it with the eyes of a human who thinks they are reading code written by another human. I allow all sorts of circumstantial factors to influence my level of attention (who wrote the code? what do I know about their strengths, weaknesses, and goals? etc), and I rarely stop to think and realize that some or all of what I’m reviewing may actually have been churned out by a machine. I’m sure this awareness will come naturally to me over time, but for now I have to make a conscious effort in order to maintain that awareness.

Am I worried about losing my job?

I’m notoriously bad at predicting the future, but it seems it would be derelict of me not to at least contemplate the possibility of workforce reductions in the face of the rise of the AI juggernaut. I don’t think any LLM currently can consistently produce the kind of results I’d expect of a skilled colleague, but it’s certainly possible that that could change within a relatively short time-scale. It seems that right now the prudent cause is to judiciously use AI to get your job done faster, allowing you to focus on the parts where you can clearly add more value than the machine can.

At the moment, LLMs are nowhere near being able to do the hard parts of my job, precisely because those parts require me to keep and access a huge amount of context that is not readily accessible to the machine itself. In my daily work, I routinely have to analyze and understand information coming from local sources (source code files, diffs) and other sources spread out spatially and temporally across Git repos (commit messages from different points in history, files spread across repositories and organizations), pull requests, issues, Google Docs, Slack conversations, documentation, lore, and many other places. It’s only a matter of time before we’ll be able to provide our LLMs with enough of that context for them to become competitive with a competent human when it comes to those tricky bug fixes, nuanced feature decisions, and cross-cutting changes that require awareness not just of code but also of how distributed systems, teams, and processes are structured.

It’s quite possible that, as with other forms of automation, AI will displace humans when it comes to the low-level tasks, but leave room "up top" for human decision-makers to specialize in high-leverage activities. That is, humans getting the machines to do their bidding, or using the machine to imbue them with apparent "superpowers" to get stuff done more quickly. Does this mean that the number of programming jobs will go down? Or that we’ll just find new things — or harder things — to build with all that new capacity? Will it change the job market, compensation levels, supply and demand? I don’t have the answer to any of those questions, but it makes sense to remain alert and seek to keep pace with developments so as not to be left behind.

Where will this take us all?

There have been some Twitter memes going around about how AI is capable of churning out essentially unreadable code, and how we may find ourselves in a future where we no longer understand the systems that we maintain. To an extent, it’s already true that we have systems large and complicated enough that they are impossible for any one person to understand exhaustively, but AI might be able to build something considerably worse: code that compiles and apparently behaves as desired but is nevertheless not even readable at the local level, when parts of it are examined in isolation. Imagine a future where, in the same way that we don’t really know how LLMs "think", they write software systems for us that we also can’t explain. I don’t really want to live in a world like that (too scary), although it may be that that way lies the path to game-changing achievements like faster-than-light travel, usable fusion energy, room-temperate superconductors and so on. I think that at least in the short term we humans have to impose the discipline required to ensure that LLMs are used for "good", in the sense of producing readable, maintainable code. The end goal should be that LLMs help us to write the best software that we can, the kind of software we’d expect an expert human practitioner to produce. I am in no hurry to rush forwards into a brave new world where genius machines spit out magical software objects that I can’t pull apart, understand, or aspire to build myself.

The other thing I am worried about is what’s going to happen once the volume of published code produced by LLMs exceeds that produced by humans, especially given that we don’t have a good way of indicating the provenance of any particular piece — everything is becoming increasingly mixed up, and it is probably already too late to hope to rigorously label it all. I honestly don’t know how we’ll train models to produce "code that does X" once our training data becomes dominated by machine-generated examples of "code that does X". The possibility that we might converge inescapably on suboptimal implementations is just as concerning as the contrary possibility (that we might see convergence in the direction of ever greater quality and perfection) is exciting. There could well be an inflection point somewhere up ahead, if not a singularity, beyond which all hope of making useful predictions breaks down.

Where would this take us all in an ideal world?

At the moment, I see LLMs being used for many programming-adjacent applications; for example, AI-summarization. There is something about these summaries that drains my soul. They end up being so pedestrian, so bland. I would rather read a thoughtful PR description written by a human than a mind-numbingly plain AI summary any day. Yet, in the mad rush to lead the race into the new frontier lands, companies are ramming things like summarization tools down our throats with the promise of productivity, in the hope of becoming winners in the AI gold rush.

Sadly, I don’t think the forces of free-market capitalism are going to drive AI towards the kinds of applications I really want, at least not in the short term, but here is a little wish list:

• I’d like the autocomplete on my phone to be actually useful as opposed to excruciating. Relatedly, I’d like speech-to-text to be at least as good at hearing what I’m saying as a human listener. Even after all these years, our existing implementations feel like they’ve reached some kind of local maximum beyond which progress is exponentially harder. 99% of all messages I type on my phone require me to backspace and correct at least once. As things currently stand, I can’t imagine ever trusting a speech-to-text artifact without carefully reviewing it.
• Instead of a web populated with unbounded expanses of soulless, AI-generated fluff, I want a search engine that can guide me towards the very best human-generated content. Instead of a dull AI summary, I’d like an AI that found, arranged, and quoted the best human content for me, in the same way a scholar or a librarian might curate the best academic source material.
• If I must have an AI pair-programmer, I’d want it to be a whole lot more like a skilled colleague than things like Copilot currently are. Right now they feel like a student that’s trying to game the system, producing answers that will get them the necessary marks and not deeply thinking and caring about producing the right answer^[1].
• AI can be useful not just for guiding one towards the best information on the public internet. Even on my personal computing device, I already have an unmanageably large quantity of data. Consider, for example, the 50,000 photos I have on my laptop, taken over the last 20 years. I’d like a trustworthy ally that I can rely on to sort and classify these; not the relatively superficial things like face detection that software has been able to do for a while now, but something capable of reliably doing things like "thinning" the photo library guided only by vague instructions like "reduce the amount of near-duplication in here by identifying groups of similar photos taken around the same time and place, and keep the best ones, discarding the others". Basically, the kind of careful sorting you could do yourself if only you had a spare few dozen hours and the patience and resolve to actually get through it all.

I’m bracing myself for a period of intensive upheaval, and I’m not necessarily expecting any of this transformation to lead humanity into an actually-better place. Will AI make us happier? I’m not holding my breath. I’d give this an excitement score of 4 out of 10. For comparison, my feelings around the birth of personal computing (say, in the 1980s) were a 10 out of 10, and the mainstream arrival of the internet (the 1990s) were a 9 out of 10. But to end on a positive note, I will say that we’ll probably continue to have some beautiful, monumental, human-made software achievements to be proud of and to continue using into the foreseeable future (that is, during my lifetime): things like Git, for example. I’m going to cherish those while I still can.

---

1. And yes, I know I’m anthropomorphizing AI agents by using words like "thinking". At the moment we have only a primitive understanding of how consciousness works, but it seems clear to me that in a finite timespan, machines will certainly pass all the tests that we might subject them to in order to determine whether they are conscious. At that point, the distinction becomes meaningless: what is conciousness? It’s that thing that agents who appear to have conciousness have. ↩︎