Experimentation Is the Real Skill in the Age of AI

Joel Spolsky wrote in his Guerrilla Guide to Interviewing: "You’re going to see three types of people in your interviews. At one end of the scale, there are the unwashed masses, lacking even the most basic skills for this job. They are easy to ferret out and eliminate, often just by asking two or three quick questions. At the other extreme you’ve got your brilliant superstars who write lisp compilers for fun, in a weekend, in Assembler for the Nintendo DS. And in the middle, you have a large number of “maybes” who seem like they might just be able to contribute something. The trick is telling the difference between the superstars and the maybes, because the secret is that you don’t want to hire any of the maybes. Ever."

Even though people can and do change (for better or worse), most of the time, they don't, so Joel has truth in what he's saying.

It's fair to think: "I need to do X, we have John on the team, but John can't handle it reliably (he failed last time), I might just do it myself instead of wasting my time and energy going back and forth with him", and probably you need to fire John or not even hire him in the first place.

The mistake senior knowledge workers and managers tend to make these last years is to apply the same algorithm to AI assistants.

Our mind goes like this: "I need to do X, we have to access to AI, but AI can't handle it reliably (it failed last time), I might just do it myself instead of wasting time and tokens".

But with the fast improvements in AI models and tools, it takes deliberate effort to adjust your expectations and delegate tasks that older models/tools failed to tackle successfully to newer models/tools that MIGHT do a better job.

Experimentation takes time, effort, and some money, but it pays off later. And always remember that innovation is not productive short term.

If you experiment intentionally and regularly, you might get scared seeing all the progress with your own eyes and think of switching careers to harder-to-automate occupations, or seize the opportunity to achieve more either in your current role or in a more advanced role (e.g. building one-man successful businesses from scratch with the help of AI).