Under Scott Aaronson’s recent post “Dispatches from the possibly last days of human relavance,” a user named Bud wrote the following comment that articulated something I had been thinking about:
I find this existentially upsetting. On the one hand we’re going to see results and advancements come thick and fast and to god knows what end. On the other hand, I feel like everything I’d ever worked to understand and to is now moot.
The basic question is: what do we do with ourselves when our intelligence is literally unnecessary? I see myself shriveling into nothing.
Me and Bud are at somewhat opposite positions (or, vectors I guess).
For a bit on my background: I’m 18, and I am currently in the process of figuring out what I want to do with my life. I have a podcast where I interview different intellectuals and thinkers. So, I think I want to do something related to academia. If becoming a tenure track professor is a viable option, I’d do that. But it seems that becoming an “independent intellectual” (á la Alex O’ Connor) is becoming a more feasible path (although, still very difficult and luck-based).
When GPT-4o came out, I stopped doing math for a while. Before that, I would read math textbooks probably for several hours a day. But once I started to realize that an LLM could just solve all the problems I was doing, it sort of lost a lot of spark.
Of course, there is something intrinsically rewarding about doing math. Like, it’s fun. I mostly did it to challenge my brain. But also because I thought I wanted to be a math professor. And I think I realized that there is a non-trivial chance that the role of math professor may not even be an option in the future.
So, in response to this mid-1/5-life crisis, and yes it was a crisis, I paused. This was supposed to be temporary break while I figured out what I was gonna do, or if there was a point. But I ended up getting distracted by school (I realized that I wouldn’t get into college if I kept neglecting my school work in favor of doing math recreationally). And I never really returned to doing math as the rate I was before.
In my recent interview with John Baez, he brought up that getting all our needs taken care of by AI is probably one of the least of our problems. And I think this is true insofar as material needs go. But we also find meaning in being useful. I think a useful analogue is fitness.
Before the industrial revolution, the fitness industry, as we know it today did not exist. Reason being that exercise was not really a separate category from life. If you were a peasant or a hunter-gatherer or whatever, you did not have to go to Planet Fitness after work because work was already Planet Fitness, except it also killed you sometimes and had bad shoes.
And then machines took over a huge chunk of physical labor. Which was good, obviously. I’m not anti-washing machine or anything. I’m also very glad I don’t have to churn butter or die of cholera or whatever. But once the necessity of physical exertion was removed, physical exertion had to be re-invented as recreation, health, aesthetics, discipline, sport, lifestyle, etc.
And similarly, being a “strong person” changed as an identity. In a world where physical strength is necessary, a strong person is useful. In a world where physical strength is mostly unnecessary, a strong person is something else. They are, well, disciplined. Maybe vain, depending on the person. Healthy, usually. My point is that the attribute of strength changed categories from a “useful” thing to mostly being an “identity” thing.
So now we have this very strange thing where people pay money to simulate, in a controlled and mostly pointless way, the types of exertion that people used to be forced to do in order to survive. Such pointless exercises include: lifting a metal object up and down, running in place on a machine, and climbing a fake hill.
And, by the way, nobody thinks it’s stupid! Mostly because it isn’t. I mean, it can be. But fundamentally it isn’t. It is actually a pretty natural response to the fact that our bodies were built for a world that no longer needs them in the same way.
And I think that a similar thing will happen to intelligence. Just like reading went from a thing you did to a thing that some people do, and being strong went from an important thing, to a… different thing, intelligence will now undergo a similar category shift.
For most of history, being smart was/is useful in a very direct way. And because intelligence was/is useful, it acquired moral and social decoration. Smart people were/are not just people who processed information well. They were serious and deep. These were/are the people who could see further. Or at least this was the mythology, which smart people were obviously very happy to believe.
We got rid of the need to use our bodies, and then immediately had to invent a new relationship to our bodies. Or, not immediately. But eventually. The body did not stop wanting to be used just because civilization found a way around it. And I suspect the mind will be the same.
The scary thing is not just that AI will be smarter than us. It is that the identity of being a smart person may become sort of incoherent. Like, what does it mean to be smart in a world where an intelligence that is not you can explain the thing better, faster, and maybe more creatively than you can?
One possibility is that “smart” becomes like “strong.” Nobody thinks a strong person is pointless because forklifts exist. But we also do not think of strength in the same way. A strong person is not strong because he can move the heaviest possible object. Like, he obviously cannot compared to a forklift. The strong person is strong because of what the strength says about the person, and also because of what it does to the person.
So maybe a smart person will no longer be someone who can produce the best answer. That seems like a losing definition, in the same way “a strong person is someone who can beat a crane” is a losing definition. Maybe a smart person is someone who has a certain relation to thought.
There is a difference between knowing the answer to a math problem and having gone through the proof. There is a difference between reading a summary of Kant and having Kant rearrange your brain for two weeks. There is a difference between being able to say the sentence “consciousness is not reducible to computation” and actually feeling why someone might think that, and why someone else might think it is nonsense, and being able to inhabit both for a second.
AI can obviously give you the products of thought. But it is not obvious that it can give you the experience of having thought. Like, a treadmill can move your legs, but it can’t run on your behalf.
The old smart person was, roughly, the person with answers. The new smart person might be the person who is willing to be changed by the attempt to understand.
Micah Zarin's Blog 