Is transformative AGI just around the corner?

Or would we have to break the laws of physics to get there in my lifetime?

Back in the before times (before we were all forced to confront just how fragile, volatile and personality-driven the AI + Tech world has become) I was already reflecting on the extremely ‘wealthiest 1% - centric’ way we’re looking at what counts as ‘astounding new Gen AI developments’.

And the past ‘Game of Thrones -esque’ week if anything cements the banality of much of what we hear talked about. Bill Gates apparently thinks having a digital assistant comparable to a human EA would be era defining.

“Imagine that you want to plan a trip. A travel bot will identify hotels that fit your budget. An agent will know what time of year you’ll be traveling and, based on its knowledge about whether you always try a new destination or like to return to the same place repeatedly, it will be able to suggest locations. When asked, it will recommend things to do based on your interests and propensity for adventure, and it will book reservations at the types of restaurants you would enjoy. If you want this kind of deeply personalized planning today, you need to pay a travel agent and spend time telling them what you want.”

AI is about to completely change how you use computers, Gates Notes, Nov 2023

Maybe to white collar workers or busy parents in affluent areas of the world. However it doesn’t a priori seem helpful to the estimated 65% of the population of developing countries who don’t have broadband internet access yet.

Yes, I am being a wee bit harsh and yes, if you read the full note, I’m cherry picking a little. He does mention healthcare and education along with ‘productivity’ and shopping.

Although when I looked at his education example, the widely touted AI Tutor, Khanmigo from Khan Academy (an organisation I’ve admired for many years for a steadfast commitment to making world class education free for everyone … with a computer and an internet connection), I was a little taken aback to discover that the new AI tutor is not actually free or even freely available. It’s a paid premium service and even to qualify to pay for it, you have to be resident in the US. The cost makes sense of course - they might be getting the API access at cost from OpenAI (not saying they are but it would be great PR) but they’ve still got to pay the actual compute costs to the cloud provider. (The resident in the US part is a bit more ominous. Education too good to share?)

When will we get to something truely world altering?

So we’ve got this fascinating new thing, that is attracting a LOT of capital and attention and will get better - a little or a lot - over time. But how much better? And over how much time? And better in what dimensions?

Many of the problems we need to solve to give all humans on earth now and in generations to follow even the quality of life a few of us enjoy today seem to require not only next level thinking, which maybe the Gen AI of today opens a window to, but also next level compassion and altruism (just little ‘a’ altruism, nothing to do with EA). Genuinely believing that everyone, regardless of country of origin, religious belief or gender has an equal right to a high standard of living.

And the ability to not be seduced by power and money. We’d have to break / bypass / solve what thus far seems to be a very durable human trait that “power tends to corrupt and absolute power corrupts absolutely”. Unless we would like AGIs to completely replace wetbrain humans, it seems impossible to have confidence that having super smart virtual co-thinkers will solve the seemingly age old problem of power and influence.

It was in this philosophical frame of mind that I stumbled across a deeply readable and thought provoking essay by

Ari Allyn-Feuer

and Ted Sanders. Now fair warning before you click through: the essay is 114 fairly densely written pages and covers a LOT. I read and thoroughly enjoyed the whole thing. If you chose not to, you can at least sample a few of the bits I found highly intriguing in the sections below.

Not AI or AGI but transformative AGI

Ted and Ari are super clear from the outset that they are talking about quantifying the probability of having transformative artificial general intelligence by 2043 (which seems like an odd date until you realise it’s 20 years from now).

The previous not-for-profit board of OpenAI (bless their little cotton socks) in the before times when they had some power, apparently got to decide when OpenAI reached AGI

Jokes aside, my point here is to note their, perhaps unorthodox but at least quite practical definition of AGI, “a highly autonomous system that outperforms humans at most economically valuable work”.

I suspect that definition is reasonably close to what ‘most’ people are expecting when they think about ‘the cool AI that we will have when GPT 6, 7, … is released’.

But it’s likely a long way beyond what many folks in the field are talking about when they say AGI! It’s in fact pretty close to what Ari and Ted refer to as transformative AGI.

AI that can perform nearly all valuable tasks at human cost or less … is a much higher bar than merely massive progress in AI, or even the unambiguous attainment of expensive superhuman AGI or cheap but uneven AGI

Transformative AGI by 2043 is < 1% likely, Ari Allyn-Feuer and Ted Sanders

So many steps to get there!

The scientist and the builder within me LOVES that Ari and Ted don’t engage in either magical thinking or handwaving to jump over the enormous operational and technical challenges that lie between us and ubiquitous, cheap AGI doing the vast majority of all economically valuable work (transformative AGI).

And, to their credit, they bend over backwards to make their working clear, their assumptions stated and their framework available for tinkering.

Transformative AGI by 2043: what will it take?

Are they right? I don’t know! But there is a lot to be constructively curious about and to learn by taking more than a cursory look at their approach.

Many lay folk are assuming that GPTs - generative pre-trained transformers - are ‘the’ algorithm that gets us to transformative AGI. As at time of writing, I don’t sit in that camp as I can’t see a solution to hallucinations coming without something new and hallucinations sink most economically useful applications of Gen AI today. Ari and Ted (notably a researcher at OpenAI when this essay was published) don’t think so either.

For transformative AGI to be achieved by 2043, it will require something new, something we don’t have. A type of AI development that is truly general in character, to handle in a graceful way the edge cases that have stymied narrow AIs, attain traction on problems that current NNs do not get traction on, and thereby permit the development of algorithms for many diverse tasks without ideal conditions like giant datasets of task-specific information, high fidelity simulations for active learning, or billions of dollars of task-specific engineering effort over decades.

They’re a lot closer to the coal face than most commentators. And note that they are actually quite bullish (60% chance) on making those additional algorithmic breakthroughs in 20 years.

On the bearish front, and this is me talking not Ted and Ari, I can’t help wondering if this is also where we might start to see the extraordinary commercial success of AI today (Gen AI) play against advances in the field. If you can make close to $US1MM a year iteratively improving on the transformer architecture, many of the human minds most suited to working on algorithmic improvements will. Not all but a fair chunk. So the mass migration we’ve seen over the past decade from non-commercially focussed university research labs to very commercially motivated for profit company labs might just slow things down. Maybe that’s in the estimate already.

But look at all the other things you need!

So, a 60% chance of awesome new breakthroughs with algorithmic development that enable really AGI-level AI.

This is where the essay starts to get really interesting!

Look at all the other things you need before our world as you know it changes and we all get to lead lives of leisure on UBIs while the machines do all the work. (Cause that’s what happens right? Right?)

All the dimensions are super interesting but I’m going to touch on just three.

(#1) AGI inference costs drop below $25/hr (per human equivalent)

Much is made of the fact that only one AGI needs to learn to do the thing and then all the other AGI clones will be able to do it too. Likely true! But it’s not going to be free.

AGIs will be software which runs on hardware; very complex software which will require powerful hardware to run. Even if all the algorithmic and data problems are solved, and AGI then becomes possible, in order to be transformative AGI, the substrate to train human scale AGIs for many tasks, and perform many billions of hours of inference, must be available and cheap.

As we’ve seen with LLM powered products over the last 12 months, actually running them is surprisingly expensive. That’s why Khanmigo is only for the (comparatively) wealthy until further notice.

Ari and Ted go into a lot of detail to estimate upper and lower bounds on the compute power required for inference of a human / superman level AGI. They come up with eye watering numbers that suggest that we would need to gain five orders of magnitude in cost of computation per unit AGI. And that’s to get costs down to $25/hr, which is of course a whole lot more than most humans working today actually make.

We need to get that five orders of magnitude drop in cost from

• getting more computation for every dollar spent

• driving down the amount of computation needed to do the work

Trying to get the cost of computation down, you run into the fact that Moore’s law is done.

… over 10 years now, the disparity between Moore’s law is 100 times versus four times, and in 15 years, it’s 1,000 times versus eight.

We could keep our head in the sand, but we have to acknowledge the fact that we have to do something different. That’s what it’s really about. If we don’t do something different and we don’t apply a different way of computing, then what’s going to happen is the world’s data centers are going to continue to consume more and more of the world’s total power.

Jensen Huang, NVIDA CEO

You also confront the fact that electricity is just not free. And until we crack a biological substrate for computing we’re going to need substantial amounts of electricity to run AGI, both for compute and for cooling.

The only Hail Mary that feels remotely capable of dropping energy costs by orders of magnitude within years is rapid deployment of breakthrough nuclear fusion. However, even if fusion is made to work (very unlikely), made to work cheaply (very unlikely), and made to work in a way that can quickly reach planet scale (extremely unlikely), there’s still an issue: 100M AGIs each consuming~1GW implies energy consumption of 1e17W, the same order of magnitude as all solar energy hitting the Earth. Such heat would rapidly overheat the planet …

Lastly, even if fusion energy were literally free (which it won’t be), the costs of transmission and distribution and literal data center cabling would still place a non-negligible floor on the retail price paid for electricity.

Transformative AGI by 2043 is < 1% likely, Ari Allyn-Feuer and Ted Sanders

Not small challenges. Fascinating to think about. And not being discussed.

(#2) We massively scale production of chips and power

To have all those AGIs doing all that economically valuable work, you need a lot of the substrate for all the ‘artificial thinking’ to occur on. As far as we know, that will be patterned silicon wafers i.e. computer chips. Where do we make computer chips? In hugely expensive, hugely complex semiconductor fabrication plants or ‘fabs’. And the more compute you need, the longer it takes, as you hit the step function points in scaling, i.e. when one fab reaches capacity and you just have to … build another one.

For individuals…, buying compute is easy. You can sign up for an account on Google Cloud, and within minutes run an A100 GPU for ~$1/hr.

But a lead time of minutes is a privilege that vanishes as soon as your order exceeds inventories. The larger your order, the longer it will take complex semiconductor supply chains to fulfil.

At large enough scales, demand for compute will require construction of extra fabrication plants, which take 3–5 years to build. At the gargantuan scales that might be needed for transformative AGI, expanding the industry by multiples could take more than a decade.

And remember, building fabs and power plants is a logistical challenge rather than an academic one. We know how to do it, it just requires a lot of work, materials and coordination. So proto AGI or even pre transformative AGI seems unlikely to be able to sweep these bottlenecks aside.

(#3) We avoid derailment from wars

This is a super interesting one and leads us back to the challenge that humans are humans and we’re not constrained only by our inability to think faster but also by our messy social interactions.

Semiconductor supply chains are highly complex, highly global, highly entwined with Taiwan and the United States and South Korea (e.g. TSMC, NVIDIA, Samsung, etc.).

China seems pretty keen on absorbing Taiwan already. A massive uptick in demand for semiconductor fabrication isn’t going to cool off that dynamic.

Or say one large, economically dominant nation is seemingly making great progress towards AGI. That seems very likely to freak the heck out of other large nations.

If a US company were on track to put US workers out of their jobs, it’s plausible that the US government would force redistribution of wealth, or even nationalize the AGI technology. But if you were the leader of Russia, how confident would you be that your geopolitical foe would altruistically redistribute its AGI wealth with Russian citizens about to face mass unemployment due to cheaper AGI workers?

Another country developingAGI is a direct threat to your exports and economy at large.

Ari and Ted actually set the probability of delay by war at only 30%. For me, I just can’t un-see this problem. Having spent some time considering the geopolitics of fab placement and control, I’m unsettled and planning to learn more. It doesn’t actually have to get to war to be pretty nasty and disruptive.

But doesn’t the development of AGI just make all these problems go away?

As noted above, better thinking machines won’t just sweep away physical and logistical challenges. And history tells us that it is not what happened with new general purpose technologies in the past. Steam, electricity, computer chips. All amazing! All took decades to have their full effect.

We expect AGI takeoff to be similarly gradual, for a constellation of reasons:

• We expect early AGIs to be extremely expensive and frustratingly unreliable, in ways that impede fast takeoff.

• We expect companies to take years to figure out how to take full advantage of improving AI technology.

• All technologies are limited by physics. Even an AGI that figures out how to build amazing digital metaverses or cancer-curing drugs will still face the same physical constraints when it comes to physical, capital-intensive industries like semiconductor fabrication, or electricity production.

Harder than we thought

I’ve definitely only scratched the surface of the essay in this blog post so next time you have a long trip coming up, I encourage you to download the PDF and dive in.

To close, I love this observation from the essay conclusion

We started this project … with a suspicion that when we reckoned the cascade, the value would be below 10%, but we were surprised to see a small fractional percentage.

… the broad pattern result, p<3%, is a pretty clear consequence of our shared way of thinking about this problem, once systematized.

… it does mean that we have a fair degree of certainty that the fundamental problems around alignment and X-risk are not going to press down on us within the next twenty years; we have time for this critical work to proceed.

It also means that to the extent any form of AI alignment research depends on identifying and halting dangerous AGI work, rather than rendering it fundamentally impossible or disincentivized, the capital-intensive, highly visible events we contemplate here would be a good way to think about the task of identifying dangerous AGI work.

This level of rigour in thinking is rare. I’m personally very appreciative of it and learned a heap from the journey.

Instead of feeling both mildly adversarial and deeply skeptical when folks around me are sure that transformative AGI will emerge in 2025 OR that we have to stop everything now lest we’re all out evolved in a decade, I’m going to have a quick chat with them about semi conductor fabs, global politics and the laws of physics.

Until next time, wishing you all a relaxing and curious week out there in the world.