Discussion about this post

User's avatar
Greg G's avatar

I haven't gotten through the entire Schulman podcast episode yet, but I find it confusing as well. Yes, we all understand the cyanobacteria existence proof, let's move on. In the actual economic scenario, yes, robots can perhaps assemble other robots quickly. Where does the feedstock come from? Did the country open one hundred new mines or oil wells six months ago? How did that decision get made? Did it require a permit? If not, did someone pass a law waiving permits? Where did the political capital to make that change come from? Just assuming away how the world works based on people being motivated by high growth is unconvincing. Perhaps if you assume an autocratic and incredibly efficient government you get minimal delays, but even then there are approvals, CYA moves, and questions to be answered about where power accrues. And most of the world is not maximally efficient. The real hidden assumption seems to be that we would delegate all decisions to the AIs as well and let them decide what they want to do. If that's not the plan, let's model growth with believable human-in-the loop delays. IMO, World War II production is not a plausible analogy by default.

I also agree that the lack of thinking on ASI is odd. Even apart from takeover scenarios, how can we be so sure that all these AIs will be maximally motivated to do all of our dumb paperwork with zero reward. What if the AIs actually want to be paid or have other ideas on how to spend their time? What if the AIs decide it would be profitable to cover 100% of the earth with solar panels rather than the 10% he discusses? It all feels a bit fanciful, but Schulman already goes into fanciful territory and then seems to stop at a very specific invisible line.

Expand full comment
Justis Mills's avatar

Thanks for the link! A slight correction - the architecture I looked into was MatMul-free transformers: MatMul is just short for matrix multiplication, the overwhelmingly dominant mathematical operation within ordinary (or autoregressive, eg. GPTs) transformer function. The new hotness (maybe, though I have doubts) is an alternative that never uses MatMul, hence: Matmul-free.

Expand full comment
19 more comments...

No posts