The RAM Shortage Comes for Us All

It’s been pointed out by others that price is part of Apple's marketing strategy. You can see that in the trash can Mac Pro, which logically should have gotten cheaper over the ridiculous six years it was on sale with near-unchanged specs. But the marketing message was, "we're selling a $3000 computer."

Those fat margins leave them with a nice buffer. Competing products will get more expensive; Apple's will sit still and look even better by comparison.

We are fortunate that Apple picked last year to make 16gb the new floor, though! And I don't think we're going to see base SSDs get any more generous for a very, very long time.

* okay I do remember that Macbook Airs could be had for $999 for a few years, that disappeared for a while, then came back

Even lower end GPUs are getting more expensive even if they are not really useful for AI. But they still contain <some> chips and ram which is in high demand.

So yes, Apple will likely also have to pay higher priceses when they renew their contracts.

https://upload.wikimedia.org/wikipedia/commons/d/df/Mac_Mini...

That square is the whole M1 package, Apple's custom die is under the heatspreader on the left, and the two blocks on the right are LPDDR packages stacked on top of the main package.

What I can think of is that there may be a push toward training for exclusively search-based rewards so that the model isn't required to compress a large proportion of the internet into their weights. But this is likely to be much slower and come with initial performance costs that frontier model developers will not want to incur.

With a bit of luck OpenAI collapses under its own weight sooner than later, otherwise we're screwed for several years.

For training, their models have a certain number of memory needed to store the parameters, and this memory is touched for every example of every iteration. Big models have 10^12 (>1T )parameters, and with typical values of 10^3 examples per batch, and 10^6 number of iteration. They need ~10^21 memory accesses per run. And they want to do multiple runs.

DDR5 RAM bandwidth is 100G/s = 10^11, Graphics RAM (HBM) is 1T/s = 10^12. By buying the wafer they get to choose which types of memory they get.

10^21 / 10^12 = 10^9s = 30 years of memory access (just to update the model weights), you need to also add a factor 10^1-10^3 to account for the memory access needed for the model computation)

But the good news is that it parallelize extremely well. If you parallelize you 1T parameters, 10^3 times, your run time is brought down to 10^6 s = 12 days. But you need 10^3 *10^12 = 10^15 Bytes of RAM by run for weight update and 10^18 for computation (your 120 billions gigabytes is 10^20, so not so far off).

Are all these memory access technically required : No if you use other algorithms, but more compute and memory is better if money is not a problem.

Is it strategically good to deprive your concurrents from access to memory : Very short-sighted yes.

It's a textbook cornering of the computing market to prevent the emergence of local models, because customers won't be able to buy the minimal RAM necessary to run the models locally even just the inferencing part (not the training). Basically a war on people where little Timmy won't be able to get a RAM stick to play computer games at Xmas.

I'm assuming you wouldn't see it as fine if the corporation was profitable.

> How did we get here?

We've always been there. Not that it makes it right, but that's an issue that is neither simple to fix nor something most law makers are guaranteed to want to fix in the first place.

Nothing in the rules stops you from cornering most markets, and an international companies with enough money can probably corner specific markets if they'd see a matching ROI.

Says a lot about human psychology.

I'll do the engineering so we're good on that front. Just need investors.

According to my research, these machines can etch around 150 wafers per hour and each wafer can fit around 50 top-of-the-line GPUs. This means we can produce around 7500 AI chips per hour. Sell them for $1k a piece. That's $7.5 million per hour in revenue. Run the thing for 3 days and we recover costs.

I'm sure there's more involved but that sounds like a pretty good ROI to me.