Microsoft CTO Says He Wants to Swap Most Amd and Nvidia Gpus for Homemade Chips

I was ranting about this to my friends; Wallstreet is now banking on Tech firms to produce the illusion of growth and returns, rather than repackaging and selling subprime mortgages.

The tech sector seems to have a never ending supply of things to spur investment and growth: cloud computing, saas, mobile, social media, IoT, crypto, Metaverse, and now AI.

Some useful, some not so much.

Tech firms have a lot of pressure to produce growth, it's filled with very smart people, and wields influence on public policy. The flip side is the mortage crisis, at least before it collapsed, got more Americans into home ownership (even if they weren't ready for it). I'm not sure the tech sectors meteoric rise has been as helpful (sentiment of locals in US tech hubs suggests a overall feeling of dissatisfaction with tech)

Yes, in the sense that this is at least partially inspired by Apple's vertical integration playbook, which has now been extended to their own data centers based on custom Apple Silicon¹ and a built-for-purpose, hardened edition of Darwin².

¹ https://security.apple.com/blog/private-cloud-compute/ ² https://en.wikipedia.org/wiki/Darwin_(operating_system)

Arguably that's a GPU? Other than (currently) exotic ways to run LLMs like photonics or giant SRAM tiles there isn't a device that's better at inference than GPUs and they have the benefit that they can be used for training as well. You need the same amount of memory and the same ability to do math as fast as possible whether its inference or training.

Maybe if you restrict it similarly to the Deepseek paper to "Gemini uses TPU for the final successful training run and for scaled inference" you might be correct, but there's no way that GPUs aren't involved for at minimum comparability and more rapid iteration reasons during the extremely buggy and error prone point of getting to the final training run. Certainly the theoretical and algorithmic innovations that are often being done at Google and do make their way into Gemini also sometimes using Nvidia GPUs.

GCP has a lot of, likely on the order of at least 1 million GPUs in their fleet today (I'm likely underestimating). Some of that is used internally and is made available to their engineering staff. What constitutes "deep learning for gemini" is very up to interpretation.

Not really and for the same reason Chinese players like Biren are leapfrogging - much of the workload profile in AI/ML is "embarrassingly parallel", thus reducing the need for individual ASICs to be bleeding edge performant.

If you are able to negotiate competitive fabrication and energy supply deals, you can mass produce your way into providing "good enough" performance.

Finally, the persona who cares about hardware performance in training isn't in the market for cloud offered services.

* a bet on storing the entire model (and code) in 900MB of SRAM. Hell of a lot of SRAM but it only really works for small models and the world wants enormous models.

* Blew it's weirdness budget by a lot. Everything is quite different so it's a significant effort to port software to it. Often you did get a decent speedup (like 2-10x) but I doubt many people thought that was worth the software pain.

* The price was POA so normal people couldn't buy one. (And it would have been too expensive for individuals anyway.) So there was little grass roots community support and research. Nvidia gets that because it's very easy to buy a consumer 4090 or whatever and run AI on it.

* Nvidia were killing it with Grace Hopper.

GC3 was way more ambitious and supports a mountain of DRAM with crazy memory bandwidth so if they ever finish it maybe they'll make a comeback.

I don't know. The transformer architecture uses only a limited number of primitives. Once you have ported those to your new architecture, you're good to go.

Also, Google has been using TPUs for a long time now, and __they__ never hit a brick wall for a lack of CUDA.

You do not need most of CUDA, or most of the GPU functionality, so dedicated chips make sense. It was great to see this theory put to the test in the original llama.cpp stack which showed just what you needed, the tiny llama.c that really shows how little was actually needed and more recently how a small team of engineers at Apple put together MLX.

Is anyone else getting crypto flashbacks?

analog neural network hardware

physical neural network hardware

Put "this paper" after each one to get academic research. Try it with and without that phrase. Also, add "survey" to the next iteration.

The papers that pop up will have the internal jargon the researchers use to describe their work. You can further search with it.

The "this paper," "survey," and internal jargon in various combinations are how I find most CompSci things I share.

It’s never come even close to penciling out in practice.

For small models there are people working on this implemented in flash memory eg Mythic.

We do know: ads, spyware and rounding corners of UI elements.

If their processor work like their software, i really feel pity for people who use it.

https://azure.microsoft.com/en-us/blog/azure-maia-for-the-er...

Will they equip the new Microsoft Vacuum Cleaner with it ? /s