We Bought the Whole Gpu, So We're Damn Well Going to Use the Whole GPU

It's not just being sure exactly what the hardware is, in demos you have the additional luxury of not being interactive. So you can plan everything exactly out in advance.

Consoles are pretty heterogeneous IRL too, though. You have multiple SKUs (regular and Pro, for example), not to mention most games will also target multiple consoles (PlayStation + Xbox + Switch is a common combo).

So in reality the opportunities to really code against a specific piece of hardware are few and far between...

Heck, then you get into multiple operating modes of the same hardware - the Nintendo Switch has a different perf profile if it's docked vs. not.

That's what you got with BeOS...throw out backward compatibility and build to current Best practices...it's ability to extract performance out of an 133 Mhz processor was amazing.

>Sometimes I dream of what the world would do if we were mystically stuck on exactly the processors we have today, for twenty years.

Reminds me of the old American cars in Cuba - https://en.wikipedia.org/wiki/Yank_tank

> The problem is that even for things like consoles, it's usually more "cost efficient" to write normal fast-to-write code that isn't maximally effective, let the compiler do its magic, and call it good enough.

Given all the time and money, there's also a skills gap.

I've wondered sometimes what software would look like if a crisis took out the ability to build new semiconductors and we had to run all our computing infrastructure on chips salvaged from pregnancy tests, shoplifting tags, cars, old PCs, and other consumer electronics. We'd basically move backwards about 20 years in process technology, and most computers would have speeds roughly equivalent to 90s/00s PCs.

But then, this still wouldn't incentivize building directly to the hardware, because of the need to run on a large variety of different hardware. You're still better off preferencing portability over performance, and then making it up by cutting scope and ease of development.

Optimizing for the hardware you are on is demonstrably an effort and skill issue. Everyone understands that with enough time and engineers, any piece of software could be optimized better. If only we had large volumes of inexpensive "intelligence" to throw at the problem.

This is one of my back-of-mind hopes for AI. Enlist computers as our allies in making computer software faster. Imagine if you could hand a computer brain your code, and ask it to just make the program faster. It becomes a form of RL problem, where the criteria are 1) a functionally equivalent program 2) that is faster.

  The problem is that even for things like consoles, it's usually more "cost efficient" to write normal fast-to-write code that isn't maximally effective, let the compiler do its magic, and call it good enough.

This wasn't always the case. I have a friend who used work on games for the original Playstation. I remember him telling me that part of his job (or maybe his whole job) was optimizing the machine code output by the C compiler.

Rust jobs would actually touch more hard tech rather than being concentrated in crypto scams.

For hardware that isn't pre-framebuffer, demos seem to be mostly about hyperoptimizing the code in a portable way, much less optimizing to specific hardware timings and quirks.

Unless it happens to something like a PS3 or Saturn.

> it's usually more "cost efficient" to write normal fast-to-write code that isn't maximally effective, let the compiler do its magic, and call it good enough.

For the last six years my full time job has largely been optimizing games where most of the team has been working with this mindset. Sometimes someone spends a few days of just getting things done, followed by others building on top of it. This leads to systems which are not fast enough and take me weeks or even months to optimize.

We even got together at my last job and created a series of lectures on performance and best practices for everyone, including artists, to get ahead of this type of issues. It was apparently very appreciated, especially among the non technical staff who said it was valuable and they had no idea.

I listened to that episode, by chance, last week. It was well worth the time to listen.

Already been done: https://www.nationalgeographic.com/pages/article/131003-boha...

Awesome! We truly have Transgressed the Boundaries.

(And I'm curious... The way you said "Ben here" makes me wonder if I know you?)

Ben, as in Benjamin Spector? (As noted below by sciurus.)

Please accept my sheepish apology for taking an unwarranted potshot at your writing style.

OTOH, if I hadn't, none of us would have known about that remarkable connection!

I'm very willing to believe that. When I hear that they just don't have enough staff for it I get the impression is that they set their hiring bar for engineers too high. Optimising CUDA is quite different from having experience training LLMs.

This is what killed IBM PowerPC in the ML market. Tried to get in with a faster CPU with NVLink embedded hoping that would win market share. But what won wasn't a faster machine or better architecture. A platform with more developers that has fewer bugs and everyone knows wins almost all the time. ML/AI developers are less rare today but still rare.

Very real.

https://www.usenix.org/system/files/usenixsecurity24-guo-yan...

https://www.sciencedirect.com/science/article/pii/S016740482...

Contra another comment: fairly low. (Or at least my search-fu has not been able to find any CVEs or published papers about breaking isolation between MIG instances. MPS should be generally be used only by one user so multiple of their own CUDA apps can attach to one (v)GPU.)

MIG is used a lot in HPC and multi-tenancy cloud, where isolation is important. See Figure 1 and §6.2:

* https://docs.nvidia.com/datacenter/tesla/mig-user-guide/

The card is actually sliced into different instances (show up as different /dev/nvidiaXs), each with their own SMs, L2, and DRAM, that are isolated between each one. (MPS is for the same user to share a GPU instance: allows multiple CUDA apps to attach and time-slicing occurs.)

MIG is low, the exploit would be exotic.

MPS should only be used where all the workloads trust each other. It is similar to running multiple games on your computer simultaneously.

You cannot use NVLink with MPS or MIG, it is not isolated, and malformed NVLink messages can be authored in userspace and can crash the whole GPU. Some vendors, like Modal, allow you to request NVLink'd shared GPUs anyway.

MIG only makes sense for cloud providers. MPS only makes sense for interactive (read: not ML) workloads. Workloads needing more than 1 GPU cannot use either.

I remember a few years ago my hardware security professor suggested we try to implement Rowhammer on GPU. I ended up doing something else, but it looks like someone got there: https://arxiv.org/abs/2507.08166

> MIG virtualization is IMHO weak sauce.

I know several HPC sites that use it: they (e.g.) ordered cookie-cutter server designs/models to simplify logistics, but not all of their users need the complete capabilities, and so they slice/dice some portion into smaller instances for smaller jobs.

E.g.:

* https://hpc.njit.edu/MIG/

* https://www.rc.virginia.edu/2025/07/hpc-maintenance-aug-12-2...

> Only seven slices. Seven?

At some point the slices because so small that they stop being useful. An A100 can have as 'little' as 40G of memory, and you're now down to 5G per instance:

* https://docs.nvidia.com/datacenter/tesla/mig-user-guide/#a10...

> Extremely limited hardware support.

It's a reasonable argument that you'd only need it at the top-end of the hardware: the number of workloads that need all that compute and memory are not that common, so downshifting some hardware to resource slices that are more typical is not crazy. Of course you then upshift when needed: but if you had purchased 'smaller' cards because that's what you thought you (initially) needed, then you're stuck at that level. There's no way for you to upshift/de-downshift.

> Difficult to configure - like the early days of CUDA.

How hard is it to run nvidia-smi?

> Meanwhile, don’t forget that if your workloads are cooperative, you can put all the processes you want on a single GPU and they’ll happily multitask. No security boundary of course, but who knows how good MIG is at that.

The security boundary of MIG is lot better than MPS, which basically has no security. I know several folks running HPC clusters that use it to isolate the Slurm workloads of different users. And my search-fu has found no CVEs or published papers jailbreaking out of MIG instances.

> I’d greatly prefer better tools for cooperative GPU sharing like per process memory limits or compute priority levels. Also seems like it should be way easier to implement.

This is what MPS is for:

* https://docs.nvidia.com/deploy/mps/index.html

* https://man.archlinux.org/man/extra/nvidia-utils/nvidia-cuda...

It should definitely be possible already using CUDA or computer shaders. From a theoretical view computer graphics is signal processing but with a signal consisting of up to four color channels across two dimensions. This is the view taken in a lot of papers and practical implementations. After all, a lot of computer graphics is about applying filters (post-processing) such as color grading, anti-aliasing, etc. to this signal.

So, in a very real sense, signal processing is exactly what the GPU is built for and primarily used for.

Yeah, but they could be.

I won an GPU hackathon back in 2019 doing something very similar to this; although the other way around, I was compressing weights using hardware modules.

It seems like this is indeed possible using video codecs: https://arxiv.org/abs/2407.00467v1