Why 32GB of Blackwell decides which models we run
The hardware under the network is two GPUs and a lot of compromise. What the VRAM ceiling actually rules out, and the CUDA trap that cost us a week.
Six products, a video pipeline, a local model stack and an agent fleet all run on two machines. Neither is a datacenter. The interesting constraint is not raw speed. It is memory, and what it quietly forbids.
What is actually here
The main workstation runs an MSI RTX 5090 Suprim Liquid: 32GB of GDDR7 on a Blackwell die, with a 600W board limit. It sits on an i9-13900K with 24 cores and 64GB of DDR5-6000.
The second machine is a RTX 4070 with 12GB, on an older i7 with 60GB of system memory, running Ubuntu. It handles anything that needs to be up when the main box is busy or rebooting.
There are two laptops. One is still macOS, one has been converted to Ubuntu.
That is the whole fleet. It is a small amount of hardware for the amount of work, and every architectural decision below follows from that.
32GB is the real ceiling
A 5090 sounds like it removes constraints. It does not. It moves them somewhere less obvious.
Video generation is where this bites hardest. A 45-second music generation run peaked at 31.2GB of the available 32. That is not close to the limit, that is the limit, and the run succeeded because nothing else was resident. A single browser tab holding a WebGL context at the wrong moment would have taken it down.
So the rule is that heavy jobs take an exclusive lock. Not a scheduling preference: an actual lock, refusing a second job with a 409 rather than letting two things discover halfway through that they cannot both fit. The alternative is two dead jobs instead of one queued one.
What the ceiling rules out
It rules out the large open-weight models at full precision, which is the obvious part. The less obvious part is that it rules out holding two models at once, and that reshapes pipelines.
The tempting design for a generate-then-critique loop is to keep both models resident and pass data between them. At this memory budget you cannot. So pipelines are staged: load, run, unload, load the next. That costs seconds per stage and buys the ability to run at all.
It also means quantisation is not an optimisation, it is the entry ticket. Anything that does not fit at 4-bit does not run here.
The trap that actually cost time
Blackwell is compute capability sm_120, and that number breaks assumptions.
The specific trap: installing a package that quietly downgrades your GPU framework to a build that predates the architecture. The install succeeds. The import succeeds. Then every kernel fails at runtime with an error that reads like a driver problem, because it is reported as one.
The cause is a transitive dependency pinning an older framework version, and the fix is to refuse that package entirely rather than to fight the resolver. We keep a short list of packages that are banned outright on this hardware for exactly that reason, and a first-run check that asserts the framework build matches the architecture before anything else happens.
The general lesson, which is not specific to this card: when a new architecture ships, the failure mode is not "unsupported", it is "installs cleanly and then lies to you."
Two CUDA majors, on purpose
The box carries two CUDA toolkits side by side. Newer applications want the current one. Several audio and vision runtimes still link against the previous major, and the shared libraries are version-suffixed, so the two genuinely coexist rather than conflicting.
This looks like mess and is actually the cheap answer. The expensive answer is forcing every application onto one major, which means either holding back the new work or rebuilding dependencies that upstream has not migrated.
If you are setting up a machine like this, check which major each application expects before installing it, not after. The libraries are named with the version in them, and a missing one produces an error that names a file rather than the problem.
What we would change
More memory, obviously, but that is not an interesting answer. The interesting one is that we would have built the exclusive lock on day one instead of after the second pair of jobs killed each other. Contention for a single scarce resource is not an edge case on a one-GPU box. It is the normal case, and designing for it later means retrofitting it into every pipeline at once.