The Engine With No CUDA

Part of a series on building Vinver, solo, from the manifold to the microcontroller. The cognition posts showed a mind that reasons in geometry. This is the GPU engine underneath it, and the reason the whole stack runs on any GPU.

My entire AI stack has never run a line of CUDA.

The manifold, the encoders, the frozen language model that speaks from geometry, all of it runs on an AMD gaming GPU, and it would run on an Apple laptop or an Intel chip without changing a line, because I wrote the compute engine on wgpu. That was not an aesthetic choice or a protest. It was the only way to get the two things I actually needed, and I want to be honest that for most people it would be the wrong choice.

You probably should not write your own engine

Let me say the sensible thing first. Writing your own inference engine is almost always a mistake. vLLM, llama.cpp, and TensorRT exist, they are fast, and they are maintained by people whose full-time job is making them faster. If you are running a standard transformer on Nvidia hardware, use one of them and do not look back.

I did not, for two reasons, and only because both were true at once.

Reason one: CUDA is a strategic cage

Almost all AI infrastructure assumes Nvidia. The fast kernels are CUDA, the tutorials are CUDA, the cloud instances are CUDA. That means the entire compute layer of whatever you build inherits a single-vendor dependency: one manufacturer's supply, one manufacturer's pricing, one manufacturer's roadmap. For a company whose whole reason to exist is a computing substrate, handing the floor of that substrate to one hardware vendor is a bet I did not want to make.

wgpu compiles to Vulkan, Metal, and DX12. The same engine runs on AMD, Apple, Intel, and Nvidia, and points toward the browser through WebGPU. I develop on an AMD Radeon, and the portability is not a slide; it is where I work every day. The day a different accelerator is cheaper or more available, the stack moves with one recompile, not one rewrite.

Reason two: the stack is not a standard transformer

This is the bigger reason, and the one that actually forced my hand. Off-the-shelf engines do one thing extremely well: serve a standard decoder transformer. My stack is not one model. It is a text encoder, a protein-sequence encoder, a protein-structure encoder, a three-dimensional molecular encoder, a graph encoder, and a frozen language model, six different model architectures, all feeding one geometric representation. That a protein and a sentence land in the same space is the thesis, not a stretch: a molecule is physical geometry, shape and position, and flattening it into a text string is the same mistake, one domain over, as flattening a concept into a token. It is the disease this whole architecture exists to cure. On top of the encoders sit the manifold operations and the conditioning that makes the language model think from geometry rather than from a prompt.

No off-the-shelf engine runs that combination. And none of them lets me reach into the model the way the cognition requires, to condition its internal state rather than its input. So the engine was not a preference. It was the price of the architecture.

What it actually is

The engine is wgpu plus a library of compute kernels written in WGSL, the shader language. The ordinary ones are there: matrix multiply, layer normalization, the activation and pooling and embedding operations. The hard ones are the point: flash attention written by hand, with quantized variants so a large context fits in a consumer GPU's memory, and the recurrent kernels for the linear-attention model that the cognition conditions. Six model architectures are implemented on top of that shared set of kernels. A thin runtime wraps a GPU command queue and a pool of scratch memory; each model composes the kernels into a forward pass.

One engine. Six model families. No CUDA, no cuDNN, no vendor library underneath any of it.

The hard parts, which are the proof

Writing this is not gluing libraries together, and the difficult stretches are where the real engineering lives.

Flash attention in WGSL is one. There is no cuDNN to call; you write the tiled, memory-aware attention yourself, and then you write the quantized versions so the context fits at all. The compression scheme there is not mine to claim: it is TurboQuant, the online vector quantization Google published in 2025. What I can claim is the port, getting that scheme into a portable shader so the KV-cache compression the paper ran on a data-center H100 runs on a consumer card from any vendor. Lifting a fresh method off arXiv and onto whatever GPU you happen to have is its own kind of engineering, and it is the kind worth doing. The recurrent kernels are another: the running-state recurrence that the cognition posts kept referring to has to execute on the GPU, portably, which is not a path the shader ecosystem has paved for you.

The quietest hard part is numerical parity. The same model has to produce the same answer as a trusted reference, which means matching a normalization epsilon, the exact error function inside an activation, and the order of floating-point reductions, across vendors whose hardware reorders those reductions differently. A kernel that is correct on one GPU and subtly wrong on another is the kind of bug that does not crash and costs a week, which is a theme I have written about before.

And then the integration problem: one engine serving six genuinely different model shapes without forking into six engines.

What it buys

The payoff is the thing this whole series is about. The entire stack, the cognition, the encoders, the GPU operations the memory layer needs, runs on one consumer GPU, on whatever vendor I happen to have, from one codebase. That is what makes a solo full stack physically possible. I am not maintaining a CUDA build and a Metal build and a ROCm build and praying they agree. I am maintaining one engine that targets all of them, and when I fix a kernel it is fixed everywhere.

What I will not pretend

This is not free, and a careful reader should hear the cost from me.

On an Nvidia GPU running a standard transformer, a vendor-tuned CUDA engine will beat mine. wgpu trades some performance for portability, and I have not chased every last kernel optimization, because for my workload, a two-billion-parameter model plus the encoders plus the manifold on one consumer card, it is already fast enough: on the AMD card I develop on, the conditioned model generates at roughly sixty to seventy tokens a second, faster than I can read. If your job is serving a seventy-billion-parameter model across a fleet of H100s, this is not your engine; use the vendor stack and accept the cage.

The tooling is younger, too. CUDA has twenty years of profilers and libraries. With WGSL you write more by hand and debug with less. That is part of the price of not being owned.

Why it is here, under everything

The engine is the floor. The manifold runs on it, the frozen model runs on it, the databases' GPU work runs on it. It is the least glamorous layer and the one that makes every layer above it portable. It is also a small act of independence: the compute layer of an AI company should not be the property of a GPU vendor, and mine is not.

Next, the memory this engine serves: two databases I built from scratch, because an off-the-shelf index throws the geometry away in exactly the way cosine does.

If you build GPU infrastructure, write shaders, or work in Rust and wgpu, or you're hiring someone who has built an inference engine that runs on anything with a modern GPU, I want to hear from you. [email protected]