The B200, level two

Level one showed that for one user, the chip spends almost all its time reading weights and almost none doing math. Here is the rule behind that.

Every weight read from memory is used once per user. In 8-bit, where a weight is one byte, one user gets two operations (a multiply and an add) out of every byte read. With a hundred users, the same byte buys two hundred, because the weight is multiplied by a hundred different inputs before it is thrown away. Engineers call this ratio arithmetic intensity.

The chip has a ratio of its own: its peak speed divided by its memory speed. For the B200 in 8-bit, that is 4.5 quadrillion operations a second over 8 trillion bytes a second, which is about 560 operations per byte. Each user buys two per byte, so the chip fills up at about 280 users. Below that, memory is the limit. Above it, the tensor cores are. The picture of this is called a roofline.

The crossover comes at 281 users in every format. That is not a coincidence. Each step down in bits halves the bytes per weight and doubles the peak, so the two ratios move together. Switching to 4-bit makes every user's reply faster, but it does not change how many users it takes to fill the chip.

Training is on the flat part from the start. A training step pushes thousands of examples through the model at once, so every weight is used thousands of times per read. That is the simplest reason training and serving look different from outside: one keeps the tensor cores busy, the other keeps them waiting.

For verification Power draw follows the tensor cores. A B200 waiting on memory draws well under its 1,000 watt maximum. A B200 training draws close to it, hour after hour. A data center's power meter is a crude but honest witness to which side of the roofline its chips are on.

NVIDIA's own page for the B200 gives its speed as several different numbers, from about 2 to 20 quadrillion operations a second. All of them are true. They differ in three choices, and a spec sheet does not always say which it made.

The first is the number format, which level one covered. The second is sparsity. If a model is trimmed so that two of every four weights are exactly zero, the tensor cores skip the zeros and go twice as fast. Most models are not trimmed this way, but the sheet often quotes the doubled number. The third is which product the chip is in. The same silicon in the NVL72 rack gets more power and better cooling and runs about ten percent faster than in the air-cooled HGX server.

The biggest number and the smallest differ by a factor of nine. When someone says "a B200 does 20 petaflops", they mean 4-bit, sparse, in the rack. When a researcher says "we trained at 2.2 petaflops per GPU", they mean 16-bit, dense, and they are describing the same chip.

For verification Any rule written in operations per second has to fix all three choices, or it is nine rules. The US export rule does this: it takes the dense number, multiplies by bits, and keeps the largest result, which for the B200 is the same in every format, as level one showed. The EU threshold counts operations actually performed in training, which are almost always dense 16-bit or 8-bit. Neither counts sparsity, which is the right call: a sparse operation that skips a zero is not an operation.

Level one called the SM a workshop. Here is the floor plan.

The chip does not run programs one instruction at a time. It runs warps, groups of 32 threads that march in lockstep, each thread holding a different number. An SM is built in four quarters. Each quarter has a scheduler that picks which warp runs next, a bank of registers, a tensor core, and a row of ordinary cores. The four quarters share one block of fast memory and, new in Blackwell, the tensor memory.

The SM's trick is the same one as batching, at a smaller scale. It keeps up to 64 warps in play at once. When one warp asks for a number from memory and has to wait, the scheduler switches to another in one tick of its clock, a fraction of a nanosecond. There is no cost to the switch, because every warp's registers stay where they are. This is how an SM hides memory delay: not by making memory faster, but by always having something else to do.

Blackwell's tensor cores run asynchronously. A warp hands a block multiply to the tensor core and goes to fetch the next block while the multiply runs. The running total lands in tensor memory, not in registers, which is why the registers are free for the fetch. On Hopper the total lived in registers, and a group of four warps had to stop and wait for it.

For verification The counters that level one mentioned live here. Each SM keeps tallies: instructions issued, tensor core cycles, bytes moved, time stalled. NVIDIA's tools read them for performance tuning. They are the most detailed record of what a chip did, and today they are readable by the chip's owner and nobody else.

Level one said some dies on every wafer come out faulty. Here is how much that matters.

Defects land on a wafer more or less at random, perhaps ten in every hundred square centimetres on a mature process, fewer on a good day. A small die is likely to miss them all. A die the size of a B200's, about 8.6 cm², is not. At one defect per ten square centimetres, a B200 die has about a 42 percent chance of being clean, and the odds fall off fast as the defect rate rises. (The formula is e to the minus defects-per-cm² times area; the calculator shows it.) Try it:

Both inputs are estimates. TSMC does not publish either, and the true defect rate for a mature 4 nm process is probably better than one in ten. The point survives any reasonable inputs: a B200's two dies cost NVIDIA somewhere in the high hundreds of dollars to a little over a thousand, before memory, packaging, testing, design, or profit. The chip sells for something like $30,000 to $40,000. The rest is the eight memory stacks, the package that joins ten pieces of silicon, the years of design, and a margin that is NVIDIA's to set while it has no rival.

For verification The fab is the narrowest point in the whole supply chain. One company makes every B200 die, on machines from one supplier, and it knows how many wafers it started and how many good dies came off. Every serious proposal for counting the world's AI chips starts at the fab for this reason, not at the data center.

A model with 70 billion weights fits on one B200. Training it on one B200 would take years. So training uses many chips at once, and the simplest way to do that is to give every chip a full copy of the model and a different slice of the training data.

Each chip runs its slice and works out, for every weight, which way to nudge it. That is 70 billion numbers, called gradients. The chips then have to average their gradients, so that every copy of the model moves the same way. Every chip needs the average of every number from every other chip, and it needs it before the next step can start. This is the moment that decides whether a cluster is one machine or seventy-two.

The standard way to do it is a ring: each chip sends about twice its gradients' worth of bytes and receives the same. The time that takes depends on how fast one chip can push bytes to its neighbours.

Inside the rack, NVLink moves 900 GB a second out of each chip (level one's 1.8 TB/s counts both directions), and the averaging takes a fraction of a second. Over a good ordinary network it takes eighteen times longer, and the chips spend most of each step waiting. That gap is the whole reason the NVL72 exists. Large runs use several racks, with NVLink inside each and a slower network between them, and arrange the work so that the slow hops carry as little as possible.

For verification The 2022 US export rules used link speed as one test for a controlled chip. NVIDIA built the H800 with its links slowed to get under the line, and the 2023 rules dropped the test, because it was easy to design around. Link speed is still the clearest physical difference between a chip built for training and one built for serving. It just makes a poor line on its own.

The EU AI Act treats a model as a possible source of what it calls systemic risk if its training used more than 10²⁵ operations. The number sounds abstract. It is not. Training compute is close to six times the number of weights times the number of tokens the model read. Two for reading each token (the multiply and the add from level one), and four more for working out which way to nudge each weight. Pick a model:

The calculator assumes the chips run at 40 percent of their peak, which is about what good training runs manage after all the waiting on memory and on each other. The US rules briefly used 10²⁶, ten times higher, before the order that set it was withdrawn.

For verification A threshold in operations becomes, in practice, a threshold in chip-days. A rule that knows how many racks a lab has, and how long they ran, can estimate the operations without seeing the model. That is why so many proposals are about counting chips and metering time, not about inspecting software.

Level one showed attestation as four steps. The real chain has more links, and the links are where the trust lives.

When the GPU powers on, a small processor inside it, the GSP, boots first from code burned into the chip. It loads each piece of firmware in turn and takes a measurement of each before running it. The measurements pile up into a report.

The driver on the host then opens an SPDM session with the GSP. The GPU proves its identity with a certificate that chains back to NVIDIA's root key, and signs the report. Whoever wants to trust the GPU then needs two more things from NVIDIA: the list of approved measurements, a signed file called a RIM, and a check that the certificate has not been revoked. NVIDIA's attestation service does both and hands back a signed token. The customer's software checks the token, and only then releases the keys that decrypt the model and data.

On Hopper, everything that crossed the wire to the GPU had to be encrypted by the CPU first and decrypted by the GPU, through a staging buffer, and that cost speed. Blackwell adds TEE-I/O, which encrypts the link itself, so the GPU can reach the protected memory directly. That is the "little slowdown" from level one.

For verification Today's attestation answers a customer's question: is this cloud provider running my job on a real, unmodified GPU? A verifier asks the opposite question: is this owner doing what they said with their own GPU? The machinery is the same. The trust relationships are reversed, and the hardware was not designed for the reversed case.

Level one showed one checking server. A fast reply proves the chip is near; a slow one proves nothing. With several servers, each fast reply cuts the map down, and the pieces that remain are where the chip can be.

Here are three servers, in northern Virginia, Frankfurt and Singapore. Set the round-trip time each one measures, and the list shows which cities are still possible. The bound uses light in fiber at 200 km each millisecond, so it is generous; a real network is at least a third slower.

Three servers are enough to tell continents apart, and often countries. They cannot tell a data center in Shenzhen from one in Hong Kong, 30 km away, and no number of servers can, because the network's own delays are larger than the difference. The method rules out, it does not pin down.

For verification The chip has to be the one answering. If a stand-in near the server could answer for it, every reply would be fast and the map would lie. NVIDIA's software signs each reply with the key that never leaves the chip, which stops that. What nothing stops is slowness: a chip can always answer late. So the method can prove a chip is near a server. It can never prove a chip is far from one.

Everything above is what exists. These are the main proposals for what could exist, and what each would need from the chip.

Every row has the same shape. The chip already produces the evidence, or could. What is missing is a way for someone other than the owner to receive it and believe it. That is a problem about keys and institutions more than about silicon, which is why people with no engineering background can work on it.