Llama 70B at full precision needs more memory than one 80GB data-centre GPU has. The alternative is more than one GPU, plus the computer they sit in. That is the case where a Mac you already own can use less electricity. The wattage printed on a chip does not tell you which side used less for the job you actually ran.
The wider picture, including the jobs where a GPU server uses less, is on energy use. This page is only this one job.
The job
Both sides run this same job. The table is the job, not a result.
| Item | What has to match |
|---|---|
| Model | Llama 70B |
| Precision | FP16 weights in memory, not a quantized stand-in |
| Prompt and output | One fixed prompt and a fixed length of output, recorded next to the watt-hours. The length is part of the figure, so it is not guessed here. |
| Mac side | A meter at the plug on the Mac you are using, from the start of the job to the last token. Time spent waiting for work is counted separately. |
| Other side | The smallest multi-GPU server that holds the same full-precision weights, measured the same way, host computer included. |
| Cooling | Cooling for the data hall is its own line. It is not assumed to be zero, and it is not added to the Mac unless that Mac is in a cooled room. |
What that figure does not cover
- Training, or a fine-tune you chose for speed. This is one inference job.
- Llama 7B, or Llama 70B at a lower precision. An H100 is faster on those, and a faster chip can use less electricity per token. See the throughput rows on the Apple silicon comparison.
- Carbon on its own. If you need carbon, apply the grid factor for where the machine is plugged in, and date it. A data centre on a renewable contract can come out lower.
- A reason to buy a new Mac. The saving is a machine you already have, used while it would otherwise be off.
For this class of model, one Mac you already own can use less electricity than the multi-GPU server. How much less depends on the job. A plug reading for this job will show the watt-hours next to the prompt and the length of output.