Generation cost falls with compression

Measured forced-length greedy-generation latency on one CPU core, with analytic generated-trace attention FLOPs (key-value cached) and generated-trace cache bytes at the measured mean trace length. The input prefix is fixed at length 64 for this demonstration.

series
setting
measured latency (normalized) attention FLOPs (normalized) 1/c and 1/c² reference