Load BearingNegm Tech Notes

All field tests

Is AI latency really fixed? Taking NCM for a spin

Yes, NCM's latency is fixed, but because every answer has the same length, not because of its new method. That buys a cost you can forecast, and it costs quality on long inputs.

Field test, , 8 min read

The bottom line

Next-Concept Modeling (NCM, from Stanford, October 2026) is a promising new approach to generating text, and its cost is almost the same for every request: on my AMD workstation, 99% of requests finished within 8% of the median. But that predictability doesn’t come from how NCM generates text. It comes from a fixed-size input and a fixed-length answer, and an ordinary small language model (SLM) with the same limits is just as predictable. What NCM does bring is speed at that fixed length, with quality matching the paper. The price is the hard limits: a third of the news articles I tested didn’t fit the input window, and their summaries were clearly worse.

The paper’s claims partly hold up: the quality matches, but its explanation for the speed doesn’t show in the released code. My verdict: worth watching for high-volume, fixed-format, fixed-output tasks, work you need to capacity-plan; definitely not ready where a wrong summary costs you, or where variable-length input can’t be ignored.

What I tested

The question I was asking about NCM: can a model give you a cost per request you can forecast, and where does that predictability come from? With today’s large language models you know what a request will cost after the request is served, simply because you don’t know in advance how long the answer will be. We can argue that output tokens are limited per model, but that is the maximum ceiling rather than your planned average. This means capacity planning is focusing on the tail of the distribution.

NCM writes text coarse to fine: it drafts the gist of an answer in a few rough strokes and refines it in a fixed number of rounds, instead of writing word by word. The paper claims 10–31× faster generation than diffusion language models at comparable or better quality (p. 3). I decided to compare it with today’s autoregressive models instead, as they are the de facto standard for LLMs. I ran the authors’ released code and model for news summarization (XSum, 11,334 test articles) on an AMD Radeon iGPU, one request at a time, and compared it with an SLM: a small, publicly available summarizer that writes word by word (T5). The setup details are under the hood, at the end.

What I found

1. The latency is fixed because the length is fixed. Over 1,000 requests, NCM’s p99 was 8% above its median. The SLM forced to always write 48 tokens was even flatter (1%). Left to stop on its own (no forced output length), the same SLM was cheaper at the median, but its p99 was about 76% above it, and one request in a thousand ran away to 382 ms. This emphasizes the variability in expected output with today’s autoregressive models.

Forced to a fixed length, the ordinary model is as flat as NCM. Left to decide its own length, it’s cheaper at the median and unpredictable in the tail.

Forced to a fixed length, the ordinary model is as flat as NCM. Left to decide its own length, it’s cheaper at the median and unpredictable in the tail.

2. At a fixed length, NCM is faster. About 98 ms per summary against 150 ms for the word-by-word model forced to the same length, and about 80 ms with the compiler optimizations the paper used. That’s the real advantage of the method: a fixed number of big rounds (NCM’s coarse-to-fine rounds) beats the many small steps of an autoregressive model.

3. The autoregressive model’s cost is simple, and that’s the problem. The SLM costs about 10 ms plus 3 ms per word-piece, so it beats NCM for answers under about 30 tokens and loses above. You don’t know which you’ll get until it’s done, and its longest answers were repetition loops.

The word-by-word model’s cost follows its answer length exactly. Predictable once you know the length; the length isn’t known in advance.

The word-by-word model’s cost follows its answer length exactly. Predictable once you know the length; the length isn’t known in advance.

4. Quality matches the paper, and still isn’t production-ready. ROUGE-L 27.93 against the paper’s 28.0 on all 11,334 articles: a match within 0.1 points on different hardware. But when I read 50 summaries drawn at random, 48 had a visible flaw (e.g. broken grammar, repeated words), and about 3 in 5 hallucinated (stated wrong or made-up information). The paper’s own examples score well above this average. On equal terms, the small public model was about level on quality (28.3 against 27.6).

5. The fixed window costs quality on long inputs. NCM reads only the first 512 tokens of an article. A third of the test articles fill that window, and their summaries score 24.8 against 29.6 for shorter ones: on long inputs, the cost is paid in quality.

Summary quality falls with article length; the third of articles that fill the window score worst.

Summary quality falls with article length; the third of articles that fill the window score worst.

What it means for your platform

  • Predictable cost is a design choice you can make with any model. Fix the input size and the output length, and cost per request becomes a constant you can multiply: requests per second × cost gives the capacity you need, especially since p50 ≈ p99. When someone claims “predictable costs”, the question becomes, “where does it come from?” or “what are the constraints applied?”.
  • Cap output length as a capacity control. For a word-by-word model, the maximum answer length isn’t just a product setting; it bounds your worst-case cost. My worst request cost about 4.4 times the median. (Findings 1, 3)
  • Where NCM-style generation fits today: high-volume, fixed-format tasks with short inputs, where you’d rather plan capacity than optimize the average. Where it doesn’t: long inputs, open-ended answers, and anything where a fluent but wrong summary is expensive. (Findings 2, 4, 5)

In general, NCM offers a fast, unique approach if the task at hand is already well defined (in terms of inputs and outputs). Some examples: incident titles, where a single alert or a short log excerpt becomes one line for the on-call channel; ticket triage, where every infrastructure or support ticket gets a fixed-length summary for routing; training-job status, where a job’s latest metrics become a one-line report for the team; and change summaries, where a config change or pull request becomes one line in the change log. I can see NCM taking on such tasks, with platforms carving out dedicated capacity per task to get its speed and predictability.

Under the hood

Setup

HardwareGMKtec EVO-X2: AMD Ryzen AI Max+ 395, Radeon 8060S iGPU (40 compute units, gfx1151), 128 GB unified memory
SoftwareUbuntu 26.04, kernel 7.0.0-43, Python 3.14.4, PyTorch 2.15.0+rocm10.0, transformers 5.23.0, rouge-score 0.1.2
Paper codeHazyResearch/ncm at commit c702742, with three small patches: BF16, a timing mode, and an optional torch.compile
Baselinepki/t5-small-finetuned_xsum, 60.5M parameters, commit 2239ec3
Compute time1.6 h over 17 measured runs
Cloud cost€0

Method

  • Quality: the released evaluation on all 11,334 XSum test articles, in float32 (ROUGE-1/2/L 34.92 / 12.71 / 27.93) and BF16 (27.94), against the paper’s 35.0 / 12.8 / 28.0 (Table 2, p. 10). Decoding is deterministic: repeat runs gave identical summaries.
  • Latency: one article per request (batch size 1), BF16, the same 1,000 randomly drawn test articles for every model, 60 s of discarded warm-up, 3 runs per configuration in rotating order (the compiled runs back to back), torch.cuda.synchronize() around every timed span. Percentiles over 1,000 requests per run:
p50p99maxROUGE-L, same articles
NCM96–100 ms104–109 ms107–114 ms27.6
T5, stops on its own80–87 ms142–153 ms354–382 ms28.3
T5, forced to 48 tokens150–164 ms152–166 ms153–166 ms25.3
NCM, compiled80–82 ms84–88 ms236–251 ms27.7
  • The baseline: T5’s input was cut or padded to 512 tokens like NCM’s, so only the output length varies; both models were scored against the same references with the same scorer.
  • Reading 50 summaries: drawn at random (fixed seed) from the full run, each read next to its reference and marked for repetition, broken grammar, contradicting the reference, or being off topic. Example of the worst kind: “Team Sky rider Sergio Henao has withdrawn from the sport after failing three drugs tests”, where the reference says the team withdrew him over biological-passport concerns.

The paper’s claim, checked

The paper explains its speed with “lower per-step cost from shorter coarse levels” (p. 3): NCM “processes very few tokens at the coarse levels of its hierarchy, saving compute at these early steps” (p. 11). In the released code for this task, it doesn’t: every round re-processes the whole 512-token article (there’s no cache of the work already done), so all 9 rounds cost about the same.

The smallest rounds aren’t cheaper: the cheapest takes 88–90% of the time of the largest. If the cost followed the amount of new text, the bars would follow the dashed line.

The smallest rounds aren’t cheaper: the cheapest takes 88–90% of the time of the largest. If the cost followed the amount of new text, the bars would follow the dashed line.

Two smaller mismatches: the paper counts 8 generator calls per summary, the released code makes 9; and the first round is the slowest, for a reason I couldn’t find. The paper’s timing code isn’t released; if it cached the article, its early rounds could be cheaper. My finding is about the released code.

Caveats

  • One task (news summarization), one GPU (an AMD iGPU, not the paper’s H100), one request at a time (the paper timed batches of 16).
  • My comparisons ran without torch.compile for fairness; the paper used it. Compiled, NCM was 16–18% faster with the same flatness and quality.
  • The two models use different tokenizers, so “48 tokens” means slightly different amounts of text.
  • A padded 512-token input isn’t how you’d serve the word-by-word model; it would be cheaper with its natural input length. The padding isolates the effect of output length.
  • The quality-by-length result is a correlation: long articles may be harder to summarize whether or not they’re cut.

Tagged ncm, ai-cost, capacity-planning, amd-gpu, latency