StockDriftsUltra-fast inference: Latent Space’s big theme from Hot Chips
Latent Space hosts a long-form discussion with Cerebras CTO and co‑founder Sean Lie the day after the Hot Chips conference, positioning it as a snapshot of the “golden age” of AI hardware as of September 2026. Lie argues that what the industry used to consider fast inference – roughly 100–200 tokens per second – is rapidly turning into a “new batch mode” suitable mainly for offline or heavily parallel workloads.
According to Lie, the central shift is from merely being able to process prompts quickly to enabling fully interactive, real‑time applications and agentic workflows. He frames ultra-fast inference as the next competitive frontier, where speed directly translates into more reasoning loops, richer user experiences, and qualitatively different products.
In this conversation, Lie walks through Cerebras’s CS4 launch, previews its CS5 roadmap, and reacts to other Hot Chips announcements including OpenAI’s Jalapeno, Groq’s new chip, D-Matrix, and memory packaging moves from Samsung. Throughout, he stresses that these comments reflect Cerebras’ perspective and his own computer-architecture lens, not neutral industry consensus.
Cerebras’s thesis: wafer-scale for frontier, ultra-fast workloads
Lie tells Latent Space that Cerebras designed its CS4 system and new Nexus rack platform to make wafer-scale computing “mainstream” at hyperscale data centers. He says the Nexus-based CS4 doubles power delivery to the wafer, doubles interconnect bandwidth, and halves latency versus the prior generation, allowing higher density per rack and substantially higher inference throughput.
Cerebras positions itself, in Lie’s words, as the “undisputed leader” in ultra-fast inference, aiming squarely at frontier‑scale, low‑latency workloads rather than smaller or purely batch jobs. He describes a strategic bet on heterogeneous, disaggregated inference systems, where different hardware types specialize in distinct sub‑tasks such as prefill, decode, or expert routing.
Looking ahead, Lie says CS5 is co‑designed with the same Nexus platform and targeted for release “next year” (relative to the September 2026 recording). He claims CS5 aims to roughly double CS4’s performance again, with goals like running medium‑sized models such as GPT‑class open models or Gemma at up to 10,000 tokens per second, and some frontier‑type models like Kimi, DeepSeek, or GPT‑series variants at up to 5,000 tokens per second.
Evidence and numbers: CS4, CS5 and OpenAI’s ultra-fast launch
Lie cites several concrete performance claims from Cerebras’s Hot Chips demos and its collaboration with OpenAI. In a CS4 demo, he says Cerebras showed an open GPT‑style model (which he refers to as GPT‑OSS) running at more than 4,400 tokens per second, a figure he characterizes as “mind‑blowing” and almost “feels like it’s fake.”
He also tells Latent Space that, in a joint ultra-fast offering with OpenAI, Cerebras is already running what he describes as OpenAI’s “largest, most capable, most intelligent model” at roughly 14x the speed of OpenAI’s “normal GPU speeds.” Lie presents this as proof that wafer‑scale hardware can materially change how frontier models feel in production settings.
For the roadmap, Lie explains that CS4 on Nexus delivers about a 2x performance gain over the previous platform, and that CS5 is architected to add another 2x on top, while staying within the same modular rack and “backpack” server infrastructure. On that basis, he asserts that targets like 10,000 tokens per second for certain medium‑sized models and around 5,000 tokens per second for some frontier models are realistic goals for the next generation, though he does not present independent benchmarks in this interview.
Risks, limits, and competitive counterpoints Sean Lie acknowledges
Lie repeatedly notes that Cerebras is effectively sold out of the capacity it can build and is deploying “every single megawatt in the most strategic way possible.” He describes this as both an opportunity and a constraint, because a large share of that capacity is committed to OpenAI, which he says is using the ultra-fast hardware internally for high‑stakes functions like incident response and core research.
On industry power and efficiency debates, Lie tells Latent Space that many conversations at Hot Chips revolved around performance per watt rather than raw tokens per second. He does not present detailed Cerebras power metrics in this interview, leaving that as a partially addressed concern and an area where investors would need more data.
He also highlights potential architectural limitations of competing SRAM-centric designs. In particular, he questions why Groq’s latest chip was only shown on a roughly 30–31 billion parameter model and why previously touted attention/FFN disaggregation features were not highlighted in production performance numbers. He speculates that non‑wafer‑scale SRAM designs may struggle with memory capacity for multi‑trillion‑parameter frontier models and predicts that such vendors might gravitate toward smaller‑model niches. Those comments are explicitly framed as his interpretation, not confirmed information from Groq or others.
Forward signals: Jalapeno, heterogeneous systems, and China’s open model surge
For future catalysts, Lie singles out OpenAI’s Jalapeno GPU as one of the most important Hot Chips announcements. While many observers focused on Jalapeno’s performance versus traditional GPUs, he is more interested in the “AI‑first” design methodology, which he believes let OpenAI build a significantly better GPU faster than expected. He calls this design approach “100% the future of our industry.”
Lie argues that when Jalapeno becomes available “next year” alongside Cerebras CS5, OpenAI and Cerebras together could offer a portfolio of fast‑inference options that is “substantially different and better” than current GPU‑based baselines. He also suggests room for deeper integration, such as explicitly separating prefill and decode across different hardware types, though he notes Jalapeno itself is currently optimized for throughput rather than that specialization.
Beyond OpenAI, Lie points investors toward trends in advanced memory and packaging. He mentions D-Matrix’s 3D DRAM packaging work and Samsung’s ZHBM as examples of the kind of 3D integration he thinks will unlock the next step in AI hardware, paralleling Cerebras’s own DRAM stacking program launched two years earlier. At the geopolitical level, he warns that as of this recording, “most of the big open models” are coming from Chinese labs, and he says the open‑source model market is “almost” entirely Chinese. In his view, addressing that strategic reliance requires government‑level initiatives, not just individual companies or fabs.
Frequently asked questions
What did Latent Space’s interview with Sean Lie say about Cerebras CS4 performance?+
According to Cerebras CTO Sean Lie on Latent Space, the CS4 system on the Nexus platform delivered roughly a 2x performance improvement over the prior generation and was demonstrated running an open GPT-style model at more than 4,400 tokens per second. He presents this as evidence that wafer-scale hardware can materially shift inference from batch-style processing toward ultra-fast, interactive experiences.
How fast does Sean Lie claim future Cerebras CS5 systems will be?+
Lie tells Latent Space that CS5, co‑designed with the Nexus rack, is targeted to roughly double CS4’s performance again. He says Cerebras is aiming to run medium-sized models such as GPT-class open models or Gemma at up to 10,000 tokens per second, and some frontier models like Kimi, DeepSeek, or GPT-series variants at up to 5,000 tokens per second, though those figures are presented as roadmap goals rather than independently verified benchmarks.
What did Sean Lie say about OpenAI’s Jalapeno chip?+
In the interview, Lie describes OpenAI’s Jalapeno as perhaps the most exciting Hot Chips announcement, not primarily for its token-per-second claims but for its AI-first design methodology. He argues Jalapeno represents a significantly better GPU than existing options and sees its eventual coexistence with Cerebras CS5 as enabling a much broader portfolio of fast-inference products than standard GPUs alone.
How is Cerebras working with OpenAI on ultra-fast inference?+
Lie says OpenAI is Cerebras’s largest partner and that they have a co‑design relationship intended to create a flywheel between faster hardware and more capable models. According to him, Cerebras hardware is already running what he calls OpenAI’s largest and most capable model around 14 times faster than OpenAI’s normal GPU speeds, and much of Cerebras’s current capacity is being used for OpenAI’s internal incident response and research workloads before being rolled out more broadly to enterprises.
What concerns did Sean Lie raise about Groq’s latest chip?+
Lie tells Latent Space that he finds it “odd” that Groq’s new SRAM-centric chip was only shown on a roughly 30–31 billion parameter model and without publicized results for attention/FFN disaggregation, which had been heavily discussed previously. He interprets this as a possible sign that non-wafer-scale SRAM designs may face memory-capacity challenges on multi‑trillion‑parameter frontier models, though he stresses that this is his reading of the situation rather than confirmed information from Groq.
How did Sean Lie describe China’s role in open-source AI models?+
Lie says that as of this recording, the open‑source model market is “almost” entirely driven by Chinese labs, and he estimates that most high‑quality large open models are coming from China. He views this as strategically challenging for the United States and argues on Latent Space that maintaining U.S. leadership will require government-level, national-interest initiatives alongside corporate efforts.
