Introduction
“How big is the market for slow search?”
Andrew Feldman, Cerebras Founder and CEO, Q1 2026
Over the past decade, a new category of specialised AI hardware has been quietly developing that is now coming to the forefront: fast inference accelerators. The appeal of these systems is fairly intuitive (Figure 1). Higher inference speeds improve the productivity of both humans and agents by allowing more work to be done in less time; they also deliver the instant, low-friction responses that users expect and prefer. Two of the most well-known companies in this space are Cerebras and Groq, with Cerebras claiming its system can deliver inference speeds up to 15 times faster than leading GPU-based alternatives. The company recently went public, raising over US$6 billion in the largest semiconductor IPO of all time and has a multi-year deal with OpenAI worth more than US$20 billion. Groq has also received major validation through a licensing agreement with Nvidia, whose CEO Jensen Huang estimated that the new Groq product would add an incremental 25% to Nvidia’s revenue.
Figure 1: Exchange between Paul Graham and Sam Altman on the need for fast inference
Source: X
In this note, we first discuss the inference bottlenecks that fast inference architectures are designed to address and how this is leading to an unbundling of AI hardware accelerators. We then profile Groq and Cerebras, outlining the foundational technologies behind their approaches, as well as their commercial partnerships and prospects. Finally, we discuss the size of the opportunity.
Unbundling AI Hardware
“There’s only two ways I know of to make money – bundling, and unbundling.”
Jim Barksdale, former Netscape CEO, 1995
AI inference has two separate stages: prefill and decode (Figure 2). Prefill is the input stage, where the model processes the user’s prompt. This stage is well suited to standard AI accelerators such as Nvidia GPUs because much of this work can be done in parallel, which is where they thrive. Decode is the output stage, where the model generates the answer one token at a time. Because each new token depends on the one before it, the process is inherently sequential and cannot be parallelised. At this stage, the bottleneck shifts from compute to memory bandwidth and latency, often referred to as the memory wall: the system is limited by how quickly it can move data between memory and compute.
Figure 2: Prefill and decode
Source: Cerebras
Several companies, most of which remain private, are currently working to address this bottleneck. Groq and Cerebras are two of the more well-known companies in this space, which are building their systems around a particular type of memory called SRAM. SRAM has significantly more bandwidth than HBM, the primary memory used in GPUs. SRAM also sits on the compute chip, whereas HBM sits off the chip, meaning data has to travel much shorter distances. This reduces both latency and power use.
However, SRAM has historically been difficult to use as primary memory because it takes up far more silicon area than HBM, which has meant SRAM capacity has been too limited. As a result, SRAM has traditionally been used only as a small, fast cache sitting above much larger DRAM/HBM main memory. This new generation of AI accelerators has been able to alleviate this capacity constraint, unlocking SRAM's bandwidth advantage and resulting in their systems achieving far higher token output speeds than GPUs.
This, in turn, has also allowed inference to be disaggregated across hardware types, where GPUs can handle the prefill stage, while SRAM-based accelerators target the decode stage. Each architecture can therefore focus on the part of the workload it is best suited to. As a result, inference hardware is beginning to unbundle.
Groq – The Software-First Approach
Groq was founded in 2016 by Jonathan Ross, a former Google engineer who helped start what became Google’s Tensor Processing Unit (TPU). Its core technology is the Language Processing Unit (LPU). Unlike the versatile GPU, which can handle many different compute tasks, the LPU was designed specifically for linear algebra calculations, which is the primary requirement for AI inference. The Groq team also took a software-first approach, with execution decisions made in software rather than hardware. The team even developed the software (compiler) before they began chip design. Because all execution planning happens in software rather than hardware, this allows for a simpler and more efficient hardware architecture that executes predetermined scripts. This determinism helps reduce delays and improve latency stability, while the simpler hardware architecture frees up space for additional memory bandwidth and transistors for performance.
Figure 3: Rubin GPU vs Groq 3 LPU comparison – Capacity vs Bandwidth trade-off
Source: Nvidia
However, even with the gains noted above, the amount of SRAM that fits on a single LPU remains limited. The Groq 3 LPU still only has a 500 MB SRAM, a tiny amount relative to 288 GB of HBM, an Nvidia Rubin GPU (Figure 3). This is far less than large models require, which means many LPUs must be connected together to create a larger effective pool of memory. This introduces synchronisation challenges. Groq solves this in two ways: its software maps out in advance exactly when each chip sends data to the others, while its chip-to-chip protocol cancels clock drift and keeps hundreds of LPUs aligned. Together, this allows the system to behave like one coordinated chip.
In December 2025, Groq and Nvidia entered into a non-exclusive licensing agreement. As part of the agreement, members of Groq’s team, including its CEO Jonathan Ross, joined Nvidia. News outlets reported that the deal was worth US$17–20 billion. (The structure has since drawn scrutiny from Senators Elizabeth Warren and Richard Blumenthal, who questioned whether it was designed to avoid antitrust laws.)
At Nvidia’s GTC event in March 2026, Jensen Huang unveiled the Nvidia Groq 3 LPX. The Groq 3 LPX is a rack of 256 interconnected LPUs, offering 128 GB of SRAM capacity and 40 PB/s of bandwidth (640 TB/s of scale-up bandwidth across the rack), positioned as a low-latency inference layer for Nvidia’s Vera Rubin platform. By combining Rubin GPUs with Groq’s LPUs, inference is disaggregated, with Groq handling the bandwidth-heavy decode, enabling the platform to reach much higher tokens per second (TPS) (Figure 4). However, it was explained that the price for this performance boost would be set “quite high” because it has a lower TPS per MW, and was intended for high value workloads and users like software engineers who could afford it. Nvidia expects to start shipping the Groq 3 LPX in the second half of 2026.
“If you extended this chart way out here and you said you wanted to have services that delivers not 400 tokens per second, but a 1000 tokens per second, all of a sudden, NVLink 72 runs out of steam and simply can’t get there. We just don’t have enough bandwidth. And this is where Groq comes in.”
Jensen Huang, CEO of Nvidia, March 2026 (GTC 2026)
Figure 4: Vera Rubin NV72 + Groq 3 LPX enabling High-Throughput and Low-Latency
Source: Nvidia
Cerebras – The Massive Chip Approach
“Fast tokens are more valuable tokens and Cerebras tokens are the fastest.”
Andrew Feldman, Cerebras Founder and CEO, Q1 2026
Cerebras was co-founded in 2016 by its CEO Andrew Feldman on the founding bet that the age of AI would demand a new kind of compute, just as PCs needed x86, graphics needed GPUs and mobile needed ARM. Like Groq, Cerebras uses SRAM as its primary memory, but it takes a very different approach. Rather than connecting many smaller chips together, Cerebras turned an entire silicon wafer into one massive processor, called the Wafer-Scale Engine (WSE). The latest WSE-3 is 58 times larger than Nvidia’s B200, which gives Cerebras far more space to place SRAM directly alongside compute (Figure 5). In total, the chip carries 900,000 AI-optimised cores and 44GB of on-chip SRAM, resulting in 21PB/s of memory bandwidth, 2,625 times more than an Nvidia B200 package. Overall, Cerebras says this architecture enables inference speeds up to 15 times faster than leading GPU-based systems.
Figure 5: Cerebras Wafer Scale Engine 3 vs Nvidia GPU B200 – 58x larger
Source: Cerebras
The key benefit of wafer-scale is that far more communication can stay on-chip. Even when a model is too large to fit entirely on a single wafer, the large chip reduces how often data needs to travel over slower interconnects between separate chips or systems. Achieving this chip scale required Cerebras to develop two foundational semiconductor technologies:
- Multi-die interconnect: A die is a region of silicon containing an integrated circuit, individually stamped onto a silicon wafer and then normally diced into small, separate chips. Cerebras invented a technology to interconnect these otherwise independent die at the wafer level. This lets adjacent die communicate at the same bandwidth as within a single die, so the whole wafer works as one chip and avoids the slowdown of going off-chip.
- Fault-tolerant architecture: Wafers typically have defects, but normally when a wafer is diced into smaller chips, the defective ones are discarded. With a WSE this cannot be done, because the whole wafer is effectively a single chip. Cerebras' answer was to build redundancy into the design, so that flaws are recognised, shut down and routed around.
(For a deeper dive on Cerebras’ technology, see: https://www.youtube.com/watch?v=7GV_OdqzmIU.)
Cerebras initially spent much of its existence as a relatively obscure hardware company, building a radical chip the market was not ready for. That began to change in 2023, when the UAE's G42 commissioned Condor Galaxy, a series of AI supercomputers built on Cerebras hardware. Momentum has built since then, with revenue reaching US$510 million in 2025, up 76% year-on-year. In January 2026, OpenAI signed a multi-year agreement worth more than US$20 billion, under which it agreed to deploy 750MW of Cerebras compute and to co-design future models for Cerebras hardware. This partnership is already beginning to translate into customer access, with OpenAI saying its upcoming GPT-5.6 Sol model will be available on Cerebras in July 2026 at up to 750 tokens per second for select customers as capacity expands. In March 2026, Cerebras also launched a multi-year partnership with AWS to bring its fast inference to a broader enterprise and developer market.
Market Sizing
“When we look out at the space, we see the entire inference market as available to us for fast inference. I mean, who doesn’t want answers in less time?”
Andrew Feldman, Cerebras Founder and CEO, Q1 2026
In terms of market demand for fast inference, we expect this to be strong given the wide range of use cases in which it can boost productivity, from coding agents and document analysis to drug discovery. Additionally, fast inference can also improve user experiences, such as cutting dead time while users wait for responses, helping them stay in flow rather than switching context. Voice is another area where fast inference is very beneficial, as it can help address the current trade-off between response quality and speed, as even small delays can make the interaction feel unnatural.
At the March 2026 GTC, Jensen Huang estimated that Nvidia’s Groq product would add approximately 25% in incremental total revenue. For context, Huang said Nvidia had US$1 trillion in high-confidence demand for Blackwell and Rubin through the end of 2027, which means its Groq product would add an additional US$250 billion for the period. Cerebras reported US$193 million in revenue for FY26 Q1, up 92% year-on-year. Growth is expected to remain strong as the company scales production under its OpenAI and AWS partnerships, which provide significant visibility into future demand.
Figure 6: AI Cloud Semiconductor Forecast
Source: Morgan Stanley Research
Looking further out, Morgan Stanley estimates that the fast inference market could capture 20% of the inference semiconductor market by 2030, representing an US$80 billion opportunity (Figure 6). This is significantly lower than a simple extrapolation of Huang’s 25% figure across the broader AI semiconductor market. However, given that the commercialisation of fast inference is still in its infancy, forecasts should be treated with wide error bars. There are still major unknowns, including how quickly supply can scale, what capacity can be secured, how price competitive fast inference becomes and how much customers are willing to pay for speed. There is also uncertainty around the pace of improvement in SRAM-based architectures relative to GPUs and other inference solutions. Ultimately, the question is not whether customers will want faster inference, but whether supply and demand converge at a point where fast inference becomes ubiquitous, similar to fast internet, or instead occupies a smaller segment that is only affordable for high-value workloads.
Limitations and Risks
Large on-chip SRAM architectures such as Groq and Cerebras excel in low-batch, latency-sensitive decode scenarios where memory bandwidth is the primary constraint. While demand for faster inference is robust in interactive and agentic workloads, there are scenarios where their advantages are eroded. Model efficiency improvements represent the most significant long-term risk. Techniques such as distillation, aggressive quantisation, Mixture-of-Experts routing, and speculative decoding can materially reduce both model size and the memory bandwidth required during decode. As these methods mature, a growing share of inference workloads may achieve acceptable latency on conventional GPU infrastructure at lower cost and higher utilisation, narrowing the addressable market for specialized SRAM-based systems.
Furthermore, large-scale batch inference, offline processing, and throughput-oriented jobs continue to favor GPUs, which offer superior utilisation and more mature software ecosystems. Extremely long context lengths can also erode the advantage, as KV cache requirements eventually exceed even the substantial on-chip SRAM capacity, forcing data movement over slower interconnects.
Finally, power efficiency and total cost of ownership remain important considerations. Both Groq and Cerebras architectures carry higher power draw relative to their performance in certain configurations, and the premium pricing required to justify this may limit adoption outside high-value use cases.
Conclusion
“We don’t know all the things that can be done with fast AI because we haven’t had it yet.”
Jonathan Ross, Groq founder, March 2026 (Nvidia GTC 2026)
In large, growing markets, unbundling is a natural part of the cycle as specialists emerge to solve persistent pain points. Although GPUs remain foundational to AI infrastructure, the memory-bandwidth constraint has created an opening for specialised SRAM-based architectures. Groq and Cerebras are both pursuing this approach but through meaningfully different designs: Groq relies on a software-first, deterministic model across many smaller chips, while Cerebras uses a monolithic wafer-scale architecture to maximise on-chip bandwidth. Both appear well positioned, supported by strong commercial agreements, though competition is intensifying as more players emerge. Nvidia, the incumbent, has also moved quickly to bundle these capabilities back into the broader AI infrastructure stack in order to defend its dominant market position. Given the nascent stage of the market, there is still a limited line of sight on how large it could ultimately become. However, we expect demand to be robust, given the productivity gains and improved user experiences that fast inference can deliver, as well as its potential to unlock new AI use cases that have yet to be imagined.
At AlphaTarget, we invest our capital in some of the most promising disruptive businesses at the forefront of secular trends; and utilise stage analysis and other technical tools to continuously monitor our holdings and manage our investment portfolio. AlphaTarget produces cutting edge research and our subscribers gain exclusive access to information such as the holdings in our investment portfolio, our in-depth fundamental and technical analysis of each company, our portfolio management moves and details of our proprietary systematic trend following hedging strategy to reduce portfolio drawdowns. To learn more about our research service, please visit https://alphatarget.com/subscriptions/.
