AI Chip Memory Costs Now Dominate: Why HBM Is Reshaping the Semiconductor Landscape

For the better part of a decade, the semiconductor industry’s AI narrative centered on compute. More TFLOPS. Bigger dies. Denser transistors. The unspoken assumption was that if we could build enough arithmetic muscle, the rest would follow. Memory would scale. Bandwidth would keep pace. The limiting factor was always raw processing power.

That assumption just shattered. AI chip memory costs have exploded. According to Epoch AI, memory now accounts for roughly two-thirds of AI chip component costs — up from about half at the start of 2024. High-Bandwidth Memory (HBM) didn’t just become expensive; it became the single largest line item on the bill of materials, eclipsing the compute die itself. Total AI chip component spending more than doubled from $22 billion in 2024 to $52 billion in 2025, and HBM alone was responsible for roughly $20 billion of that ~$30 billion increase.

We are no longer living in the era of the compute bottleneck. Welcome to the era of the memory bottleneck.

The Two-Thirds Shock: What the Data Shows

Epoch AI’s data on AI chip memory costs is unmistakable. Their newly launched AI Chip Components Explorer tracks how much global supply of logic wafers, HBM, and advanced packaging is consumed by the four largest U.S. chip designers: NVIDIA, AMD, Google, and Amazon. The dataset covers Q1 2024 through Q4 2025, and the trend line is unambiguous.

In early 2024, memory already represented the largest slice of component costs, but it was roughly balanced with logic and packaging. By the end of 2025, memory’s share had climbed to approximately 63% of total component spend. Logic and advanced packaging, while still growing in absolute terms, shrank as a proportion of the pie.

The raw numbers are staggering. Here’s a simplified breakdown of what the data reveals:

Metric	2024	2025	Change
Total AI chip component spend	~$22B	~$52B	+136%
HBM share of that spend	~50%	~63%	+13 pp
YoY increase in total spend	—	~$30B	—
HBM-driven portion of increase	—	~$20B	~67%

NVIDIA’s B300, for example, carries 288 GB of HBM3E — more than double the 141 GB on the H200. Google’s TPU v7 follows the same trajectory. Every generation of frontier AI chips is slurping up more memory stacks than the last, and the suppliers simply cannot keep up.

Perhaps the most revealing statistic from Epoch AI’s analysis is the supply-chain asymmetry: in 2025, the top four chip designers collectively consumed ~90% of global CoWoS packaging capacity and HBM supply, but only about 12% of advanced logic die production. Logic fabrication, for all the headlines about TSMC’s 3nm node, was actually a softer constraint. The real chokepoints were memory and packaging.

That is a profound inversion. For decades, the CPU or GPU die was the crown jewel. Now it is increasingly a co-star.

Why HBM Became the Dominant Line Item

To understand why AI chip memory costs have surged, you need to understand what HBM actually is — and why it is so fiendishly difficult to manufacture at scale.

The Technical Primer

High-Bandwidth Memory is not your laptop’s DDR5. It is a 3D-stacked DRAM architecture in which multiple memory dies are layered vertically using through-silicon vias (TSVs) and bonded to a logic base die. The entire stack is then placed adjacent to the GPU or AI accelerator die on a silicon interposer, creating a wide, ultra-fast data highway between compute and memory.

A single HBM3E stack can deliver over 1.2 TB/s of bandwidth. The latest HBM3E variants push that even higher. For transformer-based AI models, which are essentially memory-bandwidth-bound during both training and inference, that bandwidth is the difference between a chip that flies and a chip that starves.

The Yield and Capacity Trap

The problem is that HBM manufacturing is extraordinarily complex. Each stack requires:

• Leading-edge DRAM process nodes (currently 1α or 1β class from suppliers like SK hynix, Samsung, and Micron)
• Thousands of TSVs drilled through ultra-thin wafers with near-zero defect tolerance
• Advanced packaging houses — primarily TSMC’s CoWoS line — to integrate logic and memory dies together

Every additional layer in the stack increases yield risk. If one die in an 8-high or 12-high stack fails, the entire stack may be discarded or downgraded. That geometric yield loss translates directly into cost. When you combine that with exploding demand — industry analysts estimate NVIDIA alone has grown to consume roughly 70–75% of global HBM supply by 2025, up from approximately half two years earlier — you get a classic supply-demand squeeze.

Epoch AI notes that the memory bottleneck is expected to continue through 2027. All three major suppliers are effectively sold out. Samsung’s HBM4 is sold out for the year. SK hynix is completely sold out. Micron has sold its 2026 HBM supply to U.S. customers.

When buyers are reportedly offering to finance SK hynix’s expansion directly — including covering the cost of ASML’s EUV lithography machines — you know the market is structurally broken.

The Workload Shift

There is a deeper technical reason behind the surge. Modern AI workloads have shifted from compute-bound to memory-bound. Training large language models requires enormous parameter counts, but once a model is trained, inference is increasingly dominated by KV-cache bandwidth and decode throughput — both intrinsically memory-intensive operations. Long-context windows and agentic workflows only exacerbate the demand.

In other words, even if logic performance continued to scale perfectly with Moore’s Law, the memory subsystem would still be the limiting factor. The “memory wall” is no longer a theoretical computer architecture concept. It is the line item on a $52 billion invoice.

Foundry Fallout: Who Wins and Who Loses

The HBM cost revolution is reshaping competitive dynamics across the semiconductor landscape. AI chip memory costs are now the primary driver of foundry strategy, and the winners and losers are not always who you expect.

For more on Samsung’s broader semiconductor strategy, see our analysis of Samsung’s $73B AI chip investment.

The Memory Triopoly: SK Hynix, Samsung, Micron

Three companies control approximately 96% of DRAM and effectively 100% of HBM. Wing Venture Capital calls it “The Memory Triopoly.” SK hynix has pulled ahead as the early leader in HBM3E, securing NVIDIA as a marquee customer. Samsung is aggressively pushing HBM4 and has reportedly aligned with AMD for next-generation stacks. Micron is racing to close the gap with its own HBM3E and upcoming HBM4 offerings.

The structural implication is that every successive AI generation becomes more bandwidth-constrained, which means every successive AI generation makes these three suppliers more powerful. Memory has gone from roughly 20% of GPU BOM on the A100 to over 50% on the B300 and rising further on NVIDIA’s upcoming Rubin architecture.

TSMC: Still Indispensable, But Under Pressure

TSMC remains the world’s most advanced logic foundry, but its role is increasingly bifurcated. It fabricates the AI compute dies, yes — but it also operates the CoWoS packaging lines that stitch logic and memory together. In late 2024 and early 2025, CoWoS was the primary bottleneck. TSMC expanded capacity aggressively through 2025, easing the packaging constraint. Yet the top four designers still consume roughly 80–85% of total CoWoS supply.

As NVIDIA’s Rubin and future chips migrate to 3nm process nodes, TSMC’s logic capacity may tighten again. But for now, the uncomfortable truth is that the world’s most valuable chip foundry is not the main constraint on AI growth. The memory vendors are.

NVIDIA: Margin King, Cost Victim

NVIDIA illustrates the paradox better than anyone. The B200 — the flagship of the Blackwell generation — costs an estimated $6,400 to manufacture, nearly double the H100’s ~$3,320 BOM. HBM memory now represents 45% of total COGS, up from 41% on the H100. At the rack level, the GB200 superchip pushes manufacturing costs to roughly $13,500 per unit, with HBM alone costing ~$5,800 and CoWoS packaging ~$2,200.

And yet NVIDIA reports chip-level gross margins around 82%, thanks to selling prices of $30,000–$40,000 per GPU. The company has pricing power that would make a luxury watchmaker blush. But the cost structure underneath tells a different story: NVIDIA is increasingly a memory reseller with a GPU attached, not the other way around.

That dynamic creates long-term vulnerability. If memory suppliers ever decide to extract more value — or if geopolitical tension disrupts supply — NVIDIA’s extraordinary margins compress from the bottom up.

The Challengers: Cerebras, Groq, and SRAM Machines

Not everyone is playing the HBM game. Cerebras — which we covered in our IPO analysis — and Groq have built “SRAM machines” — accelerators that dedicate massive silicon area to on-chip SRAM rather than off-chip HBM. For certain inference workloads, particularly those with smaller models or massive batch sizes, SRAM-based architectures can bypass the memory bottleneck entirely.

The tradeoff is capacity. SRAM is fast but tiny and power-hungry per bit. HBM is slower (relative to on-chip memory) but dense and relatively efficient. For training trillion-parameter models, HBM still wins. But for inference-at-scale, the SRAM rebels are gaining ground — precisely because HBM has become so expensive and scarce.

From CapEx to OpEx: What It Means for Buyers

If you are a cloud provider, an enterprise AI lab, or a startup training foundation models, AI chip memory costs change how you budget, plan, and compete.

Memory-Bound Budgeting

Traditionally, AI infrastructure budgets were framed in terms of “how many GPUs can we buy?” The implicit assumption was that the compute die was the expensive part. Now, when memory represents the majority of component cost, budgeting becomes a memory-sizing exercise first and a compute-sizing exercise second.

Buying a GPU cluster is no longer just about peak TFLOPS. It is about:

• How much HBM capacity per chip? (determines model size that fits on a single device)
• How much HBM bandwidth per chip? (determines throughput for memory-bound layers)
• What is the memory-to-compute ratio of the workload? (determines whether you are overpaying for arithmetic you cannot use)

Epoch AI’s finding that the top four designers consumed 90% of HBM supply in 2025 has a chilling corollary: if you are not one of those four, you are competing for scraps. Enterprise buyers without multi-billion-dollar orders are increasingly forced into secondary markets, older-generation hardware, or cloud rental models where the provider has already locked in supply.

The Rise of Memory-Centric Negotiations

Major cloud providers are reportedly negotiating directly with HBM suppliers, not just with NVIDIA. When AWS, Google, or Meta places a multi-year order for custom silicon (Trainium, TPU, MTIA), the memory allocation is often the first — and longest — lead-time item. Logic design takes months. Memory supply commitments take years.

This is one reason why custom silicon momentum continues to build. If memory is the dominant cost anyway, and everyone buys HBM from the same three suppliers, the differentiation shifts to how efficiently you use that memory. Custom architectures that minimize KV-cache waste, optimize sparsity, or compress weights can extract more useful inference per HBM dollar than a general-purpose GPU.

Model Training Economics

Training costs are often quoted in “GPU-hours,” but that metric obscures the memory reality. A significant fraction of the hourly cost is paying for HBM amortization — not just the logic die’s depreciation. As HBM content per chip rises, the effective cost of training on the latest frontier hardware increases super-linearly.

For enterprises fine-tuning smaller models, the advice is increasingly: buy last-generation hardware with cheaper HBM. An H100 with mature HBM3 supply may deliver better price-performance for your use case than a B200 with scarce, expensive HBM3E.

Looking Ahead: Will Memory Stay King?

The HBM shortage is not a temporary blip. It is a structural feature of the AI hardware landscape for at least the next several years. The future of AI chip memory costs depends on whether supply can keep pace with exploding demand, and whether technology can outrun the bottleneck.

HBM4 and HBM4E

The next-generation memory standards, HBM4 and HBM4E, are already on the roadmap. HBM4 is expected to roughly double per-die bandwidth compared to HBM3E and expand stack heights. But the transition will not be smooth. Samsung’s HBM4 is sold out before it has even ramped to volume production, and yield challenges on new DRAM process nodes typically persist for 12–18 months after initial shipments.

As Atlas Peak Research notes, HBM4 and HBM4E are not merely better memory products. They are enabling infrastructure for the next phase of generative AI — long-context models, inference-heavy architectures, and agentic systems. If memory does not scale, those workloads do not scale. Period.

Co-Packaged Optics (CPO) and Interconnect Innovation

One way to reduce the HBM burden is to move data more efficiently between chips, reducing the pressure on per-chip memory capacity. Co-Packaged Optics (CPO) and advanced interconnect fabrics could allow model parallelism across more devices with less data movement overhead. If a cluster of GPUs behaves more like a single shared memory pool, each individual chip needs less HBM.

This is the bet behind NVIDIA’s NVLink and custom interconnect fabrics from Broadcom and Marvell. It will not eliminate HBM demand, but it could slow its growth rate.

Geopolitical Supply Concentration

The HBM market is not just concentrated technologically; it is concentrated geographically. SK hynix (South Korea), Samsung (South Korea), and Micron (United States) dominate production. South Korea’s strategic importance to AI infrastructure now rivals Taiwan’s, because without HBM, the most advanced logic die in the world is a very expensive paperweight.

U.S. export controls have already reshaped the HBM map. Chinese demand for HBM had been accelerating rapidly ahead of the restrictions, with NVIDIA alone previously projected to ship roughly 1.4 million H20 GPUs to China in 2025 — each carrying multiple HBM stacks — before the December controls took effect. That demand did not disappear; it was simply redistributed to U.S. and allied buyers, intensifying competition among the remaining customers.

Any future geopolitical shock — a Taiwan Strait crisis, a Korean Peninsula escalation, or further trade restrictions — could freeze HBM supply chains with devastating speed. The world’s AI labs are now, in effect, betting on geopolitical stability in East Asia as a prerequisite for continued model scaling.

Alternative Architectures

Longer term, the industry may simply design around the HBM constraint. Analog in-memory compute, optical computing, neuromorphic architectures, and SRAM-heavy designs all offer theoretical escapes from the memory wall. None are ready to replace GPUs for frontier training today. But if HBM costs continue to rise as a share of total system spend, the economic incentive to invest in alternatives grows proportionally.

Cerebras’ wafer-scale approach and Groq’s LPU are early signals. They are not yet mainstream, but they prove that viable AI compute does not require HBM at all — provided you are willing to trade model scale for latency and throughput.

The Bottom Line

The semiconductor industry spent a decade optimizing for compute. It built bigger GPUs, denser transistors, and flashier TFLOPS numbers. Then AI workloads changed, and memory became the binding constraint almost overnight.

Epoch AI’s data makes the trend undeniable: AI chip memory costs now dominate, accounting for roughly two-thirds of component spend and driving the majority of year-over-year cost growth. HBM is not merely expensive; it is scarce, geopolitically sensitive, and structurally supply-constrained through at least 2027.

For chip designers, AI chip memory costs represent the new battleground. For cloud buyers, it means budgeting for HBM first and compute second. For the industry at large, it means the three companies that control HBM supply — SK hynix, Samsung, and Micron — now hold leverage that rivals TSMC’s.

The era of compute-centric AI hardware is over. The era of memory-centric AI hardware has just begun. And if you are not paying attention to AI chip memory costs, you are not paying attention to the real cost of intelligence.

References and further reading

• Epoch AI — AI Chip Components Explorer
• Epoch AI — AI Chip Components Dataset
• SK hynix
• Samsung
• Micron
• TSMC
• NVIDIA
• AMD
• ASML
• Groq
• Cerebras
• Wing VC — The Memory Triopoly
• Atlas Peak Research — HBM4 Outlook
• Tom’s Hardware — Memory Shortages Through 2027
• Reuters — SK Hynix Flooded With Offers
• SEDaily — Samsung HBM4 Sold Out
• TechSpot — SK Hynix Sold Out
• Yahoo Finance — Micron Sells 2026 HBM Supply
• AInvest — AMD HBM4 Partnership

---

• ← Previous Post
• Next Post →

---

---