Trends in AI Hardware Demand

NVIDIA reported earnings yesterday, and, as is tradition, they crushed expectations, guided conservatively, and the stock promptly fell 3% because when you’re priced for perfection, even dominance is a mild disappointment. But let’s ignore the stock market tantrum for a moment and parse Jensen Huang's earnings call commentary for industry context: 🚀 AI Demand is Still in Hyper-Growth Mode. Data Center revenue surged to $35.6B (up 93% YoY). Blackwell is NVIDIA's fastest-ramping product ever—$11B in its first full quarter, not even a year after it was first announced. Jensen notes "It will be common for Blackwell clusters to start with 100,000 GPUs". 🧴 Inference is the Bottleneck. Reasoning models like OpenAI's GPT-4.5, DeepSeek AI-R1, and Grok-3 require 100x more compute per query than their early ancestors. AI is moving beyond one-shot inference to multi-step reasoning, chain-of-thought prompting, and autonomous agent workflows. Blackwell was designed for this shift, delivering 25x higher token throughput and 20x lower cost vs. Hopper. 📈 3 Scaling Laws. Jensen identified three major AI scaling trends that are accelerating demand for AI infrastructure: (1) Pretraining scaling (more data, larger models) (2) Post-training scaling (fine-tuning, reinforcement learning) (3) Inference-time scaling (longer reasoning chains, chain-of-thought AI, more synthetic data generation). 💰 Who's Buying? Cloud Service Providers (CSPs) still make up about 50% of NVIDIA's Data Center revenue, and their demand nearly doubled YoY but many enterprises are also investing in their own AI compute instead of relying solely on cloud providers 🍟 Custom Silicon and the ASIC vs. GPU Debate. Big Tech is building custom AI ASICs (Google has TPUs, Amazon has Trainium, Inferentia) to reduce dependency on NVIDIA but Jensen dismissed the notion that custom silicon would challenge NVIDIA’s dominance. GPUs remain more flexible across training, inference, and different AI models, while ASICs are often limited in their use cases. He flagged the CUDA ecosystem as a major competitive moat. 🛰️ The Next Frontier. Jensen repeatedly emphasized “agentic AI” and “physical AI” as the next major trends. The first AI boom was digital—models that generate text, images, and video. The next phase is AI that acts and interacts with the physical world. The market may worry about Nvidia's forward guidance but its hard to discount a company that controls everything from the chips to the networking (NVLink, InfiniBand), software (CUDA, TensorRT) and system-level AI solutions.

"We find that the computational performance of AI supercomputers has doubled every nine months, while hardware acquisition cost and power needs both doubled every year. The leading system in March 2025, xAI’s Colossus, used 200,000 AI chips, had a hardware cost of $7B, and required 300 MW of power—as much as 250,000 households. As AI supercomputers evolved from tools for science to industrial machines, companies rapidly expanded their share of total AI supercomputer performance, while the share of governments and academia diminished. Globally, the United States accounts for about 75% of total performance in our dataset, with China in second place at 15%. If the observed trends continue, the leading AI supercomputer in 2030 will achieve 2 × 1022 16-bit FLOP/s, use two million AI chips, have a hardware cost of $200 billion, and require 9 GW of power. Our analysis provides visibility into the AI supercomputer landscape, allowing policymakers to assess key AI trends like resource needs, ownership, and national competitiveness." Good work from Konstantin F. Pilz James M. S. Robi Rahman Lennart Heim

**🚨 The Rise of Dedicated Servers in AI: Not Just a Trend – It’s a Shift That’s Reshaping the Cloud Industry 🚨**  Today, we’re witnessing a fascinating pivot in enterprise IT infrastructure—a quiet yet undeniable revolution in how businesses are managing AI workloads. 🚀  For years, public cloud providers dominated conversations around scalability and innovation. The "pay only for what you use" model became the gold standard. But the landscape is changing rapidly, especially in the AI space, where dedicated servers are no longer a niche option but are emerging as a critical business enabler. 📈  The reason? AI workloads are uniquely demanding—they require formidable computing power, massive storage capacity, and real-time performance optimization. Public clouds, while still valuable for innovation and scalability, often present enterprises with ballooning costs, hidden inefficiencies, and unpredictable performance due to multitenancy and shared resources.  In contrast, **dedicated servers** provide:   - **Cost Predictability:** No surprise fees or pay-as-you-go spikes.   - **Performance Optimization:** Greater control to fine-tune AI infrastructure, especially for critical applications like real-time analytics and autonomous systems.   - **Data Security & Compliance:** Essential for industries like finance, healthcare, and government, where strict regulations like HIPAA and GDPR demand it.  This isn’t just a transient trend—it’s overtaking significant chunks of the existing cloud business. With nearly half of IT professionals expecting dedicated servers to become integral by 2030, the future looks hybrid: a strategic combination of public clouds for rapid, experimental scaling and private infrastructures for mission-critical, cost-sensitive workloads. Enterprises are no longer blindly chasing “all-in cloud strategies.” They’re building nuanced, hybrid models that align infrastructure with their unique workloads and business goals. Companies are leveraging colocation or managed dedicated services to mimic cloud ease while maintaining the control and performance benefits of private hardware. As we look ahead, this fundamental shift is redefining the role of cloud providers. It’s time to recognize that dedicated servers are no longer the silent underdogs. They’ve become the backbone of AI-driven innovation.  The question isn’t “if” this change will impact your enterprise but *how quickly*. Are you ready for the new era of hybrid infrastructure?  #ArtificialIntelligence #CloudComputing #HybridCloud #DedicatedServers #EnterpriseIT #DataArchitecture #DavidLinthicum 

Since the development of the personal computer became a desktop standard in the 1980s, you'd think processors would become a "solved problem" by now. But guess not—our greed for faster, more capable systems seems only to cause an increasing race for more computing. Obviously, the AI gold rush is driving the need for more silicon "picks and shovels," meaning more processors, both CPUs and GPUs. The advent of artificial intelligence (AI) and machine learning (ML) has only intensified this quest. As AI applications become more sophisticated, they require an ever-increasing amount of computational power. The semiconductor industry is at the heart of this technological revolution.  The global artificial intelligence chip market, valued at $14.9 billion in 2022, is projected to reach a staggering $383.7 billion by 2032, growing at a 38.2% CAGR. This demand is not just about speed; it's about the ability to process vast amounts of data quickly and efficiently. Central Processing Units (CPUs) have been the backbone of computing for decades, handling a wide range of tasks. However, parallel graphics processing units (GPUs) make them particularly well-suited for the matrix and vector computations fundamental to AI and ML workloads. That's why NVIDIA is the hottest publicly traded stock in tech. This has led to a surge in demand for GPUs, transforming them from niche components for gamers into critical hardware for AI research and deployment. As the demand for computing power continues to grow, so does the need for energy efficiency. Data centers, where much of the AI processing takes place, are notorious for their high energy consumption. This has led to a focus on sustainable chip design, optimizing power consumption, and exploring using recyclable materials. The semiconductor industry increasingly prioritizes sustainability initiatives, recognizing the opportunity to consume less energy and lower carbon emissions. The limitations of general-purpose chips in meeting the specific needs of AI workloads have led to the development of specialized AI chips. These chips, including GPUs, Field-Programmable Gate Arrays (FPGAs), and Application-Specific Integrated Circuits (ASICs), are optimized for the high-speed, parallel computations required by AI algorithms. T Looking ahead, the landscape of chip design is poised for significant change. Innovations such as 3D-IC technology, which allows for the stacking of integrated circuits, are expected to improve the efficiency and speed of electronic systems. Additionally, adopting open standard instruction set architectures like RISC-V is gaining momentum due to its energy efficiency and customizability. Marc Andreessen is famous for the saying, "Software is eating the world." but today, "AI is eating processors." and is doing so in gluttony.

Nvidia Supplier Ibiden Weighs Faster Expansion for AI Demand Ibiden Co., a century-old Japanese company, is scaling up its production of chip package substrates amid unprecedented demand. Substrates are essential for connecting Nvidia's AI chips to circuit boards, enabling seamless signal and power transmission while managing heat dissipation. Currently, the only mass producer capable of delivering these sophisticated substrates with high yields, Ibiden has cemented its position as Nvidia's key supplier while also serving Intel, AMD, and TSMC. AI semiconductor sales already account for over 15% of its revenue. Ibiden is racing to expand capacity by 2026, but concerns persist about whether this will meet surging demand. CEO Koji Kawashima emphasized Ibiden’s enduring partnership with Intel while positioning the company to support a broader range of AI players like Google and Microsoft as next-gen AI chips become the battleground for innovation. My Take  Ibiden’s dominance highlights a critical bottleneck in the AI supply chain: the ability to produce high-performance substrates at scale. For AI hardware manufacturers, cultivating resilient supply relationships with specialized partners like Ibiden will be crucial for ramping up to meet the escalating demands of AI adoption. #AI #Semiconductors #Nvidia #ChipSubstrates #Innovation #SupplyChain #EdgeAI #Ibiden Link to article: https://lnkd.in/eaD22TY3 Credit: Bloomberg This post was enhanced with AI assistance, thoroughly reviewed, edited, and reflects my own thoughts. Get Ahead with the Latest Tech Insights!  Explore my searchable blog: https://lnkd.in/eWESid86

A lot has changed since my #LLM inference article last January—it’s hard to believe a year has passed! The AI industry has pivoted from focusing solely on scaling model sizes to enhancing reasoning abilities during inference. This shift is driven by the recognition that simply increasing model parameters yields diminishing returns and that improving inference capabilities can lead to more efficient and intelligent AI systems. OpenAI's o1 and Google's Gemini 2.0 are examples of models that employ #InferenceTimeCompute. Some techniques include best-of-N sampling, which generates multiple outputs and selects the best one; iterative refinement, which allows the model to improve its initial answers; and speculative decoding. Self-verification lets the model check its own output, while adaptive inference-time computation dynamically allocates extra #GPU resources for challenging prompts. These methods represent a significant step toward more reasoning-driven inference. Another exciting trend is #AgenticWorkflows, where an AI agent, a SW program running on an inference server, breaks the queried task into multiple small tasks without requiring complex user prompts (prompt engineering may see end of life this year!). It then autonomously plans, executes, and monitors these tasks. In this process, it may run inference multiple times on the model while maintaining context across the runs. #TestTimeTraining takes things further by adapting models on the fly. This technique fine-tunes the model for new inputs, enhancing its performance. These advancements can complement each other. For example, an AI system may use agentic workflow to break down a task, apply inference-time computing to generate high-quality outputs at each step and employ test-time training to learn unexpected challenges. The result? Systems that are faster, smarter, and more adaptable. What does this mean for inference hardware and networking gear? Previously, most open-source models barely needed one GPU server, and inference was often done in front-end networks or by reusing the training networks. However, as the computational complexity of inference increases, more focus will be on building scale-up systems with hundreds of tightly interconnected GPUs or accelerators for inference flows. While Nvidia GPUs continue to dominate, other accelerators, especially from hyperscalers, would likely gain traction. Networking remains a critical piece of the puzzle. Can #Ethernet, with enhancements like compressed headers, link retries, and reduced latencies, rise to meet the demands of these scale-up systems? Or will we see a fragmented ecosystem of switches for non-Nvdia scale-up systems? My bet is on Ethernet. Its ubiquity makes it a strong contender for the job... Reflecting on the past year, it’s clear that AI progress isn’t just about making things bigger but smarter. The future looks more exciting as we rethink models, hardware, and networking. Here’s to what the 2025 will bring!

𝗧𝗟;𝗗𝗥: As AI evolves especially with LLMs, the hardware necessary to train and use them (inference) is also evolving fast. While GPUs are the foundation for AI models today, there are several hardware options emerging that could complement or even replace GPUs. 𝗚𝗣𝗨𝘀 have been the key enabler for AI models since Alexnet in 2012 (https://lnkd.in/eCQ8C7FW). 𝘘𝘶𝘪𝘤𝘬 𝘳𝘦𝘮𝘪𝘯𝘥𝘦𝘳 𝘸𝘩𝘺 𝘎𝘗𝘜𝘴 𝘢𝘳𝘦 𝘨𝘰𝘰𝘥 𝘧𝘰𝘳 𝘕𝘦𝘶𝘳𝘢𝘭 𝘕𝘦𝘵𝘸𝘰𝘳𝘬𝘴: 𝘩𝘵𝘵𝘱𝘴://𝘣𝘪𝘵.𝘭𝘺/3𝘈𝘙8𝘺𝘝𝘣. NVIDIA has been advancing GPUs rapidly but GPUs were not designed originally for AI use cases! Recent innovation (and investment) has focused on improving speed and lowering cost using AI focused chips and systems which will be focus of this post. But, first some basics on LLMs inferencing. 𝗟𝗟𝗠𝘀 are a unique class of AI models – They are 1/Very large (Billions of parameters) and hence need lots of memory and 2/Autoregressive which means for each word generation 𝘁𝗵𝗲 𝗲𝗻𝘁𝗶𝗿𝗲 𝗟𝗟𝗠 𝗻𝗲𝗲𝗱𝘀 𝘁𝗼 𝗽𝘂𝗹𝗹𝗲𝗱 𝗳𝗿𝗼𝗺 𝗺𝗲𝗺𝗼𝗿𝘆 𝘁𝗼 𝗚𝗣𝗨 which requires massive memory bandwidth. I wrote about this earlier: https://bit.ly/4dOuUFa. So how do you speed up model inference while lowering cost? Lets see how some do it: Cerebras Systems Inc. – Cerebras is one of the fastest AI hardware systems today (445 tokens per sec for Llama 3.1 70B). 𝗛𝗼𝘄 𝗶𝘁 𝘄𝗼𝗿𝗸𝘀: They just 𝗯𝘂𝗶𝗹𝗱 𝗯𝗶𝗴 𝗰𝗵𝗶𝗽𝘀 𝗮𝗻𝗱 house 𝗺𝗲𝗺𝗼𝗿𝘆 (𝘄𝗵𝗲𝗿𝗲 𝗺𝗼𝗱𝗲𝗹 𝗶𝘀 𝘀𝘁𝗼𝗿𝗲𝗱) 𝗰𝗹𝗼𝘀𝗲𝗿 𝘁𝗼 𝘁𝗵𝗲 𝗚𝗣𝗨, with the memory bandwidth at a staggering 21 Petabytes/s which is 7000x of the Nvidia H100! Some incredible hardware engineering.  𝗠𝗼𝗿𝗲 𝗵𝗲𝗿𝗲: https://bit.ly/3ZaxEbG Groq – 7 year old Groq found great product market fit in the last year & been doing some great work.   𝗛𝗼𝘄 𝗶𝘁 𝘄𝗼𝗿𝗸𝘀: Groq is all about a (very) smart 𝗰𝗼𝗺𝗽𝗶𝗹𝗲𝗿 combined with a 𝘀𝗶𝗺𝗽𝗹𝗲 𝗵𝗮𝗿𝗱𝘄𝗮𝗿𝗲 architecture with 𝗻𝗼 𝗸𝗲𝗿𝗻𝗲𝗹! Using very advanced 𝗗𝗮𝘁𝗮𝗳𝗹𝗼𝘄 architecture they can map out when an execution needs to be computed all on a deterministic compute layer. Groq afaik does not have a lot of on-chip memory which means for large models they will need LOTS of chips/racks but its all abstracted via Groq Cloud run by the awesome Sunny Madra and team.  𝗠𝗼𝗿𝗲 𝗵𝗲𝗿𝗲: https://bit.ly/3ZhNnFE. SambaNova Systems – SambaNova recently announced really fast throughput 114Tps for the large Llama 405B model   𝗛𝗼𝘄 𝗶𝘁 𝘄𝗼𝗿𝗸𝘀 – They are also a DataFlow architecture but they have memory on board which allows them to support larger models with (potentially) fewer chips & racks.  𝗠𝗼𝗿𝗲 𝗵𝗲𝗿𝗲: https://bit.ly/3XtbUGD Of course Amazon Web Services (AWS) has Trainium and Inferentia.  𝗠𝗼𝗿𝗲 𝗵𝗲𝗿𝗲: https://bit.ly/47gfiIo Many more good companies like d-Matrix, Tenstorrent, Etched in this space. 𝗔𝗰𝘁𝗶𝗼𝗻 𝗳𝗼𝗿 𝗖𝗧𝗢𝘀, 𝗖𝗔𝗜𝗢𝘀: Have a GPU diversification strategy to reduce risk, cost and improve perf!