How to integrate neural shading into graphics pipeline

Excited to share this piece I wrote for the NVIDIA Developer technical blog, breaking down the basics of neural shading, and covering the resources you need to get started using these techniques in your own code!

Following my foundational dive into Vulkan (and the low-level connections I shared last time), I've moved into the next critical chapter: Shaders—the dynamic heart of the GPU pipeline! It's been a truly insightful phase working directly with GLSL and compiling it into SPIR-V using glslc.exe. This intermediate representation is the key to Vulkan's brilliant design, enabling its strong cross-platform and cross-architecture compatibility. Along the way, I’ve been exploring powerful optimisation techniques essential for achieving real-world performance: •Specialisation Constants: A smart way to adapt shader logic at pipeline creation time without costly full recompilation. •Element Index Buffers: Crucial for memory efficiency by eliminating unnecessary vertex duplication. •Primitive Restarts: Enabling efficient rendering of multiple disconnected primitives within a single, optimised draw call. These low-level insights underscore the careful engineering required for every rendered frame. Next up, I'm eager to connect the dots and explore how these shader systems integrate into the graphics pipeline's initial stages: focusing on vertex input, assembly, and rasterisation. Learning Vulkan continues to be a highly rewarding challenge in my developer journey, offering a deeper understanding of the architecture beneath modern graphics APIs. #Vulkan #GraphicsProgramming #LowLevelProgramming #SPIRV #Rendering

3D scenes from a single image or text prompt in seconds?! 🤯 MAC Lab, Xiamen University, Tencent and Yes Lab, Fudan University present "⚡ FlashWorld: High-quality 3D Scene Generation within Seconds" Nice TL; DR from authors: "FlashWorld enables fast (5~10 seconds on a single GPU) and high-quality 3D scene generation across diverse scenes, from a single image or text prompt." Checkout the details in the comments below #machinelearning #computervision #3Dreconstruction #generativemodel #foundationalmodel #futurism #gaussiansplatting #3DGS

Surprises an limitations of LLMs. The 3D video generation is new surprises from LLMs labs. What are the limitations? Generally, the length of the context window is limited by a combination of factors: computational resources (quadratic complexity), the volume of device memory, and the need to retrain the model to work with new, larger positions. Therefore, enlarging the window infinitely is unprofitable and difficult, and the industry is parallel exploring other approaches - for example, storing knowledge in external bases and selective searching for information instead of providing the whole context together, but these are all external crutches. Integrating AI into commercial and business applications is impossible with a limited and extremely unstable context window, but NO company has provided an effective solution. These are basic but not all limitations of transformers. Memory gap: the most serious limitation is that transformers do not have permanent, long-term memory. They are not able to train on the summer in the course of interaction with the user. Every new fact or skill requires an expensive process of retraining or a complete retraining of the model. This radically distinguishes them from biological intelligence, which is taught continuously and incrementally. A context window is just a temporary buffer, not a mechanism for accumulation and integration of knowledge. Now LLMs are a completely isolated "black box" from the outside world, architecturally NOT able to self-learn and at its core cannot be considered intelligence, because the first sign of intelligence is the ability to learn. The problem of "grounding": models are trained on texts, not on interaction with the real world. Their "understanding" is a statistical analysis of patterns in the data, and not a meaningful correlation of symbols with real objects, actions and their consequences. LLMs are unable to construct abstract ideas of how the world works. This is guaranteed to lead to hallucinations. This restriction can only be circumvented in part of the so-called "physical AI," Huang mentioned, but it requires a series of separate posts to reveal this direction. Inborn inflexibility: the architecture of the transformer is static. After completing the training, the weight of the neurons are fixed. A model cannot dynamically create new connections ("synapses") or alter its structure in response to new experiences as the human brain does. This lack of plasticity means that LLMs are not truly adaptive systems. The failure of cognitive functions. "Today's architectures suffer from a limited ability to reason clearly and understand causal relationships." They statistically predict the next word based on patterns in the data, but lack innate "common sense" or true understanding of the world Overall, these constraints show that transformer architecture, despite all its power, is a dead end to the creation of universal intelligence.

🚀 PlayCanvas Engine v2.13.0 is live! This update delivers major enhancements for 3D Gaussian Splatting on the web - expanding both performance and creative flexibility. ✨ Highlights ✅ Streamed LOD system - dynamically load and manage massive splat datasets. 🎨 Splat shader effects framework - new base for reveal, hide, tint, and bloom effects. 🌍 Globally sorted splats - improved rendering quality and stability for large-scale scenes. Plus a long list of refinements across the engine - from shader customization and GPU optimizations to lighting and input fixes. 🔗 Full release notes: https://lnkd.in/eYYsB2ty 🎮 Live examples: https://lnkd.in/e2shW8ST Huge thanks to our amazing contributors and community for continuing to push what’s possible in real-time web graphics. #PlayCanvas #WebGPU #WebGL #3DGS #GaussianSplatting #Web3D #OpenSource #GameDev #RealtimeGraphics

Hi all! Been experimenting with transparency in OpenGL, and came across an approach presented by Nvidia called "Depth Peeling". Usually transparency needs to be rendered back-to-front so you can "see" through it, but with this approach (Depth Peeling) it is order independent. The way it works, or how I understand it, is you render the transparent objects in several passes with each one only drawing the front-most surfaces. Then repeat X times, drawing the next front-most surface. Then at the end you simply draw each peel layer back-to-front. It does need to be said that this does *not* generally improve performance, it is simply a way to avoid having to sort transparent fragments. Also, depth-peeling would be more helpful with complex, self-intersecting geometry like a knot where sorting the fragments would be very difficult. Unlike the simple scene presented here. #opengl #cpp #gamedev

Stop debating between CPU rendering and GPU rendering. You're likely both correct. Often, the "CPU vs. GPU" discussion misses the objective, which is not which is better - it's which is best for the task. This is not just a speed difference - it is a fundamental difference in architecture designed for different computational physics. See it this way: 🔹 A CPU is your master chef: detail-oriented, precise, and excellent at performing sequential, complicated tasks like high-fidelity path tracing. 🔹 A GPU is the factory and assembly line of rendering: a parallel beast with thousands of cores, and optimized for the notoriously heavy, repetitive task of real-time rasterization. Ultimately, understanding why they are different - all the way down to the physics of light, like Fresnel Equations and Lambert's Cosine Law is the key to an optimized pipeline. I put out a deep dive that tackles the physics, the math, and gives practical application for artists in Blender, Maya, and Unreal. I made it a klittle heavy than usual so that people can understand a little how the computational works behind the scene. Read the full analysis: https://lnkd.in/dcVsApU4 #CPUvsGPU #3DRendering #VFX #GameDev #Animation #ComputerGraphics #RayTracing #RealTimeRendering #Blender #UnrealEngine

New Project Started — Building a Real-Time Ray Tracer I’ve started working on a GPU-based ray tracer using OpenGL, SDL3, and GLSL shaders. Currently, I’ve implemented: 1. Camera setup & ray generation 2. Multiple object (sphere) intersection logic 3. Surface normals visualization. Next, I’ll be working on diffuse lighting to make the shading more realistic. This project is helping me deeply understand computer graphics, coordinate systems, and how light interacts with surfaces — beyond just rendering frameworks. 🔗 GitHub: https://lnkd.in/gTt9Y-cM #OpenGL #RayTracing #ComputerGraphics #ShaderProgramming #LearningJourney

Neural shading integrates trainable models directly into the graphics pipeline, allowing for higher fidelity beyond traditional computational limits. But how do we integrate this approach? Our latest blog walks you through the basics, lays out some techniques for optimization, and has resources to get started.