Download to read offline
Uploaded byYoss Cohen
Infrared simulation and processing on Nvidia platforms
This document discusses infrared simulation and processing on Nvidia platforms. It covers using 3D simulation to generate training data for computer vision and AI models since capturing and labeling real infrared data can be expensive. The simulation includes generating temperature maps, distance maps, and glitter effects. It also discusses challenges with infrared cameras like noise and data transfer. Sensor simulation models noise, lens effects, motion blur, and other factors. Frame processing techniques like inverse point spread function and non-uniformity correction are covered. Target acquisition and tracking algorithms like the double filter and Kalman filtering are also summarized. Finally, it discusses using Nvidia technologies like Rivermax for high bandwidth streaming of raw infrared video to GPUs for edge processing.
More Related Content
Similar to Infrared simulation and processing on Nvidia platforms
![Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticcommunityseries-day3-cfd-251120170304-ddef8112-thumbnail.jpg?width=640&height=640&fit=bounds)
Infrared simulation and processing on Nvidia platforms
- 1. Infrared simulation andprocessing On Nvidia Platforms
- 2. Neuronics • Developing projectsin • Computer vision & AI • 3D generation and simulation • Nvidia Edge and server platforms and SDKs • Defense / Military • UAV / Autonomous vehicles • Autonomous vassals • robots • Medical COMPUTER VISION & AI NVIDIA PLATFORMS 3D GENERATION & SIMULATION
- 3. Agenda •Solve IR cameraprocessing challenges
- 4.
- 5. Camera connection •Camera Link-> Capture card •GMSL – Fiber ( for distanced camera) •USB •GIGE •HDMI •RAW Transfer
- 6. Noise •DSNU - DarkSignal Non-Uniformity ~1% bias •PRNU - Photo-Response Non-Uniformity ~1% gain variability •Thermal noise •Sensor •Gain •Temp
- 7. Data problems •Capture &Label Data – expensive / not possible •Augmentation - Sensitive •Just changing brightness like in RGB images wont work! •Simulation •3D World simulation (Real, Blender, Omniverse, Unity) •Thermal assignment (simple) • Glitter simulation / Ray tracing •Atmospheric simulation (Frequency, distance, angle)
- 8. Simulation – Basic 3DWorld simulation Temperature Map Sensor Simulation Distance Map Glitter Map Shared config
- 9. Sensor Simulation –Advance Temperature to electrons Lens Shading PSF ) Point spread function( Motion Blur PRNU / DSNU NUC Simulation Shot Noise Electrons to sensor out Temperature map Distance Map Glitter Map Per frame exposure Distance Map Wave Gliter to sensor out The simulation is not standalone since exposure is dynamic and changed by the processing pipe
- 10. Frame Processor Inverse PSFs FixNUC Exposure estimation To sensor simulation
- 11. Target Acquisition Double Filter (Hadar) Morph operations Targetto track association Kalman filtering Main target selection + reporting
- 12. IR Recording andtransfer •We DON’T compress IR sensor output and pixel targets •Store in raw format 8/10/12 bit!!! •Sometimes we have problem in streaming it •Lossless JPEG, Lossless AV1 •Process at the edge - > Jetsons!!! •Stream Raw over fiber using NVIDIA Rivermax
- 13. Nvidia RiverMax •Nvidia architecturefor stream 10s-100s of GB direct to GPU •Used to stream RAW 8/10/12bit video 4K •Broadcasting quality
- 14. NVIDIA Video Analyticsflow Sensors Actions Capture and Decode Pre-processing & Batching AI Inference Tracking Composition Business Rules and Analytics … … H.265
- 15. Processing Pipe requirements •Process stream/s of video (and audio/other data synchronously) • Block based for each unit/function • Easy connection between camera / socket / encoder / decoder / file / parser • Ability to reuse and enhance existing modules (“Inheritance” + open source) • Fix all the threads / buffers on its own • Runs on Linux
- 16. Edge Processing Architecture •DeepStreamNvidia Standard processing architecture – Not used •Python CPP Processing •Basic filtering using OpenCV/CUDA , RAPIDs • Double Filter – DoG • Inverse PSF • Neural Networks • Segmentation • Binary Operations •Tracking – MTT with Kalman/EKF/UKF
- 17.
- 18.