Skip to content
Theme
OnDevice Tools

SoftwareDeveloper Tools

NVIDIA Omniverse developers use frontier AI agents to build simulations

NVIDIA highlights projects where developers pair frontier AI models with Omniverse libraries to assemble, test and refine simulations, from factory digital twins to hospital robot trials.

Illustration of developers guiding AI agents to build 3D simulations on NVIDIA Omniverse

Key takeaways

  • NVIDIA's blog showcases developer projects pairing frontier AI models such as GPT-6 Astra and Sol with Omniverse libraries for simulation work.
  • Falkor Surface turns CAD and engineering data into facility-scale digital twins, running on Azure with RTX PRO 6000 Server Edition GPUs.
  • A voice-controlled VR demo was built over a weekend, with the Claude Code agent working about eight hours.
  • In a 256-ward hospital simulation, 766 of 768 Microduck robots stayed upright after 14 simulated seconds.
In this article
  1. What NVIDIA is showing
  2. Digital twins for industrial operations
  3. Robot and VR demos built with agents
  4. Testing robot walking policies at scale
  5. 3D asset creation with Houdini
  6. Why it matters

What NVIDIA is showing

NVIDIA has published a roundup of developer projects that combine frontier AI models with its Omniverse libraries, according to the NVIDIA Blog. The idea is that turning a simulation concept into a working application involves several stages: gathering 3D assets, wiring up physics and rendering, and confirming the scene behaves correctly. AI agents guided by plain-language instructions can now help with much of that work.

The projects span industrial planning, robotics, virtual reality and 3D content creation. NVIDIA says it will add further examples from its own teams and from developers across the ecosystem over time.

Digital twins for industrial operations

One example comes from Falkor, a KONGSBERG company. Its Falkor Surface platform converts customers' computer-aided design and engineering data into facility-scale digital twins, which are virtual replicas used to test plans and decisions before acting in the real world. The platform runs on Microsoft Azure using NVIDIA RTX PRO 6000 Server Edition GPUs and Omniverse libraries for physics simulation, with the GPT-6 Astra model powering the agents.

According to NVIDIA, these agents can read dense technical documentation, understand spatial relationships and apply real-world physics. Engineers can rehearse plans in simulation before travelling to a site, which the company says reduces costly delays and mistakes, and can inspect equipment and check clearances in a shared view.

Robot and VR demos built with agents

Illustration: AI, VR headset, robot, code and the labels Claude Code, Digital Twins, Isaac Sim

NVIDIA developers Dillon Bailey and Rishabh Chadha built a pick-and-place simulation in the Isaac Sim framework using assets from the SimReady Central library. They directed an agent running the GPT-6 Sol model to assemble the scene and generate the controls for a robot moving a can. Testing covered joint movement, contact behaviour and whether the gripper could hold, move and release the can, giving a baseline for further adjustments.

Community developer Michael Wagner took a different route. He used GPT-6 Astra with Blender to model a study environment from two reference images, then used Claude Code to build a voice-controlled VR application over a weekend. The agent worked for roughly eight hours, and the app ran on Omniverse libraries including ovstage for scene sharing, ovrtx for rendering, ovxr for VR and ovphysx for physics, tested on an RTX 6000 Ada workstation with a Meta Quest 3 headset.

Wagner's main difficulty was verifying what happened inside the headset. Automated tests missed issues he could feel, such as objects shaking when grabbed. Reporting the symptoms and guiding the agent through changes fixed the problem, and he advises giving agents renders and logs to inspect, then checking the experience directly. He has released the demo as open source.

Testing robot walking policies at scale

The assets carried Newton physics settings, so no manual collision geometry was needed, and the Newton physics engine let hundreds of ward copies run together on a single RTX PRO 6000 GPU.

Treating furniture as static collision objects allowed larger runs within the solver's limits. In a 256-ward run, 766 of 768 robots were still upright after 14 simulated seconds. Panahi said the agent's work took about two hours, and the demo is open source.

3D asset creation with Houdini

Nick Scarcella, a senior technical artist on NVIDIA's BondLab team, directed agents, mainly using Claude, to work with CUDA libraries and Isaac Sim. In one experiment he guided Astra to recreate an OP-1 portable synthesizer from product images and manuals, exported it to the OpenUSD framework, tested it in Houdini and Blender, then refined materials and textures for ovrtx rendering.

Scarcella also worked with a publicly shared robot-hand checkpoint in Isaac Sim, developing motion, materials and lighting in Houdini with Claude's help. With team member Steven Dvornik, the team produced an Echos visualisation in which translucent copies of the hand overlap and merge.

Why it matters

The roundup suggests a shift in how simulation software gets built: instead of hand-assembling every asset and control, developers describe what they want and steer agents through review and correction. The examples also show where human judgement remains essential, particularly in verifying experiences that automated tests cannot capture.

For teams working on robotics, digital twins or VR, the published demos include starter prompts, checks and build instructions, offering a practical entry point. Readers should note the projects come from NVIDIA's own blog, so the results are the company's and its partners' own accounts rather than independent measurements.

Sources

  1. Into the Omniverse: How Developers Turn Ideas Into Simulations With Frontier AI AgentsNVIDIA Blog

Tags

Spotted a mistake? See our corrections policy. Read how News by OnDevice works in our editorial policy.