REST3D: Reconstructing Physically Stable 3D Scenes from a Single Image

avaer

Interested in how this generalizes outside of Isaac Gym/PhysX, because the devil is in the floats. For a stable simulation you want your initial conditions to be stable, that's what this model optimizes for. But stability tends to be highly dependent on your settings, timestep, etc. Mostly because physics engines are designed for speed, not perfect accuracy. I've only tackled this problem with shitty low-stakes (AI generated games) but I found heuristic post-optimization to be "good enough" -- essentially just stabilizing the simulation manually during scene bake. And predictably I had to rejigger the v-hacd and constants when I switched physics engines. So the real usefulness of something like this is how well it generalizes to the heterogeneous simulation ecosystem, but the paper doesn't seem to cover that.

BugsJustFindMe

> We first introduce an agentic physical scene understanding technique that constructs a scene-tree representation capturing object physical states and inter-object relationships from a gravity-support perspective, providing a structural prior for reconstruction. What an obnoxiously convoluted way of saying "We know that rooms have floor, walls, and ceiling, so we identify whether an object should be on the floor, wall, or ceiling." This could easily be a post-processing step on any other algorithm.