Planning Where, How, and What

Abstract

Text-driven human motion generation in a 3D scene is usually cast as one mapping from prompt, scene and goal position to motion, and that framing carries two costs. The user has to supply the interaction goal, and the model has no internal structure, so it can answer a request only when that combination of instruction, scene and goal was already in its training data: what limits generalization is coverage rather than capacity.

We supply the missing structure, factoring the task into three planning stages that a diffusion synthesizer renders — a goal planner for where the interaction happens, grounding object names in the room’s geometry; a trajectory planner for how the body travels there; and a motion-state planner for what it does, blended from a learnable bank of reusable motion states. What passes between the stages is an explicit quantity a person can read — a point, a path, a motion state — and not a latent vector, so the three can be recombined at inference on configurations never seen together.

We build three hold-outs on LINGO isolating novel action compositions, novel rooms and novel object categories, retrain every baseline under each, and add a scene-contact measure that is not a frame average. Ours takes 19 of the 21 best scores — the lowest translation and orientation error on every axis, the least time spent inside scene geometry — with a third of the strongest baseline’s parameters. Our model can also predict the interaction goal instead of receiving it, which no baseline supports; even in that harder setting it reaches lower MPJPE and translation error than the baselines that are handed the goal.

Method

Four inputs — the global scene, the instruction, the local scene and the previous motion — are encoded once and consumed by three planners. Each planner emits a quantity a reader can inspect: a goal position, a path, and a motion-state anchor that initialises the diffusion denoiser.

Framework: encoders, goal position planner, trajectory planner, motion-state planner, diffusion

swipe the figure sideways

The motion-state planner reads a learnable bank of motion states through multi-head cosine attention, mixes the retrieved values with temporal queries, and produces the anchor that seeds the denoiser. The bank is updated only during training.

Fidelity does not buy generalization

Each axis is the mean of seven per-metric scores — FID, MPJPE, translation, orientation, contact and depth as distance to ground truth, and sustained penetration time — each mapped onto [0, 1] over the six values it takes across three methods and two regimes. One shared range per metric is what makes the diagonal readable: a point on y = x scores the same on a held-out axis as on the split it was fitted to. ReMoGen matches our standard-split score to 0.001 and loses 0.52 under hold-out, where we lose 0.02.

Scatter of in-distribution fidelity against held-out generalization; ours sits on the diagonal

Qualitative comparison

Every case below is held out. All four methods are rendered from the same camera; the green ring marks the start and the red ring the goal position. Baselines were retrained under the identical hold-out.

BibTeX

@inproceedings{anonymous2026planning,
  title     = {Planning Where, How, and What: Generalizable Text-Driven
               Human--Scene Interaction Synthesis},
  author    = {Anonymous},
  booktitle = {Under review},
  year      = {2026}
}