Planning Where, How, and What:
Generalizable Text-Driven Human–Scene Interaction Synthesis

Anonymous Authors

Affiliation withheld

Under review · 2026

Paper (coming) arXiv (coming) Code (coming) Video results ↓
Three generalization settings: training row, ReMoGen row, our row

swipe the figure sideways to see all three columns

Grey: motion the training corpus contains, one panel per constituent action. Red: ReMoGen on the held-out request. Colour: ours on the same request, room and camera, so the rows differ only in the motion. Left, an instruction composing actions the corpus demonstrates only apart; centre, a room whose layout no training scene resembles; right, a goal on an object category that never appears as an interaction goal in training, with a related category kept.

Abstract

Text-driven human motion in a 3D scene is usually cast as one mapping from prompt, scene and goal position to motion. The goal is then the user’s to supply, and a mapping with no internal structure answers only requests whose combination of instruction, scene and goal it has already seen: what bounds generalization is coverage, not capacity.

We supply the missing structure, factoring the task into three planning stages that a diffusion synthesizer renders — a goal planner for where the interaction happens, reading the room through a multi-resolution hash-grid signed distance field grounded to object names by a frozen CLIP encoder; a trajectory planner for how the body travels there; and a motion-state planner for what it does, blended from a learnable bank of reusable motion states. Each stage emits an explicit state that can be read, replaced or reused rather than a latent, so the three can be recombined at inference on configurations never seen together.

We build three hold-outs on LINGO isolating novel action compositions, novel rooms and novel object categories, retrain every baseline under each, and add a scene-contact measure that is not a frame average. Ours takes 19 of the 21 best scores — the lowest translation and orientation error on every axis, the least time spent inside scene geometry — with a third of the strongest baseline’s parameters. Asked to predict the goal rather than receive it, a setting no baseline supports, it still beats baselines that are given it.

Method

Four inputs — the global scene, the instruction, the local scene and the previous motion — are encoded once and consumed by three planners. Each planner emits a quantity a reader can inspect: a goal position, a path, and a motion-state anchor that initialises the diffusion denoiser.

Framework: encoders, goal position planner, trajectory planner, motion-state planner, diffusion

swipe the figure sideways

The motion-state planner reads a learnable bank of motion states through multi-head cosine attention, mixes the retrieved values with temporal queries, and produces the anchor that seeds the denoiser. The bank is updated only during training.

Motion-state planner architecture

Fidelity does not buy generalization

Each axis is the mean of seven per-metric scores — FID, MPJPE, translation, orientation, contact and depth as distance to ground truth, and sustained penetration time — each mapped onto [0, 1] over the six values it takes across three methods and two regimes. One shared range per metric is what makes the diagonal readable: a point on y = x scores the same on a held-out axis as on the split it was fitted to. ReMoGen matches our standard-split score to 0.001 and loses 0.52 under hold-out, where we lose 0.02.

Scatter of in-distribution fidelity against held-out generalization; ours sits on the diagonal

Qualitative comparison

Every case below is held out. All four methods are rendered from the same camera; the green ring marks the start and the red ring the goal position. Baselines were retrained under the identical hold-out.

BibTeX

@inproceedings{anonymous2026planning,
  title     = {Planning Where, How, and What: Generalizable Text-Driven
               Human--Scene Interaction Synthesis},
  author    = {Anonymous},
  booktitle = {Under review},
  year      = {2026}
}