Affiliation withheld
Under review · 2026
Generating human motion that follows a natural-language instruction inside a 3D scene is usually cast as a single mapping from a prompt, a scene and a goal position to a motion sequence. Two limitations follow. The interaction goal has to be supplied by the user, which returns the hardest part of scene understanding to the person holding the instruction, and a mapping with no internal structure can answer only those requests whose combination of instruction, scene and goal it has already seen.
We factor the task into three planning stages, each predicting an explicit intermediate quantity, and render the motion from them with a diffusion synthesizer. A goal position planner predicts where the interaction happens, compressing the room with a multi-resolution hash-grid signed distance field and grounding object names through a frozen CLIP encoder, so no goal need be specified by hand. A trajectory planner predicts how the body travels there, and a motion-state planner predicts what it does by retrieving and blending entries of a learnable bank of motion states. Because every stage exposes a readable quantity rather than a latent, the stages recombine at inference on configurations never seen jointly in training.
We evaluate on three zero-shot hold-outs of LINGO that isolate novel action composition, novel rooms and novel object categories, retraining every baseline under the identical hold-out and scoring every method against one shared ground truth. The distribution metrics divide three ways, but the metrics that measure whether the interaction actually happened do not: our method attains the lowest translation and orientation error on all three axes and the best contact F1 on two of them, holding 18 of the 21 best scores across seven metrics and three axes. Removing the learnable memory raises FID by 4.4× on novel action combinations and 2.3× on novel rooms, and lowers it on novel goal categories — the one axis whose held-out category the memory cannot have stored.
Four inputs — the global scene, the instruction, the local scene and the previous motion — are encoded once and consumed by three planners. Each planner emits a quantity a reader can inspect: a goal position, a path, and a motion-state anchor that initialises the diffusion denoiser.
The motion-state planner reads a learnable bank of motion states through multi-head cosine attention, mixes the retrieved values with temporal queries, and produces the anchor that seeds the denoiser. The bank is updated only during training.
Every case below is held out. All four methods are rendered from the same camera; the green ring marks the start and the red ring the goal position. Baselines were retrained under the identical hold-out.
@inproceedings{anonymous2026planning,
title = {From Matching to Planning: Generalizable Text-Driven
Human--Scene Interaction Synthesis},
author = {Anonymous},
booktitle = {Under review},
year = {2026}
}