Given an instructional origami video, FoldingAgent reconstructs the full folding process: from a sequence of keyframes to a sequence of 3D geometric states, enabling re-rendering, editing, and analysis.
For each transition we show the source keyframe next to the state our method recovers, in three views: a rendered diagram of the paper, where front- and back-side colouring exposes the order the layers stack in; a layer-count map, giving the number of sheets of paper at each point of the model; and the crease pattern accumulated so far over the unfolded square, with mountain creases (the paper ridges up) and valley creases (it sinks down) marked separately.
Rather than trying to directly infer the full crease pattern (CP), a rigid, hard-to-edit representation, we reformulate the problem as predicting the transition between two states. We define a simple, intuitive action language (add_vertex, fold, unfold, rotate, flip), and build a deterministic simulator that takes a geometric state and an action, and returns the updated state.
The task is now narrow enough to state in one line: given the source keyframe, the target keyframe, and the source state, choose an action or short sequence of actions. The simulator executes it, produces the next state, and the agent commits that state as the answer for that keyframe before moving on. Working transition by transition is what makes the hidden part of the state recoverable. The layer structure is never visible in any frame, but it is derivable: each action's effect on the layers is determined, so following the video's temporal order builds the internal configuration up incrementally instead of asking a model to guess it from a single-view.
The agent is a frontier vision-language model that works through a tool library: It can view all keyframes and the motion between them, apply the transitions to the current state or render via the simulator, it has access to the memory of previously saved states, and a critic that judges whether a rendered state matches the target frame.
Verification is delegated to a separate vision-language model. When the agent thinks a transition is done, the critic receives the real source and target keyframes, and the rendered diagrams of the corresponding states, and returns Match, Mismatch, or Extreme Divergence, with a written account of the discrepancies.
The reconstruction becomes a search over a tree of paper states rather than a single forward pass. Since folding errors often only reveal themselves several steps after they occur, the agent can backtrack through this tree to the earliest point of divergence and re-plan from there, returning error correction from a per-step requirement into a property of the search itself, which is what makes long, multi-step sequences tractable.
@inproceedings{moriya2026foldingagent,
title = {FoldingAgent: Inferring Parametric Origami Procedures from Demonstration Videos},
author = {Moriya, Maya and Raab, Sigal and Vinker, Yael and Dekel, Tali},
booktitle = {SIGGRAPH Asia 2026 Conference Papers},
year = {2026}
}