Residual Path Integrals for Re-rendering

Most of the frame did not change

Before my PhD, I worked on an in-house offline renderer for interior design. One day, I was browsing our company database and found piles of images that artists had discarded after changing a scene design. Most of those renderings were simply wasted. That made me wonder whether we could reuse them when the scene changed instead of rendering the next image from scratch.

The same pattern appears far beyond interior design. In movies, animation, and interactive scene design, a frame often differs from the previous one in only a small region. Path tracing ignores that fact and starts over. The image difference has much lower energy than the full image, but only if we can estimate it without wasting samples on paths whose contribution is exactly zero.

Render only the affected paths

A path is dynamic when it contains a vertex on a moving object or an edge that crosses the object's old, ghost position. Every other path cancels in the residual integral. This turns re-rendering into a smaller sampling problem.

Why path mapping matters

Sampling an affected path is only half of the problem. We still need to evaluate corresponding paths in the old and new scenes. Dynamic geometry makes this difficult: a path can hit the moved object, cross its ghost position, become occluded, or diverge after the edit.

We spent substantial effort on path mappings related to shift mappings in gradient-domain rendering. We extended existing operators to dynamic scenes, introduced mappings tailored to our samplers, and used reconnection, rejection, and two-way mapping to keep the estimator correlated and unbiased. We call them path mappings because the mapped path is not produced by shifting the target pixel.

The operators, Jacobians, and failure cases are detailed in Section 5 of the paper.

From a residual to the new frame

We combine dynamic and ghost sampling techniques with multiple importance sampling, estimate the signed residual, and add it to the old frame. The framework also handles material edits as a simpler case where the dynamic and ghost objects coincide.

←

Moving geometry: Ninja The residual follows the moved ninja and shurikens while preserving their global lighting changes.

1 / 3 →

Rendering dissection

Limitations and open directions

The method is strongest when the edit occupies a small part of path space. Each limitation also points to an open problem.

Current limitations

• Large moving regions. As dynamic objects affect more paths, the advantage shrinks. In the worst case, every path is dynamic.
• Camera and light motion. The current formulation targets rigid-object movement and material edits, not dynamic cameras or lights.
• Uneven noise. Residual energy is spatially concentrated, so noise can be non-uniform and may need a renderer-aware denoiser.
• Glossy visibility. Visible glossy objects remain difficult, especially when small geometric changes alter high-frequency transport.

Open directions

• Sample proportionally to the actual difference integrand.
• Design better path and shift mappings, and optimize sampling and mapping jointly.
• Extend the framework to deformable motion, moving cameras, and changing lights.
• Combine residual sampling with real-time path reuse and denoising.
• Combine residual path sampling with the ReSTIR line of work.

Abstract

Conventional rendering techniques are primarily designed and optimized for single-frame rendering. In practical applications, such as scene editing and animation rendering, users frequently encounter scenes where only a small portion is modified between consecutive frames. In this paper, we develop a novel approach to incremental re-rendering of scenes with dynamic objects, where only a small part of a scene moves from one frame to the next. We formulate the difference (or residual) in the image between two frames as a (correlated) light-transport integral which we call the residual path integral. Efficient numerical solution of this integral then involves (1) devising importance sampling strategies to focus on paths with non-zero residual-transport contributions and (2) choosing appropriate mappings between the native path spaces of the two frames. We introduce a set of path importance sampling strategies that trace from the moving object(s) which are the sources of residual energy. We explore path mapping strategies that generalize those from gradient-domain path tracing to our importance sampling techniques specially for dynamic scenes. Additionally, our formulation can be applied to material editing as a simpler special case. We demonstrate speed-ups over previous correlated sampling of path differences and over rendering the new frame independently. Our formulation brings new insights into the re-rendering problem and paves the way for devising new types of sampling techniques and path mappings with different trade-offs.

Assets

BibTeX reference

@article{Xu:2023:ResidualPathIntegral,
  author  = {Bing Xu and Tzu-Mao Li and Iliyan Georgiev and Trevor Hedstrom and Ravi Ramamoorthi},
  title   = {Residual Path Integrals for Re-rendering},
  journal = {Computer Graphics Forum (Proceedings of EGSR)},
  year    = {2024},
  volume  = {43},
  number  = {4}
}