Abstract
Conventional rendering techniques are primarily designed and optimized for single-frame rendering. In practical applications, such as scene editing and animation rendering, users frequently encounter scenes where only a small portion is modified between consecutive frames. In this paper, we develop a novel approach to incremental re-rendering of scenes with dynamic objects, where only a small part of a scene moves from one frame to the next. We formulate the difference (or residual) in the image between two frames as a (correlated) light-transport integral which we call the residual path integral. Efficient numerical solution of this integral then involves (1) devising importance sampling strategies to focus on paths with non-zero residual-transport contributions and (2) choosing appropriate mappings between the native path spaces of the two frames. We introduce a set of path importance sampling strategies that trace from the moving object(s) which are the sources of residual energy. We explore path mapping strategies that generalize those from gradient-domain path tracing to our importance sampling techniques specially for dynamic scenes. Additionally, our formulation can be applied to material editing as a simpler special case. We demonstrate speed-ups over previous correlated sampling of path differences and over rendering the new frame independently. Our formulation brings new insights into the re-rendering problem and paves the way for devising new types of sampling techniques and path mappings with different trade-offs.
Assets
Paper (PDF, 18M)Arxiv preprint 
CPU Code (coming in Oct!)Real-time version (scheduled)Little talks, dirty paws
We hope to also implement a real-time version. From a practical standpoint, disregarding theoretical contributions, the immediate step to make this potentially feasible for the industry is to integrate it with 1) real-time solutions for camera movements which are also common senarios for applications; 2) various types of path reuse; and 3) a denoiser, preferablly talored for the residual rendering. The current residual-path-integral framework is orthogonal to those directions, but those puzzle parts will make it shippable to be a complete real (toy) product. Welcome any thoughts or collaborations if you are interested!Acknolwedgements
This work was funded in part by NSF grants 2105806, 2212085, 2110409, gifts from Adobe, Google, the Ronald L. Graham Chair and the UC San Diego Center for Visual Computing. We especially thank Shilin Zhu for countless discussions in the early stages of this project. We thank Rui Tang, Lifan Wu, Ling-Qi Yan, Fujun Luan, Alexandr Kuznetsov, Yash Belhe for earlier discussions; Xuanda Yang, Yang Zhou for later performance discussions; Varun Munagala for early visualization experiments, and the anonymous reviewers for their constructive feedback. Ninja model is modified from the model by blendermagician on Blendswap; Shuriken model is created by RadialRonnie; Book model is created by AnyRPG and Lamp model is modified from the original one created by Manan Dhiman. Sponza model courtesy of Marko Dabrovic and Mihovil Odak. Other scenes courtesy of Benedikt Bitterli.BibTeX reference
@article{Xu:2023:ResidualPathIntegral,
author = {Bing Xu and Tzu-Mao Li and Iliyan Georgiev and Trevor Hedstrom and Ravi Ramamoorthi},
title = {Residual path integrals for re-rendering},
journal = {Computer Graphics Forum (Proceedings of EGSR)},
year = {2024},
number = {4},
volume = {43}
}