NeuSample: Importance Sampling for Neural Materials

A neural material is not renderer-ready until it can sample itself

Neural materials can reproduce complex spatially varying appearance, including displacement, anisotropy, and layered reflectance. But a Monte Carlo renderer needs three operations: evaluate the material, sample a direction, and evaluate that sample's probability.

The first operation received most of the attention. Without the other two, a renderer falls back to cosine sampling or a rough analytic approximation and spends rays in directions where the material contributes little.

Three ways to learn where to sample

←

Analytic-lobe mixture A small MLP predicts one Lambertian and two anisotropic Gaussian lobes.

1 / 3 →

Limitations and open directions

• Per-material training. The sampler must currently be trained and precomputed for each neural material.
• Simple materials. Extra inference can outweigh better sampling when an analytic lobe already describes the material well.
• Rapidly changing lobes. Histogram mixtures can miss rare, abrupt changes across nearby material conditions.
• Beyond BRDFs. Extending the projected-hemisphere formulation to full BSDFs remains open.
• Beyond materials. The same conditional sampling ideas may help path guiding, complex luminaires, and portal sampling.

Abstract

Neural material representations have recently been proposed to augment the material appearance toolbox used in realistic rendering. These models are successful at tasks ranging from measured BTF compression, through efficient rendering of synthetic displaced materials with occlusions, to BSDF layering. However, importance sampling has been an after-thought in most neural material approaches, and has been handled by inefficient cosine-hemisphere sampling or mixing it with an additional simple analytic lobe. In this paper we fill that gap, by evaluating and comparing various pdf-learning approaches for sampling spatially varying neural materials, and proposing new variations of these approaches. We investigate three sampling approaches: analytic-lobe mixtures, normalizing flows, and histogram prediction. Within each type, we introduce improvements beyond previous work, and we extensively evaluate and compare these approaches in terms of sampling rate, wall-clock time, and final visual quality. Our versions of normalizing flows and histogram mixtures perform well and can be used in practical rendering systems, potentially facilitating the broader adoption of neural material models in production.

Recent work on neural material sampling

Recent projects explore closely related neural-material representations and sampling strategies.

Assets

BibTeX reference

@article{Xu:2023:NeuSample,
  author  = {Bing Xu and Liwen Wu and Milo\v{s} Ha\v{s}an and Fujun Luan and Iliyan Georgiev and Zexiang Xu and Ravi Ramamoorthi},
  title   = {NeuSample: Importance Sampling for Neural Materials},
  journal = {ACM Transactions on Graphics (Proceedings of SIGGRAPH)},
  year    = {2023},
  volume  = {42},
  number  = {4},
  doi     = {10.1145/3588432.3591524}
}