Today, viewers expect crisp, richly detailed images in computer-generated films. But a decade ago, these images were much more challenging to produce. Creating high-quality images quickly without noise — a grainy speckling that obscures the underlying image — was a problem that plagued computer graphics. 

This is a particularly significant problem when trying to render a lot of images. Animated films from studios like Disney and Pixar require twenty-four frames a second; for a ninety-minute movie, that means close to 130,000 frames, most of which are rendered multiple times throughout the process of making the film. “When they were trying to generate all these images so quickly,” said UC Santa Barbara electrical and computer engineering (ECE) professor Pradeep Sen, “the algorithm they were using to estimate how light flows in the scene produced extremely noisy results, which are unacceptable for a feature film.”

In July 2015, Sen, Nima Kalantari, then an ECE PhD student, and fellow PhD student Steve Bako published a paper with their solution: applying machine learning, a nascent approach at the time, to speed up the process of generating images by removing the noise.  

In the paper, which appeared in the journal ACM Transactions on Graphics, the researchers described how they could train a neural network to produce high-quality, denoised images in just a few seconds.

More than a decade later, the influence of that idea is still being felt. The Association for Computing Machinery’s Special Interest Group on Computer Graphics and Interactive Techniques (ACM SIGGRAPH) — an international organization of scientists, artists, filmmakers, and industry members that hosts the world’s premier conference on computer graphics and interactive techniques — has honored the researchers’ paper with its Test of Time Award, which recognizes work that has had a lasting impact on computer graphics.

“I’m really honored that SIGGRAPH has recognized our work,” said Kalantari, the paper’s lead author and now an associate professor of computer science & engineering at Texas A&M University. “It’s incredibly rewarding to see it recognized more than a decade later.”

Speed v. Quality

“I was always fascinated by images and the process of generating images,” said Kalantari, and as a PhD student, he saw how time-consuming and challenging this process was. 

In the early 2010s, Sen’s research group was studying ways to render images more quickly with Monte Carlo rendering, a powerful rendering algorithm that traces multiple paths of light through a scene to compute photorealistic images. The more paths are calculated, the more realistic and accurate the image becomes, but also the longer it takes to produce. 

Speeding up the process meant computing fewer light paths, which resulted in lower-quality images filled with visual static — a result of the missing path information.  

“At the time, people had tried to use advanced image-processing filters to remove some of the noise,” Sen said. This worked, but it also blurred out much of the important detail in the images  he said, because the noise level in Monte Carlo rendering is inherently high. “People don’t want noise, but they don’t want to look at blurry images, either.”

Sen had done some initial work with another student that was that was the first to show that high-quality Monte Carlo denoising was indeed possible, by leveraging internal information computed by the rendering system. But their initial algorithm was manually implemented, making it brittle and difficult to use in production environments, where rendering tools must work consistently and robustly for thousands of frames.

Then in 2012, Geoffrey Hinton’s group published their now-famous “AlexNet” paper that showcased the power of deep learning for image classification. Sen immediately pivoted his group’s work to begin to look at applying these nascent technologies to problems in rendering. 

The researchers used machine learning to design a novel algorithm for Monte Carlo denoising, and trained the resulting neural network with a set of noisy rendered images. Then, they used their trained network to process new noisy images, finding that their approach produced images of much higher quality than the state-of-the-art approaches at the time. 

The results showed that the researchers could train a neural network to remove noise while preserving more image detail than conventional approaches.  

"We demonstrated for the first time that machine learning could be used for robust, high-quality denoising,” Sen said. “It was basically opening the door to be able to use these algorithms in production.”
 

Ripple Effects

After the paper was published, the idea quickly moved beyond the research lab. Sen and Bako, now an engineer at Aurora Innovation, worked with Pixar to incorporate their denoising approach into its production pipeline, while researchers and other studios developed and refined their own approaches. 

“Today, every single final frame of a computer-generated film is denoised,” Sen said. “And every one of those frames is computed partly using ideas that were first proposed in this paper.”

And movies aren’t the only place you’ll see what has blossomed from this paper. Architects designing new homes, online entrepreneurs developing appealing products, interior designers creating realistic room representations, and artists bringing their ideas to life all benefit from this learning-based approach to denoising. “The impact wasn’t just on movies and animation, but anywhere that you use rendering, ” Kalantari said. “I’m really proud that our team at UCSB was able to do work that has had such a lasting impact.”