Beyond Photorealism

Gaussian Splatting achieves photorealism using thousands or millions of tiny points of light and color to replicate a real-life scene from novel perspectives. It’s the secret sauce that powers Apple’s new spatial photo and reframing features, and it works similarly to pointillism: your eye doesn’t see the individual nodes, it sees the gestalt of everything together.

I’ve been using splats for a few years as part of my mobile diorama codebase, which runs on the web. Since the work targets phones I have a strict technical budget to meet.

To maximize total number of points rendered, early versions of my rendering codebase trimmed out the spherical harmonics, which are the complex view-dependent lighting effects that make reflections and surfaces in splats feel “real.” This means my splats have always had a sort of lo-fi painterly look.

Since I’ve been showing them on small screens from a distance, it was never much of an issue, but I’m currently working on a diorama which uses a Google Cardboard lens to put the viewer’s eyes right up against the screen, which means that the details of the splats are a lot more apparent. I wanted to work with what I had within my technical budget, so I decided to push the “painterly” aspect further.

Geometry

In a standard splat, the shape of each node is a soft radial ellipsoid gradient. I added a macro to control splat shape, ranging from a rhombus to the original ellipse to a square. This lets me change the feel of the “brush strokes,” and a second variable controlling the falloff lets me shift from a “watercolor” feel to a “pop-art” feel.

Texture

Mathematically perfect gradients feel very sterile; painting usually has a little more mess. I added a screen-space noise function to crisp up the high frequencies and help make the splats feel a little less mechanical.

I had originally experimented with splat-space noise, but it had issues on rotation. Screen-space noise makes sure that the noise pattern stays consistent as the camera rotates or moves through the scene.

The noise affects both the alpha transparency and the color of each pixel, breaking up the splat shapes and adding some tasty grit to the scene.

Chromatic Fringing

I tend to only half-mix my colors when I’m painting, so I get streaks of base colors in my painted strokes.

For a similar effect, I modulate a rotation in hue space by distance from the center of a single node. As you move from the center to the edge of a single node, the color shifts, similar to chromatic aberration or a half-mixed pigment. This gives me more control over color harmony and more artistic control over the scene. Colors can be tinted however I want, naturalistic or not, depending on my needs.

Depth of Field

At the suggestion of @hypha (thank you!!), I added depth of field to the codebase, inspired by DOF-GS, which blurs each splat by its circle of confusion. My version diverges in a few places:

• Authorable focus planes instead of a lens model. DOF-GS derives blur from a physical camera’s aperture and focal distance. I place focus planes by hand in the editor, and blur ramps up and saturates with distance from them. I can also place “unfocus” planes that blur things near them. It’s less physically accurate, but this is a painting, not a photograph, and I want to decide where the eye goes.
• Render-time only. DOF-GS learns defocus during training. Mine is applied at render time to splats that were trained normally, so I can add it to any existing scan.
• Energy-conserving blur. Blur grows each splat’s footprint, so I dim its alpha to match. Otherwise blurry areas get brighter and more opaque instead of softer, and they cost more to draw.

This caused issues, because higher blur is more work to compute and drops frames, so I added a framerate-reactive splat culling system which tries to balance splat quality with framerate. This helped this feature, but it also expanded the number of devices capable of running the dioramas in the first place, and raised the ceiling on render quality for viewers on something like a high-power computer. That, plus the use of the SOG format, which stores much of the splat data as WebP images, meant that I was also able to bring back spherical harmonics in the renderer. Yay!

Conclusion

In the pursuit of technical excellence and pushing the envelope, it can be easy to lose sight of the imperfections that make art pieces unique. Let this be a sort of call to arms to add systems to your shaders that let you perturb values inside them, especially when the source data comes from real-life objects.