Gaussian splatting,
explained.

This rung supersedes rung 9 on the axis rung 9 lost on, and it does it by reversing the representation. Rung 9's cost is structural: every sample along every ray is a network evaluation, so rendering cost is the product of pixels, samples per ray and the size of the network. No amount of optimisation removes a term from that product.

The change is to represent the scene as millions of 3D Gaussians — each with a position, an anisotropic extent and orientation, an opacity and a view-dependent colour — and to render by projecting them onto the image plane and compositing them front to back. That is rasterisation: exactly the operation graphics hardware was built to do, at enormous rates. There is no network to query while rendering, because the scene is data rather than a function.

Crucially it keeps rung 9's method while discarding its representation. The rasteriser is differentiable, so the Gaussians are still fitted by rendering and comparing against the photographs, with the optimiser splitting and cloning Gaussians where detail is missing and pruning transparent ones — so the scene's complexity is discovered rather than fixed. Stated in rung 9's units: training in minutes rather than days, rendering at over a hundred frames per second rather than seconds per frame, at comparable or better quality. That is not an incremental win; it is the difference between a research artefact and something usable interactively.

Neural radiance fields (NeRF) represent a scene implicitly: a small neural network that, given a 3D point and a viewing direction, returns a color and density. To render one pixel you cast a ray into the scene and march along it, querying that network at many sample points and integrating the results. Gorgeous quality — but that's dozens or hundreds of network evaluations per pixel, per frame. Rendering is seconds per image, nowhere near interactive.

Gaussian Splatting asks: what if the scene were explicit — a concrete geometric thing you could draw directly, the way GPUs love to draw triangles — instead of a function you have to probe? Then rendering becomes rasterization, which hardware does astonishingly fast. The trick is choosing a primitive that's explicit and drawable, yet soft enough to represent fuzzy real-world appearance and be optimized smoothly. That primitive is the 3D Gaussian.

The scene is a collection of 3D Gaussians, each a soft ellipsoidal blob defined by four things: a position (its center), a covariance (its shape and orientation — how stretched, in which directions), a color (view-dependent, via spherical harmonics), and an opacity. A single Gaussian is a smooth lump of colored, semi-transparent fog; a million of them, arranged in space, can reconstruct a whole scene in fine detail.

The softness matters. Because each Gaussian falls off smoothly from its center — its footprint is densest in the middle and fades outward — the representation has no hard edges to alias, and every parameter is differentiable. That means the whole scene can be optimized by gradient descent (section 04), which a mesh of hard triangles can't be nearly as gracefully. Explicit enough to rasterize, soft enough to optimize: that combination is the whole design.

To render a view: project every 3D Gaussian onto the image plane (a "splat" — a 2D Gaussian footprint), sort them by depth, and blend them front-to-back with alpha compositing. A pixel's color accumulates each Gaussian's contribution, weighted by how much light is still unblocked by everything in front of it — the transmittance:

Transmittance T_i is the running product of "what got past" all the nearer, more opaque splats, so a fully opaque Gaussian in front (α = 1) drives T to zero and occludes everything behind it — exactly as physical over-painting should. It's the same volume-rendering equation NeRF integrates, but applied to a sorted, discrete set of Gaussians you can rasterize in one fast pass, not sampled along rays through a network. The runnable version below computes this compositing and shows why depth order matters.

How do you get the millions of Gaussians in the first place? You optimize them to reproduce your input photos. The pipeline is fully differentiable, so gradient descent can tune every Gaussian's position, shape, color, and opacity to minimize the difference between rendered views and the real photos:

That last step — adaptive density control — is what lets the scene allocate detail where it's needed: splitting Gaussians in under-reconstructed regions, removing ones that contribute nothing. After roughly six minutes (the 7K-iteration model) to about forty (the full one) of optimization you have a scene that renders novel viewpoints in real time. Training is fast, rendering is faster.

Kerbl, Kopanas, Leimkühler and Drettakis start from the same inputs as NeRF, posed photographs, and replace the network with an explicit set of anisotropic 3D Gaussians optimized through a differentiable rasterizer.

1. Initialize from structure-from-motion. COLMAP, run to calibrate the cameras, also produces a sparse point cloud for free. Each point becomes a Gaussian with an isotropic covariance whose radius is the mean distance to its three nearest neighbors. For the synthetic NeRF scenes, random initialization inside the scene bounds also works.
2. Parameterize each Gaussian. A Gaussian has a center μ, an opacity α stored through a sigmoid, spherical-harmonic color coefficients up to degree 3 (16 per color channel, 48 in all), a scale vector s stored through an exponential, and a rotation quaternion q. The covariance is built as Σ = R S S^^T R^^T rather than optimized directly, because gradient steps on a raw covariance matrix can leave the set of valid, positive semi-definite matrices.
3. Project to the screen. For a camera with viewing transform W, each Gaussian's 2D covariance is Σ′ = J W Σ W^^T J^^T, where J is the Jacobian of the local affine approximation to perspective projection, the EWA splatting formula of Zwicker and colleagues.
4. Cull and bin into tiles. The screen is split into 16×16-pixel tiles. Gaussians whose 99% confidence ellipse misses the view frustum are culled, a guard band rejects ones near the camera plane whose projection would be unstable, and each survivor is duplicated once for every tile its footprint overlaps.
5. Sort once. Each duplicate gets a key with the tile index in its high bits and view-space depth in its low 32 bits, and a single GPU radix sort over all keys yields a depth-ordered list per tile. Ordering is per splat, not per pixel, so blending is approximate where splats interpenetrate.
6. Blend front to back. One thread block per tile loads its splats into shared memory, and each pixel accumulates color and opacity front to back, stopping once its opacity saturates.
7. Backpropagate. The backward pass walks each tile's list back to front. Rather than storing every intermediate opacity, it keeps each pixel's final accumulated value and recovers intermediate ones by dividing out each splat's α, so any number of splats per pixel can receive gradients. Gradients for all parameters are derived explicitly rather than by automatic differentiation.
8. Optimize. The loss is (1 − λ) L₁ + λ L_D-SSIM with λ = 0.2, minimized with Adam, with an exponentially decaying learning rate for positions. Training starts at a quarter of the image resolution, upsampled twice early on, and adds one spherical-harmonics band every 1,000 iterations, starting from the constant term.
9. Control density. Every 100 iterations, Gaussians whose average view-space position gradient exceeds 0.0002 are densified: small ones are cloned and moved along the gradient (under-reconstruction), large ones split into two with scales divided by 1.6 and positions sampled from the original (over-reconstruction). Nearly transparent and overly large Gaussians are removed. Every 3,000 iterations all opacities are reset close to zero, and Gaussians that do not recover are pruned.
10. Train and render. The full model trains for 30,000 iterations, and a 7,000-iteration model is reported too. A novel view is rendered by steps 3 to 6 alone, with no network to evaluate.

• R S S^T R^T — equal to (RS)(RS)^^T, so positive semi-definite for any s and q: the optimizer cannot produce an invalid covariance.
• W, J — the camera's viewing transform and the Jacobian of the affine approximation to perspective projection at the Gaussian's center.
• α′i(p) — the Gaussian's opacity at pixel p: its learned α scaled by its 2D footprint.
• ci(d) — color from the spherical harmonics, evaluated at the viewing direction.
• 𝓛D-SSIM — structural dissimilarity, which rewards matching local structure and contrast rather than only pixel values.

Extremes: a scale component near zero flattens a Gaussian into a disc, which is how thin surfaces are represented, and α near 1 makes a splat occlude everything behind it. The affine approximation in J holds for footprints that are small relative to their distance and fails for Gaussians very close to the camera, which is what the guard band excludes.

Evaluation covered 13 real scenes, the nine of the Mip-NeRF 360 dataset plus two from Tanks and Temples and two from Deep Blending, and the synthetic Blender scenes. Mip-NeRF 360 was the quality baseline and Plenoxels and Instant-NGP the speed baselines; the headline claim is real-time rendering, at least 30 frames per second, at 1080p. On the Mip-NeRF 360 scenes:

Three things in that table cut against the headline. On its own dataset Mip-NeRF 360 keeps the best PSNR, 27.69 against 27.21; Gaussian Splatting wins on SSIM and LPIPS, so "matches or exceeds" depends on the metric. The fast 7,000-iteration model is roughly Instant-NGP quality, not Mip-NeRF 360 quality. And memory goes up about 85-fold, 734 MB against 8.6 MB, the trade that makes speed possible. On Tanks and Temples and Deep Blending the full model matched or beat Mip-NeRF 360 on all three quality metrics.

Ablations, run on selected scenes, removed one component at a time: SfM initialization (random initialization degraded mainly the background), splitting, cloning, anisotropy (isotropic Gaussians lost fine and thin structure), spherical harmonics, and unlimited gradient flow (capping how many splats per pixel receive gradients hurt quality severely). Each removal cost quality. The authors also note visible artifacts in poorly observed regions and peak training memory that can exceed 20 GB in their unoptimized prototype.

The cost is memory and file size. Millions of primitives, each with position, covariance, opacity and colour coefficients, is orders of magnitude larger than a small network's weights — so rung 9's representation was compact and slow and this one is bulky and fast, which is a clean trade rather than a free win, and it matters for anything that has to be transmitted.

There are characteristic artefacts: elongated Gaussians fitted to one set of views can look like visible streaks from angles the optimisation never checked, and the representation has no surface — it is a cloud of translucent blobs, so extracting a mesh, computing a collision, or relighting the scene needs additional machinery it does not provide. View-dependent colour is represented by a low-order expansion, which handles gentle shine and not sharp mirror reflections.

How it held up: it became the default for real-time novel-view synthesis almost immediately, and the pendulum it swung is the ladder's closing lesson. An explicit representation with a hardware-friendly differentiable renderer beat an elegant implicit one — which is the same finding as flash attention on the serving ladder, arrived at independently: the win came from matching the computation to what the hardware does well, and the neural formulation was carrying cost that the problem did not require.

Gaussian Splatting spread through graphics and vision almost overnight, spawning a huge wave of follow-ups: dynamic and animatable scenes, avatars, SLAM, compression, generative 3D. The reason is the same one that made it fast — an explicit, differentiable, GPU-friendly primitive sits at a sweet spot the field had been circling, and once someone showed it worked at real-time speed with NeRF-level quality, everyone could build on it.

Its deeper lesson is about representation choice. NeRF's implicit network was a beautiful idea whose cost was baked into the representation itself — you couldn't make querying-a-network-per-sample cheap. Switching to an explicit set of soft primitives didn't just optimize the old approach; it changed the computational character of the problem from "evaluate a function millions of times" to "rasterize millions of things," which is exactly what graphics hardware was built for. Sometimes the biggest speedup comes from picking a representation the hardware already loves.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Ten papers from the first latent-variable generator to video and real-time 3D. The diffusion half is strictly cumulative — each rung removes a cost the last one paid.

1. 1VAE2014
2. 2GANs2014
3. 3VQ-VAE2017
4. 4DDPM2020
5. 5DDIM2021
6. 6Classifier-Free Guidance2021
7. 7Latent Diffusion2022
8. 8Sora2024
9. 9NeRF2020
10. 103D Gaussian Splatting2023

All 110 papers, in 19 ladders →