NeRF,
explained.

Rungs 1 to 8 generate pictures. This rung does something different enough that it starts the ladder's second movement: given a set of photographs of one real scene, produce new views of that scene from camera positions nobody photographed. The comparison is against classical reconstruction, which builds an explicit model — a mesh, a point cloud, a voxel grid — and struggles with transparency, thin structures and view-dependent shine.

The change is to represent the scene as the weights of a small network. It maps a 3D position and a viewing direction to a colour and a density, and that is the entire scene: no geometry, no texture maps, just a function you can query anywhere. To render a pixel, march along the ray through it, sample the network at points along the way, and integrate — accumulating colour weighted by how much light each point contributes and how much is already blocked in front of it.

Because that integration is differentiable, the whole thing is trained by rendering pixels and comparing them to the photographs. There is no 3D supervision at all — the geometry is inferred entirely from the requirement that the scene look correct from every camera you have. Two details are essential rather than incidental. Feeding raw coordinates to the network produces blurry results, because such networks are biased toward low frequencies; encoding the coordinates with sinusoids across many frequencies is what lets it represent sharp detail. And taking view direction as an input is what allows a surface to look different from different angles, which is how specular highlights and reflections appear at all.

Classic 3D representations are explicit: meshes of triangles, clouds of points, grids of voxels. NeRF's radical idea is to represent a scene as a continuous function — and to store that function in a neural network. The function takes a 3D location and a viewing direction and returns what you'd see there: a color and a density. A whole scene, in all its detail, becomes the weights of a small multilayer perceptron.

Why is that powerful? Because a continuous function has no resolution limit — you can query it as finely as you like — and it can capture view-dependent effects like specular highlights, since color depends on the viewing direction. And it's learnable: you can fit the function to a set of photos by gradient descent, because the entire rendering process is differentiable. The scene is optimized to explain the images you have, and then it explains images you don't.

The network is a simple MLP. Its inputs are a 3D point (x, y, z) and a viewing direction; its outputs are an RGB color and a scalar density σ (roughly, how much "stuff" is at that point — how opaque). A careful detail: density depends only on position (a point in space is as solid from any angle), while color also depends on direction (so a shiny surface can look different as you move) — the architecture wires the inputs to enforce this.

That's the entire scene representation: query this network at a point, get color and density. Everything else is about turning many such queries along a ray into the color of one pixel — which is the classic problem of volume rendering.

To color a pixel, cast a ray from the camera through it and sample points along the ray. At each sample, get its color c_i and density σ_i from the network. Convert density over a small segment of length δ_i into an opacity, and accumulate colors front-to-back weighted by how much light survives to reach each sample (the transmittance):

A high density makes α → 1 (opaque — a surface), zero density makes α = 0 (empty air). The transmittance T_i is the fraction of light that got past everything nearer, so once the ray hits something solid, T collapses and points behind it stop contributing — occlusion, for free. This is a discretized version of the physics of light through a participating medium, and it's differentiable end to end, which is what lets NeRF learn the density and color fields from photos. The runnable version below computes exactly this integral.

There's one problem left, and its fix is the paper's most-cited trick. Feed raw (x, y, z) coordinates to an MLP and the result is blurry — neural networks have a well-known bias toward smooth, low-frequency functions, and a scene's fine texture is high-frequency. The solution: positional encoding. Map each coordinate through a bank of sines and cosines at exponentially increasing frequencies before feeding it in:

Now two nearby points that raw coordinates would place almost on top of each other get very different high-frequency features, so the network can tell them apart and assign them different colors and densities. This is what turns NeRF from blurry blobs into crisp, detailed reconstructions — the same insight that reappears (in a different guise) in the transformer's positional encodings. The runnable version below shows two points 0.01 apart, separated into a gap of ~1.4 by the highest-frequency band.

Mildenhall and colleagues optimize one network per scene from posed photographs alone. Everything below happens per scene: nothing is pretrained and nothing transfers to the next scene.

1. Collect posed images. The input is a set of photographs with known camera intrinsics and poses. For real scenes the poses come from the COLMAP structure-from-motion package; for the synthetic Blender scenes they are exact. Each pixel defines a ray r(t) = o + t·d between near and far bounds, and forward-facing real scenes are first mapped to normalized device coordinates so their unbounded depth fits a finite range.
2. Encode the inputs. Each coordinate of the position, normalized to [−1, 1], goes through sines and cosines at L = 10 octave-spaced frequencies, giving 60 numbers; the unit viewing direction gets L = 4, giving 24.
3. Query the MLP. γ(x) passes through 8 fully connected ReLU layers of 256 units, with γ(x) concatenated again into the fifth layer's input as a skip connection. The trunk outputs the density σ through a ReLU, so it cannot be negative, plus a 256-dimensional feature, which is concatenated with γ(d) and passed through one more 128-unit layer to give RGB through a sigmoid. Density never sees the direction: geometry is view-independent and only color can change with viewpoint.
4. Sample coarsely. Split each ray's range into N_c = 64 equal bins and draw one point uniformly at random inside each. This stratified sample covers the whole range and moves every iteration, so the network is effectively trained on a continuous ray. A coarse network is queried there and composited into a coarse pixel estimate.
5. Resample where it matters. The compositing weights w_i = T_i(1 − e^−σ_iδ_i) say how much each bin contributed. Normalized, they form a piecewise-constant probability density along the ray, and N_f = 128 more points are drawn from it by inverse transform sampling, so they cluster near visible surfaces.
6. Render finely. A second, fine network is evaluated at all 64 + 128 = 192 points, and compositing them gives the final pixel color.
7. Train both networks. The loss is the squared error of the coarse render plus that of the fine render, against the true pixel color, over batches of 4,096 rays drawn from all training images; supervising the coarse render is what keeps its weights a useful sampling density. Adam, with the learning rate decaying exponentially from 5×10^⁻⁴ to 5×10^⁻⁵. A scene typically took 100,000 to 300,000 iterations, about one to two days on a single NVIDIA V100. For real scenes, random Gaussian noise added to the raw density output during training acts as a regularizer, which the paper found slightly improves novel views.
8. Render a novel view. For a new camera, run steps 4 to 6 for every pixel: 256 network evaluations per ray, which is why a frame took about 30 seconds on a V100.

Why the coarse pass is not wasted

Stratified sampling alone would spend most of its 192 queries in empty air or behind the first opaque surface, where transmittance is near zero and nothing affects the pixel. The coarse pass is a cheap survey: a third of the queries find where along the ray the color comes from, and the fine pass spends the rest there.

• T(t) — transmittance: the probability the ray travels from the near bound to t without being stopped.
• σ — density, the rate at which the ray is stopped per unit length; it depends on position only.
• c(r(t), d) — emitted color, which also depends on the viewing direction.
• wi — the probability the ray stops in bin i; the same numbers composite the pixel and, normalized, decide where to sample next.
• Ĉc, Ĉf — the coarse and fine renders, both supervised by the same pixel color C(r).

Extremes: if all the weight sits in one bin, an opaque surface, nearly every fine sample lands in that bin; if the ray crosses empty space and the weights are flat, the fine samples spread evenly and the hierarchy buys nothing. The weights sum to 1 − T_N+1, the probability the ray is stopped anywhere in the sampled range.

Three datasets: Diffuse Synthetic 360° (four Lambertian objects at 512×512), Realistic Synthetic 360° (eight Blender objects with complex geometry and non-Lambertian materials, 100 training and 200 test views each, at 800×800), and Real Forward-Facing (eight scenes shot with a handheld phone, 20 to 62 images each, one eighth held out, at 1008×756). Baselines were Scene Representation Networks (SRN), Neural Volumes (NV) and Local Light Field Fusion (LLFF), scored by PSNR and SSIM (higher is better) and LPIPS (lower is better).

The one metric NeRF did not win is LPIPS on the real forward-facing scenes, where LLFF scored 0.212 against NeRF's 0.250, even though NeRF led on PSNR and SSIM; LLFF was designed for exactly that capture setup. Neural Volumes was not evaluated on the forward-facing scenes. Storage runs the other way: a NeRF scene is about 5 MB of weights, roughly 3,000 times less than LLFF needs and less than the input images themselves.

Ablations on Realistic Synthetic 360° measured each component against the full model's 31.01 PSNR. Removing positional encoding, view dependence or hierarchical sampling each cost PSNR, the first two by a few decibels and hierarchical sampling by about one. Training on fewer images, 50 or 25 instead of 100, degraded results steadily. And raising the position encoding from L = 10 to L = 15 brought no gain, which the authors attribute to 2^^L then exceeding the highest frequency present in the sampled images.

It is slow at both ends, and the paper says so. Training takes one to two days per scene on a single GPU, and rendering a single frame takes seconds, because every pixel requires many network evaluations along its ray and there are a great many pixels. It is also per-scene: the weights are that scene and nothing transfers to another. And it needs accurate camera poses, supplied by a separate structure-from-motion step whose errors it inherits.

Quality degrades with sparse or poorly-distributed views, since the geometry is only constrained where a camera saw it, and the representation can satisfy the photographs with geometry that is wrong in unobserved regions.

How it held up: it created a field, and then lost its central architectural choice. The explosion of follow-up work attacked the speed from every angle — caching, explicit grids, hash-based feature encodings — and the direction of every one of those fixes was the same: put more of the scene into an explicit data structure and less into network weights. That trend is what rung 10 completes. The differentiable rendering framing — optimise a scene representation against observations — is the durable contribution and survives the change of representation entirely.

NeRF opened a field. Within a couple of years there were NeRFs for dynamic scenes, giant outdoor scenes, few-image and single-image reconstruction, relighting, editing, generative 3D, and real-time variants. The core template — a differentiable rendering function fit to photos — became a standard way to do 3D from images, and "radiance field" entered the vocabulary.

Its enduring idea is differentiable rendering as a bridge between 2D images and 3D structure. Because the whole pipeline from scene representation to rendered pixel is differentiable, you can pose 3D reconstruction as gradient descent against photos — no explicit geometry estimation, just "make the renders match." That reframing, more than the specific MLP, is NeRF's lasting contribution; even Gaussian Splatting, which replaced almost every component, kept exactly that differentiable-rendering-fit-to-photos skeleton.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Ten papers from the first latent-variable generator to video and real-time 3D. The diffusion half is strictly cumulative — each rung removes a cost the last one paid.

1. 1VAE2014
2. 2GANs2014
3. 3VQ-VAE2017
4. 4DDPM2020
5. 5DDIM2021
6. 6Classifier-Free Guidance2021
7. 7Latent Diffusion2022
8. 8Sora2024
9. 9NeRF2020
10. 103D Gaussian Splatting2023

All 110 papers, in 19 ladders →