NeRF (Neural Radiance Fields, Mildenhall et al., 2020) represents a 3D scene as a small neural network that maps a 3D position and a viewing direction to a color and a density. To render a novel view, it casts a ray through each pixel, samples points along the ray, queries the network at each, and integrates the colors weighted by density and transmittance (volume rendering). Trained only on a set of input photos, it can synthesize photorealistic images from new viewpoints.

How does NeRF render a pixel?

By volume rendering along the pixel's ray. It samples points, gets each point's color and density from the network, converts density over a segment into an opacity alpha = 1 - exp(-sigma*delta), and accumulates color = sum of transmittance times alpha times color. Transmittance is the product of (1 - alpha) for nearer samples, so opaque regions block what's behind them. The result is the pixel color that a camera would see.

Why does NeRF use positional encoding?

A plain MLP on raw 3D coordinates produces blurry results because networks are biased toward low-frequency functions. Positional encoding maps each coordinate through sines and cosines at many frequencies, so nearby points get very different high-frequency features. This lets the network represent sharp, high-frequency detail like fine textures and edges, dramatically improving quality.

How does NeRF compare to Gaussian Splatting?

NeRF stores the scene implicitly in a neural network and renders by querying it many times per pixel along rays — high quality but slow, often seconds per frame. 3D Gaussian Splatting stores the scene explicitly as millions of Gaussians and renders by rasterizing them, reaching real-time speeds with comparable quality. Gaussian Splatting was in large part a response to NeRF's rendering cost while keeping the same volume-rendering math.

Paper Breakdowns / NeRF

Paper 68~11 min readECCV 2020worked math + runnable code

Paper Breakdown

NeRF,
explained.

Give NeRF a few dozen photos of an object and it learns to conjure the object from angles no camera ever saw — reflections, transparency, fine detail intact. The astonishing part is where the 3D scene is stored: not as a mesh or a point cloud, but inside the weights of one small neural network that answers, for any point in space and any direction you look, "what color, and how solid?" Render by asking that question along every ray. This is the paper that launched the radiance-field era. Here's the network, the rendering integral, and the trick that makes it sharp.

Video breakdown

The animated walkthrough is in production.

Read the full breakdown below in the meantime ↓

Prefer video? Browse published breakdowns →

A scene as a function Color and density Volume rendering Positional encoding Run it yourself The results What came next

A scene as a function

Classic 3D representations are explicit: meshes of triangles, clouds of points, grids of voxels. NeRF's radical idea is to represent a scene as a continuous function — and to store that function in a neural network. The function takes a 3D location and a viewing direction and returns what you'd see there: a color and a density. A whole scene, in all its detail, becomes the weights of a small multilayer perceptron.

Why is that powerful? Because a continuous function has no resolution limit — you can query it as finely as you like — and it can capture view-dependent effects like specular highlights, since color depends on the viewing direction. And it's learnable: you can fit the function to a set of photos by gradient descent, because the entire rendering process is differentiable. The scene is optimized to explain the images you have, and then it explains images you don't.

The one-sentence version

Store a 3D scene as a neural network mapping (position, direction) → (color, density), and render new views by integrating that network's output along camera rays.

Color and density

The network is a simple MLP. Its inputs are a 3D point (x, y, z) and a viewing direction; its outputs are an RGB color and a scalar density σ (roughly, how much "stuff" is at that point — how opaque). A careful detail: density depends only on position (a point in space is as solid from any angle), while color also depends on direction (so a shiny surface can look different as you move) — the architecture wires the inputs to enforce this.

That's the entire scene representation: query this network at a point, get color and density. Everything else is about turning many such queries along a ray into the color of one pixel — which is the classic problem of volume rendering.

Volume rendering

To color a pixel, cast a ray from the camera through it and sample points along the ray. At each sample, get its color c_i and density σ_i from the network. Convert density over a small segment of length δ_i into an opacity, and accumulate colors front-to-back weighted by how much light survives to reach each sample (the transmittance):

α_i = 1 − exp(−σ_i δ_i) · C = Σ_i T_i · α_i · c_i , T_i = exp(−Σ_j<i σ_j δ_j)

A high density makes α → 1 (opaque — a surface), zero density makes α = 0 (empty air). The transmittance T_i is the fraction of light that got past everything nearer, so once the ray hits something solid, T collapses and points behind it stop contributing — occlusion, for free. This is a discretized version of the physics of light through a participating medium, and it's differentiable end to end, which is what lets NeRF learn the density and color fields from photos. The runnable version below computes exactly this integral.

Positional encoding

There's one problem left, and its fix is the paper's most-cited trick. Feed raw (x, y, z) coordinates to an MLP and the result is blurry — neural networks have a well-known bias toward smooth, low-frequency functions, and a scene's fine texture is high-frequency. The solution: positional encoding. Map each coordinate through a bank of sines and cosines at exponentially increasing frequencies before feeding it in:

γ(p) = [ sin(2⁰πp), cos(2⁰πp), …, sin(2^L−1πp), cos(2^L−1πp) ]

Now two nearby points that raw coordinates would place almost on top of each other get very different high-frequency features, so the network can tell them apart and assign them different colors and densities. This is what turns NeRF from blurry blobs into crisp, detailed reconstructions — the same insight that reappears (in a different guise) in the transformer's positional encodings. The runnable version below shows two points 0.01 apart, separated into a gap of ~1.4 by the highest-frequency band.

RUN IT YOURSELF

Volume rendering, and why encoding matters

NeRF's two mathematical cores in a handful of lines. Volume rendering converts density to opacity with α = 1 − exp(−σδ) — zero density gives 0, a dense sample gives ~1 — and integrates C = Σ Tᵢ·αᵢ·cᵢ so an opaque surface returns its own color and occludes what's behind. And positional encoding: (x,y,z) at L=10 becomes a 60-dimensional feature, and two points just 0.01 apart, nearly identical to raw coordinates, are pushed a full ~1.4 apart by the highest-frequency band — which is exactly what lets the MLP render sharp detail. Change the densities or the encoding and watch it hold.

CPython · WebAssembly

import math

# NeRF stores a scene as an MLP (position, direction) -> (color, density). Render a
# pixel by marching a ray and integrating: alpha = 1 - exp(-sigma*delta), and
# C = sum_i T_i * alpha_i * c_i with transmittance T_i = exp(-sum sigma*delta in front).
# Positional encoding maps coordinates to many frequencies so the MLP can fit detail.

def alpha_from_density(sigma, delta):
    return 1 - math.exp(-sigma * delta)          # opacity of a sample from its density

def transmittance(densities, deltas, i):
    return math.exp(-sum(densities[j] * deltas[j] for j in range(i)))

def volume_render(colors, densities, deltas):
    C = 0.0
    for i in range(len(colors)):
        T = transmittance(densities, deltas, i)
        a = alpha_from_density(densities[i], deltas[i])
        C += T * a * colors[i]
    return C

def pe_dim(input_dim, L):
    return input_dim * 2 * L                       # sin & cos at L frequencies per coordinate

def positional_encoding(p, L):
    out = []
    for k in range(L):
        out.append(math.sin(2 ** k * math.pi * p))
        out.append(math.cos(2 ** k * math.pi * p))
    return out

print(f"alpha: zero density {alpha_from_density(0, 1)}  dense {alpha_from_density(100, 1):.4f}")
print(f"opaque surface returns its color: {volume_render([0.7], [100.0], [1.0]):.4f}\n")
print(f"PE dim for (x,y,z) at L=10: {pe_dim(3, 10)}")
gap = abs(positional_encoding(0.10, 10)[18] - positional_encoding(0.11, 10)[18])
print(f"points 0.10 vs 0.11: raw gap 0.010 -> high-freq gap {gap:.3f}  (separated)")

The results

NeRF set a new bar for novel-view synthesis — generating photorealistic images of a scene from viewpoints not in the training set:

Result	Significance
Photorealistic novel views	From only ~dozens of input photos, with reflections, transparency, and fine detail.
View-dependent effects	Specular highlights and glossy surfaces rendered correctly as the camera moves.
Compact	An entire scene fit in a few megabytes of network weights.
The catch: speed	Rendering meant hundreds of network queries per pixel — seconds per frame, not interactive.

That last row defined the next several years of research. NeRF's quality was undeniable and its rendering cost impractical, so a flood of follow-ups attacked the speed — precomputing, caching, using explicit grids — culminating in Gaussian Splatting, which kept the volume-rendering math but swapped the implicit network for explicit primitives to hit real time.

What came next

NeRF opened a field. Within a couple of years there were NeRFs for dynamic scenes, giant outdoor scenes, few-image and single-image reconstruction, relighting, editing, generative 3D, and real-time variants. The core template — a differentiable rendering function fit to photos — became a standard way to do 3D from images, and "radiance field" entered the vocabulary.

Its enduring idea is differentiable rendering as a bridge between 2D images and 3D structure. Because the whole pipeline from scene representation to rendered pixel is differentiable, you can pose 3D reconstruction as gradient descent against photos — no explicit geometry estimation, just "make the renders match." That reframing, more than the specific MLP, is NeRF's lasting contribution; even Gaussian Splatting, which replaced almost every component, kept exactly that differentiable-rendering-fit-to-photos skeleton.

Worth knowing

NeRF uses hierarchical sampling — a coarse network finds where the surfaces likely are, then a fine network samples densely there — so the expensive queries concentrate where they matter instead of wasting samples on empty space.

Frequently asked

Quick answers

What is NeRF in one line?

A neural network storing a 3D scene as a function from (position, direction) to (color, density), rendered into novel views by volume rendering along rays.

How does it render?

Sample points along each pixel's ray, convert density to opacity (α = 1 − exp(−σδ)), and accumulate color weighted by transmittance — the volume-rendering integral.

Why positional encoding?

MLPs favor low frequencies and render blurry; mapping coordinates through sines/cosines at many frequencies lets the network capture sharp, high-frequency detail.

NeRF vs Gaussian Splatting?

NeRF is implicit and slow (query a network per sample); Gaussian Splatting is explicit and real-time (rasterize primitives), using the same volume-rendering math.

Finished this one? 0 / 101 Paper Breakdowns done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

ML Foundations

NeRF,
explained.

A scene as a function

Color and density

Volume rendering

Positional encoding

Volume rendering, and why encoding matters

The results

What came next

Quick answers

What is NeRF in one line?

How does it render?

Why positional encoding?

NeRF vs Gaussian Splatting?

Explore the topic

More Paper Breakdowns

Explore more from Vibe Engines

NeRF,explained.

A scene as a function

Color and density

Volume rendering

Positional encoding

Volume rendering, and why encoding matters

The results

What came next

Quick answers

What is NeRF in one line?

How does it render?

Why positional encoding?

NeRF vs Gaussian Splatting?

Go deeper

3D Gaussian Splatting →

GPU Fundamentals →

Vision Transformer →

Explore the topic

More Paper Breakdowns

Explore more from Vibe Engines

Get the next one in your inbox.

NeRF,
explained.