50 LLM
Interview
Questions

An LLM is a deep neural network trained on massive text corpora to predict the next token in a sequence. At its core, it learns statistical patterns of language — grammar, facts, reasoning patterns, and stylistic nuances — by processing billions of text samples.

The "large" refers to both parameter count (often billions to trillions) and training data scale. Modern LLMs like GPT-4, Claude, and Llama use the Transformer architecture and are trained with self-supervised learning: they predict masked or next tokens, building rich internal representations of language.

After pretraining, they're typically fine-tuned with RLHF (Reinforcement Learning from Human Feedback) or similar alignment techniques to follow instructions and be helpful.

Classic models were essentially lookup tables — n-gram models computed P(word | previous n-1 words) and that was it. LLMs are learned, generalizable representations that can handle tasks never seen during training.

A foundation model is a large model trained on broad data at scale, designed to be adapted to a wide range of downstream tasks. The term was coined by Stanford's HAI center.

Key properties: Generality (diverse training data), Adaptability (fine-tune or prompt for specific tasks), and Emergent capabilities (behaviors not explicitly programmed). Examples: GPT-4 (language), CLIP (vision-language), Stable Diffusion (image generation).

This paradigm is both powerful (fewer models to train) and risky — a single model's biases propagate everywhere it's deployed.

The jump was less about architectural novelty and more about scale, data quality, and alignment. GPT-4 unlocked use cases like document analysis, complex code generation, tutoring, and professional-grade writing that GPT-3 couldn't reliably handle.

Tokenization converts raw text into discrete tokens (subword units) that the model processes. Modern LLMs use subword tokenizers like BPE (Byte-Pair Encoding), WordPiece, or SentencePiece.

Why it matters for behavior: The tokenizer determines what atomic units the model sees. Poor tokenization of certain languages (e.g., non-Latin scripts) means more tokens for the same content, degrading performance and effective context capacity.

Why it matters for costs: API pricing is per-token. Non-English text or unusual variable names can tokenize 2–5× less efficiently than standard English, directly inflating costs.

Hidden gotcha: Tokenization artifacts explain why LLMs can't reliably count letters in a word — they see subword chunks, not individual characters.

Embeddings are dense vector representations that map discrete tokens (or sentences, documents) into a continuous vector space where semantic similarity = geometric proximity.

Classic example: king - man + woman ≈ queen. Good embeddings capture semantic relationships geometrically. Modern contrastive learning produces embeddings where similar meanings are close and dissimilar ones are far apart.

Modern LLMs effectively eliminated the OOV (out-of-vocabulary) problem through subword tokenization.

The tradeoff: rare words get split into more tokens, consuming more context and giving the model less direct "understanding" of them as atomic units. This is why LLMs struggle with very rare proper nouns — they see fragments, not whole words.

Seq2seq (sequence-to-sequence) maps an input sequence to an output sequence of potentially different length. Originally proposed by Sutskever et al. (2014) using RNNs.

Problems it solves: Machine translation, summarization, question answering, dialogue — any task where input and output have different lengths and structures.

Key limitation: The fixed-size context vector is an information bottleneck. Long inputs get compressed into the same size vector as short ones, losing information. This is exactly what attention mechanisms (and later Transformers) were designed to fix.

Evolution: Seq2seq + attention → Transformer encoder-decoder (T5) → decoder-only LLMs (GPT). The paradigm still lives in T5, BART, and mBART.

Encoder-decoder: Uses both. The encoder builds a representation of the input; the decoder generates the output while cross-attending to the encoder's representations. Examples: T5, BART, original Transformer.

Modern trend: Decoder-only models (GPT, Claude, Llama) dominate because they're simpler to scale and can handle both understanding and generation in one architecture.

Hybrid approaches: XLNet uses permutation language modeling for bidirectional context with autoregressive generation. T5 uses span corruption as a middle ground. The field converged on autoregressive for large-scale LLMs.

MLM is BERT's core training objective. During training, ~15% of tokens are randomly selected:

Why it works: Bidirectional context forces the model to use both left and right context, building richer representations. It's fully self-supervised — no labeled data needed.

Limitation: The [MASK] token never appears at inference time (train-test mismatch). Also, only ~15% of tokens provide learning signal per example — less sample-efficient than autoregressive training where every token is a target.

NSP was introduced alongside MLM in the original BERT paper. The model is given two sentences and must predict whether sentence B actually follows sentence A in the corpus.

Purpose: Help the model understand inter-sentence relationships, useful for QA and natural language inference.

Why it fell out of favor: RoBERTa showed that NSP doesn't help and can even hurt performance. The task is too easy — the model can distinguish random pairs using topic mismatch alone, without learning deep discourse relationships.

What replaced it: RoBERTa dropped NSP entirely. ALBERT replaced it with Sentence Order Prediction (SOP) — both sentences are from the same document but may be swapped, forcing actual coherence understanding.

Five reasons Transformers won:

1. Parallelization: RNNs process tokens sequentially — Transformers process all positions in parallel via self-attention, enabling massive GPU utilization.

2. Long-range dependencies: In RNNs, information degrades over distance. In Transformers, self-attention connects every position to every other — path length is O(1) regardless of distance.

3. No information bottleneck: RNN seq2seq compressed the entire input into one fixed-size vector. Transformers maintain per-token representations throughout.

4. Scalability: Parallel nature makes it scale efficiently. This enabled training on orders of magnitude more data.

5. Cleaner gradient flow: Residual connections + layer normalization give much cleaner gradient paths than deep RNNs.

Self-attention is permutation-invariant — it treats the input as a set, not a sequence. Without positional information, "dog bites man" and "man bites dog" would produce identical representations.

The choice of positional encoding significantly impacts a model's ability to handle long contexts. RoPE has become the dominant choice for modern open-source LLMs because it naturally captures relative positions and supports context length extension via techniques like NTK-aware scaling.

Attention is a mechanism that lets each token dynamically focus on the most relevant parts of the input when computing its representation. It computes a weighted sum of values, where weights are determined by the compatibility between queries and keys.

Types of attention in Transformers:

Self-attention: Tokens in a sequence attend to each other — every token can interact with every other token.

Cross-attention: Decoder tokens attend to encoder representations — used in encoder-decoder models.

Causal attention: Restricted self-attention where tokens only attend to past positions — used in all decoder-only LLMs.

Multi-head attention runs several attention operations in parallel, each with its own learned projection matrices, then concatenates and projects the results.

Why multiple heads matter:

One head might focus on syntactic relationships (subject-verb agreement), another on semantic relationships (coreference), and another on positional patterns. A single head can only capture one type of interaction per layer.

Practical details: If model dimension d=512 and we use h=8 heads, each head operates in d/h=64 dimensions. Total compute ≈ same as single-head at d=512, but representation capacity is much richer.

Research finding: Not all heads are equally important — some can be pruned with minimal quality loss, while others are critical for specific capabilities.

The full formula: Attention(Q,K,V) = softmax(QKᵀ / √d_k) · V

Why divide by √d_k? The raw dot product grows with dimension (expected value increases with d_k). Without scaling, softmax would saturate on high-dimensional vectors, producing near-zero gradients and making training unstable.

Softmax serves three critical purposes in the attention mechanism:

Alternatives being explored:

Linear attention: Removes softmax for O(n) complexity instead of O(n²), but loses sparsification benefit. Sigmoid attention: Some recent work replaces softmax with sigmoid for more independent per-head attention. Sparse attention: Uses top-k or local windows to approximate softmax more efficiently (Longformer, BigBird).

The dot product appears as the similarity function between query and key vectors: score(q, k) = q · k = Σ(qᵢ × kᵢ)

The scaling: Raw dot products grow with dimension — expected value ∝ d_k. Without the ÷ √d_k scaling, softmax saturates at high dimensions, producing near-zero gradients. That's why QKᵀ / √d_k is always seen together.

In practice, QKᵀ is a single matrix multiplication computing every query's dot product with every key simultaneously — the core of efficient Transformer inference.

Modern practice: Most LLM apps use sampling (temperature + top-p) rather than beam search, because beam search tends to produce generic text. Beam search remains important for structured output tasks with a narrow correct answer space.

Temperature is a scalar that modifies logits before softmax: softmax(logits / T)

Both are filtering strategies applied before sampling to avoid drawing from the long tail of low-probability tokens.

Problem with top-k: If k=50 but 3 tokens have 95% of probability mass, you're still sampling from 47 near-zero-probability tokens — wasting candidate slots on bad choices.

Best practice: Many APIs apply both — top-k to cap maximum candidates, then top-p within that set. Common combination: top-p=0.9, top-k=50.

Adaptive softmax is an approximation technique for the output softmax layer when vocabulary is very large (100K+ tokens).

The problem: Computing softmax over the entire vocabulary requires computing logits for every token — a huge matrix multiplication that is the computational bottleneck during training.

Efficiency gain: Most of the time, the model only needs to compute the full softmax over the small head cluster. Tail clusters are only evaluated when needed, providing 2–10× speedups on the output layer.

Modern relevance: With BPE vocabularies around 32K–100K and faster hardware, adaptive softmax is less critical than before — but the concept of variable-capacity allocation based on frequency remains influential in efficient NLP.

Cross-entropy loss measures the difference between the model's predicted probability distribution and the true distribution: L = -log(P(correct_token))

Perplexity formula: PPL = exp(H(p,q)) — the standard evaluation metric for language models. Lower is better.

Embedding layers are lookup tables (matrices) where each row corresponds to a token. During training:

Sparse gradients: The backward pass computes gradient only for the rows corresponding to tokens that appeared in the current batch. This is why gradients are sparse — only the rows corresponding to tokens that appeared get updated.

Key implications:

• Rare tokens learn slowly — fewer gradient updates, less refined embeddings. This is why models struggle with very rare proper nouns.

• The embedding matrix is often the largest single parameter block in the model (vocab_size × hidden_dim).

• Weight tying: Many models share the embedding matrix with the output projection (lm_head), which acts as a regularizer and reduces parameter count.

The Jacobian is the matrix of all first-order partial derivatives of a vector-valued function. For f: ℝⁿ → ℝᵐ, the Jacobian J is an m×n matrix where J_ij = ∂fᵢ/∂xⱼ.

Conditioning: Well-conditioned Jacobians (singular values ≈ 1) lead to stable training. Layer normalization and careful initialization aim to maintain this throughout the network.

The chain rule is the mathematical foundation of backpropagation. For composition f(g(x)), the derivative is f'(g(x)) · g'(x).

Why backprop is efficient: Computing the gradient for layer 1 in an L-layer network is O(L) multiplications — not O(L!) (which you'd get by naively applying the chain rule). It reuses intermediate results from the forward pass.

Practical concerns: The chain of multiplications can cause vanishing/exploding gradients. Residual connections (x + f(x)) mitigate this by providing a direct gradient highway — the gradient of x + f(x) w.r.t. x is I + ∂f/∂x, so even if ∂f/∂x vanishes, the identity term I preserves the gradient.

ReLU (Rectified Linear Unit): f(x) = max(0, x). Despite its simplicity, it revolutionized deep learning training.

Why ReLU was a breakthrough: The gradient is 1 for x > 0, enabling constant gradient flow in the positive region. Unlike sigmoid/tanh which saturate at both extremes, ReLU maintains a constant gradient, enabling much deeper networks.

Why modern LLMs use GELU/SwiGLU: ReLU has a "dying ReLU" problem (neurons stuck at 0) and isn't zero-centered. GELU provides a smooth approximation; SwiGLU adds a gating mechanism that empirically outperforms both.

Root cause: During backpropagation, gradients are multiplied by the Jacobian at each layer. If these Jacobians have spectral norms consistently < 1 (from saturating activations), the gradient shrinks exponentially with depth.

In PCA (Principal Component Analysis) and spectral methods, eigenvalues and eigenvectors provide the geometric decomposition of the data's variance structure.

Connection to LLMs:

• LoRA: Exploits the fact that weight update matrices have low intrinsic rank — SVD decomposition finds the eigenvectors of the update subspace

• Embedding visualization: PCA projects 768-dim BERT embeddings to 2D for visualization

• Attention analysis: Eigendecomposition of attention matrices reveals what the model focuses on

KL divergence (Kullback-Leibler divergence) measures how one probability distribution P differs from a reference distribution Q:

KL(P||Q) = Σ P(x) · log(P(x)/Q(x))

Key uses in LLMs:

RLHF constraint: During RLHF training, a KL penalty prevents the fine-tuned model from diverging too far from the base model. This preserves fluency while improving alignment. Without it, the model would exploit the reward signal and produce gibberish that scores high.

Knowledge distillation: Student model is trained to minimize KL divergence between its output distribution and the teacher's soft outputs.

VAE training: KL divergence regularizes the latent space toward a prior (Gaussian) distribution.

Why it works: Weight updates during fine-tuning have low intrinsic rank — most of the change concentrates in a small subspace. LoRA parameterizes the update as a low-rank product, capturing this efficiently.

Catastrophic forgetting occurs when a neural network trained on new data loses its ability to perform well on previously learned tasks. New gradients overwrite the weights that encoded old knowledge.

Why it's especially problematic for LLMs: Fine-tuning on a narrow domain can destroy the model's general capabilities. A model fine-tuned heavily on legal text might lose its ability to write code or do math.

PEFT (Parameter-Efficient Fine-Tuning) is a family of methods that update only a small subset of model parameters while keeping most pretrained weights frozen.

Why PEFT prevents forgetting by construction:

• Frozen base = preserved knowledge (mathematically — frozen weights can't change)

• Small parameter budget = optimization landscape constrained — model can't drift far from original

• Composability: train separate PEFT modules for different tasks, swap at inference. Base model stays pristine.

Tradeoff: Slightly lower peak performance on the target task vs full fine-tuning, but preservation of general capabilities makes this worthwhile in practice.

Knowledge distillation transfers knowledge from a large "teacher" model to a smaller "student" model by training the student to mimic the teacher's output distribution, not just match hard labels.

Why soft labels work better than hard labels: The teacher's probability distribution encodes which wrong answers are "almost right" — richer signal than a one-hot label. This dark knowledge helps the student learn the relationships between classes.

Applications in LLMs: Model compression (distilling GPT-4 quality into smaller models), synthetic data generation (large model generates training data for smaller), speculative decoding (small draft model verified by large model).

Limitation: Capacity gap — a 7B model can't fully absorb a 70B model's knowledge. Distillation also can't transfer emergent abilities that require scale.

In LLM fine-tuning, overfitting means the model memorizes the fine-tuning data rather than learning generalizable patterns.

Symptoms: Training loss keeps decreasing but validation loss plateaus or increases; the model generates verbatim training examples.

Evaluation strategy: For LLMs, you need task-specific evaluation beyond just loss — measure actual generation quality on held-out examples. Perplexity on training set vs validation set is a quick diagnostic.

Modern blur: LLMs like GPT are generative models that perform discriminative tasks (classification, NLI) by generating the answer. "Is this review positive or negative?" → GPT generates "Positive." This generative approach to discriminative tasks has been surprisingly effective, blurring the traditional boundary.

Business model difference: Discriminative AI tends to save costs (automating classification). Generative AI tends to create new value (enabling new capabilities). Best products often combine both: generate content (generative) then filter/rank it (discriminative).

Prompt design fundamentally determines what part of the model's learned distribution you're sampling from. Small changes can dramatically shift output quality.

System-level prompt engineering: In production, you design system prompts that set behavioral constraints, inject context, and define output schemas. This is where prompt engineering becomes system design.

Chain-of-thought (CoT) prompting encourages the model to show its reasoning steps before giving a final answer.

Variants: Tree of Thought (explores multiple reasoning paths), Self-Consistency (sample multiple CoTs, take majority vote), Zero-shot CoT (just add "Let's think step by step").

RAG (Retrieval-Augmented Generation) augments LLM generation with retrieved external knowledge.

Evaluation metrics: Retrieval quality (Precision@k, recall, MRR), Generation quality (faithfulness, groundedness), End-to-end (user satisfaction).

Knowledge graphs (KGs) store information as structured triples (entity → relationship → entity) and can significantly enhance RAG pipelines beyond what vector search alone provides.

GraphRAG: Microsoft's approach combines vector retrieval for broad context with graph traversal for precise, structured facts. LLMs are used to build a community-based KG from documents, then queries it with graph algorithms + LLM generation.

MoE replaces the dense feedforward network (FFN) in each Transformer layer with multiple "expert" FFNs and a learned router that activates only a subset per token.

Challenges: Load balancing (ensuring all experts get used — auxiliary loss required), communication overhead in distributed training, expert collapse (all tokens routing to same experts), higher memory (all experts must fit in RAM).

Examples: Mixtral 8x7B, Switch Transformer, Grok (xAI), rumored GPT-4.

Zero-shot means the model performs a task without any task-specific training examples in the prompt. It relies entirely on pretraining knowledge and instruction-following ability.

Why LLMs can do zero-shot: During pretraining on diverse text, the model encounters millions of implicit task demonstrations. It learns the pattern of instructions and responses. Instruction tuning (fine-tuning on instruction-following data like InstructGPT) dramatically improves zero-shot performance.

Limitations: Zero-shot performance varies wildly. Simple classification and translation work well; complex structured output or domain-specific tasks often need examples or fine-tuning.

Few-shot learning provides a small number of input-output examples directly in the prompt before the actual query.

Best practices: Use diverse, representative examples. Order can matter (recency bias — later examples weigh more). More examples help up to a point, then returns diminish. For structured tasks, consistent formatting is critical.

The context window is the maximum number of tokens a model can process in a single forward pass — the model's "working memory."

Practical limits beyond raw size: Self-attention is O(n²) — doubling context quadruples compute. Effective context < maximum context (needle-in-a-haystack tests show degraded recall as context grows). Latency scales with context length.

Best practice: Use RAG to retrieve and inject only relevant context rather than stuffing the entire window. Quality of context >> quantity of context.

Three types of hyperparameters:

Architecture: Number of layers, hidden dimension, number of attention heads, vocabulary size, FFN intermediate size — define the model's capacity.

Training: Learning rate, batch size, number of epochs, warmup steps, weight decay, dropout rate, gradient clipping — control the training dynamics.

Generation: Temperature, top-p, top-k, max tokens, repetition penalty — control the model's output at inference time.