The Sierra Agent
Engineer interview

The current posting titles the job Software Engineer, Agent and frames it as owning the Agent Development Life Cycle, “from initial pilot through deployment and continuous iteration”, for customers including The GAP, Rocket Mortgage, SoFi, Sutter Health and SoftBank.first-party⁴²

What differs from a generic FDE role is the depth expected on one specific technical surface. You are expected to reason precisely about tool schemas, escalation paths, guardrails, failure handling, evaluation of non-deterministic behaviour, and what happens when the agent is confidently wrong in front of a customer’s customer.

The phone screen became system design, because “vibe-coding an app is easy” and production is the hard part;⁴¹ the aggregated guide reports an agentic subscription-cancellation flow as one prompt.²⁵ The debugging pilot has you improve a colleague’s draft PR with coding agents, and interviews run in pairs against criteria that do not depend on what you build.⁴¹

Cells hold a number or a formula such as =A1+B1+2. Agree that small a grammar out loud, so the hour goes on the graph rather than on a parser.

import re

REF = re.compile(r"[A-Z]+[0-9]+")

class CycleError(ValueError):
    pass

class Sheet:
    def __init__(self):
        self.raw = {}      # cell -> "7" or "=A1+B1+2"
        self.deps = {}     # cell -> cells its formula reads
        self.users = {}    # cell -> cells whose formulas read it
        self.value = {}    # cell -> cached number; get() never computes

    def _reaches(self, start, target):
        # stage 2: can we walk from start to target along deps? (iterative DFS)
        stack, seen = [start], set()
        while stack:
            c = stack.pop()
            if c == target:
                return True
            if c not in seen:
                seen.add(c)
                stack.extend(self.deps.get(c, ()))
        return False

    def set(self, cell, raw):
        new = set(REF.findall(raw)) if raw.startswith("=") else set()
        if any(self._reaches(d, cell) for d in new):  # reject before mutating anything
            raise CycleError(f"{cell} would depend on itself")
        for d in self.deps.get(cell, ()):
            self.users[d].discard(cell)
        for d in new:
            self.users.setdefault(d, set()).add(cell)
        self.raw[cell], self.deps[cell] = raw, new
        self._recompute(cell)

    def _eval(self, cell):
        raw = self.raw.get(cell, "0")
        if not raw.startswith("="):
            return int(raw)
        return sum(self.value.get(t, 0) if REF.fullmatch(t) else int(t)
                   for t in (p.strip() for p in raw[1:].split("+")))

    def _recompute(self, start):
        # stage 3: only cells downstream of the edit, each evaluated once, in dependency order
        affected, stack = {start}, [start]
        while stack:
            for u in self.users.get(stack.pop(), ()):
                if u not in affected:
                    affected.add(u)
                    stack.append(u)
        waiting = {c: sum(d in affected for d in self.deps.get(c, ())) for c in affected}
        ready = [c for c in affected if waiting[c] == 0]   # Kahn's algorithm on the affected subgraph
        while ready:
            c = ready.pop()
            self.value[c] = self._eval(c)
            for u in self.users.get(c, ()):
                if u in affected:
                    waiting[u] -= 1
                    if waiting[u] == 0:
                        ready.append(u)

    def get(self, cell):
        return self.value.get(cell, 0)

Cost: get is O(1); a write costs one depth-first search plus O(A + E) over the A downstream cells and the E edges among them. Running Kahn’s algorithm on the affected subgraph only is what makes a diamond — A1 feeds B1 and C1, both feed D1 — evaluate D1 once rather than twice.

An agent framework hides exactly the parts that break in production. Build one by hand and you are forced to make every decision the framework was making for you:

import json

MODEL = "the-model-your-brief-names"
POLICY = "You are Acme's support agent. Look orders up before answering. Never promise refunds."
MAX_STEPS, MAX_RESULT_CHARS = 8, 2_000
HANDOFF = "I'm handing you to a teammate who can help with this."

def lookup_order(order_id):
    return {"order_id": order_id, "status": "shipped", "eta": "2026-10-24"}  # the client's API goes here

TOOLS = [
    {"type": "function", "name": "lookup_order", "strict": True,
     "description": "Fetch one order by id. Call this before answering anything about an order.",
     "parameters": {"type": "object", "additionalProperties": False, "required": ["order_id"],
                    "properties": {"order_id": {"type": "string", "description": "Like 'A-10293'."}}}},
    {"type": "function", "name": "escalate_to_human", "strict": True,
     "description": "Hand off to a person: refunds, anything policy forbids, or when you are unsure.",
     "parameters": {"type": "object", "additionalProperties": False, "required": ["reason"],
                    "properties": {"reason": {"type": "string"}}}},
]
HANDLERS = {"lookup_order": lookup_order}

def run_turn(client, history):
    """One customer turn. Extends history in place; returns (reply, status)."""
    for _ in range(MAX_STEPS):
        resp = client.responses.create(model=MODEL, instructions=POLICY, input=history, tools=TOOLS)
        history += resp.output                    # the model's calls (and any reasoning) stay in context
        calls = [item for item in resp.output if item.type == "function_call"]
        if not calls:
            return resp.output_text, "answered"   # exit 1: the model answered
        handoff = None
        for call in calls:
            try:
                args = json.loads(call.arguments)
                if call.name == "escalate_to_human":
                    handoff, output = args["reason"], "handed off"
                else:
                    output = json.dumps(HANDLERS[call.name](**args))[:MAX_RESULT_CHARS]
            except Exception as e:                # a failed call is an observation, not a crash
                output = json.dumps({"error": type(e).__name__, "detail": str(e)[:200]})
            history.append({"type": "function_call_output", "call_id": call.call_id, "output": output})
        if handoff:
            return HANDOFF, "escalated"           # exit 2: a person takes over
    return HANDOFF, "step_limit"                  # exit 3: the budget ran out; never loop forever

Three exits, each with a status you can count; MAX_RESULT_CHARS as a crude bound on what enters context (summarise or paginate in a real build); an except that turns a timeout, malformed arguments or a hallucinated tool name into an observation; strict schemas whose descriptions say when to call a tool; a history you should say how you would trim; and escalation as a tool, so handing off is a decision rather than a crash.

Two details interviewers notice. Every function_call is answered by a function_call_output with the same call_id, the escalation included, so the transcript stays valid next turn. And history += resp.output keeps the model’s own items in context, which OpenAI’s guide requires for reasoning models.⁴⁵

Then measure it the way Sierra does: run each scenario several times. Sierra Research introduced pass^k with τ-bench — the chance that all k attempts at a task succeed — and found state-of-the-art function-calling agents succeeding on under 50% of tasks, with pass^8 below 25% in retail.first-party⁴³ One working demo is the evidence that metric was built to replace.

For the three spreadsheet stages, Time and Space mean the usual thing. Here V is the number of cells and E the number of formula references, and A is the set of cells downstream of an edit. For the agent knobs, Time is latency and Space is what enters the context window.

• Recursive cycle detection in Python

  A recursive DFS crashes with RecursionError on a chain longer than about 1,000 cells, Python’s default recursion limit. That is why _reaches uses an explicit stack. Say this out loud when you write it.

• Retrying a tool that changes things

  A timeout does not mean the action failed. Retry a refund or a cancellation blindly and the customer may be charged or credited twice. Send an idempotency key, or look up the state before you retry.

• Slicing a JSON string

  [:MAX_RESULT_CHARS] can cut through the middle of a value, so the model reads broken JSON. That is acceptable in a drill. In a real build, pick the fields the task needs before you serialise, or return one page of results and a cursor.

• Fixing a schema bug in the prompt

  When the model passes a name where an order id belongs, adding more rules to POLICY treats the symptom. Fix the parameter description and give an example value, as in "Like 'A-10293'.". The page notes that most “the model is bad” reports turn out to be schema bugs.

• Stopping at the first bug

  The debugging round reportedly plants about three bugs. Fixing one and announcing that you are done shows you stopped looking. After each fix, check the intended-behaviour diagram again and look for the same class of bug elsewhere.