Builder Developer III

About the Role

You are a systems engineer for non-deterministic systems, comfortable with setting clear expectations for demos and defining the criteria for success. You are an expert in AI-Native Workflow, setting the standard for speed and safety, and proactively removing bottlenecks. You excel in Agentic Architecture, building end-to-end agents and designing the underlying state management, tool interfaces, and failure semantics. You are adept at Legacy-to-Modern Sequencing, prioritizing modernization efforts based on blast radius, data gravity, and measurability. You champion Measured Reliability, defining "reliable enough" through acceptance thresholds, regression suites, human-in-the-loop checkpoints, and rollback paths, understanding that the engineering goal is a measured error rate with a containment strategy.

The Three Things You Own

• The Guardrail Layer: Input and output validation, tool-permission scoping, financial and PII data handling, sandboxed execution, human approval gates on irreversible actions, kill switches, and blast-radius containment. You design with the assumption that agents may err, aiming for minor logging impacts rather than critical system failures.
• The Eval Framework: Building golden datasets from real ERP edge cases, implementing offline evals in CI as a merge gate, and conducting online evals in production. This includes using LLM-as-judge calibrated against human labels, detecting calibration drift, and identifying regression when models, prompts, or retrieval indices change. The goal is to move from subjective "prompt vibes" to objective metrics that can fail a build.
• The Cost & Performance Envelope: Modeling routing, caching, context budgeting, and latency budgets. You track token cost per completed business transaction and can articulate the cost drivers of workflows and identify opportunities for optimization.

Elective Depth (be deep in one, fluent in all six)

• Agentic Architect (Systems): State management and tool-calling interfaces for long-running business processes across complex ERP schemas.
• UX/Interaction (Product): Generative UIs and browser-use agents, designing human-in-the-loop integration points.
• Data & Context (Data): RAG and knowledge-graph architecture, including vector stores, chunking strategy, retrieval evals, and data relationships.
• Reliability & Eval (QA): Programmatic prompt optimization (e.g., DSPy), red-teaming agents, and adversarial testing of financial logic.
• Infra & Agent-Ops (DevOps): Agent deployment environments, including tracing, logging, monitoring AI "thoughts," and versioned deployment/rollback of prompts and models.
• Domain Logic Translation (Business): Translating complex business regulations into structured system prompts and agentic constraints.

What You'll Actually Do

• Define and enforce the standard for when an agent can interact with production data, embedding these standards in code.
• Build and integrate the eval harness into CI as a merge gate.
• Design guardrails for regulated and financial logic, including human approval and rollback mechanisms.
• Conduct design reviews to identify and mitigate architectural issues early.
• Own agent incidents by identifying root causes, pulling traces, and implementing evals to prevent recurrence.
• Make build-vs-buy decisions on frameworks, vendors, and models, and re-evaluate them as needed.
• Mentor and grow junior developers through code reviews, design reviews, and pairing.

The Stack

• Python, SQL (Postgres/SQL Server), TypeScript
• OpenAI/Anthropic
• pgvector/Qdrant
• LangChain/LangGraph
• DSPy
• dbt
• Apache Hop/Airflow
• Docker
• Weights & Biases
• OpenTelemetry
• Orchestration layer: eval and tracing platforms (e.g., LangSmith / Langfuse / Braintrust), policy and guardrail tooling, model routing, and caching.

Who You Are

• You have direct experience shipping agentic systems to production, learning from unexpected failures, and iterating on architecture based on those learnings.
• You prioritize rigor over hype, valuing evaluation metrics over unproven concepts, and understanding the distinction between demos and production-grade financial agents.
• You are an agent-first thinker, focusing on capabilities and failure modes rather than just functions.
• You are a manual-task hater, automating repetitive tasks to increase efficiency.
• You are technically fearless, viewing complex and undocumented systems as challenges to be solved.
• You are a multiplier, aiming to enhance the team's output through your contributions to design and reviews.