Why AI Teams Are Ditching "Move Fast and Break Things" for Evaluation-First Development

The phrase "move fast and break things" was never a software engineering charter. It was a growth-stage slogan, popularized by Mark Zuckerberg in the early 2010s, that made sense when the cost of a bug was a broken newsfeed and the cost of waiting was irrelevance. In the era of large language models, that trade-off has inverted. A bad prompt update can now cost a company millions in liability, compute, and trust, as a handful of publicly documented incidents show. That is why a growing number of AI teams are abandoning the move-fast ethos and adopting evaluation-first development: a discipline that treats model and prompt changes the way mature engineering organizations treat code changes, with tests, baselines, and a gate before production.

This article is grounded in public, verifiable facts. Where the evidence is qualitative, such as tooling and practitioner guidance, this article says so plainly rather than inventing precision.

What Evaluation-First Development Actually Means

Evaluation-first development is the practice of building a measurable test harness for an LLM application before, or alongside, the features it powers. Instead of judging a model by how a handful of prompts feel in a chat window, teams define what good looks like, encode that definition into automated checks, and refuse to ship when the checks fail.

The core techniques are well established. Golden datasets are small curated sets of inputs with expected outputs, used to catch regressions when a model, prompt, or retrieval layer changes. Semantic similarity checks compare the meaning of a generated answer with a reference answer using embeddings and cosine similarity, rather than requiring exact string matches. LLM-as-a-judge systems use one model to grade another model's output; the approach was formalized in the 2023 paper "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" by Zheng and colleagues, which found that a strong judge model agreed with human preferences more than 80 percent of the time, while also documenting real biases including position bias, verbosity bias, and self-enhancement bias. Retrieval pipelines get their own evaluation layer, building on the retrieval-augmented generation framework Lewis and colleagues introduced in 2020.

None of this is exotic. It is the LLM equivalent of unit testing and regression testing, applied to a probabilistic runtime. The reason it feels new is that the underlying systems are new: a non-deterministic API call can return different answers for the same input, so traditional test suites alone cannot guarantee behavior. Evaluation-first teams close that gap by testing the behavior, not just the code.

The Evidence That Ignoring Evaluation Is Expensive

The strongest argument for evaluation-first development is not a vendor survey. It is a set of publicly documented failures.

In August 2012, Knight Capital deployed untested software to its trading systems. Within 45 minutes, the firm lost roughly 440 million dollars and was forced to sell itself days later. The failure was not a market shock; it was a release process that bypassed validation. In February 2024, a Canadian tribunal ordered Air Canada to honor a refund policy its chatbot had invented, a liability of 812 Canadian dollars, because the airline was responsible for the information its AI presented to customers. In January 2024, a delivery company's customer-service chatbot began swearing at a customer and criticizing the company; the company disabled the AI and called it a mistake. These are the shape of the risk, financial, legal, and reputational, and each one is a failure mode an evaluation harness can catch before customers see it.

The stakes scale with deployment. IBM's Cost of a Data Breach Report for 2024 put the average cost of a data breach at 4.88 million dollars, with an average of 258 days to identify and contain it. Gartner estimated in March 2024 that poor data quality costs organizations an average of 12.9 million dollars per year, predicted that more than 80 percent of enterprises would have deployed generative AI APIs or models in production by 2026, and warned that 30 percent of generative AI projects would be abandoned after proof of concept by the end of 2025. McKinsey has estimated that generative AI could add the equivalent of 2.6 trillion to 4.4 trillion dollars in annual value to the global economy. None of these figures is about evaluation specifically, but together they describe a market in which the difference between shipped-with-tests and shipped-blind is measured in millions.

Why Evaluation Is the New Engineering Discipline

LLM applications fail in ways that classic software rarely does. Models exhibit position bias, favoring information at the start or end of long contexts; the 2023 paper "Lost in the Middle" by Liu and colleagues at Meta showed that models perform measurably worse on information in the middle of long prompts. Models drift: the same prompt can behave differently after a provider updates the underlying model. And models hallucinate: they produce fluent, confident statements that are not grounded in the source material. Each of these failure classes is discoverable only by testing against a baseline.

That is why the practitioner playbook has converged on a few durable patterns. Version and pin model versions in production. Build a golden dataset from real traffic, not invented examples. Establish a baseline score before any change, and a pass-or-fail threshold enforced in CI/CD. Track evals over time so a model update that quietly degrades quality is caught by a diff, not by a customer complaint. Measure semantic similarity, retrieval quality, and latency together, because an evaluation that looks at only one dimension will miss the others.

The Tools Landscape: Mature, and Still Rapidly Evolving

The tooling ecosystem for LLM evaluation is crowded and maturing quickly. The major providers ship evaluation primitives inside their platforms, and a wave of dedicated tools has emerged, including LangSmith, Langfuse, DeepEval, Ragas, Promptfoo, Braintrust, Galileo, Confident AI, Weights and Biases, Arize Phoenix, and OpenAI Evals. This article does not rank them, and it does not assign them invented performance statistics; the right choice depends on a team's stack, budget, and whether it needs offline eval harnesses, online monitoring, or both.

What matters more than the tool is the habit. The teams that benefit most treat evaluation as a practice, not a purchase: they write the test before the feature, they treat a failing eval like a failing build, and they make the eval suite the first thing a new engineer learns to read. Pricing in this market is aggressive and changing; teams should treat any published price as a point-in-time data point, not a contract.

One economic note is worth making plain: inference is cheap enough to waste. As of this writing, a high-end frontier model such as Claude 3.5 Sonnet costs roughly 3 dollars per million input tokens and 15 dollars per million output tokens. OpenAI's GPT-4o, after its August 2024 price cut, costs 2.50 dollars per million input tokens and 10 dollars per million output tokens. DeepSeek-V3 is dramatically cheaper at 0.27 dollars per million input tokens and 1.10 dollars per million output tokens, with 0.07 dollars for cached input. Prompt caching, OpenAI offers 50 percent off cached input and Anthropic up to 90 percent, means re-running evaluation suites against cached prefixes is not the cost center it once was. Wasted inference from repeated, unmeasured prompt tuning is a real operating cost, and evaluation-first teams spend less of it because they measure before they iterate.

How to Build Your Evaluation Practice in 30 Days

Week one: collect. Pull a sample of real production inputs, including the messy, multilingual, and adversarial ones, and write expected outputs for at least a few dozen of them. This is your golden dataset, and it is the single highest-leverage artifact you can create.

Week two: measure. Pick an evaluation tool, connect it to your application, and compute a baseline score for your current model and prompts. Record it. You cannot improve what you have not measured, and you cannot prove a regression without a baseline.

Week three: gate. Set a pass-or-fail threshold, for example a minimum semantic-similarity score on your golden dataset, and wire it into your CI/CD pipeline so that a prompt or model change cannot reach production while failing. Add checks for the failure modes that matter to your domain: hallucinated citations, off-policy answers, broken tool calls, poor retrieval.

Week four: widen. Add evaluation to your review process for every prompt change, and start tracking quality over time so that provider model updates show up as visible diffs. From here, evaluation-first is a habit, not a project.

The Regulatory Backstop

Regulation is moving in the same direction, which gives evaluation-first teams a compliance advantage. The European Union's AI Act entered into force on August 1, 2024, with obligations phased in over the following years; the most serious violations can draw fines of up to 35 million euros or 7 percent of global annual turnover, whichever is higher, and general-purpose AI obligations began applying in August 2025, with high-risk requirements arriving in August 2026. Frameworks such as the OWASP Top 10 for Large Language Model Applications and the NIST AI Risk Management Framework, published in January 2023, give teams a shared vocabulary for risk. A team that already tests its models, documents its baselines, and can show a record of evaluation is a team that will answer a regulator's questions with evidence rather than vibes.

The verdict is not subtle. "Move fast and break things" was a reasonable slogan for an era when the thing being broken was a newsfeed. When the thing being broken is a production LLM application, with real customers, real liability, and real money moving through it, the only defensible version of the slogan is "move fast with evaluation." Ship the test before the feature. Gate every change. Measure every baseline. That is the discipline behind every AI team shipping reliably in 2026, and it is available to any team willing to start with a spreadsheet, a sample of real traffic, and thirty days of honest measurement.

— Jessica Ali, Sylt.ing

About the Author

Jessica Ali is the lead anchor of Global 1 News and a senior AI journalist at Sylt.ing. Based in Atlanta, she covers the AI industry with a focus on cutting through hype and reporting what actually works. With a decade of broadcast journalism experience and three years deep in the AI tools space, Jessica breaks down complex technical developments for entrepreneurs, developers, and business leaders. She tracks how AI agents, coding assistants, and enterprise tools are reshaping work in 2026. Find her coverage at sylt.ing/Jessica and global1.news.