The Future of Software Development Won’t Be Just One AI Coding Tool to Rule Them All
AI coding tools have impressed, but won't be replacing human developers or the software development lifecycle they work in anytime soon.
AI code generation has been the single hottest early-stage category of the past year. In Silicon Valley, it’s been nearly impossible to miss. One of ChatGPT's primary use cases is code generation, and Github Copilot stands as arguably the most successful AI application after ChatGPT itself.
The success of Copilot has sparked a new wave of AI code generation startups — including Cursor, Codeium, Cognition (Devin), Stackblitz (Bolt), and All Hands. Collectively, this group has raised over $2 billion from VCs in the last two years.
The scale of adoption is now nearing ubiquity: The market crossed $500 million in annual spend in December 2024, up from zero two years ago. Over half of organizations use AI in their development process, and for some organizations, AI generates 46% of new code.
The rapid ascent of AI code generation has sparked dramatic predictions about the category eating the software development toolchain whole, and even suggestions that it could replace human developers entirely (no doubt encouraged by some AI startups’ own marketing).
But as impressive as the first wave of AI code generation solutions have been, the idea that any platform could collapse the entire dev ecosystem – developers and all – into just a single AI code generator both overestimates the near-term capabilities of AI coding and underestimates the complexity of enterprise software development. In particular:
Generating code that works is now easy, but generating good code is much harder. The first argument is a technical one. Today's AI code can compile, execute, and pass tests, but struggles with the higher bar required for production systems – code that is maintainable, scalable, and aligned with company-specific best practices. Current AI is best at rote execution, but lacks the deeper understanding of system architectures and product roadmaps to build more robust software.
Software development is not just coding. Over time, AI will undoubtedly get better with better models, context, and more agentic designs. But the broader challenge remains: developers don’t just code, but need to understand, operate, maintain, and extend systems throughout their entire lifecycle. This complexity has led to specialized roles like frontend and backend engineering, QA, DevOps, and reliability engineering – each requiring distinct workflows and expertise today that necessitate specialized tooling.
The best code generation models will be accessible to all AI dev tool builders. The breakthrough performance gains of the past year have primarily come from model improvements led by Anthropic and other large AI research labs. As these foundational model providers continue developing better models, every AI dev tool will have access to the same underlying coding intelligence — leaving app layer AI dev tools to compete on form factor, developer ergonomics, and workflow-specific automations. This market structure naturally tends toward an ecosystem of specialized AI developer tools rather than a single dominant solution.
The Realities of the ‘46% of New Code’ Written by AI Today
AI is already writing production code today – and a lot of it. We previously mentioned that Copilot generates 46% of code in files where it is enabled. Google too said 25% of its code is now being written by AI. But what do these figures actually mean?
To understand what AI coding tools are really doing today, we should be precise about what is being measured here. Luckily, Copilot provides an answer in its telemetry that Parth Thakkar thoughtfully breaks down in his analysis of Copilot internals.
When a developer accepts a Copilot suggestion, Github checks if that code (or something similar) is still there after 15 seconds, 30 seconds, 2 minutes, 5 minutes, and 10 minutes. The platform doesn’t look for an exact match, which would be too strict as developers often make small edits over Copilot’s suggestions, but instead difs between the “final” code and Copilot’s original code. When comparing words, if the current code is at least 50% similar to the original suggestion, the suggestion is considered successfully “AI written.”
Obviously, this definition is a lot more measured than the headlines it generated would imply. But more importantly, it speaks to what AI code generation actually is today – a great tool for scaffolding an implementation and generating boilerplate code like API endpoints or class declarations. But between this and the code that developers can actually merge to main, there is still significant additional manual work in:
Evaluating AI-suggested architectural decisions
Refactoring AI code to be more understandable and maintainable
Adding edge case handling for scenarios outside the happy path
Building in proper exception handling and recovery
Ensuring idiomatic code patterns and best practices
These gaps are even more apparent when it comes to more agentic AI solutions like Cognition’s Devin that are targeted towards larger developer tasks.
In the month since Devin went GA, some of the most popular use cases for the AI coding agent involve “schlep” – or tedious code tasks that sit in the backlog and would usually get assigned to an intern or junior dev, requiring not a lot of context but instead pure execution. Examples include adding type annotations for Ramp or converting React MUI components to Tailwind using Lumos’ style.
But even for these use cases, Devin’s work can require fairly extensive human reviews at times. For Dagger.io, for instance, Devin was tasked with adding a small feature: a variable to prevent Dagger from removing old engine containers during version upgrades. Devin’s initial solution came out “functionally correct” – producing around 100 lines of code changes. But in a follow-up commit, human developers needed to make almost twice the number of manual line edits to bring code quality up to production standards, addressing issues with Devin’s original PR including:
Hardcoded string in too many places
Overly complex logic
Unnecessary logs
Manual string comparisons instead of strconv.ParseBool (a Go idiom)
No tests
Smarter Models Won’t Fix AI Coding’s Software Development Problem
The performance of today’s AI coding tools are moment-in-time snapshots in a rapidly changing space. Just over the course of the last year, state-of-the-art in AI code generation has evolved from 1.96% on swe-bench in October 2023 (by Anthropic’s Claude 2) to 49% by the upgraded Claude 3.5 Sonnet in October 2024. By the time Claude 4 is released, today’s AI coding headaches could quickly become tomorrow’s solved problems.
As we look towards the day that better models are released though, we should be careful to distinguish between things we know will improve (AI’s technical coding capabilities) and others that likely will not (the collapse of the software developing lifecycle into just the “build” phase).
First, the things that will get better:
Smarter reasoning models. We’re still in an era of rapidly improving model capabilities. Recent models like Claude 3.5 Sonnet (released June 2024, updated October) have significantly enhanced coding capabilities – a step forward that is clear to see in both academic benchmarks and increased market share among AI startups using Anthropic's models compared to Copilot (which ended up adding Claude to in late October). Compared to earlier generation models, the latest models better handle real-world challenges like multi-file operations, longer contexts, and architectural reasoning.
Better codebase context. Github Copilot was launched in 2022 with a number of smart “hacks” for retrieving context from actively edited files (i.e., your current file and possibly a few other open tabs). As it turns out, this approach rendered Copilot blind to a lot of useful information. Newer tools like Cursor improved on this by offering comprehensive codebase indexing with vector retrieval and reranking, but even today, promptly surfacing the right context from vast codebases remains a key technical challenge that AI code editors still regard as one of their top problems to solve.
More agentic AI coders. Since the beginning of 2024, leading AI coding solutions have not been models alone, but agent systems that combine AI models with software scaffolding that manages prompts, parses outputs, and handles interaction loops. When Anthropic achieved state-of-the-art swe-bench results in October with the new Claude 3.5 Sonnet, for instance, the team equipped Claude with both prompt scaffolding and specialized tools for bash commands and file operations. Empowering models with the right code scaffolding and letting them iterate in your developer environment – including approaches like Monte Carlo Tree Search or test-time inference – continues to be an exciting new vector for AI coding advancement.
With that being said, certain aspects of software development will most likely remain unchanged even with more capable AI. Better AI coding tools will handle more and more technical challenges over time, and may even develop sophisticated understandings of software architectures and product roadmaps — allowing expansion into underpenetrated spaces today like code review, security scanning, quality assurance, observability, and build/deploy systems. (Greg Foster, the CTO of the AI code review platform Graphite, goes deeper on the path for AI expansion in his excellent piece on the topic here.)
But the reality is that coding represents just one part of the broader software development lifecycle. For enterprises, this process encompasses planning and design before the build phase, and testing, deployment, maintenance, and expansion after it. Leading software companies already account for this reality through the development of specialized roles: product managers, designers, QA engineers, security specialists, DevOps engineers, and site reliability engineers.
Many of these technical roles involve limited hands-on-keyboard coding. As such, they'll likely remain beyond the scope of automation for any single AI code generation tool. Instead, success in these segments will come from purpose-built, role-specific AI tools, creating a diverse ecosystem of AI software development solutions.
Software Development in the Age of AI
AI coding is eating the software development process – but not as much as the headlines would have you believe. Given the current technical and market realities of AI coding and the complexity of the software development process, the future will not simply be a single AI code generation platform but instead a robust ecosystem of specialized AI dev tooling instead.
In fact, we’re already seeing the first greenshoots of these powerful new platforms emerge at the earliest stages – including Graphite for code review; Ranger for QA; Traversal for site reliability engineering; Modelcode for code migration; and Semgrep for security analysis.
In future articles, we’ll explore each of these emerging areas in greater detail.