– or Are We Again Stuck in Exploding Budgets, Unclear Product Goals, and Unexplained Complexity?
The software industry is in a constant state of flux, evolving from one paradigm to another. We often hear about how we’ve “learned from the past” and how modern methodologies—whether agile, DevOps, or microservices—are the solutions to the problems of yesterday. But when you look at many large-scale projects today, one can’t help but wonder: have we truly learned from the software crisis of the past, or are we once again stuck in a loop of exploding budgets, vague product goals, and unpredictable complexity?
The Software Crisis Revisited
The software crisis, as it was termed in the 1960s, refers to the struggle of managing the growing complexity of software systems. Software development was growing at a pace far outstripping our ability to manage it effectively. This led to bloated projects, runaway costs, and software that was unreliable and difficult to maintain. As Fred Brooks put it:
“The software crisis is a crisis of software development, not a crisis of technology. The major problem is that we don’t know how to build large, reliable software systems.” — Fred Brooks
For example, the Apollo space program faced challenges managing the software complexity needed for its lunar missions. Despite rigorous testing, software issues were a significant concern, illustrating early on the difficulties of handling large, complex systems.[1] Similarly, the Therac-25 radiation therapy machine disaster in the 1980s[2] was a tragic reminder of the consequences of software failures in critical systems. Modern examples, like the failed launch of Healthcare.gov[3], show that even with advanced methodologies, large-scale software projects can still face massive technical setbacks. And the Y2K bug[4] in the late 1990s demonstrated that even seemingly small coding decisions can lead to widespread chaos. Fast-forward to today, and we see similar problems manifesting, though they often come in new forms.
Despite the advent of Agile methodologies, DevOps practices, and modern architectural styles like microservices, the software industry still faces many of the same issues that plagued the 60s and 70s. In fact, the complexity of modern systems may actually be accelerating, not decreasing, especially as we try to break systems apart into microservices or use sophisticated frameworks that are intended to make us more agile.
The paradox of the software crisis is that complexity never really goes away—it just morphs into different, sometimes even more insidious, forms.
In response to these challenges, the field of software engineering emerged, aimed at applying systematic, disciplined, and quantifiable approaches to the development, operation, and maintenance of software systems. As Tom DeMarco said:
“Software engineering is the application of engineering principles to software development in a methodical way.”— Tom DeMarco
This shift was crucial in moving the industry from an art to a more structured and reliable field, emphasizing planning, testing, and maintenance as foundational principles.
Exploding Budgets and Mismanaged Expectations
Take a look at any major tech project; chances are you’ll find cost overruns and schedule slippage. We’ve all seen the headlines about multi-billion dollar software failures. Even with modern agile frameworks, we continue encountering exploding budgets and missed deadlines—patterns that feel eerily familiar to the software crisis days.
Let’s take a step back and consider what happens when a project begins: there’s often an initial vague sense of urgency and enthusiasm to solve a complex problem. Business goals are loosely defined, and everyone dives into the project with high hopes. But as teams begin to build, the scope balloons. Features multiply. Teams grow, and with each new person added to the project, communication overhead increases.
In microservices, for example, the intention is to divide and conquer—breaking down a large system into smaller, more manageable services. But each service introduces additional complexity, interdependencies, and integration challenges. More services mean more teams, more coordination, and, inevitably, higher maintenance costs.
But it’s not just about money—often, business goals themselves are poorly defined. A product team might want a “revolutionary feature,” but they might not fully understand how it fits into the system or how it aligns with long-term goals. As these goals remain unclear, software teams are forced to pivot frequently, changing direction and adding functionality, which can lead to scope creep and unnecessary complexity.
The result? Projects go over budget, and business stakeholders often don’t see the promised results because the product evolved into something different than originally envisioned.
The Product Goal Conundrum
One of the biggest risks in modern software projects is the lack of clear and stable product goals. It’s easy to think that the problem lies in technical execution, but often, the heart of the issue is in how product goals are defined, communicated, and refined over time.
A product goal might start as a simple statement—“We need a recommendation engine for our e-commerce platform”—but as the project evolves, so do the requirements. Different stakeholders bring their own visions to the table. Marketing wants personalisation. The UX team wants an intuitive design. Engineering wants to break it down into microservices to scale better.
The problem isn’t necessarily with the individual goals, but with the lack of alignment and clarity across teams. What was initially intended to be a simple feature becomes a complex system of systems that requires significant time and effort to build, test, and deploy. In the worst cases, business stakeholders are left waiting for features that no longer match the original vision.
If the goals aren’t clear upfront or if they shift too often during development, the team ends up working on the wrong thing—building solutions to problems that weren’t clearly defined in the first place. And because the technical team is also working in the dark, there is no feedback loop that allows for constant alignment with the business. When the dust settles, you realise the software solution isn’t delivering the intended value.
Rising Complexity for Simple Features
Another symptom of the ongoing software crisis is increasing complexity—particularly for seemingly simple features. Many modern development practices, such as using microservices or adopting event-driven architectures, promise to reduce the friction of scaling and maintaining software. But often, these patterns introduce more complexity than they resolve.
Here’s an example: imagine a team tasked with implementing a user authentication system. In a monolithic application, this might mean creating a straightforward login page with a simple back-end service for user validation. In the world of microservices, however, this “simple” feature might require an entire set of services dedicated to identity management, OAuth tokens, distributed session storage, role-based access control, and more. These systems must be secured, integrated, and regularly updated across multiple services and platforms.
For developers still learning the ropes of these technologies, what should be a small, isolated feature now touches many parts of the system—leading to increased complexity and new bugs that weren’t anticipated. As a result, teams often underestimate the effort involved in delivering something that seemed simple in the beginning.
This explosion of complexity is not just limited to technical details; it extends to communication as well. As teams grow, coordination becomes harder. Service contracts between teams evolve, and the effort to keep them in sync increases. When simple features are split across microservices, each with its own set of dependencies, the overhead can easily spiral out of control.
Are We Back in the Same Boat?
So, have we really learned from the software crisis, or are we simply facing new flavours of the same old problems? On one hand, we have made significant strides. We’ve introduced better practices, such as agile development, CI/CD pipelines, and DevOps, to mitigate risk and improve delivery speed. But on the other hand, we’re still seeing the same underlying issues crop up: exploding budgets, unclear goals, and increasing complexity for simple tasks.
In many cases, modern practices like microservices and event-driven architectures promise a more elegant way of scaling and organising software, but they also shift complexity elsewhere. The more granular we get with our services, the more dependencies we have to manage. The more distributed we make our systems, the harder it becomes to maintain consistent communication and versioning across services.
At the heart of this dilemma is a fundamental truth: complexity is inevitable in large systems. But managing that complexity effectively requires focus—clear goals, an understanding of what technologies are truly necessary, and a commitment to simplifying when possible. If we continue to chase the latest trend without fully understanding its implications, we risk repeating the mistakes of the past.
The Path Forward: Focus on the Fundamentals
The lessons from the software crisis haven’t been completely lost, but we need to keep returning to them. Complexity must be managed, not avoided. Features must be well-defined, and product goals must be clear and aligned across all teams. More importantly, we need to recognize that no architecture, pattern, or tool will eliminate complexity altogether—it’s about how we approach it.
Instead of getting lost in the buzzwords of the day, we need to ask ourselves: What problem are we actually trying to solve? Are we improving the system, or are we creating new complexity that ultimately makes the software harder to maintain? The answer to this question might be the key to avoiding the same fate as the software crisis: spiralling costs, misunderstood goals, and runaway complexity.
Ultimately, the most important thing we’ve learned is that software is never “done”. It’s an ongoing process that needs constant attention, review, and refactoring. If we’re willing to stay vigilant and grounded, perhaps we can finally leave the ghosts of the software crisis behind.
References
[1] NASA's Apollo Space Program. Smithsonian Magazine. https://www.smithsonianmag.com/smithsonian-institution/margaret-hamilton-led-nasa-software-team-landed-astronauts-moon-180971575
[2] Therac-25 Radiation Therapy Machine Disaster on Wikipedia. https://en.wikipedia.org/wiki/Therac-25
[3] The Failed Launch Of www.HealthCare.gov. Harvard Business Review. https://d3.harvard.edu/platform-rctom/submission/the-failed-launch-of-www-healthcare-gov
[4] Y2K bug | Definition, Hysteria, & Facts". Encyclopedia Britannica. Encyclopedia Britannica. https://www.britannica.com/technology/Y2K-bug