Sep 13, 2007

Waterfall vs. Agile Development

Waterfall vs. Agile Development
Criticism of the waterfall model of software development is in vogue these days, while the relative benefits of agile methods are extolled - with the primary assertion being that "requirements are unknowable" and continually evolving, and must be learned by progressive discovery through trial and error.

I assert that the value proposition of agile methodology has nothing to do with evolutionary requirements discovery, and everything to do with listening carefully to the customer and building a better understanding of their needs.

Agile proponents postulate that fully understanding requirements is impossible, holding up failures of the past as anecdotal evidence. The conclusion has been that the lifecycle model is at the root of these failures. It's more tolerable to blame the lifecycle model than confess a lack of knowledge, skill and discipline in eliciting requirements.

The real success of agile - that many have missed - is the subtle insertion of frequent validation of developer assumptions with users. It is this novel concept of actually listening to the user that is the real key to success in agile. But frequent validation was not invented with agile methods - and has been a mainstay of successful software projects from the beginning.

The reality is that, although we live in a very turbulent time, business needs actually evolve rather slowly. And while there are some technologies that are evolving rapidly, most software is developed for more "mature" needs.

At the same time agile is reveling in the spotlight, a quiet revolution is taking place in thoughtful, lean-thinking software companies. They are targeting poor requirements practices as a source of waste, and are focusing on proactive discovery of users needs as the solution. This approach differs from agile in one important way: it recognizes that requirements are not as hopelessly volatile as suggested by agile/anti-waterfall extremists - that requirements volatility is more an effect of evolving understanding of user needs, than of changes in them.

A major source of methods in this space is an unlikely, but logical, discipline: human factors - not the UI designers focused on layouts and widgets, but the psychologists who look at the systems within which humans interaction with computer systems. ACM's SIGCHI is common ground for many of these folks.

One method in this space with which I've had incredible personal success is called Contextual Design. It teaches developers to learn how system users think about their work. One of its activities, Contextual Inquiry, puts software developers in the cubicles of prospective customers to observe, understand, and record the details of the work. What they learn is taken back and shared with the development team. One of the most remarkable things about Contextual Inquiry has been its effectiveness and efficiency in building the deep understanding needed by developers to make informed day-to-day design decisions. This is a huge leap past the practice, shared by traditional and agile methods, of unwittingly depending upon erroneous personal assumptions about the users' world.

Other elements of Contextual Design provide efficient up-front validation at critical decision points along the way that minimize waste, an approach to definition and construction of incremental releases, each contributing recognizable customer value.

Instead of spending energy on variations and iterations on the agile theme, developers need to focus on learning and applying new skills to capture and share a more robust understanding of the environment and work of targeted users. If we did that, we'd be developing more relevant software with less waste and more profit. And we'd be having a lot more fun doing it.