YDD: The Future of Agile

Cockburn also noted how his understanding of agile principles, values and practices were colored by his reading of Naur. He was not alone, most of the other agile innovators had similar insights, they just remained implicit.

Numerous agile practices directly support the creation of theory, of conceptualization. Conceptualization is theory building; and it begins with telling stories about the world, the things in the world, what they do, why they do it, what other things they associate with and why and when they form those associations. This is the product backlog.

Metaphor is our most powerful tool as we seek to describe and understand what we do not know in terms of what we do know. Kent Beck once advocated the System Metaphor as a formal practice but has generalized the role of metaphor throughout the development process - even down to metaphor as a principle for proper naming of classes and methods.

Story cards are an "invitation to a conversation" and evocative memory of conversations had. On-site Customer insures a continuing evolution of theory as it is reformulated in terms of what is built - theory, like working software, is developed incrementally. At each stage the existing theory is evaluated and revised in terms of new understandings embedded in what we have built.

Most of the practices also support the development of consensual team theory building. Daily stand-ups, big visible charts, Collective Code Ownership, wall-to-wall white boards, communal work space, Whole Team–all of these ensure that everyone involved shares the same theory.

Reinhard Keil-Slawik is the author of a chapter in Software Development as Reality Construction , edited by Christiane Floyd. In that chapter he shows how "artifacts," like models on whiteboards, function as a kind of "external memory" for the humans working together developing software. Not only is this memory external–it is shared. Not only is it shared, it is integrated to the humans that create and use that external memory. External memory is not documentation and it cannot be passed to others as a means of sharing the theory. This fact is also noted and documented by Naur, and accounts for the difficulties encountered when developers are asked to alter or maintain the work of others.

The foundations for a theory driven approach to software development are clearly present in agile as it is practiced today. For YDD to become a reality those practices must be rearticulated stressing how they contribute to theory development and how they directly address what Brooks' called "The Essential Difficulty" of software development.

Existing practices must be augmented by techniques, heuristics, and practices that stress incorporate diverse subjects (e.g. the ethnography of ‘thick description' from anthropology, metaphor-based reasoning, and naturalistic philosophy) into the family of agile values, principles, and practices.

YDD will also need to address and overcome several obstacles.

Obstacles to Realizing YDD
There are two major obstacles that need to be addressed by YDD. Both transcend Agile and apply to all approaches to developing software.

One: almost all software developers today already operate on the basis of a "default theory." Unfortunately this theory is wrong and counterproductive. Briefly stated, developers, especially those trained in classic computer science programs, use the implicit theory, "the world is a computer."

Things learned from mathematics and the physical sciences allowed, and continue to allow, us to make advances in the computers and the basics of computing. However, as Brooks noted, those same tools do little to help us understand the essential complexity of the applications and systems that employ computers. This is the primary reason I objected to Model Driven Development earlier. MDD (or