need not be the case. As Kent Beck pointed out, "XP does include design, you design a little, code a little, and repeat."
Another problem (for all developers, not just agilists) is the fact that design is not a subject most are comfortable with. Design is not part of typical computer science curricula and design requires knowledge of far more than the structure and modeling of programs.
A third problem arises from the fact: software development is a team activity. This means that the entire team must, somehow, share the same theory!
Agile already has the means and the tools to address these difficulties.
Starting with a kind of philosophical foundation. Alistair Cockburn included Naur's essay on theory building as an appendix in his book, Agile Software Development. Cockburn also noted how his understanding of agile principles, values and practices were colored by his reading of Naur. He was not alone, most of the other agile innovators had similar insights, they just remained implicit.
Numerous agile practices directly support the creation of theory, of conceptualization. Conceptualization is theory building; and it begins with telling stories about the world, the things in the world, what they do, why they do it, what other things they associate with and why and when they form those associations. This is the product backlog.
Metaphor is our most powerful tool as we seek to describe and understand what we do not know in terms of what we do know. Kent Beck once advocated the System Metaphor as a formal practice but has generalized the role of metaphor throughout the development process - even down to metaphor as a principle for proper naming of classes and methods.
Story cards are an "invitation to a conversation" and evocative memory of conversations had. On-site Customer insures a continuing evolution of theory as it is reformulated in terms of what is built - theory, like working software, is developed incrementally. At each stage the existing theory is evaluated and revised in terms of new understandings embedded in what we have built.
Most of the practices also support the development of consensual team theory building. Daily stand-ups, big visible charts, Collective Code Ownership, wall-to-wall white boards, communal work space, Whole Team–all of these ensure that everyone involved shares the same theory.
Reinhard Keil-Slawik is the author of a chapter in Software Development as Reality Construction , edited by Christiane Floyd. In that chapter he shows how "artifacts," like models on whiteboards, function as a kind of "external memory" for the humans working together developing software. Not only is this memory external–it is shared. Not only is it shared, it is integrated to the humans that create and use that external memory. External memory is not documentation and it cannot be passed to others as a means of sharing the theory. This fact is also noted and documented by Naur, and accounts for the difficulties encountered when developers are asked to alter or maintain the work of others.
The foundations for a theory driven approach to software development are clearly present in agile as it is practiced today. For YDD to become a reality those practices must be rearticulated stressing how they contribute to theory development and how they directly address what Brooks' called "The Essential Difficulty" of software development.
Existing practices must be augmented by techniques, heuristics, and practices that stress incorporate diverse subjects (e.g. the ethnography of ‘thick description' from anthropology, metaphor-based reasoning, and naturalistic philosophy) into the family of agile values, principles, and practices.
YDD will also need to address and overcome several obstacles.
Obstacles to Realizing YDD