YDD: The Future of Agile

models of complex phenomena, deriving properties from the models, and verifying those properties by experiment. This paradigm worked because the complexities ignored in the models were not the essential properties of the phenomena. It does not work when the complexities are the essence.

Much of the complexity that [the software developer] must master is arbitrary complexity, forced without rhyme or reason by the many human institutions and systems to which his interfaces must conform. These differ from interface to interface, and from time to time, not because of necessity but only because they were designed by different people, rather than by God.

It is no wonder that Brooks considers software development to be hard. Given all the obstacles to the formation of adequate conceptual constructs–and the absolute need for such a construct - what reason do we have that it is even possible?

We do have some existence proofs–software entities that clearly are grounded in a well-formed conceptual construct. We also have existence proofs that both individuals and groups can formulate and share conceptual constructs of the same order of complexity as any software entity. I speak, of course, of things like culture and cosmology.

As humans we have internal conceptual constructs of the universe, our place in it, and of all the interactions and consequences of those interactions that take place in it. We can successfully apply this mental model even though it would be impossible to articulate it, model it, or reduce it to a set of formal rules.

We also have several prominent developers who hold that the establishment of a conceptual construct is the essential act required to successfully develop software. Peter Naur (" Programming as Theory Building") , for example, holds that software development IS conceptual construct creation–what he calls theory building.

"I shall use the word programming to denote the whole activity of design and implementation of programmed solutions. ... the activity of matching some significant part and aspect of an activity in the real world to the formal symbol manipulation that can be done by a program running on a computer.

... If it is granted that programming must involve, as the essential part, a building up of the programmer's knowledge, the next issue is to characterize that knowledge–the programmer's knowledge should be regarded as a theory, Very briefly, a person who has or possesses a theory in this sense knows how to do certain things and in addition can support the actual doing with explanations, justifications, and answers to queries about the activity of concern. [Naur 1985]

Theory Building and Agility
"Theory building" is also known as "design" and design has always been a bit problematic for agile developers. The first problem is the traditional belief that design is something done "up-front;" before any actual work (testing and coding) can take place. But this need not be the case. As Kent Beck pointed out, "XP does include design, you design a little, code a little, and repeat."

Another problem (for all developers, not just agilists) is the fact that design is not a subject most are comfortable with. Design is not part of typical computer science curricula and design requires knowledge of far more than the structure and modeling of programs.

A third problem arises from the fact: software development is a team activity. This means that the entire team must, somehow, share the same theory!

Agile already has the means and the tools to address these difficulties.

Starting with a kind of philosophical foundation. Alistair Cockburn included Naur's essay on theory building as an appendix in his book, Agile Software Development.