Evidence for Evolution

A few years ago, I heard Richard P. Gabriel give a talk on the evolution of the programming language Lisp, from its creation in the 1950s up to today. He stressed the long and painful path to ideas that seem obvious to us now. How, we wonder, could our predecessors not have seen it? They couldn’t see it in part because the very words they used—"variable," "function," and so on—meant different things than they do now. The history of Lisp and other programming languages is the history of figuring out what those words ought to mean in a domain where people write text to be executed by stupid, fast machines.

I was on the periphery of Lisp's evolution in the early ‘80s. From that and from Gabriel's talk, I think that evolution depended on these things:

A hothouse community : There were, arguably, two foci of Lisp activity: the MIT and Stanford artificial intelligence labs. They were populated by young people who obsessed about the problem of communicating with the computer to solve hard problems. They fed each other's obsession at work, at dinner, and at play.

Written expression and discussion : Despite geographical locality, the people involved were heavy early users of the predecessor to the Internet (the ARPANET). For many people, writing clarifies thinking, and traditions like the MIT AI lab memos presented ideas rigorously enough that email discussions could be productive.

Reflective use : In many cases, the same person developed an idea, implemented it in the language, and used it in a substantial program—or it was done by people who spent a lot of time together. Therefore, lessons from implementation and use had an immediate conduit into the brain of someone who cared about improving Lisp.

Domain hopping : Lisp is an extremely abstract language, favoring programmer expressiveness over machine efficiency far more than most languages of the day (and even today). And yet the very same people who did the best thinking about Lisp were also fluent in assembly language, the language "closest to the machine." Their discussions could bounce between two relevant domains, generating ideas along the way.

Software is a language for manipulating some domain, such as a business. Most such software is disappointing because it never comes to grips with the domain in the way Lisp hackers came to grips with programming. Because of that, the program as it is doesn't help anyone understand the program as it should be. It and the development team are just passive recipients of requirements. Those requirements typically just replicate the surface complexity of the domain; they're neither a result nor a cause of inspiration.

To have success like the Lispers, we need to find and nurture the people who have the best understanding of both the business and system internals. On many projects, these are the testers and analysts. It's their job to find the right words to describe the domain, use those words precisely, and help others on the project become fluent as well.

That work would be aided by writing user documentation early. By "user documentation," I don’t mean page after page describing what menu items do. I mean documentation that helps the user make sense of why the program does what it does. Writing that documentation is an excellent way of learning that, actually, it doesn't make sense.

Tests are also a good way of finding the words to describe the domain—but only when the tests are written in terms of the business, not in terms of the user interface or code. I would expect to find the words in the tests