Pre-agile helps one to understand agile

One of the greatest mistakes of the agile community is to compare agile with waterfall. There is an assumption that before there was agile, there was waterfall, and that most projects were waterfall. That is not my experience.

It is not fair to compare a well run agile project with a highly dysfunctional project using waterfall, yet that is the comparison that is routinely made.

During the 1980s I was on a string of very successful IT projects, and none of them were waterfall. On the other hand, I was on one project that was not successful, and it was a waterfall project.

The projects that were successful (all non-waterfall) were at two companies: Intermetrics, and CAD Language Systems. These companies built compilers and other advanced tools that were used to design hardware systems. This was major league programming. Our projects were characterized by small teams (3-15 people), lots of collaboration, evolutionary design, and lots of other practices that agile claims credit for.

These experiences have helped me enormously to understand agile, because I can look at agile practices and compare them to earlier practices that I saw work well - even though they were often done slightly differently - so I can discern what really makes those agile practices important. I can also discern that certain agile practices are not critical, because I saw projects be successful without those practices. Standups for example: of all of the successful projects that I was on during the 1980s, none used the practice of standups, and so I am confident in saying that standups are not important. Another practice that is a red herring is the team room: during the 1980s, programmers had their own offices (at least they did everywhere I worked), yet we collaborated continually - separate offices were not an impediment, as long as we were co-located. Co-location was important. And I distinctly recall closing my door from time to time so that I could maximize the quiet to think deeply about a problem, and then open it again when I had finished thinking. Thus, the ability to shut out the world to think deeply was also important. The open door was a universal signal that you were open to someone walking in to discuss something: the closed door was the reverse.

Each of those projects that were successful had a person who was responsible for making sure that everything fit together: someone who was charged with thinking about the entire system in an end-to-end manner. That was essential, and when there was no such person, or when the person was incompetent (that was the case on the waterfall project) things went wrong very quickly. I have also seen agile projects flounder for lack of such a person. The theory that the entire team is responsible for design is kind of like communism: it is a nice egalitarian theory, but in practice it seldom works - I won't say "never" because there are always exceptions. Generally speaking, there needs to be a qualified person whose main job is to think end-to-end, even if that person also does coding. The real issue is what type of person that should be, because at other times in my career I have had nightmare project managers or technical leads who almost single-handedly made everything go wrong (the waterfall project was like that).

In the course of these 1980s projects, the two things that I found to really make a difference in a project were:
    ▪    Small team: if there are so many people that they cannot keep track of what they are each working on, then communication breaks down and code diverges.
    ▪    Servant leadership: Someone who provides gentle leadership to the team: not someone dictatorial, but someone who keeps track of what everyone is doing and what challenges they have on a daily basis; ensures that people discuss issues that need to be discussed; asks hard questions, listens, and proposes solutions but rarely dictates them; and who also understands all of the issues - someone who is knowledgeable about what the team is working on and how it will work. In my experience, self-organization cannot substitute for a good servant leader.

From there, things kind of take care of themselves! With good servant leadership, you will end up with continuous daily regression testing (we did), you will have information radiators on testing results and on the evolving design (we did), you will have a continuous feature-driven or story-driven process with testable features or stories (we did), you will have continual design discussions as needed throughout the project (we did), the team members will feel empowered to work in their own way and contribute ideas and innovation (we did and did), and there will be a sense of harmony, order, and calm rather than an atmosphere of crisis and frustration. Servant leadership is really the key: everything else will follow, as long as the project is not hamstrung from the beginning by having a team that is too large or by having other poisonous situations imposed from the outside.

Even the practice of developing requirements incrementally is not new: circa 1980 I worked at American Electric Power as a nuclear physics simulation engineer, and there was a programming team that supported us, and the method in which we interacted with them was such that the programmer would sit with us and talk about what we wanted, they would go away and develop some of that, then come back and show us to get feedback, then go away and build some more - sound familiar?

So when I reflect on the Agile Manifesto today - or when I did after it was published - I see it as a rejection of the wrong paths that some projects - waterfall projects - took before that and a return to what worked. It was not new, but rather it was a validation of key things that had worked in the past, and that historical perspective helped me to understand the motivation behind each value and each principle and what its intent really was. And yes, agile does add some tweaks to some of those historical practices: that is valuable contribution, but the historical perspective is just as - I would say more - valuable.