Agile: A Long History

By: Dennis van der Spoel

Have you ever heard a song on the radio that was a cover of an old pop-song? And have you ever pointed out to a bunch of teenagers that this was already a hit when you were young? Remember the response you got? As if you were stealing their mojo.

As a middle-aged agile consultant, trainer, and coach, I regularly get similar responses when I’m conducting my daily business. I often get the feeling that the people I work with, both young and old, believe that an agile mindset is the prerogative of the younger generations, and that it is a recent ‘invention’. Nothing could be farther from the truth.

While agile is popular, even a hype, these days, there is nothing ‘recent’ about it. Get ready for a long journey.

From 'Friktion' to 'Auftragstaktik'

​Carl von Clausewitz observed in October 1806 how two French forces, acting almost as one, destroyed the Prussian Army in the twin battles of Jena and Auerstedt. Clausewitz knew first hand what it was like both to plan an action and to conduct one, and the experience got him thinking and writing. In 1832, after his death, his wife published his work titled ‘Vom Kriege’ (On War) in which he showed remarkable insights into the nature of war (and other complex endeavors), where everything is simple, but the simplest thing is difficult. He introduced the concept of ‘Friktion’ (Friction) as the concept which covers in fairly general terms what it is that makes the difference between the real thing and the thing on paper. One leading Clausewitz scholar has summarized the concept of Friction as referring to the totality of uncertainties, disagreements, errors, accidents, variabilities, technical difficulties, the unforeseen and their effect on decisions, morale, and actions.

Clausewitz’ second major contribution is in describing complex endeavors as nonlinear. Today there is a whole realm of scientific studies into nonlinear dynamics, also known as ‘chaos theory’, complexity or systems thinking, but at the time it was revolutionary. While his contemporaries, like Von Bülow still promoted a scientific approach to organizing and controlling an army, Clausewitz was reaching towards the idea of the organization as an organism. While the scientific school sought to eliminate human factors to make the organization as machine-like as possible, Clausewitz sought to exploit them.

His thinking inspired others to change the ways of the Prussian army. General David Scharnhorst observed that commitment through conviction (now known as ‘Theory Y’) trumps compliance through compulsion (now known as ‘Theory X’). He also concluded that of the three fundamental variables in war – force, space, and time – lost forces (resources) could be replaced and lost space (market share) could be recaptured, but lost time could never be made good. It was essential to take actions which were about right quickly, rather than waiting to be told in detail what to do.

Prince Friedrich Karl of Prussia in an essay from 1860 tells the story of a staff officer dutifully carrying out an order without question, only to be pulled up short by a high-ranking general with the words: “The King made you a staff officer because you should know when not to obey.” The Prussians didn’t allow themselves to be hemmed in with rules and regulations, but give rein to the imagination and exploit every opportunity opened up by unexpected success.

Field Marshal Helmuth Carl Bernhard Graf von Moltke, was the main builder of the German Army which emerged from the Prussian Army. He invented a concept called ‘Auftakstaktik’ (Mission Command), which he published in 1869 and is still used by NATO forces today. He understood that reality doesn’t stick to the plan. Von Moltke is often quoted as follows:

“No plan of operations extends with any certainty beyond the first contact with the main hostile force. Discipline means being ready to act on your own initiative and involves not commanding more than is strictly necessary, nor planning beyond the circumstances you can foresee. Circumstances change very rapidly, and it is rare indeed for directions which cover a longer period of time in detail to be fully carried out. The rule to follow is that an order should maintain all, but also only, what subordinates cannot determine for themselves to achieve a particular purpose.”

So, Von Moltke would have supported EDUF (Enough Design Upfront) and the use of user stories. He would also agree with Daniel Pink on the topics of Purpose, Mastery, and Autonomy. He broke the compromise (that existed in the army at the time) between alignment and autonomy. He demanded high autonomy and high alignment at one and the same time by distinguishing between ‘what and why’ (intent / purpose) and ‘how’ (autonomous actions) when providing directives. He rigorously trained his commanders in these principles.

Chief of the General Staff, Hans von Seekt, expanded on Von Moltke’s ideas. Whereas Von Moltke had restricted the use of these directives to the higher levels of command, in 1933 their use was pushed right down the hierarchy. To further this and create greater levels of trust, all NCOs were trained as officers, and officers were expected to master the tasks of two ranks higher up the hierarchy and to take their place if needs be. The effect of this was that during WWII, despite being completely outnumbered and outproduced, the German Army outperformed the Allies on every occasion. In 1977 US Army Colonel Trevor Dupuy reluctantly concluded:

“On a man for man basis, the German ground soldier consistently inflicted casualties at about a 50% higher rate than they incurred under all circumstances. This was true when they were attacking and when they were defending, when they had superior numbers and when they were outnumbered, when they had air superiority and when they had not, when they won and when they lost.”

Meanwhile in business, we were still stuck with the ideas of Frederick Winslow Taylor, who published ‘The Principles of Scientific Management’ in 1911. It was not until the 1980s that some writers began to suggest that business organizations were more like organisms than machines and that they contained people with brains as well as hands and legs. Finally, in the 1990s and 2000s many distinguished professors and consultants like Richard D’Aveni, Robert Ogilvie, Jorge Vasconcellos è Sa, Donald Krause, Gao Yuan, Harro von Senger, William Cohen, Richard Barrons, Deborah Tom, Stephen Bungay, Tiha von Ghyczy, Bolko von Oetinger, Christopher Bassford, Robert Kiyosaki, Paul de Ruijter, and Jaap Jan Brouwer translated military strategies and doctrines to the business environment in their publications.​​

Japan in the Aftermath of WWII

I started my career in 1985. Those were exciting times. In that same year, Hirotaka Takeuchi and Ikujiro Nonaka co-wrote an article for the January 1986 edition of Harvard Business Review. Called ‘The New New Product Development Game’, this article was instrumental in revolutionizing the discipline of Project Management. Takeuchi and Nonaka gave us a new way of thinking about how to develop products and deliver projects. And they coined an evocative sporting metaphor for their process, which has stuck: ‘scrum’. They followed this article up with a 1995 book, ‘The Knowledge-Creating Company: How Japanese Companies Create the Dynamics of Innovation’. This looks at the way Japan became a major economic power, especially in the automotive and electronics industries. This way has evolved into what most of us now recognize as Lean, Kanban, and Agile.

Nonaka was originally hired by the Japanese government after World War II to help analyze why they lost the war. His research showed that Japan was simply outproduced by the United States, as Germany was outproduced by the joint efforts of Russia and the US. So, the Japanese took interest in American production methods. Taiichi Ohno, the inventor of the Toyota Production System says everything he knows he first learned at Ford. Then all he did was go back to Japan and remove waste. The story of his work is summarized in ‘The Machine That Changed the World’ where Womack discusses what Ohno did after he returned from studying mass production at Ford:

“Back at Toyota City, Ohno began experimenting. The first step was to group workers into teams with a team leader rather than a foreman. The teams were given a set of assembly steps, their piece of the line, and told to work together on how best to perform the necessary operations. The team leader would do assembly tasks as well as coordinate the team, and, in particular, would fill in for any absent worker—concepts unheard of in mass production plants… Ohno next gave the teams the job of housekeeping, minor tool repairs, and quality checking. Finally, as the last step when teams were running smoothly, he set time aside periodically for the team to suggest ways collectively to improve the process.”

Another important factor must be the physicist and statistician Walter Shewhart of Bell Labs, who in the 1930s began applying Plan-Do-Study-Act (PDSA) cycles to the improvement of products and processes. Shewhart taught this iterative and incremental-development methodology to his mentee, W. Edwards Deming, who used it extensively in Japan in the years following World War II. Toyota hired Deming to train hundreds of the company’s managers, eventually capitalizing on his expertise to develop the famous Toyota Production System - the primary source of today’s ‘lean’ thinking. Iterative and incremental development methods were also a major contributor to the successful creation of the X-15 hypersonic jet in the 1950s.

This cross functional team process and continuous improvement was observed not only at Toyota, but in many of the best companies worldwide by Takeuchi and Nonaka while they taught at the Harvard Business School in the early 1980s. The teams at Toyota and elsewhere reminded them of the game of rugby and they called this style of project management ‘Scrum,’ a short form of the term ‘scrummage’ where the game is restarted when the ball has gone out of play.

Takeuchi and Nonaka have written many books and papers about Toyota, Honda, and other lean companies, yet they never talk about lean. They talk about ‘scrum’, which means to them cross-functional teams engaging in the dynamic conflict of ideas that generates ‘ba,’ the energy flow that surfaces knowledge that forms new products. It's the innovation they are interested in and what westerner's call lean are a bunch of context dependent techniques that are side effects of knowledge generation.

What Takeuchi and Nonaka saw at Toyota, Honda, Canon, Fuji-Xerox, 3M, HP and other high performing organizations is ‘scrum project management’, which, to them, means teams that are autonomous, motivated by transcendent purpose, and engaged in cross learning. Short iterations combined with these team dynamics facilitate a knowledge generation cycle that leads to innovation, faster time to market, and higher quality. The ‘lean techniques’ touted by Western observers are side effects of what Takeuchi and Nonaka see as the root cause of performance. And that root cause are self-organizing teams engaged in continuous improvement.

To Nonaka, scrum is only indirectly related to software. It is directly related to leadership and running the top companies in the world. See the recent HBR paper on ‘Wise Leadership’ by Takeuchi and Nonaka. Takeuchi teaches scrum in his classes by reviewing the case studies taught at the business school and showing how success was always due to cross-functional teams working intensely together generating continuous improvement. This is scrum to Takeuchi.

An Evolution in Project Management

Meanwhile, project management began to evolve into a separate discipline and profession within information technology. And with a new profession came new frameworks and methodologies. In the 1970s, two models emerged: waterfall (Winston W. Royce, 1970) and adaptive (E.A. Edmonds, 1974). Waterfall (predictive) proved best for capital-intensive projects (requiring multiple rounds of investments), while the adaptive (iterative) model proved best for projects involving a lot of knowledge workers. When, over time, more and more IT-projects required capital investments as well knowledge workers a dilemma was born. So, in the 1990s the race was on to solve this dilemma. A lot of new approaches were launched, most prominently:

• Rapid Application Development (RAD) – James Martin, 1991
• Dynamic Systems Development Method (DSDM) - DSDM Consortium, 1995
• eXtreme Programming (XP) – Kent Beck, Ward Cunningham, Rom Jeffries, 1996
• PRojects IN a Controlled Environment v.2 (PRINCE2) – The Office of Government Commerce (OGC), 1996

Of this short list, PRINCE2, has received a lot of fame as well as a lot of blame. The best-practice is still widely used, although commonly misunderstood and improperly applied, which accounts for a lot of the blame.

The approach, however, that took the world by storm is another one. In 1993 Jeff Sutherland faced what seemed like an impossible task: Easel Corporation, a software company, needed to develop a new product to replace its legacy offerings in less than six months. Sutherland already had experimented with methodologies such as rapid application development (RAD). He then began learning everything he could about maximizing organizational productivity. Reading hundreds of papers and interviewing leading product-management experts, he found himself intrigued by several provocative ideas. One came from a Bell Labs article on the Borland Quattro Pro team, suggesting that short daily team meetings increased group productivity dramatically. But the capstone concept for Sutherland was the discovery of Takeuchi’s and Nonaka’s rugby approach, even though it focused on manufacturing rather than software. Borrowing many of the HBR article’s key ideas and filling in specific operational practices, Sutherland created a new way of developing software; honoring the rugby imagery, he dubbed his approach ‘scrum.’ Scrum methods enabled him to finish his seemingly impossible project on time, under budget, and with fewer bugs than any previous release. He then collaborated with longtime colleague Ken Schwaber to codify the approach, and in 1995 the pair presented scrum to the public for the first time.

Of course, Sutherland and Schwaber weren’t alone in their search for innovative methods. The Information Age was exploding. Disruptive technologies were terrorizing slow-footed competitors. Start-ups and incumbents alike sought better ways to adapt to the unfamiliar and turbulent environment. Software was becoming an integral part of nearly every business function, and many creative software developers were working hard on better methods of programming to increase adaptability.

In 2001, 17 software developers met in Snowbird, Utah, to share their ideas. Sutherland and other proponents of scrum were among them. But the group also included advocates of several competitive approaches, including extreme programming (XP); crystal; adaptive software development (ASD); feature-driven development (FDD); and the dynamic-systems-development method (DSDM). All these approaches were often known as ‘lightweight’ frameworks because they used fewer, simpler rules to allow faster adaptation to rapidly changing environments. Not many of the attendees found the ‘lightweight’ terminology flattering.

Although they disagreed on much, the group eventually settled on a new name for the movement: agile. The word was suggested by one attendee who had been reading the book Agile Competitors and Virtual Organizations: Strategies for Enriching the Customer. The book gave 100 examples of companies — including ABB, Federal Express, Boeing, Bose, and Harley-Davidson — that were creating new ways of adapting to more turbulent markets. Name in hand, attendees then forged consensus on a call to arms dubbed the “Manifesto for Agile Software Development,” which spelled out four key values that everyone agreed on. Later in the meeting, and continuing over the next few months, they developed 12 operating principles, called “Principles Behind the Agile Manifesto.” From 2001 on, all development frameworks that aligned with these values and principles would be known as agile techniques.

Meanwhile, agile methodologies continued to evolve. In the late 1980s and early 1990s, researchers from MIT had begun to study Japanese manufacturing systems, especially the Toyota production system. They coined the term ‘lean’ to describe the system’s methods of improving productivity by eliminating waste (‘muda’) through reductions in uneven work flows (‘mura’) and destructive overburdening (‘muri’).  Although lean methodologies were not presented as agile frameworks in Snowbird, formal lean and kanban software-development systems emerged during the 2000s (e.g. Lean Software Development, Tom & Mary Poppendieck, 2003). At first, some agile purists refused to recognize these approaches as agile methodologies. But lean advocates intensified their focus on customer collaboration, and eventually more agile practitioners came to accept lean, kanban, and their hybrids (such as scrumban and lean scrum) as legitimate applications of agile values and principles.

As early agile methodologies were designed to optimize cooperation and flow in a single development team, the challenge then became to scale the agile way of working across the enterprise. A new race was on and various groups uncovered and codified how to scale the agile way of working in their specific context. The multitude of new frameworks fall in two categories:

1. Product Development as a Unique Assignment, which means temporary cross-functional project/program organizations;
2. Product Development as Business as Usual, which means permanent cross-functional line organizations.

Examples of the first group are PRINCE2 Agile and AgilePM. Examples of the latter group are Scaled Agile Framework (SAFe), Nexus, LeSS, Spotify, and S@S.

A Final Word and Credits Due

Success has many fathers, and agile innovation has a colorful heritage. While agile’s complex family tree sometimes provokes passionate debates among agile practitioners, two things are clear: first, agile’s roots extend far beyond information technology and, second, agile’s branches will continue to spread to improve innovation processes in nearly every function of every industry. And I was there to live it, contribute to it, and watch history evolve. And I will hopefully do so for many years to come.

Much of this post is not my own, as I took ideas and copied snippets of text from articles, blogs, and books written by others to create this unique history of agile. Main sources used are:

• Book: The Art of Action – Stephen Bungay
• https://en.wikipedia.org/wiki/Agile_software_development
• HBR article: The Secret History of Agile Innovation by Darrell K. Rigby, Jeff Sutherland and Hirotaka Takeuchi
• ScrumInc blogpost: Takeuchi and Nonaka: The Roots of Scrum by Jeff Sutherland

​Dennis van der Spoel Consulting helps organizations to become more agile while staying true to their cause. Our firm specializes in the transition to a modern way of working with respect for existing practices. Please address any questions you might have to us via the contact form on this website.