What Drove the Creation of Agile? Not a Search for a Better Way!

I was thinking about what drove the creation of Agile. Having lived through that period, I guess I was too close to it to really see what was happening, but now, with the power of hindsight it suddenly dawns on me that Agile wasn't a search for a better way. Here is what I think happened, or at least this is how I saw it in my little patch of Developer Land....

By the closing years of the 20th century the role of business software had moved past simply enabling back-end database tasks and into the role of enabling faster business processes. Software moved from supporting business processes to providing competitive advantage.

However this change came with some problems. The software development processes were set up to support fixed, slow changing requirements. Software was now trying to support fast-moving and constantly changing business needs.

Constantly changing business processes led to constantly changing project requirements. A string of high-profile failures dogged the industry in the late 1980s and throughout the 1990s.

In response, the phrase “you can’t develop on quicksand” was used to justify increasingly strict adherence to documented specifications – the software system was built and delivered in accordance with the specification and required changes were entered into a change log for future releases - and then Version 2 was developed to deliver the required change. I remember that "Version 2" became the answer to every change request.

Consequently, the systems delivered were never a reflection of the current business requirement.

Software developers became frustrated by this problem – characterised by the phrase “The Project was a success, but the business need died” – and they started experimenting with shorter cycle times and more fluid requirements processes.
 
And so the key principals of Agile were born, not as "a better way" - the previous way worked until it didn't - rather, Agile was born as a response to a change.

Blueprint For An Agile Guild in a large Corporate Environment

Vision Statement

As an Agile Guild, we want to engage, excite and educate Agile Coaches, so that Agile flourishes in [corporation X].

Implementation

Iterating down from the Vision to a proposed implementation plan in support of the vision, there are three parts (engage, excite, educate):-



1. Engage. Create a sub group within the Guild who will undertake to:
• Identify Agile Coaches and stakeholders, within [corporation X] and within vendors and suppliers, onshore and (if required) offshore.
• Connect with them. Construct an informal hierarchy or pathway that allows vision and strategy to flow from the Agile Guild to vendor coaches and and (if required) offshore coaches and development teams. This structure should allow goals to flow out and team-proposed implementation details/impediments to flow back. Follow the basic Agile principle - goals from the business, implementation details from the team.
• Identify potential Agile projects at [corporation X] and ensure that [corporation X] stakeholders, vendor and (if required) offshore coaches (identified above) are engaged and connected.
• Ensure that vendor coaches are involved in planning, retrospectives, education sessions etc. Ideally, the vendor coaches need to be so engaged that they identify themselves primarily as part of the [corporation X] Agile Team, not as part of the vendor team.


2. Excite. Create a sub group within the Guild who will undertake to improve engagement…. by exciting our stakeholders.
In order for people to buy in, we need to “sell” Agile. This means we need to answer the “What’s in it for me?” question for a variety of audiences/stakeholders. I’m not a sales person, but here are some answers I use for a few different types of audience:
• What is in it for developers – A level of self-determination, control and success. You get to decide your own actions and estimates, while ensuring that your customer is happy with the decisions.
• What is in it for upper management – This one is a no-brainer. Numerous studies from Gartner, Forrester, IBM, etc have shown that Agile gives them a Competitive Advantage. If they do not go down this path they give the competitive advantage to their competitors. Most of them would rather disembowel themselves with a spoon than give an advantage to the competition.
• What is in it for Vendors. This is a tougher sell, as you are outside of your organisation. The contractual model really needs to support your story. Split Contracts (a fixed price “Discover” phase followed by a Capped T&M “Evolve” Phase) or a “Pipeline” style contract, combined with the knowledge that other vendors are going down the same path, will strongly encourage the right behaviours. Within your vendors, make sure you form very close ties with your vendor coaches, and encourage them to do a lot of the “Selling” at the various levels.
• What is in it for Middle Management… Demonstrating self-interest for the “Frozen Middle” is difficult.  I try to hit two points. Firstly, they get to do something new and exciting. Secondly, Agile is currently hot in the market. So the skills they develop are very “marketable” either within [corporation X], or outside. Make sure you form very close ties with your vendor coaches, and encourage them to do a lot of the “Selling” at the vendor middle manager level.


3 Educate. Create a sub group within the Guild who will undertake to:
• Find out what our coaches are good at. We need experts in the nitty-gritty of Agile - Kanban/Scrumban, Lean, XP Technical practices (CI Automated Regression Testing, TDD, etc) and supporting tools, BDD, Backlog generation, etc, etc. And we need experts in the more esoteric or theoretical side of Agile – Queuing theory, the Theory of Constraints, Systems Thinking, Scaling Agile, Complexity Theory, Vanguard, etc.
• Schedule Cross-Training based on the list developed above. Everybody has something to give and everybody has something to learn.
• Find out who is coming out for Agile Coferences and find Corporate Sponsors who will get them to come into [corporation X] and speak.
• Make sure Vendor coaches are invited and/or recordings of the sessions are shared with them – your Vendor Coaches need to feel like involved equal partners.
• Connect people up with Special Interest Groups, Meet Up groups, Conferences, etc. Get our people recognised as experts, so that we can attract the best.
• Develop standard packages for running repeating task and ceremonies (such as Discover Workshops and even Retrospectives). This is intended for inexperienced practitioners - encourage experienced practitioners to improvise and adapt, as Retrospectives and similar ceremonies get boring and unproductive if you always follow the same pattern. The standard packages should include instructions and a physical package of things they will need (sharpies, paper, etc).
• Develop minimum standards for developers involved in Agile Projects – run courses. These standards apply to Vendor developers too!!!! Get your vendor coaches to develop courses, or do a “Train the Trainer” course for them, and let them use your course. The course needs to be run just before the project starts. Earlier than that, and the developers forget.

Strategic Kanban in Practice - Rolling Wave Planning Guided By Hypothesis Driven Development

Here is an approach to Strategic Portfolio Kanban that I have used with some success.

1.   Rolling Wave Planning Allows Planning in Increments.

I have suggested in the past that SAFe’s Portfolio Kanban methodology is applicable in almost any Enterprise that needs to manage multiple teams, but that SAFe’s  Agile Release Train (ART) planning methodology may be too heavy-weight for many situations.

However this raises a question – if you have no ART (Agile Release Train) planning process, what feedback mechanism is used to keep your Portfolio Kanban realistic?

And here is my answer, based on experience in recent engagements: Your Strategic or Portfolio-level plans should be incrementally reviewed and updated based on real-world feedback around team capacity and team progress - so use Rolling Wave Planning regularly updated from your Iteration or Sprint planning process.

In using Rolling Wave Planning remember the basic Agile Tenet that Estimates come from the team – so the planning process needs to feed a prioritised Backlog to your teams, and get a realistic work commitment based on capacity back from your teams. In this respect, it is similar to the ART planning process.

A rolling wave planning approach differs from an ART planning approach in that Rolling Wave Planning does not require all teams to run simultaneous, synchronised planning sessions. Rolling Wave Planning allows you to update your plan incrementally, rather than commit to the longer-term Release Plan cycles that an Agile Release Train plan tends to require.

 

2.   Hypothesis Guided Product Vision.

However Rolling Wave Planning is only one part of the Portfolio Kanban puzzle. Your planning comes from the Rolling Wave…. But where does your strategic direction come from? What guides the goals that shape the plan? SAFe suggests that you need a Vision Statement – but this just begs the question: What guides your vision?

And here is my answer: The Lean Start-Up approach provides Hypothesis-driven development (or Experiment Driven Development) as an aid to guiding your strategic vision.

But the Enterprises we work in are not Start-Ups. How, in an Enterprise environment, do you generate testable Hypotheses? (Yes, I had to look up the plural of hypothesis before I wrote that.)

The answer is easier than you might think. In an Enterprise environment you have two big advantages:

- First you probably have access to numerous sites from which to choose a few Pilot Sites.

- And second, you probably have a good user base, from which you can select representative users.

In my current engagement we have a user integrated into the team (echoes of my old XP days).  This product user is NOT the Product Owner. The product user has been brought in from the field, and is proving to be invaluable in two ways:

i.    The user can describe the reality of how the product is used (this has proved to be an education for the Product Owner – who is too high level to have a grasp of the fine-grained details).

ii.  The user can talk to other users to develop hypotheses about where the product might go.

The value of that last point cannot be overstated. If you want to develop a good hypothesis, don’t just bring users into a product review and ask them what they think. Instead, ship your product to the first pilot sites and let it settle in for a while. Then send your user out to talk to the people using the product. Don’t send a BA or a manager – your user has to be a peer. The pilot-site users will be much more likely to give extended, honest and thoughtful responses to a peer – because they know that the peer will soon be using the product.

You can equip your user with questions, but it has to be a user that asks them.

When your user comes back into the office, you can start generating hypotheses. At this time, compare the user findings and hypotheses to your Product Vision. Adjust Vision as required.

After reviewing your Product Vision, you may need to send the user back out to the pilot site to validate or further refine the hypotheses list (keep your list short). Once you have your list of hypotheses developed and finalised, you are ready to test.

You may not need to write code for your tests – consider using physical mock-ups, wireframes, or simple what-if scenarios. Remember the Agile dictum – the simplest thing that works.

Once you have tested your various hypotheses, you can enter the highest-value ideas into your Portfolio Kanban, where they can be analysed, assigned a business case, and then prioritised against other ideas to form a portfolio backlog.

Streaming to the Pipeline

Now stream your Portfolio Backlog into Pipelines of work and send the Pipeline Backlogs out to your Pipeline Teams to get their estimates and a commitment to the work. The teams can update estimates, progress and work commitment  using the usual Agile Iteration or Sprint mechanisms.

This allows you to constantly update the plan and look forward using your Rolling Wave approach.

And there you have it – your Hypothesis-driven Product Vision drives your Portfolio Kanban, which generates your prioritised backlog, which is incrementally updated in a Rolling Wave Planning process..... A practical approach to Strategic Kanban using Lean Start-Up ideas in an Enterprise environment.

 

Queuing Theory

What is Queuing Theory and Why Is It Relevant?

Our goal in an Agile Project should be to create Business Value in the shortest possible time. In undertaking this goal we are attempting to move a backlog or "queue" of tasks through a process as quickly as possible. It is here that Queuing Theory becomes relevant.

The time taken from the moment a piece of work enters our work process until it exits (Done) is known as the "Lead Time". This is a valuable metric when using Kanban. Other valuable metrics are "Cycle Time" (the time taken to actually process the work item once work starts) and "Throughput" (the rate at which work flows through the system).

Some Relevant Guidelines From Queuing Theory

To minimise the cycle time we can look for certain problems or warning signs. The following problems can increase cycle times:

•Variability in rate of arrival, or in service time (the time taken to respond to or process a task), will increase the cycle time and the queue. Reducing variability will improve flow. Variability increases cycle time.
•The arrival of large batches leads to queues and associated inefficiencies. Large batches increase cycle time.
•Very high utilization results in no spare capacity for dealing with sudden incidents or unusual batches.

Consequently, when such incidents occur, there are problems associated with increased queues, task switching and associated inefficiencies. This greatly increases cycle time. High utilization increases cycle time.
To improve cycle times and reduce queues, target an even, stable rate of arrival. Restaurants, parking stations and night clubs achieve this by applying discounts or introducing offers at certain times of the day.

Other ways to help are: create smaller batches, create stable service times, and utilize parallelism where applicable (if a task can be split into 2 atomic tasks with no interdependencies then it can be shared between 2 people, thus halving the cycle time.)

Other Guidelines include:

•Decreasing variability early in the process is better than later. The first bottlenecked station dictates the rate to the rest of the system.
•High utilization can lead to problems that are worse than just high cycle times. A system where every station is working all the time is close to collapse (also known as Goldratt's maxim). If everyone is already working all the time, the system has no time to recover when its queue grows due to upstream variability.
•Engage early. Start processing as soon as a batch starts to arrive.
•Monitor the queue. Respond if the queue grows and understand what caused it to grow - respond to the cause as well.

Applying Queuing Theory To Agile Software Development - An Example

As an example, we can apply queuing theory to a common problem in testing:
Testers in many projects do little for the first 75% of the Sprints and are then overloaded in the last part of the sprint - they work long hours (often all night or weekends) to get the work done.
Queuing theory tells us that if you can't match servers to the load (increase the number of testers on demand), then you need to create a more stable arrival rate (the work associated with testing needs to be spread out more evenly).

Based on the guidelines above we need to:
1.Decrease variability. The load should not all arrive at the end of the sprint.
2.Decrease the size of individual units of work (Decrease Batch Size).
3.Engage with a unit of work early.
4.Monitor the queue and respond.

So, in terms of Queuing theory, our solution to the problem is for testers to:
1.Decrease Variability and engage earlier with tasks. Plan and write tests earlier, as part of requirements elaboration (based on acceptance criteria), and automate as you implement. Don't wait for the developer to complete development as this leads to a large, late batch arrival. Writing tests as part of requirements elaboration smooths out the work flow and addresses points 1 and 3 above.
2.Create Small Batch Sizes (work units). Plan for tasks to be small and testable. Don't plan for large user stories, break them into smaller chunks of functionality, ideally 4 hours – use the INVEST principle.
3.Monitor Your Queue. Look for people that are sitting idle because they're waiting for something to test. Understand why they are sitting idle.

References:

Lean Tools and Queuing Theory: http://css.dzone.com/articles/lean-tools-queuing-theory

Wikipedia on Queuing Theory: http://en.wikipedia.org/wiki/Queueing_theory

The Theory Of Constraints

Overview

In conjunction with Queuing Theory, the Theory of Constraints (ToC) presents some additional ideas for making improvements in the efficiency of the Software Development process.

The Theory of Constraints was introduced into management by "The Goal", a book written by Eli Goldratt in 1984.

The basic idea is that you can always find a slowest point (the area with the slowest throughput of work) or the "limiting constraint" in a process by finding out where there is a buildup of unfinished work. This slowest point defines the limit of the amount of work that can be put through the system.

This slowest point is where you need to concentrate your efforts for improvement - because improvements made anywhere else will not improve the rate at which work can be moved through the workflow.

For example, suppose two people are cleaning dishes in a restaurant, one washing dishes and the other drying. If the number of wet washed dishes keeps building up, then the constraint for the process is the person drying. Increasing the rate at which dishes get washed won't increase the number of dishes you are able to put on tables, because those dishes aren't going to be dry. If you want to increase the rate at which you can put dishes on tables you need to increase the rate at which dishes get dried, not the rate at which they get washed.

Applying The Theory to Agile

In more complex processes such as software development, it is sometimes more difficult to see where the constraint is. So look for people who are waiting for work that is coming from someone else. Usually, the "someone else" is a constraint, so a method should be found to improve the efficiency there.

In Agile, the focus on resolving the constraint would be to provide that person with extra training or get other members of the team to assist in the work by "Swarming".

The ToC is important because it tells us where to focus our efforts. For example, if we need to improve throughput in a project, it would be a waste to increase the developer's capacity to code stories from 90 points up to 100 points per sprint if the BAs could only supply 80 points of elaborated stories and the testers could only test 70 points of stories!

The "Limiting Constraint" is defined as the point at which the lowest throughput occurs - in this case the testers, who can only throughput 70 points of stories. Work that focusses to improve any other area will be wasted as our delivery of tested code would still be limited to 70 points per sprint. The first area of concern should be the limiting constraint.

NOTE: Removing the limiting constraint (in this case the testing process, at 70 points) simply moves the constraint to a new bottleneck (in this case it would move to the BAs at 80 points). The process of identifying and removing constraints is thus ongoing.

The Theory of Constraints can be applied to measures other than rate of work throughput. There may, for example be a limiting constraint on work quality.

The ToC may have implications in prioritizing which areas to improve during a Retrospective. If an area identified for improvement does not represent a constraint to the work by any measure (volume, quality, etc) then this area may not be a priority.

Root Cause Analysis For Agile

Overview

Agile typically uses Root Cause Analysis techniques drawn from LEAN.

Root Cause Analysis using the 5 Whys

The 5 Whys is a technique developed by Sakichi Toyoda at Toyota as part of the lean methodologies that were developed there. It is an iterative question-asking technique. The primary goal of the technique is to determine the root cause of a defect or problem.

The technique starts with a statement of the problem, then the application of 5 "whys", with each answer triggering the next "why".

The classic example given to illustrate this technique is:-

The problem: The vehicle will not start.

1.Why will the vehicle not start? - Because the battery is dead. (first why)
2.Why is the battery dead? - Because the alternator is not functioning. (second why)
3.Why is the alternator not functioning? - Because the alternator belt has broken. (third why)
4.Why is the belt broken? - Because the alternator belt is well beyond its useful service life but has not been replaced. (fourth why)
5.Why has the belt passed its useful service life but not been replaced? - Because the vehicle was not maintained according to the recommended service schedule. (fifth why, a root cause)

The use of 5 questions derives from an empirical observation that 5 is typically the number of iterations required to establish a Root Cause of a problem. In practice, the number may be greater or smaller than that.

Root Cause Analysis using Fishbone Diagrams

Note that a Root Cause may not be just one factor. In Complex Adaptive Systems it is common for the Root Cause to be an interaction of factors. The primary technique used to address this level of complexity is the fishbone (or Ishikawa) diagram although a tabular format listing causes can also be used. These tools allow for analysis to be branched in order to provide multiple root causes.

 Th Fishbone Diagram divides the causes into general categories (often Equipment, people, process, materials, environment and management) and iteratively explores each one in an approach similar to the 5 Whys to understand all of the factors contributing to a problem.

Note: The shapes to create these diagrams are available in Visio, in Business->Business Process->Cause and Effect Diagrams.











(Example from http://en.wikipedia.org/wiki/File:Ishikawa_Fishbone_Diagram.svg )

Mess Maps

Lean assumes some degree of linear flow. For very complex environments it may be necessary to go outside of the LEAN methodologies - a Mess Map may be required.

Adaptive Stance: "Experiment, Inspect and Adapt"

Our linear thinking struggles with decision-making in an uncertain situation. Committing to an unknown strategy that will be guided by an undefined series of "experiment, inspect and adapt" cycles is a challenge to most leaders.

This problem is increasingly recognised in military circles. The battlefield is an extreme example of a Complex Adaptive System, with modern communication and networking enabling enemy elements to communicate instantly, sharing information and changing tactics. For a military leader, the consequences of failing to adapt quickly are extreme.

A guideline to achieving an "Adaptive Stance" mindset for military decision making is laid out in "Decision-Making" by Lieutenant Colonel Mick Say and Lieutenant Colonel Ben Pronk:


The Adaptive Stance is built within a framework of a number of key personal qualities:

• Ambiguity tolerance. There are no simple solutions to complex problems, and attempts to remove ambiguity from a situation can be very dangerous. Every effort must therefore be made to resist the urge to over-simplify the complex. Again, one must accept that messiness and sense-making are key.
• Self-reflection through ever-present consideration of the questions: ‘How would I know if I was wrong about this?’ and ‘How much would it matter?’ This characteristic encapsulates an ‘ingrained habit of thoughtful self-reflection about the effectiveness of one’s beliefs, actions and decisions’. It echoes the requirement to treat one’s own ideas dispassionately, helps to combat confirmation bias, and primes the practitioner to be constantly on the lookout for ‘Question Four’ moments.
• Decriminalisation of being wrong, openness to learning and supporting others’ learning. If one accepts that it is virtually impossible to predict the outcomes of an interaction with a CAS, and the process of adaptation entails elements of trial and error, then it becomes completely naïve to expect ‘fail-safe business plans with defined outcomes’. Toleration of failure is an ‘essential aspect of experimental understanding'