Has Agile Lost Its Way?

What is Agile?

I have been practicing Agile for over a decade. I used to blog about it (that’s why this blog is here), but the last time I posted an Agile article was nearly ten years ago. In that time-span, I have worked with quite a number of teams in companies large and small, helping them to implement Agile. I have done training, coaching, setting up their toolsets (mostly JIRA and Confluence – more frequently now, Azure Boards) and helping them to optimize their processes. I felt it was time to start blogging again so that I might share a bit about what I have learn in the interim.

It might be useful for you, the reader, to know that my motivation was never to be a trainer, project manager or an Agile Coach. I am still, primarily an architect and developer. My motivation sprang solely out of my desire to deliver better software sooner and my frustration with the ineffective or nonexistent processes in most organizations that I worked with. To do that, I had to learn about process. I have tried a number of methodologies in the past three decades. About a dozen years ago, I ended up at the doors of Agile – I walked in and haven’t left.

Before we get into answering the question of whether or not Agile has lost its way, let’s remind ourselves as to what Agile is.

The whole notion of Agile Development sprang out of a meeting on February 11-13, 2001 at The Lodge at Snowbird ski resort in the Wasatch mountains of Utah, seventeen people met to talk, ski, relax, and try to find common ground—and of course, to eat. What emerged was the Agile ‘Software Development’ Manifesto. Representatives from Extreme Programming, SCRUM, DSDM, Adaptive Software Development, Crystal, Feature-Driven Development, Pragmatic Programming, and others sympathetic to the need for an alternative to documentation driven, heavyweight software development processes convened. https://agilemanifesto.org/history.html

They produced two concise components that define what Agile Software Development is:

1. The Agile Manifesto
2. The Twelve Principles of Agile

1. The Agile Manifesto

Agile Manifesto: https://agilemanifesto.org/

1. Individuals and interactions – over – processes and tools
2. Working software – over – comprehensive documentation
3. Customer collaboration – over – contract negotiation
4. Responding to change – over – following a plan

2. The Twelve Principles of Agile

Twelve Principles: https://agilemanifesto.org/principles.html

1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
2. Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
4. Business people and developers must work together daily throughout the project.
5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
7. Working software is the primary measure of progress.
8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
9. Continuous attention to technical excellence and good design enhances agility.
10. Simplicity–the art of maximizing the amount of work not done–is essential.
11. The best architectures, requirements, and designs emerge from self-organizing teams.
12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

How Are We Doing?

Where Are We Headed

In my view, we are not doing well and we are not headed in a good direction. The balance of this article is a mix of where I think we have gone wrong balanced against some things that I think can easily be done to get us back on track.

I can’t claim to have a large sample of data pulled from in-depth studies to support my position. I can speak only to my own experience and what I see and deal with the organizations that I work with.

I have discovered that people’s interpretation of what Agile is and should be, is drifting farther and farther away from the original definition of Agile. It is instead, focusing more on technology, tools, increasingly rigid processes and metrics. In effect, what I see is that what many people now call Agile is actually mini-waterfall.

When I adopted Agile, I did not buy into a set of tools or a particular process. In the Agile Manifesto, there is no mention of tools, specific rituals or any particular processes – just a general framework within which, we are free to define whatever process or processes we wish. Increasingly, however, I am seeing burdensome processes, meaningless metrics and people even equating SCRUM to Agile. In other words, they are of the opinion that SCRUM is Agile and Agile is SCRUM. This simply is not true. SCRUM is a form of project management that adopts a number of the Agile Principles, but SCRUM does not equal Agile. In my view, some of what exists in SCRUM is not particularly Agile; in fact I think that parts of SCRUM are actually anti-Agile – more about that in a moment.

The Four Tenets

When I adopted Agile, what I adopted was the Agile Manifesto and its 12 Principles. I still believe in them, but what I see in the work that I do with many companies is that they have reversed the order of the three of the four basic tenets. Specifically, this is what I see:

1. Processes and tools – over – Individuals and interactions
2. Working software – over – comprehensive documentation
3. Contract negotiation – over – Customer collaboration
4. Following a plan – over – Responding to change

The second tenet hasn’t really changed (hence me crossing it out) because the issue of too much documentation was rarely a thing before or outside of Agile, except in large corporations. Some of the companies I have worked with have actually used the second tenet as an excuse for doing absolutely no documentation at all – essentially what they did before their supposed adoption of Agile. Now they at least had a rationale for it.

Selling an Idea

Agile is an idea or a set of ideas that describe a framework for a team culture wherein a team defines how it works and how it optimizes the work that it does. The problem with an idea or a set of ideas that define principles of behaviour (Agile) is that in isolation, the ideas bring no benefit to anyone. Further, change is difficult – sometimes it is very difficult. Because of that, there is an inherent resistance and cost to change. To be of use, the ideas need to spread and be adopted. This requires people, time, effort and money to promote the ideas. This is where things get difficult. Justifying the time, cost risk and disruption of the change to a different process (even if the existing one doesn’t work very well), requires a justification. That justification is almost always in the form of money saved or money earned and that usually requires documentation, metrics and process and this is where tools and training come in – good things when they are actually serving the right purpose.

The Answer – Process, Tools and Metrics!

Actually, it’s not. It can be helpful, but its not the answer. We are in a post-industrial economy where manufacturing has given way to a service economy and an economy of information. The problem is that corporations think about people and the work they do as if we were still in a manufacturing economy where we attempt to measure every step of a process on an assembly line in order to shave pennies off of the cost of a widget. In our current information economy, we now have powerful software tools to monitor and generate mountains of data about everything, including the software that we create. But software development is not a factory process. It is an inherently human and creative process and every time a developer sits in front of a keyboard and a monitor to begin work, that person is about to do something that they have never done before, solve a problem that they have never solved before, create something that they have never created before and in the process, they will learn something and invent something. If you have never developed software before, you will not understand this. If you have gone deep into the work of software development, you will know what I mean because you will have suffered through that cycle thousands of times as you see that next Eureka! Sometimes understanding and invention come easily; more often it’s a struggle. Most importantly, it is not a repetitive, widget stamping process, so metrics that attempt to quantify software development in a factory fashion are most often pointless at best or manipulative and dehumanizing at their worst.

The metrics and heavy process often, however, provide the rationale and (specious) evidence for their adoption, which is how the process begins and it is where the vendors want to sell you their tools, training, coaching and whatever.

Rigid Sprint Plan Commitments

I am all for sprint planning and story grooming. This is an important way for team members to communicate with each other, to share, learn and become better at what they do. However, if the product of your sprint planning session is a set rigid, inflexible story point commitments for a sprint, that’s just dumb. Priorities change, your client’s requirements change, your team makeup changes, team members get sick and miss work, critical issues suddenly occur and other projects and other things around you change. To not change and adapt to these factors and constantly re-prioritize and re-evaluate is not being Agile.

Forget the rigid commitment to a predefined set of stories. All this really does is tell your developers that you don’t trust them to do what they say they are going to do. Instead, create a culture where everyone feels appreciated for doing a great job. It’s surprising how consistently productive happy and appreciated people can be. Also, programmers aren’t stupid. If you force them into a box of hard timeline commitments, they will pad their estimates game your numbers and make them even more meaningless. They will do this to ensure that they will never miss a sprint commitment and they will be unhappy about having to create defensive mechanisms when they would rather be writing great code. The bottom line here is that you will get less throughput – not more and your numbers will be consistent and consistently pointless.

Just drop strict sprint point commitments. You will have happier, more productive developers and your team will be more responsive and build more software.

Velocity

Here is something that is actually useful, but not in terms of commitments that the team must rigidly adhere to. Companies, their owners, shareholders, customers and anyone who has a stake in the company deserves some level of predictability for their investments and trust. But estimating the amount of time it takes to do develop software is very difficult to do. That being said, it is important and possible, but only for short timelines. We do this in terms of story estimation. What you really want from story estimation is the following:

1. An estimate of how long it will take to complete a body of work (a story)
2. How much work (story points) can a team sustain on a continuing basis – the sprint velocity
3. Is there anything that is affecting the team’s ability to maintain that velocity during a given sprint.

This is pretty simple to do.

1. Simply estimate each story in the form of story points
2. Count the number of story points closed at the end of the sprint
3. Keep track of a rolling average for the sprints that have closed
4. This is your velocity
5. Make the velocity public
6. In your retrospectives, compare velocity between sprints and seek to understand why it has change
7. Work with the team to find ways to improve the velocity

Include bugs in your grooming and in your velocity – it’s work that needs to be done, so include it.

If you have to introduce something into the middle of the sprint because all of a sudden it is urgent, go ahead and do it, even it if it is not estimated. When you get to your next grooming session, have the developer share with the team what was done, but do not tell the team how long it took. Have the team estimate the story as they would estimate any other story and then compare the team’s estimates to the actual. Include this in your velocity calculation. There is an awful lot that can be learned from doing this and comparing their estimates to actuals will help them to get better at their estimation. The important thing is not that they organized the sprint and ended up completing exactly that many points. What you really care about is whether or not your team is working effectively and if you can depend on story point estimations for your forecasts.

Do this well and you will have predictability for your team’s capacity and you will be able to do meaningful forecasts based on what has been done as compared to your backlog. You will also have a happier team.

Burn Down Charts

I could write an entire post (perhaps several) in this one topic, but I will be as brief as possible.

Does anyone actually think that a Burn Down Chart contributes one iota to creating Better Software Sooner? In my view, it is one of the most anti-agile things that we can do and it provides absolutely no value to anyone other than perhaps the executive suite so that they can have a nice, warm fuzzy feeling that we have a pretty chart that has a nice slope to it and it looks like the one from last sprint. Committing to a body of work for the duration of a sprint and not deviating from that plan; not introducing work into the sprint, even if it is more important (perhaps even urgent) than what was agreed to in the plan because we don’t want to break the burn down chart is nonsense and not at all consistent with the tenets of the Agile Manifesto. If developers know that they will be held to this, they will pad their estimates to make sure that they will always deliver on the burn down chart. It is wasted time and effort and it provides no value. I have discussed this with executives and CIOs and the rationale that I hear for this is that it demonstrates that the team has planned well – so what. It does not mean that they are executing well, or that they are being responsive and/or even Agile. It just means that they blindly followed a plan and likely padded their estimates to make sure that they comfortably tracked to the burndown chart. I have many times witnessed developers holding back their commits so that they do not get ahead of the chart and have empty time and end up participating in a Swarming (see the next section)

When the burn down chart becomes your gold standard of whether or not your team is functioning well, you have definitely lost your way, or more likely, you never found it in the first place.

Get rid of the burn down chart now.

Swarming

This is a bizarre one and it is directly tied to a commitment to completing a set of stories. If you have never encountered it, what it amounts to is a situation where you are approaching the end of the sprint and one or two developers have finished their work for the sprint, but others are in jeopardy of not completing theirs. Testing is almost always the blocker for this. Rather than developers picking up another high-priority story that is not in the sprint, the developers who have completed their stories swarm the other stories and do things that that may be outside of their area of expertise, like UI testing when they are back end developers – all for the sake of the hard commitment and the burn down chart. In my view, this another example of process before people. It has nothing to do with being agile or creating better software sooner.

Conclusion

I promised myself that I was going to make this a 1,000-word essay and it is now north of 2,800 words and I am just scratching the surface of this topic. I think, however, that the main theme is pretty obvious as is what to do about it. It comes down to the following:

Process, rules, metrics, charts and reports are overwhelming the Agile process, making it no longer Agile. For the things that you do with your agile teams, consistently refer to the Agile Manifesto and ask yourself “Does this ritual, this statistic, this report or anything that I or the team are doing contribute in a meaningful way to delivering better software sooner?” If the answer is no, I don’t know, or even maybe, abandon it and focus your attention on the things that you know bring real value.

Where have I landed in my process? I think it’s pretty obvious that I am not a practitioner of Scrum, although there are many great things about Scrum. Where I have landed is a sort of Scrumban methodology. I may blog about that in the future.

I hope that you found this useful.

As always, I look forward to your comments.

Michael