Analyst Report: How to unlock the promise of agile in the enterprise

1 Summary

The promises of agile — getting product in front of users faster, embracing requirements changes, choosing and trusting solid contributors — map incredibly well to today’s business environments, where disruptive technology and the ability to quickly capitalize on opportunities either make or break entire verticals. While agile methodologies have had tremendous success in task-oriented teams, many larger businesses have been slower to embrace them as a corporate standard. But agile is not a panacea, and not all projects, business processes, and corporate cultures are natural fits.

The typical enterprise will support multiple methodologies, and making them work together isn’t easy. From budget forecasting to performance benchmarking and accountability, agile presents new disruptions to traditional processes. At the same time, its more granular, responsive approach and the tools that support it can bring new efficiencies to the projects, teams, and organizations implementing them. This report will examine the current state and the future of the multi-methodology enterprise and examine procedural and technological changes that can help enterprises integrate agile methodologies into a larger ecosystem.

Key highlights include:

  • A high-level overview of the palette of project-management methodologies available to enterprises, and how and where each can provide value.
  • Discussion of the cultural, technical, and political barriers to agile adoption in the enterprise, and how focus on culture, people, and product are requirements for an organization planning on using agile.
  • An examination of the software tools available that allow teams to collaborate and correspond in an agile fashion, and an investigation of how important tool choice is for the internal perception and support of agile.
  • A look at how independent, and seemingly non-complementary methodologies can be threaded together in even large, complicated enterprise-level projects.
  • Concrete direction and advice for CIOs, CTOs, and project managers on how to either introduce or expand agile adoption at their organization.

2 Introduction

It’s no secret that the choice of project-management methodology is often the biggest contributor to a project’s success or failure. This is especially true in the enterprise where stricter stakeholder oversight is often at odds with team efficiency. The good news is that today, more than ever, all stakeholders — from project managers to management to executives — have access to a wide set of methodologies from which to choose and — especially in larger efforts at larger companies — many of them are often used in concert.

On one end of that spectrum are traditional project management (TPM) approaches such as the waterfall model, where project features, cost, and schedule are planned out, marched through, and forced back to resolution when they deviate. At the other end is the oft-misunderstood agile family of methodologies, which are often touted as leaner, faster, and more flexible.

The larger a project becomes, the more likely it will employ a variety of methodologies, leveraging each for its individual strengths where appropriate. A large enterprise software delivery often uses standup meetings, kanban boards, pairing, test-driven development, and delivery sprints in individual teams, then wraps team contributions up into a traditional approach that provides continuity and control.

The choice comes down to picking the right tool for the job, but before we choose tools or methodologies, we need to properly understand them.

3 Definitions

The two dominant schools of thought are TPM, embodied in such standards as A Guide to the Project Management Body of Knowledge (PMBOK guide), and agile project management (APM), which, although originally conceived as a solution for software projects, has since found utility in many other disciplines. Both TPM and APM are umbrella terms that encompass a whole family of concrete methodologies.

TPM is often described as a “top down” or push methodology — management defines the costs, budgets, and schedules, and then addresses deviations from any of them as the project marches toward completion. APM is in many ways TPM’s polar opposite, being a “bottom up” or pull methodology — workers self-organize, self-prioritize, and pull what they consider to be the most important tasks from a queue. They can populate it or management can.

Demonizing traditional methods like waterfall as slow and unresponsive and parading agile as the cure-all pill for delivery woes is all the rage these days. But, the practice is rarely that simple, and in an organization as complex as today’s enterprise, each really has its place. The majority of enterprises use a combination of the two. Before we open that can of worms, though, let’s dig a bit more into the specifics of what we mean by “agile” and “traditional.”

Traditional approaches came out of the understanding that certain actions must be completed before other actions. If you’re building a house, you first need a permit. Then you proceed with the foundation before you can build the first floor. The first floor must be completed before the second, and installing drywall before the rough plumbing and electrical are finished makes no sense. Organizations like the Project Management Institute (PMI) have produced and managed standards like the PMBOK guide, which lays out general guidance as well as particular methodologies to manage these projects.

Long before the Agile Manifesto was released in 2001, managers of software projects realized that the traditional methods broke down when building software. Software, maybe more than any other discipline, seems to be a moving target in which the end user thinks they tell developers what they want, then the developers run off and believe they built it. Rare is the case when they all get it right on the first go.

The main thrust behind the agile movement is constant delivery of small, working bits of functionality, ensuring quick feedback and a sort of discovery of requirements as customers and developers continue to tailor and — they hope — improve the product. Agile approaches also allow developers to “fail fast” and quickly prove (or disprove) what will resonate with end customers. Under the umbrella of agile, developers have a variety of approaches including:

Scrum — short, daily stand-up meetings where team members, individually, report on what they did yesterday, what they plan on doing today, and any hurdles in their way.

Sprints — one- to four-week releases with working product delivered at the end of every release.

Cross-functional teams, physically close teams — teams of four to eight people who, in combination, are all the resources required to deliver the product.

Visualization or kanban boards — “card walls” with columns representing the different phases of a workflow (e.g., prototype, development, testing, passed tests, etc.).

As software continues to become a central part of more and more devices and projects, and as the pace of technology accelerates across disciplines, many teams are finding agile approaches to be optimal in many projects outside of software.

Project management consultant Mark Tolbert says, “Agile methodologies are most appropriate for projects where we need to maximize creativity in designing new products and solutions where we expect a lot of volatility in requirements. In this environment, we need high-energy teams that are very empowered, motivated, self-reliant, resourceful, and use a lot of initiative.” The type of deliverable is also a heavy factor in determining optimal methodology, with agile being, in general, a better fit for virtual and software products and traditional being a better fit as the product gets closer to the physical.

Although agile may be the perfect tool for dealing with malleable, ever-changing requirements, it requires a culture open to a bottom-up, pull-based approach. As Barbee Davis, author of Agile Practices for Waterfall Projects, states, “When we talk about agile, we’re talking about a philosophy — a new way to treat customers, work with employees, and gain real business value out of projects.”

The decision to “go agile” or not is typically where most project managers start the dizzying process of determining appropriate roles, structure, and reporting. More often than not, however, that is the wrong first step. Before attempting to overlay a project management methodology on a team, a manager needs first to understand how that team currently works. Too frequently, managers fall into the trap of assuming that project-management methodology can be thought of as independent of everything else, when in reality, it’s often just a reflection of a company’s culture and beliefs.

4 Enterprise’s (slow?) adoption of agile

The culture of an organization is not something written in bulleted value statements pinned up on cubicle walls. It’s embodied in the people the company hires, promotes, and fires; it’s lived out in the dominant management style. For many enterprises, a culture of command and control and top-down hierarchies are realized in the TPM approaches to projects. Bottom-up APM approaches can be seen as threatening and as directly challenging the established culture and hierarchies.

Furthermore, many enterprises are just plain risk-averse. Many have a select handful of products or services that rake in the majority of the profits, and anyone who upsets that apple cart will at best be forever shamed in the company, and at worst get a lifetime industry blacklist. Finding IT teams at these companies defining themselves as “the department that says no” is not surprising.

But in many ways, resisting change is the most risky thing an organization can do. Volatility, competition, and opportunities have all greatly increased while shelf life, delivery cycles, and brand loyalty continue to shrink and shrivel.

“What you know about risk is that you can’t innovate without it. If you’re structuring a business around protecting golden geese, then you’re just building the buggy whips of the future,” says Dean Leffingwell, author of Agile Software Requirements: Lean Requirements Practices for Teams, Programs, and the Enterprise, and creator of the Scaled Agile Framework (SAFe), an approach for scaling agile in the enterprise.

When considering introducing or expanding agile in the enterprise, two of the biggest issues businesses will face are cultural transformation and how to make agile scale.

Historically, many organizations have had success with the “all in” approach to agile, but larger enterprises may find better luck with a slow and steady approach. Davis says, “For agile to really be effective in an enterprise, it needs to be a slow process. It’s about changing the way people think and interact with customers and product owners. You can’t come in, lay down a new set of processes, and change everything overnight.” Hiring, training, and mentoring the right people is a huge part of this transformation.

Recent years have provided several frameworks that can be used to scale agile in the enterprise effectively. Two of the most popular options appear to be disciplined agile delivery (DAD) and SAFe. The two approaches are complementary, with SAFe providing guidance on dealing with requirements, integration, and dependency issues at scale, while DAD focuses on adaptive techniques to deal with enterprise-wide peculiarities and goals, including mixed TPM and APM environments. Where SAFe provides the “what,” DAD provides the “how.”

Although enterprise-level support of agile is proven and has mature frameworks, culture-transformation challenges and manager buy-in are still listed as two of the top three reasons why enterprises are hesitant to adopt.

Figure 1. Agility at scale survey results, November 2009

Screen shot 2013-12-18 at 5.01.20 PM

 

Ambler: Agile State of the Art Survey, November 2011

5 Multi-methodology in practice

Because of the extraordinary benefits to be gained, roughly half of enterprises have reported doing “some” form of agile, and many are even reporting going as far as investigating some sort of continuous delivery work (where software is pushed live automatically after passing all functional, regression, and other tests). Since these deliverables often involve a complex mix of software and physical product, it’s very common to see multiple methodologies used in tandem by separate teams. These mixed-methodology organizations are typically reporting intra-methodology communication (TPM to APM and vice-versa), testing, and common language/understanding as some of their biggest challenges.

For organizations dipping their toes in the agile waters, what seems to work well is allowing individual teams to function as agile while reporting metrics in terms that the more traditional-based managers are used to seeing — in effect, translating progress into a language that managers understand. Even within agile-based projects, traditional metrics are what stakeholders are most interested in — they need to know if the project is on time, within budget, and on track to deliver the scope. If the manager of a team can deliver those metrics, the stakeholders will not care if the team is using APM or TPM.

The challenge, however, comes from the fundamental disparity between what TPM and APM are trying to achieve. “Management has to understand that you can have everything eventually,” says Davis. “With waterfall, you typically freeze the scope and have to adjust cost and schedule, while with agile, you must freeze the schedule and cost, while the feature set slides around.”

From this perspective, agile may seem risky, but teams prioritize the most important work first, ensuring that when a sprint ends, the most important functionality has been completed first. What management needs to understand is that the schedule will always be met, even if the feature set cannot be promised. Just as agile teams need to understand and speak traditional managers’ language, the managers need to understand and respect the independence and value of their agile teams.

Management, stakeholders, and product owners in these teams also need to be much more proactive than perhaps they are used to. If managers want status, they need to get used to coming down, looking at the visualization wall, and getting a five-minute face-to-face on where things stand.

Patrick Millar, CTO of agile-heavy Chatham Financial — the largest independent hedge advisor in the world — suggests that good architecture is the secret to APM and TPM teams coexisting nicely. Loosely coupling the code produced by APM and TPM teams allows them to begin to function independently, each pursuing the objectives most important to them. “If you can convince an existing TPM team to refactor their code just enough so that a clean, stable interface emerges,” Millar states, “you open up the possibility for APM teams to coexist with TPM teams in a mixed-methodology environment.”

6 A culture of opportunity

Once an organization has gone through this sometimes lengthy step of breaking down a larger effort into smaller chunks, its next step is creating a culture of opportunity in which individuals are given more freedom and independence to accomplish the larger goal, creating intra-team, multi-methodology interfaces that scale, and then allowing each individual team to determine the best management style for itself. Although decentralization of control may seem like too dramatic a culture change, the benefits realized by pioneering teams soon cause change to spread throughout the enterprise.

In a culture of opportunity and smaller iterations, risk can actually be decreased. Cross-pollination between agile and traditional teams can often result in even the traditional teams rethinking their approaches, perhaps even scaling back their schedules and deliverables into smaller workflows, allowing them to either succeed or fail more quickly. “Agile in a way really works well in this environment in that it’s all about small increments rather than going for the Hail Marys,” says Alex Robbio, president of the nearshore software development firm Belatrix Software.

Although many may be hesitant to view Amazon as a mixed-methodology enterprise, that is exactly what it is. To drive its software-based e-tailer website, Amazon requires data centers, shipping warehouses, and customer service centers. It’s a good bet that construction and maintenance of those buildings, along with all of their associated needs are managed with traditional methods. Amazon is also now getting into the physical product space, not only with their flagship Kindle e-reader, but also with the AmazonBasics product line of electronics and accessories. Building of these physical products is also likely being managed via traditional methods.

On the software side, Amazon is second to none in showing how far agile can go. Amazon has taken agile all the way to the continuous integration/continuous delivery (CI/CD) phase that allowed it in 2012 to deploy new code live every 12 seconds, on average, to as many as 30,000 servers at once. In this CI/CD environment, Amazon can do extremely fast A/B tests, allowing it to refine and respond constantly to changes in customer behaviors. But even the core Amazon web services product would not have come about without some top-down dictation of standards — especially the now infamous leaked “everything as an API” mandate.

Amazon has been able to do amazing things thanks to its strong, visionary management, its empowered, resourceful teams, and its culture of opportunity that lets each team choose the tools most appropriate for itself. That’s all fine and dandy, but what if your organization doesn’t have a culture that allows for more decentralized control? What if those in power view agile as a threat?

Although it’s now four years old, Netflix’s “Culture Manifesto” provides guidance for a company looking to begin building a culture that is more friendly to implementing agile methodologies. Some of the highlights from that document include the gems:

  • Freedom and responsibility
  • Context, not control
  • Highly aligned, loosely coupled
  • Don’t tolerate “brilliant jerks”

Amazon and Netflix both still think and act like small companies in many ways. Granted, they were born into an environment where agile was the norm, and that’s all they’ve ever known, but seeing how scrappier, upstart divisions inside of even old-guard enterprises could adopt many of their cultural values and reap the rewards is not difficult.

If you believe that software is eating the world, then APM will continue to steamroll. Look at even manufacturing models that allow mockups and multiple demos of physical product — thanks to computer-aided design and 3D printing. Regardless of the vertical, if a company is fighting agile tooth and nail, there is a very good chance it will not be able to adapt to the future as its competitors will. As Millar says, “In a rapidly evolving ecosystem, there’s only so long you can protect the old goose that lays the golden eggs before a fox evolves that is smarter than you.”

7 Focusing on people

Companies like Netflix and Amazon know very well that at the end of the day, every business is a people business. A key part of Netflix’s culture is its approach to hiring and retaining the best people possible in every position. If your enterprise doesn’t recruit and retain good people, and keep them challenged and engaged, someone else will.

For fully staffed enterprises looking to transition to agile, a great deal of time must first be dedicated to taking inventory and discovering who the natural APM leaders are. Once leaders are identified, agile-friendly team members must be ferreted out, and then the whole team must be trained in the methodologies that will be used. Training is foundational to success. “Sending people to (at least) scrum training, bringing in advisement consultants for the first few projects and then having a plan for moving it all out systematically — that’s where people really find the business value in agile,” Davis says.

The one unspoken fear that everyone has when moving a team to agile is also often the most unrealistic in practice: “What happens if members of our team just cannot change and adapt? Will we have to let people go?” Although some people will not be able to switch gears, in a mixed-methodology environment, at least finding them a team that more closely matches their style should be easy. Those that do convert often times convert in spades. “We have many refugees from TPM shops who turn out to be fantastic agilistas,” says Millar. “The personal growth individuals experience is often extraordinary, and the institutional knowledge that long-time TPM team members bring can become a massive asset when coupled with a renewed enthusiasm for the job at hand as they engage in APM.”

Another insider secret to successful mixed-methodology enterprises is ensuring that middle management is trained and fully understands the value and utility of agile methodologies. Even if the senior executive team is hesitant to initially embrace or adopt agile concepts, middle management alone can make a huge impact. Leffingwell says, “Middle management really holds the keys to the success of agile adoption — they create all of the procedures and policy. If middle is not on board, transformation will be shunned.”

The leaders always set the tone, and if the project-level leaders are exceptional, that excellence quickly spreads up and down.

8 The right tools

In some ways, tools can drive culture change, and choosing the proper ones can begin to grease the wheels for more efficient, transparent, and agile teamwork. If project information is scattered or outdated, determining what resources are available or ensuring you have the right people on the right projects is impossible. Unfortunately for many enterprises, this is precisely the environment they find themselves in — some communications are in emails, some on sticky notes left for drive-by requests, and some in the “formal” project management tool that everyone knows is out of date.

Communication and collaboration tools may be the third most critical element of an effective multi-methodology enterprise, directly behind a mixed-methodology-friendly culture, and cross-functional, agile-friendly team members. Just don’t fall into the trap of assuming that the right tool will make everything work properly — tool selection and implementation must happen either after or at least in parallel with instilling the right culture and identifying, training, and mentoring the proper team members.

Many tools are out there in common use for TPM-only or APM-only projects. On the agile side, nearly everyone uses a solution from one of four vendors — Rally, VersionOne, JIRA Agile (formerly GreenHopper), or IBM’s Rational Team Concert. All of these vendors offer exceptional, mature products with legions of loyal followers. On the TPM side, MS Project (famous for its Gantt charts) has a near-monopoly.

Surprisingly few tools exist for mixed-methodology enterprises. MS Team Foundation Server (TFS) and AtTask are two of the more popular. Although it has hooks to integrate with non-Microsoft products like Git and Eclipse, TFS probably works best for Microsoft-heavy houses and offers extensive integration to MS products like Visual Studio, MS Project, and SharePoint. If you’re an MS house already, TFS may be your best bet.

For everyone else, AtTask offers a multi-methodology tool that allows project managers to identify, plan, execute, and measure the outputs of multiple projects, people, and schedules, quickly and easily — all in real time, and all in one centralized place. Instead of locking companies into individual methodologies and ways of doing business, it lets individual teams, projects, and portfolios choose and use the methodologies that work for them — including both waterfall and agile frameworks. AtTask lets these two coexist side by side in one tool and enables granular management for each.

9 What to use where

At a high level, there are some very clear guidelines of which methodologies work well in which projects. Although agile will definitely see expanded adoption as “software eats the world,” certain project types will likely always be better served via traditional or mixed methodologies.

  Exemplefied in

Traditional

Agile

Hybrid

Physical product Construction, manufacturing

x

Software or services Apple, Amazon, Netflix, Google

x

Mixed product (software and hardware) Cell Phones

x

Regulation / Compliance Government / DOD, Financial, Health

x

x

x

Repeatable or known process Services

x

RISK: stagnation, missed market opportunities

x

Bottom-up culture Netflix, Google

x

x

Top-down culture Yahoo, GE

x

RISK: stagnation, missed market opportunities

x

Few competitors / monopolies eBay, Financial, Health, Government

x

RISK: stagnation, missed market opportunities

Many competitors / disruption Consumer electronics

x

Mature or established space  

x

RISK: stagnation, missed market opportunities

x

Nascent or fast-moving space  

x

x

 

10 Key takeaways

  • Most enterprises currently use a mixture of both traditional and agile project methodologies, but to succeed, they need to train their team members to speak each other’s language, and possibly be more proactive than they are used to.
  • Agile will likely continue seeing greater and greater adoption, but projects with fixed costs, known and repeatable scope, or tight links to the physical world will always likely be better served via traditional methods like waterfall.
  • The ability to iterate quickly is becoming more important even in disciplines outside of software. Agile mindsets can help many enterprises not only protect existing revenue streams but create new ones.
  • People are key — choose carefully, train religiously, and ensure that at least middle management is on board.
  • A culture of trust, transparency, and freedom is maybe the number one prerequisite for successful agile or mixed-methodology adoption. Full buy-in throughout the company is probably the second.
  • Proper tools — especially those that make communication more efficient and effective — can help drive agile-like efficiencies, mindsets, and culture, but they require the right culture and the right teams to be effective.

11 About Rich Morrow

Rich Morrow is a 20-year open-source technology veteran who enjoys coding and teaching as much as writing and speaking. His current passions are cloud technologies (mainly AWS and OpenStack) and big data (Hadoop and NoSQL), and he spends about half of his work life travelling around the country training the Fortune 500 on their use and utility. He leads the Denver-Boulder Cloud Computing group, as well as quicloud, a Boulder, CO, based cloud and big data consultancy.

12 About Gigaom Research

Gigaom Research gives you insider access to expert industry insights on emerging markets. Focused on delivering highly relevant and timely research to the people who need it most, our analysis, reports, and original research come from the most respected voices in the industry. Whether you’re beginning to learn about a new market or are an industry insider, Gigaom Research addresses the need for relevant, illuminating insights into the industry’s most dynamic markets.

Visit us at: research.gigaom.com.

13 About AtTask

AtTask is a cloud-based Enterprise Work Management solution that helps marketing, IT, and other enterprise teams conquer the chaos of excessive email, redundant status meetings, and disconnected tools. Unlike other agile tools, AtTask Enterprise Work Cloud enables teams to use the methodology that fits the job and do request management, work prioritization, agile, ad hoc work, traditional methods, and collaboration. Teams use one tool, throughout all stages of the work lifecycle, to improve department productivity and executive visibility. AtTask is trusted by thousands of global enterprises, like Adobe, Cisco, HBO, Kellogg’s, House of Blues, REI, Trek, Schneider Electric, Tommy Hilfiger, Disney, and ATB Financial.

14 Copyright

© Knowingly, Inc. 2013. "How to unlock the promise of agile in the enterprise" is a trademark of Knowingly, Inc. For permission to reproduce this report, please contact sales@gigaom.com.

Tags