Challenges in Agile Transformation

Agile practices have become increasingly popular over the last decade. At the beginning, only small enterprises invested money in transformation to agile software development. Over the last few years, agile software development is no more limited to startups and smaller software development teams. In fact, 10th annual agile survey carried out by VersionOne in 2016, reports that 24 % of respondents work for organizations with more than 20,000 employees. 

As large enterprises continue to embrace agile globally, many organizations face obstacles in agile implementation. Agile principles and core values are a certain way to ease out the usual pain of software development. Agile teams have reported increased team productivity, enhanced project visibility, increased team morale, enhanced software quality, better team management and faster time to market. In 10th agile survey, 95% of the respondents said their organizations practice agile and only 1% of the 3,880 respondents said that their agile implementation was unsuccessful.  

The key reasons of failure in agile implementation usually hinge around organizational culture, including the ability to change, general resistance to change, and management support. The majority of developers around the globe consider company culture as the primary reason for failed agile projects. Once these barriers are overcome, the next limiting factor has always been availability of personnel with the necessary agile experience. We take a deep look at some of these obstacles and some guidelines to overcome them.

10th Annual State of Agile Report by VersionOne: Agile Success and Metrics

Inadequate Experience and Knowledge of Agile

Agile principles and core values must be well understood in essence before they are put in practice. One big reason for failure of agile projects in organizations is lack of experience or knowledge by the team and the management. This is generally the case in software development teams without professional software engineers. The team may be very well trained to do a certain job in a certain way but not experienced enough to adopt agile practices. Enforcing agile in such teams will result in project failure. Team members tend to follow the same legacy waterfall approach under the shed of specific agile terms like "Sprints". This is an indication of lack of experience and knowledge by the team.

This barrier can be crossed by internal or external training of the team and hiring an experienced agile consultant. 10th annual agile survey indicates that the pool of talent and agile experience continues to grow annually. However, 41 percent respondents consider lack of experience with agile methods and 21 percent consider insufficient training as a contributory factor in failure of agile projects. 

Company Philosophy and Culture at Odds

Company philosophy and culture not in favor of agile is a big contributory factor in failure of agile projects. In the 10th annual agile survey, 46 percent of respondents consider company culture being at odds with core agile values as an important factor in agile projects failure. Agile core values and principles largely revolve around individuals and active participation. Agile faces huge resistance in organizations which are not structured to work in a democratic way. One such example is a military organization with complete autocratic top-down style of working. If the company culture is hostile to agile values, it will be isolated to small pockets of agile teams.

Broader and organization wide agile adoption is only possible by executive sponsorship. Management should understand that agile impacts organizational goals and assist in smooth transformation. Team members should leave the top down style of working behind for success with agile transformation.

Lack of Support by Management

Enterprise management is responsible to drive the successful implementation of agile attributing to project success. 38 percent respondents in 10th annual agile survey consider lack of management support for agile transition as a contributory factor in failure of agile projects. The only way to conquer this barrier requires serious involvement by managers relinquishing legacy styles of working. There can be a number of reasons for lack of management support such as:

• The real essence of agile lies in active interactions between team members. It is a democratic way of working. Managers may like to follow the traditional hierarchical autocratic style of working.
• Agile teams are self organizing. This agile principle is realized by self assignment of tasks by developers. Managers may wish to keep tasking of developers in their own control.
• Agile principles allow transparent metrics to measure and indicate actual value delivered in the project. Management may be interested in vanity metrics to exaggerate the team's achievements.
• Management may feel some external pressures to follow legacy rigid models. The organizational structure may not be well suited with agile practices. It happens in larger teams with pyramid like structures. Some managers in the pyramid may not be actively involved in the project. 

Forceful Implementation of Agile by Managers

One beauty of agile is it's inherent flexibility. It entirely revolves around motivated individuals and their interactions. Forceful implementation of agile is against the spirit of agile values. This usually happens in teams where developers depict negative behavior for personal goals. 30 percent respondents in 10th annual agile survey consider unwillingness of team to follow agile practices as a factor in failure of agile projects. 

It is pertinent to mention that agile is a complete change in working culture. Some teams are simply not ready to embrace such a change. Team members may be convinced that agile is simply not the best choice and they may be true. Teams will cease to follow agile practices once management pressure relieved. If the managers are strongly convinced on agile, they can get the team on board by hiring agile consultants and internal support team. Otherwise, team must stick to already enforced legacy process model.

Agile is not the Best Fit

Just like an appropriate process model can be effective, an in-appropriate process model can bring negative consequences. When requirements are very well understood before the start of the project, waterfall may still be the best approach. Generally for most of real world projects, this is not the case. It is nearly true that requirements will change and iterative approach is the best choice. 

Popular agile practices like early testing, active collaboration with customers, self organizing teams, focus on working software, simplicity, retrospectives and responding to change have emerged to be the best practices over the years. A team may decide to implement only a few of these practices at the beginning. Some experts believe that waterfall model was followed at a time when compiling and building software seemed to be a tedious task. Developers ensured that they spend enough time to freeze requirements and design just to avoid frequent builds of the software. Today, building software is on the click of a button. Today, software development naturally occurs in an iterative and incremental way. This makes agile to be very suitable for most software development projects.

External Pressures to Follow Legacy Models

Some organizations have no control in deciding their ultimate goals and objectives. Their goals are set by outside agencies and enterprises. In such cases, building a team which collaborates effectively poses a serious challenge. Team members may also be skilled on working with rigid process models like waterfall. 36 percent respondents in 10th annual agile survey consider External pressure to follow traditional waterfall processes as an important factor in agile projects failure. 

External pressures are commonly observed in larger enterprises with complex pyramid structures. Some groups in the enterprise follow agile while others do not. Problems arise when these groups need to interact. Sometimes, agile implementation may require modifying the team structure in a long term. 

Agile requires a complete cultural change in a software development team. Sometimes, teams may find it difficult to adapt due to external pressures. This results in a combo where some phases of software development process follows a traditional waterfall model. This combo kills the real essence of agile and the resulting process becomes adhoc. This can be overcome by executive sponsorship, slow transformation and hiring an agile consultant. Team can adopt some practices at an early stage and gradually move ahead with complete adoption.

Lack of Collaboration in Teams

A software development project may be accomplished by multiple constituent sub teams. Collaboration between multiple teams becomes a challenging task. 30 percent respondents in 10th annual agile survey consider broader organizational or communications problem and 25 percent consider ineffective collaboration as a factor in agile projects failure. Sub teams may not share common goals and objectives. Agile practices work very well with well integrated smaller teams. For larger teams, collaboration between multiple distributed sub-teams is paramount for success of agile projects. According to the 10th annual agile survey, agile is rapidly being implemented in larger enterprises. In fact, 10th annual agile survey reports that 24 % of respondents work for organizations with more than 20,000 employees.

The enterprise management should make efforts to integrate the teams and bring them on board for agile implementation. One of the 12 principles in agile manifesto states: "Business people and developers must work together daily throughout the project". This requires active collaboration to get real benefits of agile methodologies. However, this becomes thorny for cross-functional distributed teams where active collaboration may not be easy. Managers must make serious efforts to ensure collaboration between the teams for successful implementation of agile. One good solution to be opted by managers is to have a common tool across all constituent teams.

Inconsistent Agile Practices and Process

This is by far the most common reason for agile projects failure. It is strongly linked to all other barriers. 38 percent respondents in 10th annual agile survey consider inconsistent agile practices and process as a contributory factor in agile projects failure. An organization may follow inconsistent agile practices because of the following reasons:

• Developers and managers do not understand agile process model. Managers focus on installation of a certain tool instead of enforcing the process itself. They think installing a certain tool will bring a magic and everything will be agile. Agile principles and corresponding practices are the best patterns of software development evolved over time. These patterns can not be incorporated in teams with the use of tools. Tools are only used to automate certain aspects of the specific agile methodology like scrum and XP.
• Agile is considered a kind of fashion symbol and management is only interested to go with the current trends.
• Agile practices may be modified due to some external pressures or lack of experience. An organization may skip reviews and retrospectives due to time availability. These modifications will damage the effectiveness of the process.
• Roles may be skipped or totally miss-understood. For example, Scrum product owner role may be confused with a project manager role. 
• An organization may be captured in a variety of anti-patterns conflicting with agile core values and principles. Some of the common anti-patterns include analysis paralysis, group think, silos, heroics, death march, micromanagement, seagull management, loose cannon and intellectual violence. These anti-patterns can be devastating for agile projects.