Patterns for Adopting Scrum - Start Small or Go All In

Conventional, long-standing advice regarding transitioning to Scrum or any agile process has been to start with a pilot project, learn from it, and then spread agile throughout the organization. This approach is the frequently used start-small pattern in which an organization selects typically one to three teams (of five to nine people each), gets them successful, and then expands Scrum from there. As Scrum spreads through the organization, new teams benefit from the lessons learned by the teams that have gone before. There are many variations of start small, depending on how many people the organization wants to transition and how quickly they want to do it. Start small can also be applied differently based on how riskaverse or uncertain about the transition the organization is. For example, in some cases the first team or teams will finish their projects before a second set of teams even begins. Other organizations will take an overlapping approach, where the second set of teams starts only one or two sprints after the first.

The start-small pattern, while popular, is not for everyone. Salesforce.com, for example, followed the opposite pattern (Fry and Greene 2006). I remember answering my phone on October 3, 2006, and hearing Chris and Steve from

Salesforce.com tell me that they had just converted 35 teams to Scrum overnight. They asked if Fd like to help. My initial thought was that they needed a psychiatrist more than a Scrum consultant. Not one to shrink from a challenge, though, I agreed to help, packed a copy of Freud alongside my laptop, and set off for their office in San Francisco. Part of what I saw there wasn't entirely unexpected� teams and individuals in an uproar over such a sudden, far-reaching change�but I also saw other things that helped this large-scale, rapid adoption succeed. Salesforce.com was pursuing the all-in pattern, which draws its name from a poker player who bets all of his chips on one hand. Salesforce.com has a hard driving, aggressive, achievement-driven culture that would not have been a good fit for a cautious start-small approach. When key executives were presented with a proposal to adopt Scrum, they were convinced. They felt that if Scrum was worth doing for one team, it was worth doing for all teams, so they chose to go all in.

Surprisingly, the all-in and start-small patterns can be combined. An increasingly common approach is a one- to three-team pilot project followed immediately by going all in. The pilot in this case serves the typical purpose of allowing the organization to learn about Scrum and how it will function there. However, the pilot in this scenario also serves the more important purpose of increasing organizational awareness about Scrum. If you're going to transition 200 or more people all at once, it is extremely helpful to be able to point to one team who has already done it and say, "We're all going to do what they did."


Reasons to Prefer Starting Small
The start-small approach offers several advantages.

� Starting small is less expensive. An all-in transition will almost certainly cost more than starting small. Because of the greater number of people learning a new way of working all at the same time, all-in transitions generally rely more heavily on outside coaches, ScrumMasters, and trainers. The slower pace of a start-small adoption allows the organization to build internal expertise and then use that to help the teams that start later. Starting small also saves money because early mistakes affect only a subset of the organization. Tom Gilb, who was perhaps the original agilist, has written, "If you don't know what you're doing, don't do it on a large scale" (1988, 11).

� Early success is almost guaranteed. By carefully selecting the initial project and team members, you can almost guarantee the success of your first Scrum project. You may consider this cheating; I don't. When starting small, a goal of the first few projects is to generate the knowledge that will enable the successful rollout of Scrum. There may be value in starting with a project and team that make success easy and then learning from its experiences. Additionally, an early success can be vital to gaining buy-in from skeptics or fence-sitters.

� Starting small avoids the big risk of going all in. An all-at-once transition can be very risky. Small mistakes will be magnified across the entire transition effort. Perhaps the most significant risk to an all-in approach is that you will be unlikely to get a second chance. If you start to transition the entire organization, make a mistake that increases resistance, and then revert to your pre-Scrum process while figuring out how to overcome the newly discovered issues, it is unlikely that team members will give you a second chance to start the transition. Resistance by that point will likely be so entrenched that the transition effort will have failed. By contrast, if you start small and find a fatal flaw in how you've started, you can keep the next round the same size as the current one, rather than expanding, effectively restarting the transition process.

� Starting small is less stressful. Twenty-first century organizations and their employees are under constant stress. An announcement that the whole development organization is adopting Scrum, which affects so many aspects of everyday work, could be the proverbial straw that breaks the camel's back. The stress of transitioning is reduced by starting small because early adopters become coaches and ambassadors. They encourage other groups to make the transition with stories of their successes and honest discussions of the challenges they faced and overcame.

� Starting small can be done without reorganizing. Most organizations that fully adopt Scrum will eventually undergo some degree of reorganizing. This can create further stress and can increase resistance from some individuals. By starting small, the need to reorganize can be put off longer, ideally until valuable experience with Scrum has been gained.


Reasons to Prefer Going All In
Just as there are reasons to prefer starting small, there are reasons to prefer an all-in transition:

� Going all in can reduce resistance. In anything less than an all-at-once transition, there will always be some skeptics who will hold out hope that the whole effort is a pilot that will soon be abandoned. Like Cortez burning his boats at Vera Cruz to prove his resolve to his soldiers, an organization that goes all in is demonstrating both its commitment to the new process and also that it will not turn back. This level of visible commitment to the change can be beneficial in helping the change succeed.

� It avoids problems created by having Scrum and traditional teams work together. If you transition anything short of the entire company all at once, you run the risk of having some teams using Scrum and others not. This means there will be times when a Scrum team needs to coordinate work with a traditional team, which creates challenges because of the different attitudes Scrum and traditional teams bring to things like planning, deadlines, and communication. These problems go away when the entire organization adopts Scrum at the same time. Chris Fry and Steve Greene ofSalesforce.com report that "the key factor driving us toward a big-bang rollout was to avoid organizational dissonance and a desire for decisive action. Everyone would be doing the same thing at the same time" (2007, 137).

� An all-in transition will be over more quickly. One of the central tenets of this book is that an organization is never "done" becoming agile; there are always improvements to be made. However, there is definitely a time when employees can look back and say of the transition that the worst is over. An organization that goes all in can reach this point more quickly.


Choosing Between Going All In and Starting Small
As I mentioned at the start of this chapter, starting small has been the default approach recommended by most agile authors and used in most agile adoptions. The combination of this approach's low risk and high likelihood of success make it hard to find fault with. Always choose to start small when there is reluctance by leaders in the organization to fully commit to Scrum. Success, even on a small scale, can be the best way to convince the skeptics. Always start small when there is a high cost associated with failure. If the cost of failure is too high for those leading the transition, starting small is the way to go, even if it may not be best for the organization as a whole. Start small is probably not the best approach when your organization urgently needs the benefits of Scrum. (But if you do choose to start small, scale quickly.) Starting small is safe, but it's slow.

Going all in should be used in limited cases. Consider going all in if time is critical. Although an all-in approach may cost more money, it will cost less time. If time is your primary concern, all in may be the best solution. Consider going all in if you, like Salesforce.com, want to send a clear message to a small number of critics and stakeholders that Scrum is here to stay. Never go all in without enough experienced ScrumMasters to serve each team. It doesn't matter in the short term whether these ScrumMasters are internal or external; but remember that eventually you'll want all of your ScrumMasters to be internal employees. Finally, size matters. If there are only ten of you, you might as well go all in. But for teams of more than perhaps 400, going all in may not be logistically possible.

Whichever route you choose for adopting Scrum, remember that choosing this pattern is only the first of the many decisions you'll need to make when transitioning. You will next need to decide whether to make your transition public.

Source of Information :  Pearson - Succeeding with Agile Software Development Using Scrum 2010

ADAPTing to Scrum

Lori Schubring was among the first to realize that things had to change. An application development manager for a large manufacturing company, Lori realized that its development process had become "so formalized that we hindered our ability to remain flexible for the business. It got to the point where we weren't turning around project requests fast enough" (2006, 27). Aware of the need to change, Lori attended a free, half-day seminar introducing Scrum. What she saw there was a better way to develop software, a framework she thought might help her organization. As such, Lori developed the desire to change to Scrum. Next, she acquired the ability to do it by participating in a ScrumMaster class, attending an agile conference, and visiting a company that had already adopted Scrum. Lori then promoted Scrum to her boss and team, convincing them of its benefits. Finally, Lori transferred some of the implications of her team using Scrum to the rest of her company so that organizational gravity would not pull the team back to where it had started.

Lori's story encapsulates the five common activities necessary for a successful and lasting Scrum adoption:

� Awareness that the current process is not delivering acceptable results

� Desire to adopt Scrum as a way to address current problems

� Ability to succeed with Scrum

� Promotion of Scrum through sharing experiences so that we remember and others can see our successes

� Transfer of the implications of using Scrum throughout the company

Conveniently, these five activities�Awareness, Desire, Ability, Promotion, and Transfer�can be remembered by the acronym ADAPT. These activities are shows Awareness, Desire, and Ability as overlapping, whereas Promotion and Transfer repeat and occur throughout the transition effort. After you have transitioned, this cycle will continue as you continuously improve.

The five activities of ADAPT are based on ADKAR (Hiatt 2006), a general model of change that includes the steps of Awareness, Desire, Knowledge, Ability, and Reinforcement. In practice, I have found separating Knowledge and Ability to be unnecessary. In a field such as software development, knowledge without ability is meaningless. Additionally, the Reinforcement step of ADKAR is replaced in ADAPT with separate Promotion and Transfer steps, emphasizing the importance of these activities to a successful transition.

An organization that successfully adopts Scrum can be thought of as engaging in these activities at multiple levels:

� Organizationally. The organization as a whole will engage in these activities. No matter how aware one person or group is, there must be a critical mass of people with a similar awareness before the organization will be able to collectively move forward. In thinking of the ADAPT model at this level, we may speak of a company with an organizational desire to adopt Scrum. Or we may say that our organization currently lacks the ability to do Scrum.

� As individuals. Because organizations are made up of individuals, it is important to acknowledge that individuals will progress through the overall transition at different rates. For example, you personally may already have acquired the ability to do Scrum; you've learned some new skills and some new ways of thinking about software development. A colleague, on the other hand, is only starting to become aware that the current approach isn't working.

� As teams. Individuals can be aided or hindered in the transition to Scrum by their teams. Teams tend to progress through the ADAPT cycle more or less together. In the same way that studies have shown individuals are more likely to be overweight if their friends are overweight (Thaler and Sunstein 2009), you are more likely to have a desire to do Scrum if the rest of your team does as well.

� Per practice. The ADAPT model can also be applied to each new skill that is acquired as part of adopting Scrum. Consider the increased reliance on automated unit testing that is common on Scrum teams. The team and its members must first become aware that the current approach to testing isn't working. They must then develop the desire to automate more tests and to do so earlier in the process. To do this will require that some team members learn new skills. Promoting the team's success with automated testing will encourage other development teams to emulate them. Finally, transferring the implications of the team doing more automated testing to other groups ensures that forces external to the team do not prevent it from continuing with the new practice.

One of the first things you'll need to do, whether you are currently using Scrum or just starting your adoption, is to decide where your individuals, teams, and organization are in their ADAPT sequence. It could be that you are acquiring the ability to do test-driven development on a team that is promoting its success inside a department that desires to implement Scrum. The overall organization, however, may be aware of only a general need to change. This chapter will discuss not only the five ADAPT activities but also the tools you will need to encourage and develop awareness, desire, ability, promotion, and transfer throughout all levels of the organization.

Source of Information :  Pearson - Succeeding with Agile Software Development Using Scrum 2010

Why Becoming Agile Is Hard (But Worth It)

Many software development organizations are striving to become more agile. And who can blame them? Successful agile teams are producing higher-quality software that better meets user needs more quickly and at a lower cost than are traditional teams. Besides, who wouldn't want to be more agile? It just plain sounds good, doesn't it? It is almost as though one cannot be too thin, too rich, or too agile. But beyond the buzzword and hype, organizations that take becoming agile seriously by adopting a process such as Scrum are seeing dramatic benefits.

They are seeing significant gains in productivity with corresponding decreases in cost. They are able to bring products to market much faster and with a greater degree of customer satisfaction. They are experiencing greater visibility into the development process, leading to greater predictability. And for them, out-of control, will-it-ever-be-done projects have become a thing of the past.

One company to realize these benefits by adopting Scrum is Salesforce.com. Founded in 1999 in a San Francisco apartment, Salesforce.com is one of the true, lasting dot-com-era success stories. With revenue of more than $450 million and 2,000 employees in 2006, Salesforce.com had noticed the frequency of its releases had dwindled from four a year to one a year. Customers were getting less and waiting longer to get it; something needed to be done. The company decided to transition to Scrum. During the first year of making the switch, Salesforce. com released 94% more features, delivered 38% more features per developer, and delivered over 500% more value to its customers compared to the previous year (Greene and Fry 2008). In the ensuing two years, revenue more than doubled to more than $1 billion. With results like these, it is not surprising that so many organizations have transitioned to Scrum. Or at least tried to.

I say "tried to" because transitioning to Scrum and other agile methods is hard�much harder than many companies anticipate. The changes required to reap all of the rewards being agile can bring are far reaching. These changes demand a great deal from not only the developers but the rest of the organization as well. Changing practices is one thing; changing minds is quite another. It is my aim in this book to show not only how to transition well but also how to succeed long term.

I've personally witnessed several failed agile adoptions that could have been prevented. The first was in a company that had spent more than a million dollars on its transition effort. Executives brought in outside trainers and coaches and hired five people into an "Agile Office" to which new Scrum teams could turn for advice. The company's failure was the result of thinking that the implications of adopting Scrum would be restricted to only the development organization. The executives who initiated this transition thought that educating and supporting developers would be sufficient. They failed to consider how Scrum would touch the work of salespeople, the marketing group, and even the finance department. Without changes to these areas, organizational gravity pulled the company back where it had started.

For completely different reasons, Josef ultimately failed at introducing Scrum to his company. A newly promoted and first-time project manager, Josef was instantly attracted to Scrum because it fit his natural management style. Josef easily convinced his team�who had all been his peers as little as one month before�to try Scrum on their new project. The project was wildly successful, earning accolades for the team and winning Josef the chance at a much larger project. Josef introduced the new project team to Scrum, and most members were willing to try the new approach. Although those working on the project were happy to use Scrum, some of the functional managers to whom they reported got nervous about what Scrum might mean to their careers. Josef's luck ran out. The functional managers�in particular the directors of quality assurance and database development�banded together and convinced the vice president of engineering that Scrum was inappropriate for projects of the complexity and importance being done in their company.

Caroline fared a little better. A vice president of development in a large data management company, Caroline had more than 200 developers in her organization. After seeing the benefits of Scrum on one project, she excitedly launched an initiative to introduce Scrum across her division. All employees were provided with training or coaching. Within a few months nearly all teams were producing working software at the end of each two-week sprint. This was great progress. When I visited this company a year later, though, the employees had failed to make any additional headway. To be sure, teams were producing higher-quality software and doing it a bit faster than they had before starting with Scrum, but her company's gains were only a fraction of what they could have been. Caroline's company had forgotten that continuous improvement is part of Scrum.

Frightening, isn't it? Each of these failures was a well-intentioned effort to transition to Scrum.Yet all the good intentions in the world could not keep them from failing. Don't worry, though. Transitioning to Scrum may be hard, but it's entirely possible with the right approach. In this chapter we examine why transitioning to any agile development process, including Scrum, is especially difficult.

We detail some of the challenges that derailed the companies I've mentioned. Most important, though, we look at the reasons why the benefits of becoming an agile organization are more than worth the effort.

Source of Information :  Pearson - Succeeding with Agile Software Development Using Scrum 2010

Useful Software Development Tools

Here�s a quick survey of some other candidates for every programmer�s toolbox.

Testing Tools
FitNesse: http://fitnesse.org/
FitNesse is an acceptance testing tool that allows tests to be expressed as tables of input data and expected output data, described in Fit for Developing Software: Framework for Integrated Tests.

Watir: http://wtr.rubyforge.org/
Watir is an open source library for automating web browsers allowing automated testing of web applications. It started out on Internet Explorer on Windows but is in the process of being ported to other browsers.

Selenium: http://selenium.openqa.org/
Selenium is a cross-platform suite of tools to automate web application testing.

Sahi: http://sahi.co.in/
Sahi is an automation and testing tool for web applications that runs as a proxy server.

The Grinder: http://grinder.sourceforge.net/
This is an open source load testing tool in which scripts are written Jython.

JMeter: http://jakarta.apache.org/jmeter/
This is an open source load testing tool in which scripts are written in Java.

QuickTest Professional and LoadRunner: http://www.hp.com/
QuickTest Professional is an automated functional GUI testing tool, and LoadRunner is a performance and load testing product.

Peach Fuzzing Platform: http://peachfuzzer.com/
Peach is a fuzzer that is capable of performing both generation and mutation-based fuzzing.

RFuzz: http://rfuzz.rubyforge.org/
RFuzz is a Ruby library that allows web applications to be easily fuzz tested.


Runtime Analysis Tools
Valgrind: http://valgrind.org/
Valgrind is an instrumentation framework for Linux and includes, among other things, memory analysis and profiling tools.

BoundsChecker: http://www.compuware.com/products/devpartner/visualc.htm
BoundsChecker is part of Compuware�s DevPartner for Visual C++ BoundsChecker Suite. It analyzes running programs to detect memory and other issues.

Purify: http://www.ibm.com/software/awdtools/purify/
IBM�s Rational Purify detects memory leaks and corruption within running programs.

DTrace: http://opensolaris.org/os/community/dtrace/
DTrace is a highly regarded dynamic tracing framework created by Sun Microsystems for troubleshooting kernel and application problems. It is also incorporated in Mac OS X �Leopard,� including a GUI called Instruments.


Network Analyzers
If your software relies upon network communication (and it�s becoming difficult to find software which doesn�t), it can be very useful to see what�s really being transferred over the network. A network analyzer (sometimes called a packet sniffer) sits on the network capturing and analyzing all the packets crossing it. You can then filter these packets to extract only those that you�re interested in and examine their contents. Broadly speaking, a packet sniffer is a low-level tool. It can capture all the traffic on the network but doesn�t necessarily have a deep understanding of the protocol being used. So if, for example, the communication is encrypted, a packet sniffer is unlikely to be able to display the information being exchanged.

TCPDUMP: http://www.tcpdump.org/
TCPDUMP is a widely distributed open source packet sniffer.

Wireshark: http://www.wireshark.org/
Wireshark (previously known as Ethereal) is an open source tool that provides similar functionality to TCPDUMP, but it has a graphical front end and a wider selection of built-in analysis tools.


Debugging Proxies
A debugging proxy is a higher-level tool than a network analyzer, targeted to a particular protocol. You normally need to configure your software slightly differently so that it communicates via the proxy rather than directly, but having done so very often you can get a deeper analysis of the conversation. Some debugging proxies can even view encrypted data.

Charles: http://www.charlesproxy.com/
Charles is a cross-platform HTTP proxy that, among other things, supports debugging encrypted communications.

Fiddler: http://www.fiddlertool.com/
Fiddler is a Windows HTTP proxy that, as its name suggests, allows you to �fiddle� with incoming or outgoing data.


Debuggers
In most cases, your choice of debugger is going to be governed by your choice of language, IDE, or tool chain, so there�s little value in me providing a list of choices here. There is one particular debugger that I have to mention, however:

Firebug: http://getfirebug.com/
Firebug has transformed web development by providing dramatically improved client-side debugging facilities. It allows you to inspect and edit the DOM and CSS, as well as monitor and profile network activity, and it provides full JavaScript debugging support.

Source of Information : Paul Butcher - Debug it Find repair and prevent bugs

Useful Software Development Libraries

Not all tools are stand-alone�many, covered in this section, come in the form of libraries that we need to link with our own code.


Testing
The last few years have seen an explosion in the number of test frameworks, many of which are based upon the seminal JUnit. There�s no way that I can begin to cover them all here, so I will restrict myself to referencing the �big two� in the Java community:

JUnit: http://www.junit.org/
This is the library that started it all.

TestNG: http://testng.org/
This is a more recent test framework, which builds upon the ideas in JUnit but takes a few different approaches and is starting to gain a considerable following.


Debugging Memory Allocators
Debugging Memory Allocators, on in languages like C and C++ that don�t provide memory management, a debugging memory allocator is an essential tool to avoid memory leaks, corruption, and other common issues.

libcwd: http://libcwd.sourceforge.net/
This is an open source debugging support library that provides memory debugging along with other features.

Microsoft Visual C++: http://msdn.microsoft.com/visualc/
Microsoft�s Visual C++ ships with a debugging memory allocator built in. Search for Memory Leak Detection and Isolation in the documentation for further information.

Mudflap: http://gcc.gnu.org/wiki/Mudflap_Pointer_Debugging
Mudflap is a technology built into some versions of the GNU C and C++ compiler that instruments all risky pointer and array dereferencing operations, some standard library string and heap functions, and some other associated constructs with range and validity tests.

Dinkumware: http://www.dinkumware.com/
Dinkumware sells C and C++ standard libraries that include comprehensive support for memory debugging.

Electric Fence: http://perens.com/works/software/ElectricFence/
This uses virtual memory hardware to detect memory overwrites and reuse of freed memory.


Logging
Logging frameworks provide the ability for your code to contain configurable logging that can be enabled, disabled, or increased in detail, typically at runtime and by individual feature.

log4j: http://logging.apache.org/log4j/
Apache log4j is probably the best-known Java logging library, and ports exist to most major languages.

Logback: http://logback.qos.ch/
Logback was designed by Ceki G�lc�, the founder of log4j, to be its successor.

java.util.logging: http://java.sun.com/j2se/1.4.2/docs/guide/util/logging/
As of 1.4.2, Java includes a standard logging API java.util.logging, commonly known as JUL.

SLF4J: http://www.slf4j.org/
The Simple Logging Facade for Java is an attempt to tame the plethora of Java logging APIs by providing a common interface that can write to different implementations at deployment time.

syslog-ng http://www.balabit.com/network-security/syslog-ng/
syslog-ng is the most popular implementation of The BSD syslog Protocol, allowing log data to be integrated from many different systems into a central repository and rich content-based filtering.

Source of Information :  Paul Butcher - Debug it Find repair and prevent bugs

Build and Continuous Integration Tools

We�ve examined at length the benefits of automating your build process, and as you would expect, there are many off-the-shelf tools that will help you to do so.


Build Tools
The granddaddy of build tools is the venerable make. Things have moved on, however, and several much better choices are now available.

GNU Make: http://www.gnu.org/software/make/
Although based upon make, GNU Make supports a number of significant extensions allowing much more sophisticated control over the build process than has traditionally been available.

Autoconf: http://www.gnu.org/software/autoconf/
Autoconf is particularly appropriate for open source software that needs to support building in a wide range of different environments. It allows the build system to automatically determine what facilities are available on the host system and behave accordingly.

Jam: http://www.perforce.com/jam/jam.html
Jam is an alternative to make that typically requires much less configuration to build a given project.

Boost.Build: http://www.boost.org/doc/tools/build/
Built on top of Jam, Boost.Build provides a standard build system particularly appropriate to building C++ software.

SCons: http://www.scons.org/
This is a make replacement integrating autoconf-like functionality.

Ant: http://ant.apache.org/
Ant is a make replacement that has become the de facto standard build tool within the Java world.

Maven: http://maven.apache.org/
Maven is a software project management tool that does much more than simply manage the build process, bringing package management, deployment, and other facilities to the Java world and rapidly gaining mind share from Ant.

Capistrano: http://www.capify.org/
Not a build tool per se, Capistrano manages the task of deploying software on a number of different servers. Although particularly associated with Ruby on Rails, it can be used to deploy products created with any technology.


Continuous Integration Tools
Many of the proprietary systems we�ve already discussed (such as Microsoft�s Visual Studio Team System) come with their own continuous integration solutions. In addition, there are a number of open source systems available:

CruiseControl: http://cruisecontrol.sourceforge.net/
This is probably the best known open source continuous integration system. As well as the main Java implementation, there are also .NET and Ruby on Rails variants.

Hudson: http://hudson.dev.java.net/
This is an open source J2EE continuous integration server.

Source of Information :  Paul Butcher - Debug it Find repair and prevent bugs

Source Control and Issue-Tracking Systems

The problem with choosing a source control and issue-tracking system isn�t so much finding one that�s right for you as picking through the huge range available. So, what might sway your decision? Some things (not an exhaustive list) to consider include the following:

� Open source or commercial?

� Do you need to host it yourself (behind your firewall, for example), or do you want to use one of the many services that provide hosting for you?

� Do you need your source control and issue-tracking systems to be tightly integrated with each other?

� What level of support for distributed development do you need?

I can�t possibly give a complete survey of all the different source control and issue-tracking systems here, but I can give you pointers to a few of the major players and why you might consider them.


Open Source Solutions
CVS: http://www.nongnu.org/cvs/
Until fairly recently, the only real open source choice was CVS. CVS has a number of well-known limitations, however, not least of which are the fact that check-ins aren�t atomic and it doesn�t version directory structures.

Subversion: http://subversion.tigris.org/
Over the last few years, CVS has been almost entirely supplanted by Subversion, which addresses most of CVS�s obvious weaknesses and has become the default open source choice.

Git: http://git.or.cz/
Coming up fast on the rails is Git, which is gaining mind share with a number of high-profile projects switching to it, in part because of its excellent support for distributed development.

Mercurial: http://www.selenic.com/mercurial/
This is a cross-platform, distributed system with very similar goals to Git and particularly good support for branching.

Bazaar: http://bazaar-vcs.org/
This is designed to just work and adapt to your team�s workflow instead of imposing its own model.

Bugzilla: http://www.bugzilla.org/
For a long time, Bugzilla, developed as part of the Mozilla project, was the default open source choice for issue tracking. Recently a number of alternatives have become available, however.

Trac: http://trac.edgewall.org/
Trac uses a minimalist approach, aiming to keep out of the way of developers as much as possible. It�s particularly notable for tight integration with its integrated wiki.

Redmine: http://www.redmine.org/
A relative newcomer on the scene, Redmine seems to be well supported and making good progress. Where open source solutions have traditionally been weak is integration between source control and issue tracking and with development environments. The situation has improved considerably recently with IDEs such as Eclipse providing excellent Subversion support, for example.


Hosted Solutions
SourceForge: http://sourceforge.net/
SourceForge is the best known of a number of similar sites that provide hosting for open source projects, integrating a number of tools such as source control, issue tracking, documentation tools, and so on. Others include Google Code (http://code.google.com/hosting/) and language-specific sites such as RubyForge (http://rubyforge.org/).

GitHub: http://github.com/
GitHub provides Git hosting and has recently gained a lot of attention when it started hosting the Ruby on Rails project.

Lighthouse: http://lighthouseapp.com/
This is a hosted issue-tracking system with integration for Subversion and Git.

Unfuddle: http://unfuddle.com/
This is a secure, hosted project management solution providing Subversion or Git hosting together with integrated issue tracking.

Rally: http://www.rallydev.com/
Rally provides Agile life-cycle management tools.

VersionOne: http://www.versionone.com/
This is a project management and planning tool designed specifically for agile software development. This is also available for local installation as well as hosted.

Pivotal Tracker: http://www.pivotaltracker.com/
Tracker is a free, award-winning, story-based project-planning tool that allows teams to collaborate in real time.


Commercial Solutions
Perforce: http://www.perforce.com/
Perforce is a source control system that particularly concentrates on cross-platform support and performance. It also includes a simple issue-tracking system or can integrate with various open source or commercial solutions.

FogBugz: http://www.fogcreek.com/FogBugz/
FogBugz, from Fog Creek Software, is a flexible bug tracking and project planning tool, available for local installation or as a hosted solution. It�s traditionally been available for Windows, but is in the process of being ported to Linux and Macintosh.

Visual Studio Team System: http://msdn.microsoft.com/teamsystem/
Microsoft�s Visual SourceSafe has long been a favorite punch bag, criticized for a range of failings. To be fair, Microsoft can hardly complain about this given that it never seemed to use it itself despite its famous policy of eating its own dog food. Thankfully,
Microsoft�s offering in this area seems to have improved to no end recently with the introduction of Visual Studio Team System, a fully integrated source control and project management solution.

Rational ClearCase and ClearQuest: http://ibm.com/software/awdtools/clearcase/
The ClearCase source control system and its associated issuetracking solution ClearQuest used to be considered the default enterprise choice. They are expensive and complex, however, and inappropriate for anything other than large teams with dedicated
support organizations.

StarTeam: http://www.borland.com/starteam/
This is a fully integrated source control and project management system.

BitKeeper: http://www.bitkeeper.com/
This is a distributed system with similar goals to Git.

Source of Information :  Paul Butcher - Debug it Find repair and prevent bugs

Debugging Builds

Many teams find it helpful to create a debugging build, which differs from a release build in various ways designed to help reproduce and diagnose problems.


Compiler Options
Most compilers provide you with a wide range of options that allow you to control exactly how they translate your source code into object code. Often it makes sense to use a different set of options during development and debugging from those used in production. Here are a few examples:

Optimization: Modern compilers can perform wonders, generating object code that is as efficient, or better, than hand-rolled machine code. In the process of doing so, however, they often restructure things so much that the relationship between source code and object code can become muddied. This can, for example, make single-stepping in a debugger confusing or even impossible. As a result, debug builds often disable optimization. Debugging information: To be able to single step through the source, debuggers need to know how to map lines of source code to regions of object code. Typically these are excluded from a production release because they add size and may give away information we would rather keep to ourselves.

Bounds checking: Some C/C++ compilers provide an ability to add bounds checking to arrays and other data structures.

There�s more to a debugging build than just choosing different compiler options, however.


Debugging Subsystems
Sometimes it�s worth thinking about replacing an entire subsystem with a version specifically designed to make debugging easier. This can be particularly useful if we can�t easily control the behavior of the production version of the subsystem (because it�s under the control of a third-party, for example, or because its behavior has some random element).

Imagine, for example, that our software interfaces with a server provided by a third-party and we�re trying to debug a problem that occurs only when it returns a specific sequence of results. It may not be easy, or even possible, to find a way to ensure that it always returns that exact sequence on demand. Even if we can, its owners may not thank us for bombarding it with requests�especially if those requests aren�t well-formed (which is likely to be the case during debugging).

There is some overlap between a debugging subsystem and the test doubles, Mocks, Stubs, and Other Test Doubles. The difference is one of scale and scope. A test double is a short-lived object only intended for use within a single test. A debugging subsystem is normally a complete replacement for its associated production subsystem, implementing all of its interfaces and operating correctly across a wide range of use cases. It may even make sense for us to ship a debugging subsystem with the software so that end users can enable it in order to help us debug a problem in situ.

A debugging subsystem either can entirely replace its corresponding production system (emulating its entire behavior) or can be implemented as a shim that sits between the rest of the software and the production system, modifying its behavior as appropriate.

One particular subsystem you might want to consider bypassing during debugging is the user interface.


Solving the User Interface Problem
The needs of the end user and the needs of a developer are often very different. A graphical or web-based user interface might make it very easy for end users to achieve their goals, but it can get in the way during development and debugging because such interfaces are difficult to control programmatically.

For this reason (among others), it makes sense to ensure that the user interface layer is as thin as possible, just looking after the details of displaying information and soliciting input from the user. In particular, it should contain no business logic whatsoever. This should mean that you can replace it with an alternative such as a scripting language that can drive the rest of the software, which is likely to be much easier to work with from a debugging standpoint.

This might fall out in the wash if your software implements an object model such as (OLE) Automation under Windows or AppleScript support on the Mac. It might even be worth adding support for such an object model exclusively for debugging. Another subsystem commonly replaced with a debugging version is the memory allocator.


Debugging Memory Allocators
In languages like C and C++, which don�t provide automatic memory management, a debugging memory allocator can be worth its weight in gold. A debugging allocator can help you detect and solve a number of common problems:

� By keeping track of memory allocation and deallocation, it can detect memory leaks (memory that is allocated but not freed).

� By placing guards before and after allocated memory, it can detect buffer overflows and memory corruption.

� By filling memory regions with known patterns, it can detect instances where memory is used without being initialized.

� By filling deallocated memory with a known pattern and holding onto it, it can detect instances where memory is written to after it has been deallocated.


Built-in Control
As well as modifying the behavior of third-party code, we can also choose to have our own code behave differently in a debug build, building in the control that will prove useful during diagnosis. Examples include the following:

Disabling features: Sometimes your software might include features that are valuable in production but obfuscate things during debugging. Communication between one part of the application and another might be encrypted for security reasons, for example. Or data structures might be optimized to improve memory usage and execution speed. You are likely to make problems in these areas much easier to diagnose if you allow such features to be selectively disabled.

Providing alternative implementations: Sometimes there is more than one way to implement a module�one that is simple and easy to understand and another that is complex and optimized. By including both within the code and providing a means to switch between them, you can validate the results of the complex version against the simple one. This can help pinpoint whether the bug resides in the optimized version or elsewhere, and it can help with debugging even if it does lie elsewhere by making things simpler to understand.

Although we tend to talk about two different builds, debug and release, there�s nothing to stop you from building other flavors. Many teams, for example, have an integration build that acts as a halfway house between a debug and a release build. It might, for example, have debugging symbols and assertions enabled like a debug build but have optimizations enabled like a release build.

Source of Information : Paul Butcher - Debug it Find repair and prevent bugs

How Do I Become Warning Free?

If you�re starting a new project from scratch, writing warningfree code is easy. But if you�re starting from an existing codebase, it can be much less straightforward. The chances are that the first time you increase your compiler�s warning level or run a new tool, you will disappear under a tidal wave of warnings. Often these result from systemic issues with the code�common mistakes you�ve made over and over again, which have gone unnoticed until now, but each instance of which generates a warning. There are also issues that tend to �percolate� through the code generating many warnings (const-correctness in C++ is a classic example).

The solution is to be pragmatic. Most static analysis tools provide fine-grained control over which warnings are generated where (via comments embedded in the source code, for example). Very often you can get the number down to a manageable level by switching off the one or two warnings that account for the majority or by excluding a �problem� module. You can go back and fix these other warnings at a later date, but you gain most of the benefit of static analysis in the interim.

The same approach can help on the rare occasions where a buggy tool generates spurious warnings for legitimate code, where you knowingly choose to write �questionable� code, or where a third-party library generates warnings.

Source of Information :  Paul Butcher - Debug it Find repair and prevent bugs

Automated Tests as an Aid to Debugging

So, what makes automated tests valuable when debugging? They help out at all stages:

� First and foremost, well-tested code tends to have fewer bugs in the first place. The easiest bug to fix is the bug that never existed.

� The shorter the delay between a mistake being made and subsequently being discovered, the easier and cheaper it is to fix. Early testing means that most bugs are discovered very shortly (often immediately) after they�re introduced.

� Automated testing is a key enabler of continuous integration, in which code is integrated with the whole product as soon as it�s complete.  

� Automated tests allow you to frequently release new versions of the software with high confidence that the new release is functional. This means that you get end-user feedback on new features and bug fixes much more quickly than would otherwise be the case (again, reducing the time between code being written and bugs being discovered within the code). It can also reduce the need to back-port bug fixes to previous versions of the software or to release patches.

� For code to be tested, it needs to be structured in such a way as to provide access to intermediate results and internal state that might otherwise be unavailable. This kind of access turns out to be a great help during later debugging.

� Writing a test is an excellent way to reproduce a bug during the diagnostic process. Many of the techniques created to support automated testing are extremely useful for reliably reproducing bugs.

� After you�ve completed your diagnosis, automated tests provide powerful protection against the fix introducing regressions.

� If, during diagnosis, you make a habit of always writing a test that reproduces the bug, you naturally end up with a regression test that ensures that the bug won�t be reintroduced at some point in the future.

� Automated tests are a key enabler of refactoring, which is the most powerful weapon at your disposal to ensure that code remains well-structured and flexible throughout its lifetime.

Automated tests are a particularly powerful debugging tool when allied with a technique that has risen in popularity alongside them�test doubles.

Source of Information :  Paul Butcher - Debug it Find repair and prevent bugs

Effective Automated Testing

Agile software development has dramatically changed software construction through the widespread adoption of automated testing and refactoring. There�s more to effective automated testing than simply automating your tests. To achieve maximum benefit, your tests need to satisfy the following goals:

Unambiguous pass/fail: Each test outputs a single bit�pass or fail. No shades of gray, no qualitative output, no interpretation required. Just a simple yes or no.

Self-contained: No setup required before running a test. Before it runs, it sets up whatever environment it needs automatically, and just as important, it undoes any changes to the environment afterward, leaving everything as it found it.

Single-click to run all the tests: All tests can be run in one step without interfering with each other. As with a single test, the output of the complete test suite is a simple pass or fail�pass if every test passes, fail otherwise.

Comprehensive coverage: It�s easy to prove that achieving complete coverage for any nontrivial body of code is prohibitively expensive. But don�t allow that theoretical limitation to put you off�it is possible to achieve close enough to complete coverage as to make no practical difference.

Source of Information :  Paul Butcher - Debug it Find repair and prevent bugs

Beware of Heisenberg

One of the lessons of quantum physics is that the act of observing a system can change the system itself. Computer software isn�t quantum mechanical (not yet, anyway), but we still need to be wary.

Instrumenting software intrinsically involves changing it, which raises the specter of affecting, instead of simply observing, its behavior. This is dangerous during diagnosis, because introducing an unintentional change during a series of experiments can easily lead to you draw invalid conclusions.

Fundamentally speaking, there is no way that you can guarantee to avoid introducing some side effects. The fact that you�ve modified the source code means that the layout of the object code in memory and the timing of its execution will be affected. Happily, most of the time this remains a purely hypothetical problem�as long as you�re careful to avoid the more obvious side effects, you can normally ignore the issue.

Nevertheless, it is very good practice to keep the source code as close to its pristine form as possible. Don�t allow failed experiments, along with their possible side effects, to accumulate over time. Keeping things neat also helps ensure that the code remains easy (or at least, no harder) to understand and will help ensure that you don�t check in unintended changes when you eventually come to fixing the problem.

Source of Information : Paul Butcher - Debug it Find repair and prevent bugs 

What Are the Key Insights of Refactoring?

Many people�s reaction to refactoring when first exposed to it is �so what?� This is just �tidying up� the code, something that programmers have been doing for almost as long as programmers have existed. Certainly, to some extent all that Martin Fowler did when he published Refactoring was to catalog techniques that developers have been using for years.

But there�s more to refactoring than just a catalog of useful techniques. It relies on Fowler�s two key insights:

� Modifying existing code can be carried out safely only with the safety net of a comprehensive suite of unit tests.

� We should never attempt to refactor the code at the same time as modifying its behavior.

In other words, you can modify the behavior of the code, or you can refactor it. You should never attempt to do both at the same time.

Upon reflection, it�s easy to see why this is the case. Imagine that you attempt to modify both the structure of your code and its functionality at the same time, and after doing so one of your tests fails. This might indicate that you made a mistake when modifying its structure. Or it might be an expected result of the change in functionality. It�s difficult, however, to be sure which. The more complicated the change in functionality or structure, the harder it is to be certain.

By doing only one or the other, you avoid this issue entirely and can forge ahead with potentially far-reaching refactorings involving dramatic changes to the code with confidence.

Source of Information : Paul Butcher - Debug it Find repair and prevent bugs 

Should I Leave My Logging in the Code?

Some topics are guaranteed to create an argument among developers, and logging is one of them. If you�ve added logging to the code to help while tracking down a problem, it�s tempting to leave this instrumentation in place so that you can find the problem again quickly if it happens again. This is especially true if you�re using a logging framework that allows it to be enabled and disabled easily. What�s not to like?

So, why the controversy? Detractors will tell you the following:

� Logging obscures the code, making it difficult to see the wood for the trees.

� Logging can suffer from the same problems as comments�as the code evolves, often the logging isn�t updated to match, meaning that you can�t trust what it says and making it worse than useless.

� No matter how much logging you add, it�s never what you need. The next time you find yourself debugging in that area, you�ll just have to add more, and if you leave it in the code when you�re done, you just exacerbate the first two problems.

As with most disputes of this nature, the answer is to be pragmatic. Logging is a useful tool, but it can be overused. Consider implementing permanent logging if you believe that it will add value, but be disciplined about how you do so. Make sure that your logging is up-to-date and agrees with the code and that you don�t add it for its own sake.

As a general rule, the most useful logging is at the highest (strategic) level�a record of what happened, such as the access log generated by an HTTP server, for example. Lower level, more tactical logging can be of questionable long-term value, so make sure you know what it�s giving you before you decide to add it.

If you find that logging is getting in the way but you don�t want to lose its benefits, you might want to look at aspect-oriented programming, which may give you a way to separate it from the main body of the code (a good reference is Ramnivas Laddad. AspectJ in Action: Practical Aspect-Oriented Programming. Manning Publications Co., 2003.)

Source of Information : Paul Butcher - Debug it Find repair and prevent bugs

On the Nature of Software

Software is remarkable stuff. Sometimes, perhaps because we work with it all the time, we forget just how remarkable it is. Very little else in human experience is as malleable, allowing us free rein to exercise our ingenuity and inventiveness almost without limits. Also, with a very few exceptions, software is deterministic�the next state is completely determined by the current state, and (crucially) we have complete access to all of that state whenever we want it.

Compared to traditional engineering, we are spoiled. What do you think a Formula One engineer would give to be able to instantaneously stop an engine when it�s rotating at 19,000 revolutions per minute and examine every aspect of it in minute detail? To see the precise state of each component while under pressure and stress, for example, or to dynamically record the shape and position of the flame front within the combustion chambers during ignition?

It is exactly this kind of trick that we are able to perform with our software, which is why the empirical approach is particularly powerful when debugging.

Empirical Approach. Construct experiments, and observe the results. Empiricism relies upon observation or experience, rather than theory or pure logic. In the context of debugging, this means directly observing the behavior of the software. Yes, you could read the entire source code and use pure reason to work out what�s going on (and on occasion you may have no other choice), but doing so is usually inefficient and dangerous. You can track the problem down much more effectively by carefully constructing experiments and observing how the software behaves. Not only is this faster, but these observations force you to reexamine flawed assumptions in your mental model about how the software behaves. The software itself is the most powerful tool in your toolbox�allow it to show you what�s going on.

Source of Information : Paul Butcher - Debug it Find repair and prevent bugs

Debugging Is More Than �Making the Bug Go Away�

Ask an inexperienced programmer to define debugging, and they might answer that it is �finding a fix.� In fact, that is only one of several goals, and not even the most important of them.

Effective debugging requires that we take these steps:
1. Work out why the software is behaving unexpectedly.

2. Fix the problem.

3. Avoid breaking anything else.

4. Maintain or improve the overall quality (readability, architecture, test coverage, performance, and so on) of the code.

5. Ensure that the same problem does not occur elsewhere and cannot occur again.

Of these, by far the most important is the first�identifying the root cause of the problem is the cornerstone upon which everything else depends.

Understanding Is Everything
Inexperienced developers (and sometimes, unfortunately, those of us who should know better) often skip diagnosis altogether. Instead, they immediately implement what they think might be a fix. If they�re lucky, it won�t work, and all they will have done is waste their time. The real danger comes if it works, or seems to work, because now they�ve made a change to the source that they don�t really understand. It might fix the bug, but there is a real chance that in reality it is only masking the true underlying cause. Worse, there is a good chance that this kind of change will introduce regressions�breaking something that used to work correctly beforehand.

Wasted Time and Effort
Some years ago, I found myself working in a team containing a number of very experienced and talented developers. Most of their experience was with UNIX, but when I joined the team, they were in the late stages of porting the software to Windows. One of the bugs found during the port was a performance issue when running many threads simultaneously. Some threads were being starved, while others were running just fine.

Given that everything worked just fine under UNIX, the problem was clearly broken threading in Windows, so the decision was made to implement a custom thread scheduling system and avoid using that provided by the operating system. This would be a lot of work, obviously, but quite within the capabilities of a team of this caliber.

I joined the team when they were some way into the implementation, and sure enough, threads were no longer suffering from starvation. But thread scheduling is subtle, and they were still working through a number of issues that had been caused by the change (not least of which was that the changes had slowed the whole system down somewhat).

I was intrigued by this bug, because I�d previously experienced no problems with Windows� threading. A little investigation demonstrated that the performance issue was caused by the fact that Windows implements a dynamic thread priority boost. The bug could be fixed by disabling this with a single line of code (a call to SetThreadPriorityBoost( )).

The moral? The team had decided that Windows� threads were broken without really investigating the behavior they were seeing. In part, this might have been a cultural issue�Windows doesn�t have a good reputation among UNIX hackers. Nevertheless, if they had taken the time to identify the root cause, they would have saved themselves a great deal of work and avoided introducing complications that made the system both less efficient and more error-prone.

Without first understanding the true root cause of the bug, we are outside the realms of software engineering and delving instead into voodoo programming or programming by coincidence.

Source of Information : Paul Butcher - Debug it Find repair and prevent bugs