Debugging Builds

Many teams find it helpful to create a debugging build, which differs from a release build in various ways designed to help reproduce and diagnose problems.


Compiler Options
Most compilers provide you with a wide range of options that allow you to control exactly how they translate your source code into object code. Often it makes sense to use a different set of options during development and debugging from those used in production. Here are a few examples:

Optimization: Modern compilers can perform wonders, generating object code that is as efficient, or better, than hand-rolled machine code. In the process of doing so, however, they often restructure things so much that the relationship between source code and object code can become muddied. This can, for example, make single-stepping in a debugger confusing or even impossible. As a result, debug builds often disable optimization. Debugging information: To be able to single step through the source, debuggers need to know how to map lines of source code to regions of object code. Typically these are excluded from a production release because they add size and may give away information we would rather keep to ourselves.

Bounds checking: Some C/C++ compilers provide an ability to add bounds checking to arrays and other data structures.

There�s more to a debugging build than just choosing different compiler options, however.


Debugging Subsystems
Sometimes it�s worth thinking about replacing an entire subsystem with a version specifically designed to make debugging easier. This can be particularly useful if we can�t easily control the behavior of the production version of the subsystem (because it�s under the control of a third-party, for example, or because its behavior has some random element).

Imagine, for example, that our software interfaces with a server provided by a third-party and we�re trying to debug a problem that occurs only when it returns a specific sequence of results. It may not be easy, or even possible, to find a way to ensure that it always returns that exact sequence on demand. Even if we can, its owners may not thank us for bombarding it with requests�especially if those requests aren�t well-formed (which is likely to be the case during debugging).

There is some overlap between a debugging subsystem and the test doubles, Mocks, Stubs, and Other Test Doubles. The difference is one of scale and scope. A test double is a short-lived object only intended for use within a single test. A debugging subsystem is normally a complete replacement for its associated production subsystem, implementing all of its interfaces and operating correctly across a wide range of use cases. It may even make sense for us to ship a debugging subsystem with the software so that end users can enable it in order to help us debug a problem in situ.

A debugging subsystem either can entirely replace its corresponding production system (emulating its entire behavior) or can be implemented as a shim that sits between the rest of the software and the production system, modifying its behavior as appropriate.

One particular subsystem you might want to consider bypassing during debugging is the user interface.


Solving the User Interface Problem
The needs of the end user and the needs of a developer are often very different. A graphical or web-based user interface might make it very easy for end users to achieve their goals, but it can get in the way during development and debugging because such interfaces are difficult to control programmatically.

For this reason (among others), it makes sense to ensure that the user interface layer is as thin as possible, just looking after the details of displaying information and soliciting input from the user. In particular, it should contain no business logic whatsoever. This should mean that you can replace it with an alternative such as a scripting language that can drive the rest of the software, which is likely to be much easier to work with from a debugging standpoint.

This might fall out in the wash if your software implements an object model such as (OLE) Automation under Windows or AppleScript support on the Mac. It might even be worth adding support for such an object model exclusively for debugging. Another subsystem commonly replaced with a debugging version is the memory allocator.


Debugging Memory Allocators
In languages like C and C++, which don�t provide automatic memory management, a debugging memory allocator can be worth its weight in gold. A debugging allocator can help you detect and solve a number of common problems:

� By keeping track of memory allocation and deallocation, it can detect memory leaks (memory that is allocated but not freed).

� By placing guards before and after allocated memory, it can detect buffer overflows and memory corruption.

� By filling memory regions with known patterns, it can detect instances where memory is used without being initialized.

� By filling deallocated memory with a known pattern and holding onto it, it can detect instances where memory is written to after it has been deallocated.


Built-in Control
As well as modifying the behavior of third-party code, we can also choose to have our own code behave differently in a debug build, building in the control that will prove useful during diagnosis. Examples include the following:

Disabling features: Sometimes your software might include features that are valuable in production but obfuscate things during debugging. Communication between one part of the application and another might be encrypted for security reasons, for example. Or data structures might be optimized to improve memory usage and execution speed. You are likely to make problems in these areas much easier to diagnose if you allow such features to be selectively disabled.

Providing alternative implementations: Sometimes there is more than one way to implement a module�one that is simple and easy to understand and another that is complex and optimized. By including both within the code and providing a means to switch between them, you can validate the results of the complex version against the simple one. This can help pinpoint whether the bug resides in the optimized version or elsewhere, and it can help with debugging even if it does lie elsewhere by making things simpler to understand.

Although we tend to talk about two different builds, debug and release, there�s nothing to stop you from building other flavors. Many teams, for example, have an integration build that acts as a halfway house between a debug and a release build. It might, for example, have debugging symbols and assertions enabled like a debug build but have optimizations enabled like a release build.

Source of Information : Paul Butcher - Debug it Find repair and prevent bugs

How Do I Become Warning Free?

If you�re starting a new project from scratch, writing warningfree code is easy. But if you�re starting from an existing codebase, it can be much less straightforward. The chances are that the first time you increase your compiler�s warning level or run a new tool, you will disappear under a tidal wave of warnings. Often these result from systemic issues with the code�common mistakes you�ve made over and over again, which have gone unnoticed until now, but each instance of which generates a warning. There are also issues that tend to �percolate� through the code generating many warnings (const-correctness in C++ is a classic example).

The solution is to be pragmatic. Most static analysis tools provide fine-grained control over which warnings are generated where (via comments embedded in the source code, for example). Very often you can get the number down to a manageable level by switching off the one or two warnings that account for the majority or by excluding a �problem� module. You can go back and fix these other warnings at a later date, but you gain most of the benefit of static analysis in the interim.

The same approach can help on the rare occasions where a buggy tool generates spurious warnings for legitimate code, where you knowingly choose to write �questionable� code, or where a third-party library generates warnings.

Source of Information :  Paul Butcher - Debug it Find repair and prevent bugs

Automated Tests as an Aid to Debugging

So, what makes automated tests valuable when debugging? They help out at all stages:

� First and foremost, well-tested code tends to have fewer bugs in the first place. The easiest bug to fix is the bug that never existed.

� The shorter the delay between a mistake being made and subsequently being discovered, the easier and cheaper it is to fix. Early testing means that most bugs are discovered very shortly (often immediately) after they�re introduced.

� Automated testing is a key enabler of continuous integration, in which code is integrated with the whole product as soon as it�s complete.  

� Automated tests allow you to frequently release new versions of the software with high confidence that the new release is functional. This means that you get end-user feedback on new features and bug fixes much more quickly than would otherwise be the case (again, reducing the time between code being written and bugs being discovered within the code). It can also reduce the need to back-port bug fixes to previous versions of the software or to release patches.

� For code to be tested, it needs to be structured in such a way as to provide access to intermediate results and internal state that might otherwise be unavailable. This kind of access turns out to be a great help during later debugging.

� Writing a test is an excellent way to reproduce a bug during the diagnostic process. Many of the techniques created to support automated testing are extremely useful for reliably reproducing bugs.

� After you�ve completed your diagnosis, automated tests provide powerful protection against the fix introducing regressions.

� If, during diagnosis, you make a habit of always writing a test that reproduces the bug, you naturally end up with a regression test that ensures that the bug won�t be reintroduced at some point in the future.

� Automated tests are a key enabler of refactoring, which is the most powerful weapon at your disposal to ensure that code remains well-structured and flexible throughout its lifetime.

Automated tests are a particularly powerful debugging tool when allied with a technique that has risen in popularity alongside them�test doubles.

Source of Information :  Paul Butcher - Debug it Find repair and prevent bugs

Beware of Heisenberg

One of the lessons of quantum physics is that the act of observing a system can change the system itself. Computer software isn�t quantum mechanical (not yet, anyway), but we still need to be wary.

Instrumenting software intrinsically involves changing it, which raises the specter of affecting, instead of simply observing, its behavior. This is dangerous during diagnosis, because introducing an unintentional change during a series of experiments can easily lead to you draw invalid conclusions.

Fundamentally speaking, there is no way that you can guarantee to avoid introducing some side effects. The fact that you�ve modified the source code means that the layout of the object code in memory and the timing of its execution will be affected. Happily, most of the time this remains a purely hypothetical problem�as long as you�re careful to avoid the more obvious side effects, you can normally ignore the issue.

Nevertheless, it is very good practice to keep the source code as close to its pristine form as possible. Don�t allow failed experiments, along with their possible side effects, to accumulate over time. Keeping things neat also helps ensure that the code remains easy (or at least, no harder) to understand and will help ensure that you don�t check in unintended changes when you eventually come to fixing the problem.

Source of Information : Paul Butcher - Debug it Find repair and prevent bugs 

What Are the Key Insights of Refactoring?

Many people�s reaction to refactoring when first exposed to it is �so what?� This is just �tidying up� the code, something that programmers have been doing for almost as long as programmers have existed. Certainly, to some extent all that Martin Fowler did when he published Refactoring was to catalog techniques that developers have been using for years.

But there�s more to refactoring than just a catalog of useful techniques. It relies on Fowler�s two key insights:

� Modifying existing code can be carried out safely only with the safety net of a comprehensive suite of unit tests.

� We should never attempt to refactor the code at the same time as modifying its behavior.

In other words, you can modify the behavior of the code, or you can refactor it. You should never attempt to do both at the same time.

Upon reflection, it�s easy to see why this is the case. Imagine that you attempt to modify both the structure of your code and its functionality at the same time, and after doing so one of your tests fails. This might indicate that you made a mistake when modifying its structure. Or it might be an expected result of the change in functionality. It�s difficult, however, to be sure which. The more complicated the change in functionality or structure, the harder it is to be certain.

By doing only one or the other, you avoid this issue entirely and can forge ahead with potentially far-reaching refactorings involving dramatic changes to the code with confidence.

Source of Information : Paul Butcher - Debug it Find repair and prevent bugs 

Should I Leave My Logging in the Code?

Some topics are guaranteed to create an argument among developers, and logging is one of them. If you�ve added logging to the code to help while tracking down a problem, it�s tempting to leave this instrumentation in place so that you can find the problem again quickly if it happens again. This is especially true if you�re using a logging framework that allows it to be enabled and disabled easily. What�s not to like?

So, why the controversy? Detractors will tell you the following:

� Logging obscures the code, making it difficult to see the wood for the trees.

� Logging can suffer from the same problems as comments�as the code evolves, often the logging isn�t updated to match, meaning that you can�t trust what it says and making it worse than useless.

� No matter how much logging you add, it�s never what you need. The next time you find yourself debugging in that area, you�ll just have to add more, and if you leave it in the code when you�re done, you just exacerbate the first two problems.

As with most disputes of this nature, the answer is to be pragmatic. Logging is a useful tool, but it can be overused. Consider implementing permanent logging if you believe that it will add value, but be disciplined about how you do so. Make sure that your logging is up-to-date and agrees with the code and that you don�t add it for its own sake.

As a general rule, the most useful logging is at the highest (strategic) level�a record of what happened, such as the access log generated by an HTTP server, for example. Lower level, more tactical logging can be of questionable long-term value, so make sure you know what it�s giving you before you decide to add it.

If you find that logging is getting in the way but you don�t want to lose its benefits, you might want to look at aspect-oriented programming, which may give you a way to separate it from the main body of the code (a good reference is Ramnivas Laddad. AspectJ in Action: Practical Aspect-Oriented Programming. Manning Publications Co., 2003.)

Source of Information : Paul Butcher - Debug it Find repair and prevent bugs

Debugging Is More Than �Making the Bug Go Away�

Ask an inexperienced programmer to define debugging, and they might answer that it is �finding a fix.� In fact, that is only one of several goals, and not even the most important of them.

Effective debugging requires that we take these steps:
1. Work out why the software is behaving unexpectedly.

2. Fix the problem.

3. Avoid breaking anything else.

4. Maintain or improve the overall quality (readability, architecture, test coverage, performance, and so on) of the code.

5. Ensure that the same problem does not occur elsewhere and cannot occur again.

Of these, by far the most important is the first�identifying the root cause of the problem is the cornerstone upon which everything else depends.

Understanding Is Everything
Inexperienced developers (and sometimes, unfortunately, those of us who should know better) often skip diagnosis altogether. Instead, they immediately implement what they think might be a fix. If they�re lucky, it won�t work, and all they will have done is waste their time. The real danger comes if it works, or seems to work, because now they�ve made a change to the source that they don�t really understand. It might fix the bug, but there is a real chance that in reality it is only masking the true underlying cause. Worse, there is a good chance that this kind of change will introduce regressions�breaking something that used to work correctly beforehand.

Wasted Time and Effort
Some years ago, I found myself working in a team containing a number of very experienced and talented developers. Most of their experience was with UNIX, but when I joined the team, they were in the late stages of porting the software to Windows. One of the bugs found during the port was a performance issue when running many threads simultaneously. Some threads were being starved, while others were running just fine.

Given that everything worked just fine under UNIX, the problem was clearly broken threading in Windows, so the decision was made to implement a custom thread scheduling system and avoid using that provided by the operating system. This would be a lot of work, obviously, but quite within the capabilities of a team of this caliber.

I joined the team when they were some way into the implementation, and sure enough, threads were no longer suffering from starvation. But thread scheduling is subtle, and they were still working through a number of issues that had been caused by the change (not least of which was that the changes had slowed the whole system down somewhat).

I was intrigued by this bug, because I�d previously experienced no problems with Windows� threading. A little investigation demonstrated that the performance issue was caused by the fact that Windows implements a dynamic thread priority boost. The bug could be fixed by disabling this with a single line of code (a call to SetThreadPriorityBoost( )).

The moral? The team had decided that Windows� threads were broken without really investigating the behavior they were seeing. In part, this might have been a cultural issue�Windows doesn�t have a good reputation among UNIX hackers. Nevertheless, if they had taken the time to identify the root cause, they would have saved themselves a great deal of work and avoided introducing complications that made the system both less efficient and more error-prone.

Without first understanding the true root cause of the bug, we are outside the realms of software engineering and delving instead into voodoo programming or programming by coincidence.

Source of Information : Paul Butcher - Debug it Find repair and prevent bugs