Managing Code and Teams for Cross-Platform Software Projects

In this second part of a four-part series on software configuration management, you’ll learn how to manage the complexity innherent in building cross-platform software, especially with large teams. This article is excerpted from chapter three of the book Cross-Platform Development in C++, written by Syd Logan (Addison-Wesley; ISBN: 032124642X).

Item 12: Set Up a Tinderbox

Tinderbox is a tool that was developed initially at Netscape, but that is now open source software maintained by the Mozilla project. Tinderbox is designed to manage the complexity one encounters when developing software, especially in terms of large-scale cross-platform software that involves a widely distributed team of developers. Tinderbox is particularly useful in cross-platform projects, as you will see. Coupled with a system known as bonsai, the goals of Tinderbox are fairly simple:

  1. Communicate any and all changes made over time to the source code repository to the entire development team, in a centralized location, as soon as the changes have been made.
  2. Communicate the overall health of the repository by continually pulling and building the source code on each supported platform. For each pull and build cycle, a pass/fail status is reported to a centralized location. This allows developers to determine when they should update their local trees to avoid pulling source code that will not build (or run) correctly.
  3. Combining the above, Tinderbox can be used to assign accountability of the health of the tree to specific individuals and/or changes to the repository. Knowing this information helps get problems solved as quickly and accurately as possible.

Basically, Tinderbox is a group of machines that continually pull and build the contents of a CVS repository (see Item 13), and a server that retrieves and reports the status of these builds on a Web server that everyone in the organization can monitor. Tinderbox is currently supported as three versions. Version 1, perhaps the most widely used, was developed by Netscape/Mozilla, and is still in use by Tinderbox 2.0 is a rewrite of version 1, providing essentially the same feature set. The goal of Tinderbox 2.0 was to essentially clean up the implementation of version 1. Both Tinderbox 1 and Tinderbox 2 are available from Tinderbox 3 is a more recent version, available as a tarball from John Keiser, an ex-Netscape developer. Tinderbox 3 adds a number of desirable features, and strives to make Tinderbox easier to set up and administer.

Figure 3-7 illustrates the Web page displayed by a Tinderbox server. (You can access a large number of live Tinderboxen by visiting

Figure 3-7. Tinderbox

The Tinderbox in Figure 3-7 illustrates the state of Mozilla’s Seamonkey reporting the health of some of the “port” platforms that are defined by the Mozilla project. (Seamonkey was the code name used by Netscape/Mozilla during the development of the Mozilla browser suite. A port is a platform that is not considered to be tier-1 by Mozilla.)

The use of Tinderbox is pervasive in the software development community. Not only is it used by, but by other open source projects (for example, OSDL) and in commercial development (AOL, for example). Tinderbox is particularly well-suited to cross-platform development, as we discuss later.

The Tinderbox Web page consists of a table viewed as a series of mutually exclusive columns that are organized from left to right. In the first column (Build Time), each row contains a timestamp that can be used to identify the time associated with events that are represented in the remaining columns of the table. The second column identifies each check-in made by developers; the time of these check-ins can easily be determined by looking at the corresponding row in the Build Time column. The remaining columns each represent a specific platform that is being reported on by the Tinderbox. (An organization may have several Tinderboxen, each reporting a specific group of builds. You can see an example of this by visiting

Any given column represents a build machine, and a platform, and contains a series of colored boxes. Green boxes indicate a successful build of the repository for that platform on that machine. Conversely, a red box indicates a failed build, and a yellow box indicates a build that is currently in the process of being produced. Furthermore, the lower edge of any of these boxes represents the start of a pull and build cycle, and the upper edge represents the time of completion. The time corresponding to both of these events, for a particular colored box, can be determined by looking at the timestamp at the same row in column one as the upper or lower edge of that box. For example, the uppermost failed build in Figure 3-7 (Linux balsa Dep (static), represented by column four of the table) was started at about 13:07, and failed about five minutes later, at about 13:12.

Let’s take a closer look at this failed build, and see what we can infer about it. It is clear that the check-in by sspitzer at 13:09:41 did not result in the failure of the build for two reasons. First of all, the Linux balsa Dep build was already burning prior to sspitzer’s check-in. (See the red box in the same column that completed around 12:00, and also notice how the lower edge of the uppermost red box is lower than the entry for sspitzer’s check-in.) Another piece of evidence that sspitzer is not the cause of the problem is that each of the other Linux platform builds are green. (Generally, one finds that Linux builds of Mozilla are generally are either all red at the same time, or all green.) Finally, and perhaps most important, we can see that the build was previously red at noon (12:00), and had not gone through a green cycle since then. (Gray portions of the Tinderbox indicate no build was in progress, or the progress of a build was not reported to the Tinderbox server.)

For largely the same reasons cited previously for sspitzer, we can also infer that mozilla.mano is not to blame for the redness of the build, either. The tree was already red prior to his or her check-in, and the other Linux builds were not affected.

Let’s say I am not entirely sure that sspitzer is not to blame, and want to take a closer look at what the exact cause of the broken build might be. There are several other facilities provided by Tinderbox that you can use to drill down for further information. The L (or L1) link inside of the red box can be used to obtain a log of the build; this log contains compiler and perhaps linker output that should identify what caused the build to break. Clicking the L1 link gives the result shown in Figure 3-8.

Figure 3-8. L1 link output

Clicking View Brief Log results in Figure 3-9, which indicates a problem building, the shared library that contains the implementation of the Mozilla Spidermonkey JavaScript engine (which is used in Trixul; see Chapter 9, “Developing a Cross-Platform GUI Toolkit in C++”).

Figure 3-9. View brief log output

The other major source of input that I would want to consider is an understanding of what portion of the tree was impacted by sspitzer. If the check-in he made does not correlate to the error messages displayed in the log (Figure 3-9), I can eliminate him from the “blame” list and focus my search elsewhere. I can do this in two ways. First, by clicking the C link in the red box; this will display a list of checkins that were made prior to the start of the build. It is these checkins that likely would be the cause of any state change in the build (that is, going from red to green, or from green to red, which is not the case here since the tree was already red at the 12:00 hour). Figure 3-10 shows the result of clicking the C link.

Figure 3-10. List of checkins

This result clearly confirms that sspitzer is not on the blame list—and it also shows that the problem is not related to a check-in by mozilla.mano, because this change was made to a Cascading Style Sheets (CSS) source file that would have no impact on the JavaScript engine.

The final technique to eliminate sspitzer from the blame list (short of sending him an e-mail and asking him whether his check-in caused the build failure) would be to click the sspitzer link in column one and see what changes he made. Doing so gives us Figure 3-11.

Once again, these are changes to Extensible Markup Language (XML) markup and JavaScript sources that have no bearing whatsoever on the stability of the JavaScript engine, the failure of which was clearly the cause of the red tree.

Figure 3-11. List of checkins made by sspitzer

In terms of cross-platform development, it is best not to rely on a Tinderbox to find all of your portability problems. At Netscape, developers were required to ensure that any changes they were considering for checkin built and executed cleanly on each tier-1 platform, before submitting the changes to the repository (see Item 4). However, you can never be sure that all developers will follow this rule all the time, without exception. Because of this, having a Tinderbox monitor the health of the repository is an especially good idea. It isolates the problems in terms of change made, time of the change, and developer who made the change, focusing the area of investigation to a small fraction of the overall possibilities.

At Netscape, Tinderbox was a way of life (as it remains for Mozilla and many other development projects). The state of the tree was closely monitored, and acted as the focal point of development. If the tree was red, you could not checkin to the tree until it turned green. After you checked in your changes, you were added to a group called “the hook,” and as a member of the hook, you were required to watch the Tinderbox and ensure that the your changes built cleanly (that is, were green) for each platform affected. (Obviously, if the change was only made to, say, Mac-specific code, you were only obligated to see the Mac builds go green.) If the tree affected by your changes was green when you checked in, and then it went red in the first build that followed, then, as a member of the hook, you were required to help identify and fix the problem.

In addition to the hook, a sheriff was assigned by Netscape (or Mozilla) to monitor the overall state of the Tinderbox monitoring the tier-1 platforms, and to ensure that the builds all remained green. We eventually rotated the responsibility of sheriffing inside of Netscape among the various development teams, one day the responsibility would fall to members of the mail-news team, on another, it was the responsibility of the IM team, and so on. Should a problem arise, the sheriff had the power to close the tree to all check-ins (this was communicated by a simple message at the top of the Tinderbox page), which was done in an attempt to aid those trying to isolate the cause of problems. The sheriff also had the authority to contact anyone on the hook, usually by e-mail, but by phone if necessary, should a diagnosis of the problem indicate that the person being contacted was to blame for the tree going red. As sheriff, I recall numerous times calling people by phone who had left the tree burning and had gone home for the night. Not everyone was happy about the policy, but it did cause people to be more careful about their work.

In a nutshell, Tinderbox plays an important role in cross-platform development because it forces developers to confront issues that affect the portability of code being committed to the repository. Although a responsible developer will try to determine the impact of changes on other platforms before landing code in the tree, this is not always done. Tinderbox acts as a backstop, ensuring that nothing slips through the cracks. And when problems are detected, either because of a red tree or a failed QA test the next day, Tinderbox can be used as an aid in determining which changes had been made that might be the cause of the problem.

Getting a green build on all the platforms is, of course, not the end of the story. A green build does not ensure that cross-platform feature parity is met for the platform, for example. A developer could implement a cross-platform feature and check in a full implementation for, say, Mac OS X and only stub implementations for Linux and Windows, and Tinderbox will not make this fact evident. What Tinderbox does do a good job of is ensuring that code shared among platforms builds everywhere, and that no one platform can hijack the stability of the tree (that is, leave the repository in a state such that it builds cleanly for only a subset of the supported platforms). As such, Tinderbox helps ensure that when platforms are worked on at the same time, the cross-platform train is moving ahead at the same pace for all platforms involved. This in turn helps ensure that the organization will be able to release a product to the market for all the supported platforms at about the same point in time, which is something that we strived for at Netscape. Remember, however, that Tinderbox is just an aid—only by testing the resulting builds carefully can you confirm that a product or feature has achieved cross-platform parity.

{mospagebreak title=Item 13: Use CVS or Subversion to Manage Source Code}

An SCM system does exactly what the name implies; it helps you manage source code and related resources. To appreciate why SCM is important, consider for a moment what life without it might be like. Assume that you are a developer working on a new project that will consist of C++ sources files, a Makefile, and perhaps some resources such as icons or images. Obviously, these items must live somewhere, and so on day one of the project, you create a directory on your local desktop system where these files will live, and write a few hundred lines of code, storing the results in that directory. After a few weeks of hacking, you come up with version 0.1 of your creation, and after some light testing, you decide that your creation is ready to be pushed out onto the Web, with the hope of generating some user feedback from a small community of users.

After a few weeks, your e-mail inbox has accumulated several feature requests from users, and perhaps a dozen or so bug reports (in addition to a ton of spam because you gave out your e-mail address to the public). You get busy implementing some of these features and fixing the worst of the bugs, and after a few more weeks, you are ready to post version 0.2. This process repeats itself for a few months, and before you know it, you are shipping version 1.0 to an even wider audience.

The 1.0 release is a success, but isn’t without its share of problems. First off, users are beginning to report a nasty bug in a feature that was first introduced in version 0.8, and was working flawlessly until version 1.0 was released to the public. You are able to duplicate the bug in a release version of the 1.0 binary, but can’t duplicate it in a debugger. In an attempt to understand the problem, you pour over the code in search of clues; but after numerous hours of looking, you realize that you have no idea what might have caused this bug to surface. About the only way you can think of identifying the cause of the problem is to determine what specific changes you made to the codebase that might have led to the bug’s manifestation. However, all you have to work with is the 1.0 source code, and you have no way of identifying the changes you made between 0.9 and 1.0.

The second problem before you is a request. It turns out that you removed a feature that was present in version 1.0, and removing the feature has angered a lot of your users, who are clamoring for it to be reinstated in version 1.1. However, the code for this feature no longer exists, having been deleted from the source code long ago.

It is these two situations that, in my experience, make the use of a source code management system a necessity, cross-platform or not, no matter how large the project is or how many developers are involved. A good source code management system will allow you to re-create, in an instance, a snapshot of the source tree at some point in the past, either in terms of a specific date and time, or in terms of a specific release version. It will also help you to keep track of where and when changes to the source code have been made, so that you can go back and isolate specific changes related to a feature or a bug fix.

Had the developer used a source control management system, he or she could have retrieved versions of the source code starting at 0.9 and used this to isolate exactly what change(s) to the source caused the bug to first surface. And, to retrieve the source code for the feature that was removed in 1.0, the developer could have used the source code management system to retrieve the code associated with the feature, and undo its removal (or reengineer it back into the current version of the source code).

The benefits of a source code management system increase significantly as soon as multiple developers are assigned to a project. The main benefits from the point of view of a multideveloper project are accountability and consistency. To see how these benefits are realized, I need to describe in more detail how a source code management system works. Earlier, I described how a developer will typically manage a body of source code, in the absence of a source code management system, in a directory, which is usually created somewhere on the developer’s system where a single copy of the source is stored and edited. The use of a source code management system changes things dramatically, however. When using a source code management system, the source code for a project is maintained in something called a repository, which you can think of as being a database that stores the master copy of a project’s source code and related files. To work with the project source code, a developer retrieves a copy of the source code from the repository. Changes made to the local copy of the source code do not affect the repository. When the developer is done making changes to his or her copy of the source, he or she submits the changes to the source code management system, which will update the master copy maintained in the repository. It is important to realize that the repository records only the changes made to the file, instead of a complete copy of the latest version.

By storing only changes, you can easily retrieve earlier versions of files stored in the repository. The date of the change, the name or the ID of the developer who made the change, and any comments provided by the developer along with the change are all stored in the repository along with the change itself. The implications for developer accountability should be obvious; at any time, you can query the source code management system for a log of changes, when they were made, and by whom. This is a great help in locating the source of bugs and who may have caused them.

The ability to attribute a change in the repository to a developer, bug, or feature is directly affected by the granularity of check-ins made by developers on the project. Frequent, small changes to the repository will increase the ability of a developer to use the source control management system to identify and isolate change, and will also help ensure that other developers on the project gain access to the latest changes in a timely manner. A good rule of thumb is to limit the number of bugs fixed by a check-in to the repository to one (unless there are multiple, related bugs fixed by the same change).

So, now that you know the basic ideas being using an SCM, let’s talk briefly about the implications to portability. First off, using an SCM is not a magic pill that makes your project portable. Portability requires attention to a lot more than just a source code management system to happen. (If that were not the case, this book would not need to be written.) But, using a source code management system that is available on each of the platforms that your organization is supporting (or plans to support) is, in my view, a critical part of any successful cross-platform project. It does no one any good if only Windows developers are able to pull source code, but Linux and Macintosh developers are left without a solution, after all. Not only should the SCM software be available everywhere, it at least should support a “lowest common denominator” user interface that behaves the same on all platforms, and to me, that means that the user interface needs to be command line based (both CVS and Subversion [SVN] support a command-line interface).

Because cross-platform availability and a common user interface are requirements, there are only two choices for an SCM system that I can see at the time of writing this book: CVS and SVN. At Netscape, and at countless other places (open source or not), CVS is the SCM of choice. It has stood the test of time, and is capable. It has been ported nearly everywhere, and its user interface is command line based. A very close cousin of CVS is SVN. After using SVN in a professional project, I have come to the conclusion that for the programmers using it, SVN is quite similar to CVS in terms of how one approaches it and the commands that it offers, so either would be a good choice. (It is not without its quirks, however.) In this book, when I refer to an SCM, I am referring to CVS, but I could have easily said the same thing about SVN.

Besides providing a location from which Tinderbox can pull sources (see Item 12) and its support for Windows, Mac OS X, and Linux, perhaps the most important contribution of CVS to cross-platform development is its ability to create diff (or patch) files. The implications to cross-platform development of patch files are detailed in Item 14; in the following paragraphs, I describe what a patch file is and how CVS can be used to create a patch file.

A diff file, or a patch, is created by executing the cvs diff command. For example, assume I have added a method called GetAlignment() to a file named nsLabel.h in the Mozilla source tree. By typing cvs diff, I can easily identify the lines containing changes that I made:

$ cvs diff
cvs server: Diffing .
Index: nsLabel.h =================================================================== RCS file: /cvsroot/mozilla/widget/src/gtk/nsLabel.h,v retrieving revision 1.21
diff -r1.21 nsLabel.h
> NS_IMETHOD GetAlignment(nsLabelAlignment *aAlignment);
< GtkJustification GetNativeAlignment();

The preceding output tells us that a line was added around line 61 of the file nsLabel.h, and one was removed around line 71 of the file. I can take this output, mail it to others on my team, and ask them to review it for errors or comments before checking in the changes. I can also look at this patch and make sure that it contains only those changes that I intended to land in the repository. I can’t stress how important cvs diff is as a tool for identifying inadvertent check-ins before they are made.

With a lot of changes, the default output format that is shown here can be difficult to understand. A better output would show context lines, and make it more obvious which lines were added to the source, and which lines were deleted. The -u argument to cvs diff causes it to generate a “unified” diff, as follows:

$ cvs diff -u
cvs server: Diffing .
Index: nsLabel.h =================================================================== RCS file: /cvsroot/mozilla/widget/src/gtk/nsLabel.h,v retrieving revision 1.21
diff -u -r1.21 nsLabel.h
— nsLabel.h 28 Sep 2001 20:11:17 -0000 1.21
+++ nsLabel.h 1 Feb 2004 02:47:21 -0000
@@ -59,6 +59,7 @@
NS_IMETHOD SetLabel(const nsString &aText);
NS_IMETHOD GetLabel(nsString &aBuffer);
NS_IMETHOD SetAlignment(nsLabelAlignment aAlignment);
+ NS_IMETHOD GetAlignment(nsLabelAlignment *aAlignment);

NS_IMETHOD PreCreateWidget(nsWidgetInitData *aInitData);

@@ -68,7 +69,6 @@

NS_METHOD CreateNative(GtkObject *parentWindow);
– GtkJustification GetNativeAlignment();

nsLabelAlignment mAlignment;

The differences in the output are the inclusion of context lines before and after the affected lines, and the use of + and -to indicate lines that have been added, or removed, respectively, from the source. This format is generally much easier on everyone who must read the patch, and it is the format that I recommend you use. You can change the number of lines of context generated by cvs diff by appending a count after the –u argument. For example, to generate only one line of context, issue the following command:

$ cvs diff -u1
cvs server: Diffing .
Index: nsLabel.h =================================================================== RCS file: /cvsroot/mozilla/widget/src/gtk/nsLabel.h,v retrieving revision 1.21
diff -u -1 -r1.21 nsLabel.h
— nsLabel.h 28 Sep 2001 20:11:17 -0000 1.21
+++ nsLabel.h 1 Feb 2004 02:50:45 -0000
@@ -61,2 +61,3 @@
NS_IMETHOD SetAlignment(nsLabelAlignment aAlignment);
+ NS_IMETHOD GetAlignment(nsLabelAlignment *aAlignment);

@@ -70,3 +71,2 @@
NS_METHOD CreateNative(GtkObject *parentWindow);
-GtkJustification GetNativeAlignment();

Generally, you’ll want to generate somewhere between three or five lines of context for patches of moderate complexity. I use -u3 almost religiously, and it is the default number of lines for svn diff (which does context diffs by default, too). However, don’t be surprised if developers working with your patch files ask for more lines of context.

Please check back for the continuation of this article.

[gp-comments width="770" linklove="off" ]

chat sex hikayeleri Ensest hikaye