Lessons Learned Managing Software Development in Startups

This article outlines the process of setting up a source code management system, evaluating the pros and cons of Razor and CVS, and compares BugTracker and GNATS.

Introduction

I recently discovered that the original meaning of the verb “to manage” was closer in meaning to “to cope with” rather than “to direct”, as in “my house burned down and I lost my job but I managed.” Having managed software development in a startup for the past few years, I sense a great deal of truth in that description!

This is the article I wish I had read four years ago when we started our software company. While a lot has been written about the business side of startups, there is not much about the technology aspects. In this article, I will describe how the software development process evolved in my company and then present some lessons learned.

Evolution of the Software Development Process at My Company

My company works in the area of  wireless LAN equipment, specifically in the areas of security and mobility. Our products are quite complex and are aimed at corporate IT departments. As we shall see, this has had a significant impact on the way we have managed our development activities. If your company’s products are aimed at individuals or the SOHO market, it is likely that your development processes will be different from ours.

Initially, all of our developers, including myself, came from a research background. While all of us were experienced developers, not all of us had worked in formally structured software development projects. Several developers had significant Linux experience, so Linux was chosen as the platform for our product.  Since we were VC funded, we had the luxury of being able to spend some time exploring alternatives before having to begin product development in earnest.

{mospagebreak title=Source Code Management}

In the first few weeks of my company’s existence, I was given the task of setting up a source code management system. After evaluating several alternatives, I decided to go with Razor from Visible systems. The main reasons were the modest cost per seat and the relatively good integration of bug tracking with code management. A trial version was available and we tried it out for a few weeks. The results of the trial were encouraging but ultimately Razor did not go over well with our developers when put into production. The main problems at that time (mid 2000) were:

  • The CLI client tool was hard to use (compared to CVS). Since a number of our users had worked with CVS, this was an important objection.

  • The Linux GUI client was a little clunky to use.

  • It turned out that the client and server had to be mutually reachable over the network. From what I could make out, this was necessary to allow the GUI to change file status in real time. Since our developers sometimes needed to access the server through a NAT (Network Address Translation) gateway, this turned out to be a big problem.

  • One more administration annoyance was that the Razor server was locked to a particular machine. Since our IT infrastructure was far from stable, this caused a lot of problems since each transfer of the license had to be approved by Visible.

Due to these problems, and also general resistance from the developers, we switched to using CVS for source code management. Since we were still in the experimental phase, there was no urgency in setting up a bug tracking system (which we lost by abandoning Razor).

Our CVS server is configured so that every CVS commit resulted in an email containing details of the commit being sent to all developers. Developers generally save commit emails in a folder in their mail reader and use the built-in search facility as a sort of a database to search for commit data.

In addition to each developer’s private store of commit messages, we also use a tool called Bonsai that saves CVS commit messages in a relational database. A convenient web UI is available that allows one to perform complex queries against this database. Bonsai is invaluable at release time and when trying to determine whether a bug has actually been fixed.

There are many problems with CVS and these have been adequately described elsewhere, but the main issue that one needs to be aware of is a cultural one. CVS is great as long as its used in the way that its supposed to be used. In general CVS works best where the philosophy of development is along the lines of “it’s better to ask for permission than to get permission”. CVS is not a tool that works well with a “command and control” approach to managing development.  Despite this caveat, there will be times when it is necessary to lock a branch —  for example, when finalizing a release. This kind of locking can be added by using server side wrapper scripts although we have not done this so far.

The other big problem with CVS in the context of commercial development is the lack of any means to group commits together (i.e. changesets). When finalizing a release, for example, it’s important for the release manager to know exactly which bugs have been fixed and which new features have added. This problem can be overcome by having developers include a reference to a bug or feature in the commit comments. If this convention is followed diligently, it is possible to track down all commits for a particular bug or feature with the help of Bonsai. However, mistakes often happen — usually at the worst possible time. Developers may neglect to include a bug reference or may reference the wrong ones.

If we were starting today, and decided to use open source tools,  we would definitely use a better tool such as subversion. We have considered switching to subversion right away but have delayed a decision since the advantages do not appear to be compelling for now.

{mospagebreak title=Tracking Bugs}

When product development finally got underway, the need for a bug tracking system became clear. Since we were using Lotus Notes for groupware applications at that time, we decided to use BugTracker(since discontinued)  from the Eden Corporation, a Notes-based bug tracking system.

We used BugTracker through the first two major release of our product (over a period of more than a year). Several problems became apparent but we stuck with BugTracker, mostly due to inertia. The problems included:

  • The obvious lack of integration between CVS and BugTracker: A lot of manual labor was necessary to keep track of what was actually going into each release. Just because a feature or a bug fix was listed as complete did not mean that the code had been checked in.  We have one.

  • Referring to a bug became awkward since the developers (all except one) did not use Notes for email. If a developer referred to a bug, the reader would have to launch the Notes client and navigate to the bug report. Had we been a pure Notes shop, this would not have been a problem since Bugs could have been referenced with DocLinks which is a means of linking one Notes document with another. Now, a web front-end for Notes called Domino is available but its slow and the URLs are enormous. Also not all of the BugTracker facilities were available via Notes.

  • Since developers used Linux for development, it became awkward to have to maintain a separate Windows machine just to run the Notes client (although VMWare does help).

  • Finally, customizing BugTracker was hard since no one had the expertise in-house.
When we had some free time between releases, we decided to switch from BugTracker to GNATS, a popular open source bug tracking system from the OSF. Overall, GNATS has been fairly good and GNATS has both a CLI and a convenient web interface. This has proved very important since the web interface makes it much easier to refer to bugs in email and documents.  Another good feature is the tight integration with email. By including the proper address, an email thread about a bug or feature can be captured in the audit trail.

GNATS, like all bug tracking tools, needs to to be customized but this turned out to be relatively straightforward since the source code was available and we have good Perl expertise in-house. In addition to tracking bugs, we also use GNATS to keep track of feature requests — as everyone is aware, the difference between a feature and bug is small indeed! GNATS is a flexible tool and scales very well. At the current time, there are several thousand bugs and features recorded in our GNATS database.

One useful local modification that we have made is to associate a Twiki document with each bug report. We use Twiki for general project management functions.  This has been useful in a number of ways, the most important of which is that it allows to add well-structured design and background information with each bug report.

The biggest problems that we have had with GNATS are:
  • Its very difficult to track bugs that apply to multiple releases. A bug in GNATS has a single “state” attribute. What we need is a way to track the state of the bug for each release. Thus for example, a bug may be in the process of getting fixed for one release while a completed fix may already be ready for another. There are two (unsatisfactory) ways to get around this problem:

    • Add an additional release-specific attribute for each release branch. Such attributes could have names such as state_3.0. This is ultimately not scalable however.

    • Use a “clone” mechanism. This is the approach that we have adopted. If a bug applies to more than one release branch, we create one bug in GNATS for the original branch in which the bug was found. We then “clone” the bug for other release branches to which it applies. This does allow us to separately track the bug state for each branch but has the disadvantage of greatly proliferating bug reports.

  • While the audit trail is a good thing to have, it’s not searchable. We are not able to perform queries of the type: “who wrote the code for this bug fix”. This is especially annoying when a bug has been fixed by several developers.

GNATS is not a bad bug tracking system overall but, as is the case with CVS, we would probably not use GNATS if we had to do it again.  GNATS works best with scenarios where there are no long-levied maintenance branches. It’s clear that a proper database backed bug tracking system is needed in such cases.

{mospagebreak title=Release Management and Lessons Learned}

We use the standard approach to creating releases:

  • New code is added to the trunk.

  • When it’s time to create a new major release, we tag the trunk. The major release is always compiled directly from the trunk. Major releases have numbers such as 1.0, 2.0 etc.

  • Bug fix releases are numbered 1.1, 1.2 etc and are released from bug fix branches.

Due to the nature of our customers, our bug fix branches tend to be very long lived. IT organizations are generally conservative in upgrading their equipment. If the system works, all they want are bug fixes. If your product is aimed at individuals or SOHO rather than IT departments, bug fix branches are not likely to be as long lived. It may well be possible to persuade your customers to accept new features along with bug fixes. As previously mentioned, tracking the state of a bug fix in multiple branches is still a problem for us.

Lessons Learned

The main things I have learned are:

  1. If you have the money and depending on the complexity of your product and type of customers, you should consider an industrial-strength product management suite (such as that from Rational – now IBM) right from day one.  Although such tools are expensive (several thousand dollars per seat), ultimately you will wind up paying this amount one way or another. In my company we were fortunate to have talented developers who could customize open source tools such as CVS and GNATS. However, the cost of making these modifications in terms of developer time has been substantial.

  2. Tight integration of bug tracking with source code management is highly desirable. By using loosely integrated tools, release managers and developers have also had to spend a considerable amount of time tracking down the state of various features and bug fixes. On a number of occasions, this high degree of manual labor has hampered our ability to get a release out in a timely manner.
Copyright © 2004 Rangachari Anand, All Rights Reserved.
[gp-comments width="770" linklove="off" ]

antalya escort bayan antalya escort bayan Antalya escort diyarbakir escort