Are software development failures unusual?

It’s often written that a significant portion of all software development projects fail to some extent or another. The authors then usually go on to sell their solution to the problem. I remember sitting in a class on systems analysis–the old fashioned mainframe kind–I forget what the instructor was talking about but at some point he said that the great works of art did not require maintenance and he cited works like Tolstoy’s War and Peace and Shakespeare’s Romeo and Juliet. This in comparison to software which requires constant maintenance.Well, even at the time I thought that wasn’t true. How many of us can read Chaucer’s Canterbury Tales anymore? Or even Shakespeare for that matter? At least not without helpful notes on the side. And of the great works of art, how many are there? Of any list of, say, hundred greatest novels, how many have you read? How many do you want to attempt in your life time?Then consider how many millions of novels have been written. Tens of thousands of novels are written every year. Most aren’t worth reading once let alone worth considering good much less great.I heard, my memory isn’t so good, that a reporter once asked a science fiction writer why so much science fiction was crap. And the writer responded that ninety percent of everything is crap. I think the figure is probably closer to 99.9%, particuarly when we’re talking about creative efforts.And software is creative. So is it any wonder what most software is crap?But in a world where real money is involved, is this tolerable? And what can we do mentally to prepare for this situation. And what can we do organizationally?

Checklists

I’m reading Atul Gawande’s “The Checklist” in the December 10th New Yorker magazine.  I’m always behind in my reading.  Gawande writes about medicine as she is practiced, but I usually find his articles fascinating as a perspective on computer programming.  Is programming like medicine?  Like law?  Like movie making?  Who knows?  Maybe bits of each.  The point of the article, at the risk of vast over simplification, is that checklists help to reduce errors. (Favorite exclamation here.) Well, I usually make a checklist for complex processes.  But, and two crucial points, only for complex processes involving many steps and the checklist is always ad hoc.  As a planning tool.  I don’t use checklists for recurring processes.  As a programmer I would argue that each process is unique and there aren’t recurring processes that I need a checklist for.  Which probably points to the arrogance of programmers and our illusion that what we do is somehow ‘creative’.Checklists are probably a good idea in the practice of programming.  Note to managers: making up a checklist just to have a checklist for every occasion is a BAD IDEA.  

My Leopard Upgrade

Well, I finally upgraded my Mac to Leopard.  As a highly trained professional I’ve learned to wait awhile before upgrading operating systems.  Actually, I never upgrade, I let someone else do it.  Or buy a new machine with bright shiny new software.  I had been experiencing creeping crud on my Mac.  Quicktime wouldn’t run.  I couldn’t even download and install a new version.  The system would crash in the middle.  Plus I was back on OS-X 10.3 (Panther). So it was about time to start with a fresh new OS.  Being a highly trained professional I have a compilation somewhere of every logon and password I’ve ever set up, I bought a big honking disk drive for backups.  I was all set.  Version 10.5.1 came out.  It would be nicer if I got that version from the get go, but I figured I would probably have to download the patch file.  Meanwhile, my brother had already upgraded his Mac to 10.5.  The only problem he mentioned was that his WiFi network got sluggish and he had to periodically spank his Mac.Now, I know that am I anal retentive and make life more difficult for myself.   And I ask myself, does everyone make such a big deal of their upgrade or do they just pop that DVD into the drive and go?   And I ask myself, does taking all these precautions somehow amuse the Gods of Upgrades and they make everything go wrong when I try an upgrade? Well.First, I was too cheap to get a FireWire drive so I can make a bootable drive.  [I've read that USB drives are not bootable, at least not with a PowerPC mac, and only some FireWire drives are bootable as well.  Looking around the stores, it seemed like most of the FireWire drives were huge mothers intended for graphic artists and Hollywood studios.  Since I use a computer mainly for word processing all my data still fits on a single thumb drive.  Even using TimeMachine it would take decades for me to use a half-terabyte drive.  So I bought this cheapo USB drive.]  But for temporary backups and eventually TimeMachine, I figured a USB drive should be sufficient.  After all, things don’t go wrong for most people, right?I’m ready to get started.  My internet connection isn’t working right.  I’ve had this problem before, they’ve always had to come out and switch premises wires. Nowadays you need to get on the internet during the upgrade process, so I’m not about to try to upgrade until this is resolved.  And it’s the weekend.  Do they come out on weekends?  I go to bed. Surprise, the connection is fine the next morning.So I make a backup.  The backup runs for hours.  Seems to have gotten stuck around 25 GB.   One resource says to reformat and write zeros to the drive to make sure it maps out any bad sectors.  OK.  I try that.  That takes hours–I just let run all night and it’s done by monring.  Try the backup again.  Run it running  with the system monitor so I can ‘watch’.   Around 25GB the target drive goes bonkers and abruptly shows 450GB used.   So, I’m thinking, I know that my Mac has gotten squirrelly lately, remembering my problems with Quicktime.  Is that my main drive?   I run hardware diagnostics.  It says the disk drive is OK, but one of my memory banks is bad.  Run out and get memory.  Trying to open the Mac, I nearly strip off the screws because I don’t realize the screws don’t come out, they just loosen.  Which bank to change?  I try the first one.  And, lucky me, that’s the one.  [The diagnostic testing memory also runs about an hour.   I've gone through my stack of magazines already. Maybe I should start War and Peace.]Now, pause. should I replace my hard disk as well, just in case it, too, is squirrelly?  Well, I can’t buy one tonight anyway.  Run the backup again.  Surprise, the backup works, too.  I take two more backups of just my data files.  I have no separate backups of my iTunes library or stuff like that.  I got the Leopard disk late Friday.  This is now Sunday evening.  Big thrill: I pop in the Leopard disk.  I opt for the middle installation option: backup the old OS and do a clean install of the OS only.  Partway through it chokes and tells me I have to do a reformat and completely new installation.  Hm.  Well, I have all those backups.  I have tons of log notes on what I’ve done to this system.  I’ve saved all my emails where I’ve gotten software licenses and I’ve forwarded them to myself JUST IN CASE I might need them again.  I can do a full install.The install fails again.  Says it can’t write a package to a particular directory.  Damn damn damn!Maybe the drive is bad after all.   Well, I can’t get a new drive tonight.  I just try again.  And it completes.  Wow.  Giant exhale of relief.  Now I just have to reload all my applications and data redo all my accounts, set up parameters and permissions and preferences and I’m back in business. And how do I do that?  With Windows, I would reload from the original installation disks.  All those registry settings, DLLs, hidden directories even. But the Mac is different, I hope, and I start to just drag stuff from my backup of Applications to the new Applications.  The apps even work.  I drag over my documents.  I need to reconfigure my internet connection.   Going through the menus is confusing.  Am I just too tired?  Finally I run the set up disk I got from the IP how many years ago.  Will it still work?  Is it compatible with Leopard.  Wow, that worked too.  I have the internet.   I go to get my mail (I don’t have Mail working yet, but I had sent to an internet-based account for just that reason).  I get some stuff.  I got to work on Mail.  I can’t figure out how to restore my old downloaded emails from my backups.  Oh well.  And for some reason Mail retrieves all the old mail from the server and I cannot, just cannot, delete them.  I try all sorts of stuff.  Finally I find a posting that says you just “rebuild” the mailbox.  And that works.  Then I got back to the other mail system.  And all my emails there–with the license keys–are gone.  WTF?  Coincidence?  Probably.  [In truth, I can find those emails now.  I must have been punchy and was looking at the wrong mailbox.] By rooting around in various directories, I was able to find all of my licenses except TubeSock.  I was not able to carry forward my Address Book, but I was able to recreate it by converting my Palm addresses to vCards and importing that. [You may want to export your address book before converting.  You may want to archive your mailboxes before hand too.]So three days later I’m pretty back to normal.  I did not attempt to reinstall MS Office.  I don’t use Office.  Although I am a Microsoft bigot, I do think that two of the best apps ever are Excel and Access.  Access doesn’t run on the Mac and Excel for the Mac just looks weird. Excel I used primarily as a database because it had outlining, multiple columns and notes.  All of which I get in OmniOutliner.  I live in OmniOutliner.So.  If you haven’t upgraded to Leopard, I would recommend you do these things:1) Be optimistic that all will go well, but prepare just in case.2) Run the hardware diagnostics (from the original OS disks) and fix anything that needs fixing.3) Export your Address book to vCards4) Archive your Mail mailboxes5) Write down all your system and app Preferences (well, the important ones anyway)6) As you set up various accounts, write them down (too late now, but next time).  7) Backup everything.  If you want to be extra careful, backup it up twice–to different drives–just in case the first drive has itself some problems.  If you don’t have a lot of time, take the extra time to make a bootable backup.  Then if the installation goes seriously wrong, you can at least keep going from the backup.8) Now run the upgrade.  Notice how this is way down at step 8.   Be prepared mentally to take your time and not rush. There are no shortcuts–nobody does more than they have to, so there are no shortcuts to take.9) If your install went OK and you did not have to wipe out every thing , you may be done.  I wouldn’t know, I wasn’t that lucky.10) Check your apps and data and make it looks OK.   Look into upgrading your applications.  11) If things did not go smoothly or you wiped your system:  Slowly restore things one at a time and test them out.  If you need to find licenses and stuff look in  your backup in Applications, Library, System/Library, ‘YourMacintoshHD’/Users/’yourlogon’/Library.  It won’t be something obvious and you could mess your system by copying the wrong things so leave this for your nerd friend or be careful and copy one thing at a time.  Most of all, do not panic and be methodical, take your time.12) Even after everything looks fine, keep your backups around for awhile. So, was the upgrade worth it for me?  I donno yet.  Quicktime works again.  That’s a big double plus.  I needed to upgrade because some apps just didn’t run under Panther, my last OS.  But I haven’t tried them yet.   I was also interested in Spotlight and TimeMachine, neither of which I’ve explored yet.  I did not think I would use Spaces.  I had something similar when I was a Sun SysAdmin but never found it useful.  But I’m finding Spaces useful now. I put Mail and my Calendar in one, OmniOutliner in another, and everything else in a third and it seems to reduce screen clutter dramatically without any time penalty switching amongst them.  At least I’m glad that’s over.  I hope it’s over.