One Free Lesson in Every Bottle

I just got off the telephone with Benefits. I think this is the out-sourced HR benefits call center, but I don’t really know. Anyway, I was checking the premium amounts for my coverage. Next year I switch to the retiree plan. I hear the retiree plan is significantly different–more expensive–from the employee benefits plan. So, all the paperwork I’ve gotten so far says my dental premium will be the same as when I was working. I called benefits to verify this. The agent said that he could not tell what the amount was until after I enrolled. What good is that I asked him. Well, that’s all he can do. Then he offered that they can work with me afterward to fix it once we knew what the actual premium amount was.
Now, there is a lesson here for analysts and programmers (hence the entry).
But first another story. I think I got this from a former project lead. A tape drive kept failing. The IBM field engineer would come in and fix it. And then it would fail again. He could fix it each time, but he could tell it would keep failing and he couldn’t get at the root cause. So eventually he covered the damn thing with a blanket and let it destroy itself so he could just replace it.
It sounds like the benefits agent was in the same boat. The system isn’t designed with enough flexibility to do his job so he has to let it fail completely so he can work from the ashes. Unfortunately, software systems are often like this. Many software systems are not fault tolerant in that (1) the point of failure [eg the error message] has little or nothing to do with the source of the failure, (2) systems tend to fail catastrophically because they aren’t designed to fail softly [An example: steam engines are all built with safety values to vent steam pressure rather than let the whole boiler blow.], (3) systems tend to fall off a cliff once they go beyond their concept of reality [example: buffer overflow errors occur because the programer didn't allow that more data may come in that he expected.].
A side effect though, as in my benefits case, is that the system may fail but you may not realize it because systems are hard to monitor. In this case, I need to review the output after ward and then start making noises with benefits to force a correction. It would be something that the system, or benefits, will be able to detect on their own. Story: I took over a tape library system. There was a bug in the code that deleted one entry each time it ran. It ran twice a day. There were hundreds of tapes in the library. The system had been running for several years before a clerk came over and said there was something wrong with the inventory listing. I looked at the output. The report was blank. So I looked into the matter, found the bug. I asked them, didn’t they notice that the list kept getting shorter and shorter? Why did they wait until the report was COMPLETELY empty? They just shrugged. And that is a problem with insidious computer errors: people defer to the computer, even when they know they shouldn’t. Even when they know computers are filled with bugs.
Now some users get in the habit of just jamming changes to the data. This is a suboptimal fix. Benefits may be doing something similar.
But what is the lesson? There are probably several, but I do teach this: Always sit with your (real) users after the system is in use to watch how they use it. I’ve always found that clerks and operators use features of a system different from as intended. I learn (1) they often use the feature in dangerous ways that need to be guarded against, (2) they need to do things that need new features to make the software safer and make their jobs easier, (3) there are aspects to a feature that I need to be careful about changing when the time comes to modify that part of the system because I do know that they use it in ways differently from what the programmers think. You never get this information from change requests. You have to be there. And you have to be there enough that you aren’t a stranger or the Enemy.
Another lesson, is that literally everything and anything has something to teach you about analysis.