“Theres a bunch of research I’ve come across in this work, where people say that the social context is a 78-80 per cent determinant of performance; individual abilities are 10 per cent. So why do we make this mistake? Because we spend all of these years in higher education being trained that its about individual abilities.”

via What went wrong with the Hubble Space Telescope and what managers can learn from it – leadership, collaboration – IT Services – Techworld.

Former NASA Director Charlie Pellerin is now a consultant on how to prevent failures on teams charged with carrying out large scale, technically impossible projects. He’s had to face two biggies while at NASA the Challenger explosion and subsequent to that and more directly the Hubble Space Telescope mirror failure. In that time he tried to really look at the source of the failures rather than just let the investigative committees do all the work. And what he’s decided is that culture is a bigger part of the chain of failure than technical ability.

Which leads me to ask the question how often does this happen in other circumstances as well? I’ve seen the PBS NOVA program on the 747 runway collision in Tenerife back in 1977. At that time the KLM Airliner more or less start taking off before the Pan American 747 had taxied off of the runway. In spite of all the protocols and controls in place to manage planes on the ground the captain of the KLM 747 made the decision to take-off not once, but TWICE! The first time it happened his co-pilot corrected him saying they didn’t have clearance from the tower. The second time, the co-pilot and navigator both sat back and let the whole thing unfold, to their own detriment. No one survived in that KLM 747 after it crashed into the Pan American 747 and bounced down the runway. In the article I link to above there’s an anecdote that Charlie Pellerin relates about a number of Korean Air crashes that occurred in the 1990s. Similarly it was the cockpit ‘culture’ that was leading to the bad decisions being made and resulting in the loss of the airplane and passengers on board.

Some people like to call it ‘top-down’ management, where everyone defers to the person recognized as the person in charge. Worse yet sometimes the person in charge doesn’t always realize this. They go on about their decision making process never once thinking people are restraining themselves holding back questions. The danger is always once this pattern is in place, any mistake by the person in charge gets amplified over time. In Charlie Pellerin’s judgement modern airliners are designed to run by a team who SHARE the responsibilities of conducting the airplane. And while the planes themselves have many safety systems in place to make things run smoothly the assumption is always made by the plane designers of a TEAM. But when you have a hierarchy of people in charge and people that defer to them, the TEAM as such doesn’t exist and you have now broken the primary design principle of the aircraft’s designer. No TEAM, No plane, and there’s many examples that show this not just in the airline accident investigations.

In the case of the Hubble Telescope mirror, things broke down when a simple calibration step was rushed. The sub-contractor in charge of measuring the point of focus on the mirror followed the procedure as given to him. But skipped a step that threw the whole calibration off. The step that he skipped was to simply apply spray paint onto two end caps that would then be placed on to a very delicately measured and finely cut metal rod. The black spray paint was meant to act as a non-reflective surface to expose a very small bit of the rod end to a laser that would measure the distance to the focus point. What happened instead because the whole telescope program was going over budget and was constantly delayed all sub-contractors were pressured to ‘hurry up’. When the guy who was responsible for this calibration step couldn’t find matte black spray paint to put on the end caps he improvised (like a true engineer). He got black electrical tape, wrapped it on the end of the cap, cut a hole with the tip of an Xacto knife and began his calibration step.

But that one detail was what put the whole Hubble Space Telescope in jeopardy. In the rush to get this step done, the Xacto knife nicked a bit off the metal end cap and a small shiny, metal burr was created. Almost too small to see, the burr poke out into the hole cut into the black electrical tape for the laser to shine through. When the engineer calibrated it, the small burr was reflecting light back to the sensors measuring the distance. The burr was only 1mm off the polished surface of the calibration rod. And that 1mm distance was registered as ‘in spec’ and the full distance to the focus point had 1 mm added to it. Considering how accurate a mirror has to be for telescope work, and how long the Hubble mirror spent being ground and polished, 1mm would be equal to 1 mile in the real world. And this was the source of the ‘blur’ in the Hubble Telescope when it was first turned on after being deployed by the Space Shuttle. The culture of hurry up and get it done, we’re behind schedule jeopardized a billion dollar space telescope mission that was over budget and way behind schedule.

All these cautionary tales reiterate the over-arching theme of big failures are not technical, no. These failures are cultural and everyone has the capacity to do better every chance they get. I encourage anyone and everyone reading this article to read the original interview with Charlie Pellerin as he’s got a lot to say on this subject and some fixes that can be applied to avoid the fire next time. Because statistically speaking there will always be a next time.

Related articles

• Why Hubble Broke and How It Was Fixed (science.slashdot.org)
• Reddit Discussion about the Hubble Mirror problem