raganwald
(This is a snapshot of my old weblog. New posts and selected republished essays can be found at raganwald.com.)

Friday, April 11, 2008
  "Stuffy" Dowding


The other night I popped The Battle of Britain into my MacBook.



Early in the movie (where “the movie” is understood to mean “the highly sensationalized dramatic adaptation of a highly fictionalized book recounting the version of history written by the winners of WWII”)… Where was I? Oh yes, early in the movie there is a very telling scene between Harry Andrews, playing a senior civil servant, and Sir Laurence Olivier, playing Air Chief Marshall “Stuffy” Dowding.

Both are anticipating an assault on Britain by the Luftwaffe, partly to beat it into submission, partly to damage its war capabilities for strategic reasons, and most urgently to destroy Britain’s air force so that the Germans may enjoy air superiority, which would allow them to launch an invasion.

Olivier has stirred up a hornet’s nest by writing a report saying outright that the RAF is outnumbered and has too many inexperienced pilots. Andrews is pressuring him to give a more optimistic review of the situation. He explains that the report conflicts with what the Minister has told Churchill. There are hints that if Olivier does not buck up and go along with the way the rest of the chaps are running the war, he will be sacked.

In real life, Dowding was sidelined during the First World War for insisting that pilots needed rest to be effective. In WWII, he had realized that France was a lost cause, concluding that sending men and machines to their death in France would leave Britain completely undefended, although he did provide air cover for the evacuation from Dunquerque. He exercised prudence and caution religiously, as befits a commander outnumbered 4-1, and in the end his methods were vindicated by success in defending Britain.

So, an interesting semi-fictional anecdote. And?

Managing Software Development

Managers have this incredibly hard problem: they have to push (or pull, or prod, or horsewhip, or cajole, or whatever) their teams into producing the optimum performance. In most cases, they rely on the team for information about what is to be done, how it is to be done, and when it can be done. In software, this is especially true: the people doing the work have the most expertise in determining what can be done, by when.

It is almost impossible to be fired if you (a) Go along with the official plan cheerfully, (b) Are seen to work long hours, and (c) Can give plausible explanations for why people or circumstances unrelated to management are responsible for things not working out.


And then we add to the mix a very difficult social problem: The members of the team usually need the approval of the manager to keep their jobs. It is very, very difficult to have a relationship between a manager and a subordinate where the manager actively dislikes the subordinate. In almost all cases, one or the other will eventually have to go.

So most subordinates have a great deal of incentive to tell the manager what he wants to hear, whether it’s true or not. To this, you may add the fact that it’s relatively easy to explain away failure in software development after the fact, provided you are seen to have worked your ass off. “We worked around the clock, but there were too many bugs in the FizzBuzz library…”

Seriously, it is almost impossible to be fired as a programmer on a team if you (a) Go along with the official plan cheerfully, (b) Are seen to work long hours, and (c) Can give plausible explanations for why people or circumstances unrelated to management are responsible for things not working out. It is much easier to find yourself looking for work if you (a) Dispute the feasibility of the official plan, and then (b) Blame poor planning and management for the team’s failure.

So there is a problem: there is an incentive for subordinates to give falsely optimistic information to managers. And guess what? In many organizations, managers have managers too. So they take the optimistic information and make it even more optimistic when reporting to their managers. Astronomy buffs will be amused to hear that this effect has a name: Green Shift.

Information gradually changes in hue from red to orange to yellow to green on its way up the pyramid.

The Two Kinds of Managerial Behaviour

Obviously this is an oversimplification, but I am going to posit that managers exhibit two kinds of behaviour around asking the team for estimates, progress reports, feasibility checks, and any other kind of information that validates or invalidates the “official plan.”

Behaviour number one is to seek the most accurate information. The manager may still press the team to try to do difficult things. For example, the team may say that they have a lot of confidence something can be shipped by September 15th, and the manager may overrule them and insist on September 1st. Or may laugh at their optimism and put October 15th down. But either way, the manager wants to know what the team really thinks.

Behaviour number two is to manipulate the team into saying what the manager wants them to say. At KL Group, we used to call this the “Guess the date I’m think of” game. A certain manager would ask for an estimate. Then, he would reject it and ask the subordinate to think again, harder. This would go on until the subordinate gave the answer the manager wanted, and from then on the subordinate had committed to the date.

The manager, of course, could blame the subordinate for any failure: Hadn’t the subordinate volunteered the date?

This latter behaviour is quite common, and one of my epiphanies in development project management practices was realizing that there is no practice, no methodology, no tool that can fix this problem. It is a social problem that requires a social fix. For example, FogBugz has a very interesting feature called Evidence-Based Scheduling. In short, Evidence-Based Scheduling takes your estimate and combines it with reality to produce a delivery date.

For example, if Fred Fast is conservative on estimates but always delivers on time, Evidence-Based Scheduling doesn’t change his estimate. But if Sally Slow is always late, Evidence-Based Scheduling adjusts her estimate accordingly. Terrific!

But here is the question: does it solve the problem of Maligna Manager badgering her team into giving unrealistic estimates?

You might think “Yes!” You might think that if Maligna pressures her people into giving unrealistically optimistic estimates, Evidence-Based Scheduling will adjust for that and fix the estimates. That is actually true if Maligna’s own manager uses Evidence-Based Scheduling and doesn’t tell her.

If you are the CIO of BigCo, you can work on igniting positive change in your culture, incentivising your management structure to increase visibility into their process, perhaps leveraging the competencies of a strategic value-added think tank. In other words, hire consultants.


But Maligna is fully aware that the company is using Evidence-Based Scheduling, and what actually happens is this: When Maligna wants things done on September 1st, she badgers Fred into giving September 1st as his date to be done, and she badgers Sally into giving August 1st as her date to be done. Maligna doesn’t view Evidence-Based Scheduling as a system for finding out when things will actually be done: Maligna views Evidence-Based Scheduling as a system for finding out what estimates Fred and Sally need to give her to get the result she wants.

Evidence-Based Scheduling is like a formula with an unknown team estimate on the left side and an unknown delivery date on the right side. One kind of manager finds the team’s estimate, plugs it into the left side, and out pops the most likely delivery date. The other, Maligna, plugs her desired delivery date into the right side and out pops the estimates she demands from the team.

To fix this, you have to change Maligna’s motivation. There is no tweak to the algorithm that can prevent her from gaming it to get the result she believes is in her best interests.

You Own Your Own Estimates

Well, if you are the CIO of BigCo, you can work on igniting positive change in your culture, incentivising your management structure to increase visibility into their process, perhaps leveraging the competencies of a strategic value-added think tank. In other words, hire consultants to tell you what a strategic thinker you are.

The rest of us are a little like Stuffy Dowding. It’s our job to stand our ground and just say no when people pressure us to pretend we can warp the laws of space and time. I wish it were more complex than that. I would like to write what most people would like to hear about selling your boss. I would like to write an article about the three sure-fire ways to manage your manager into making realistic plans.

But reality goes a little like this: If you “play the game,” you will obtain personal reward even though some or many of your projects will suffer. If you stand your ground and politely but firmly insist on telling what you believe to be true about what will be done by when, your career will go a little like Dowding’s: you will suffer some setbacks, such as being sidelined during World War I. And you will, from time to time, save the country, as he did in World War II.

It’s really up to you. My old (and terrific) manager Steve Rosenberg used to say “We own the compiler,” meaning that no matter what product management demanded, we had the final say on what code was written. And so it is with estimates. No matter how it is couched, you own your own estimates. Your manager can write any plan, but you and you alone decide whether you agree or disagree. You and you alone answer questions like “Do you think you can accomplish X by Y?”

Managers will try every trick. One President I recall used to come up with all sorts of scenarios, “But what if this turns out to be easy? What if there aren’t that many bugs?” And I used to say, “Then we will have good news for you in a few months. But today, this is my opinion of how things will go based on my experience and what we know to be true right now.”

No matter what, I would not allow him to get me to change my estimate. He had, of course, the final say on the plan. As he should. And it was my job to execute the plan as best I could, which meant getting the team of 23 or so people to execute the plan as best they could. That’s the job, to do your best once the decision is made. Just as Dowding’s job was to execute the plan that Churchill approved.

But nothing can force you to fabricate false optimism or to allow your manager to tell you your opinion of the situation. You own your own estimates.

Stuffy, Again

It is easy to blame management, or blame certain kinds of corporate cultures for problems with unrealistic plans. That is very nice to think about, it relives us of responsibility for what happens: “I had to go along, that’s the way things work around here.”

This was Stuffy’s problem, too. His management were pressuring him to do unrealistic things like save France, or engage the Luftwaffe head on, or to promise things the RAF couldn’t deliver. There were many things the RAF had to do to win, like use Radar effectively, and take advantage of intelligence, and yes, to out-fly the enemy in combat.

The Battle of Britain was not won in that office on that day, of course. But it could very easily have been lost there.
 

Comments on “"Stuffy" Dowding:
"It is much easier to find yourself looking for work if you (a) Dispute the feasibility of the official plan, and then (b) Blame poor planning and management for the team’s failure."

I'm surprised I still have a job.

Wait, my one-on-one is in 10 minutes. Maybe I don't.
 
Great post. I have to admit that the first time I read about Joel's Evidence-Based Scheduling tool, I thought, "Wow. Cool." But you have it completely right that people would just game it, putting you right back to where you were before you bought the "cool software that was going to save us tons of money."

I would like to point out another type, than the dilbertesque PHB, the person who wants to use your estimate to sell an idea. They aren't usually evil; however, they think that your estimate will kill their pitch before it's ten-seconds out of their mouth. So, they will try to cajole you into shortening it.

I actually had a friend do this to me, and I found it very tough to stand my ground. What I wound up doing was removing features to meet the budget and time constraints he imagined he could sell upper management on. Overall, I think it was the best, ethical thing I could do with the situation.
 
Dan:

Some people would game it. My contention is that it can’t fix a cultural problem, but I honestly believe it can make a good team better.
 
That's one of the most discouraging things I've witnessed in my career. Delusional managers who aren't interested in reality and actively ignore it. My spirits are sinking right now just thinking about it. That's one of the things I always dread when starting a new job. When will I have my first "Delusional Manager Moment"? The moment when all the optimism you have for your new job begins to erode, and you start thinking about the future. It never takes long. So, so depressing.
 
Read "Here Comes Everybody." The hierarchical system was explicitly and deliberately designed to obfuscate information as it passes up the chain. It was an early relevance-oriented filtering system, designed for railway switch operations, where info overload could be deadly. The fact that it became the de facto standard for organization in the 20th century is one of those insane little ironies of history.
 
I believe you're talking about "Blue Shifting" not "Green Shifting".

http://en.wikipedia.org/wiki/Blueshift
 
I'm pretty sure I meant Green Shifting. Red and Blue shifting are astrological phenomena, green shifting is social.

Think of a traffic light: The question is, "Do we have agreen light to ship the product on September first?"

The programmers think, "No way! Red light!!" But as the message goes upwards, the colour shifts greenwards until the CTO reports, "Yes, Green Light to ship September 1st."
 
A way around getting push-back on your estimate is to figure out what your manager really wants and what you really want. If you are asked to cut your estimate in half for example, you can agree if and only if a few things are dropped from the work load. If those are precious to the manager, you'll at least be able to negotiate.

The best advice I ever got was to 'pick your battles'. If you don't want it, and your manager doesn't want it, let it go. If you really want it (or need it), trade it for something else. It would be nice if we had all of the time and resources necessary, but that's incredibly rare (and usually comes with other problems).


Paul.
http://theprogrammersparadox.blogspot.com
 
Paul:

The expression "push back on your estimate" seems ambiguous to me. There is a difference between a manager saying "I understand your estimate but I am demanding that you do more/better/do it faster" and a manager manipulating you into giving a different estimate.

The former is the start of a negotiation, which as you point out has many possible outcomes.

The latter is what I was trying to write about. In the fictional scene, teh civil servant wasn't saying "I understand you think we'll lose but fight as we direct any ways," he was trying to manipulate Dowding into giving a report that matched what the civil servant wanted to tell the Prime Minister.

Would you agree these are very different things?
 
I'm not so sure Maligna's trick would work for very long. Her first project would be easy to game, yes, but after that she would need to up the ante further to deal with the compounding of the initial error rate with her "correction".

It wouldn't take long before Maligna hit a point at which her "corrections" required estimates that were so ludicrous that they would be obvious (30%-50% error rates are indistinguishable, but 400% is obvious.. usually).

At this point either Maligna forces estimates to produced that are obviously ludicrous, which could only be explained by admitting to gaming the system, which might very well finally get her fired. Or Maligna realizes the risk in such ludicrous estimates and is forced to accept an estimate that lets the evidence based scheduling finally work.

There might be a missing piece in FogBugz if it only looks at the accuracy of the estimator, but ignores the aggregate error of the persons with influence over the estimator. Also if Maligna finds a way to switch companies, departments or otherwise somehow periodically "wipe the slate clean", she could keep the game up, but otherwise there would be a limit to how long she can get away with that game.
 
Hi Reg,

Sure, if your only choice is between the truth and a lie, you have no room to negotiate. But I have found that more often than not, the world is far less black and white than I, as a programmer, have a tendency to envision.

Projects, features, functions, etc. are rarely 'all or nothing' deals. Sometimes during estimates I'm given the time first, and then I have to work back to see what will fit. If I only have eight weeks to work with then this next round of features won't be as nice as if I had sixteen.

I try very hard to get past my own bias and see the world as many shades of gray. I've always found that I am given a bigger role in the process if I am more flexible, and I am only able to stop really bad things from happening if I have a bigger role. Otherwise I'm just along for the ride.

Paul.
 
Good read.

@Ryan:

I think Maligna could pretty well game the system, and in a twist of social engineering, show everyone in the company how faulty* FogBugz is and return to ye olde system.

* Personally, I love the idea of EBS and the FogBugz product itself.
 
I've never used FogBugz, but I read Spolsky's article about Evidence-Based Scheduling (EBS) article and found it intriguing. One thing it does which you don't mention here is forbid estimates longer than 16 hours. I think this lets the software quickly adapt to the manager's adaptations. For example, Sally Slow estimates, by herself, the first chunk of a project will take 16 hours, but it really takes 32. EBS says the next 16-hour chunk can take up to 32 hours.

Next week, the manager badgers Sally into halving her estimate of the next chunk to keep the project on track. Sally complies, but the eight-hour chunk takes 32 hours. EBS says the next eight-hour chunk can take up to 32 hours.

The third week, the manager badgers Sally into dividing her estimate of the third chunk by four. Sally complies, but the four-hour chunk takes 32 hours. EBSD says the next four-hour chunk can take up to 32 hours.

At what point does either Sally grow a backbone or does the manager stop badgering Sally into halving her estimates? Moreover, at the end of every chunk, EBS correctly predicts the release date. I think that's remarkable.
 




<< Home
Reg Braithwaite


Recent Writing
Homoiconic Technical Writing / raganwald.posterous.com

Books
What I‘ve Learned From Failure / Kestrels, Quirky Birds, and Hopeless Egocentricity

Share
rewrite_rails / andand / unfold.rb / string_to_proc.rb / dsl_and_let.rb / comprehension.rb / lazy_lists.rb

Beauty
IS-STRICTLY-EQUIVALENT-TO-A / Spaghetti-Western Coding / Golf is a good program spoiled / Programming conventions as signals / Not all functions should be object methods

The Not So Big Software Design / Writing programs for people to read / Why Why Functional Programming Matters Matters / But Y would I want to do a thing like this?

Work
The single most important thing you must do to improve your programming career / The Naïve Approach to Hiring People / No Disrespect / Take control of your interview / Three tips for getting a job through a recruiter / My favourite interview question

Management
Exception Handling in Software Development / What if powerful languages and idioms only work for small teams? / Bricks / Which theory fits the evidence? / Still failing, still learning / What I’ve learned from failure

Notation
The unary ampersand in Ruby / (1..100).inject(&:+) / The challenge of teaching yourself a programming language / The significance of the meta-circular interpreter / Block-Structured Javascript / Haskell, Ruby and Infinity / Closures and Higher-Order Functions

Opinion
Why Apple is more expensive than Amazon / Why we are the biggest obstacles to our own growth / Is software the documentation of business process mistakes? / We have lost control of the apparatus / What I’ve Learned From Sales I, II, III

Whimsey
The Narcissism of Small Code Differences / Billy Martin’s Technique for Managing his Manager / Three stories about The Tao / Programming Language Stories / Why You Need a Degree to Work For BigCo

History
06/04 / 07/04 / 08/04 / 09/04 / 10/04 / 11/04 / 12/04 / 01/05 / 02/05 / 03/05 / 04/05 / 06/05 / 07/05 / 08/05 / 09/05 / 10/05 / 11/05 / 01/06 / 02/06 / 03/06 / 04/06 / 05/06 / 06/06 / 07/06 / 08/06 / 09/06 / 10/06 / 11/06 / 12/06 / 01/07 / 02/07 / 03/07 / 04/07 / 05/07 / 06/07 / 07/07 / 08/07 / 09/07 / 10/07 / 11/07 / 12/07 / 01/08 / 02/08 / 03/08 / 04/08 / 05/08 / 06/08 / 07/08 /