Still failing, still learning
(This essay appears in the e-book What I've Learned From Failure: A quarter-century of experience shipping software, distilled into fixnum bittersweet essays)The good news: I’m still learning.
The bad news: …
from failure.
This post officially announces that my side project (originally named
cause & effect and later named
certitude) is over.
What I wanted to achieve
For those of you who weren’t subjected to one of my enthusiastic rants, here was my
Graham Question: Can we predict the outcome of a software development project with objective observation?
Although most businesspeople would soil their khakis if they had to ride a tank into action, Reis and Trout are right on when they compare the four major strategies of War—Offense, Defense, Flanking, and Guerrilla—to businesses, especially start ups. I rate this even higher than Crossing the Chasm for its ability to succinctly explain what growing companies have to do and how to do it to succeed.
After you’ve read what I have to say here about my failure, I invite you to read what Marketing Warfare has to say about Guerrilla Warfare (just three tactics to pursue!) and see if I'm a case in point.
I have always believed that the answer is
Yes. And I manage projects that way. But I also strongly believe that the only way to prove that we have an objective understanding of an algorithm is to mechanize it, to write a piece of software that executes your algorithm.
So I set out to write a piece of software that, pure and simple, would look at a software development project and show you a traffic light: a green light would mean that the project looks like it’s on track, a yellow light would mean that the project needed help, and a red light would mean that there is no hope.
I won’t go into a lot more detail, mostly because this isn’t a story about teeming hordes of customers and me not being able to finish by deadline. It’s a story of inventing a great solution to a problem nobody cares about. But if you absolutely must visualize the application, think of something that gobbles up your MS project files, your bug tracking data, even your burn down spreadsheets, and belches out that traffic light when it’s done. That’s it.
So how and why did I fail?
Project management software is social software
Reason zero, just as Paul Graham warned, my age. Really. I remember how much code I could crank at age 22, and now that I am double the age, I write an order or magnitude less. I’d like to say that my code is that much better that it makes up for the lack of volume. That might be true if I start a project with a specific end in mind, but the act of experimenting, tinkering, and exploring benefits from being able to turn large amounts of code around in short amounts of time.
This is reason zero because I wanted to get it out of the way before explaining why I think I still would have failed if I were 22. But it’s still important to understand, because I might have known that I was going to fail two years ago, instead of today.
Reason one that I failed—and this is the most important reason I failed—is that I was trying to tackle a social problem with technology. This can work—ask any dating site zillionaire—but it is a very high-risk strategy for software. Not just for making money, but for what really matters, adoption.
Project management is a social problem.
Project management is a
social problem. It is 99.5% about getting everyone who knows something about the state of the project to share what they know with everyone else. Getting all the relevant information is 99.5% of the problem, analyzing the information is 0.5% of the problem.
Joel likes to brag about how much trouble FogBugz goes to to make it easy to enter bugs, about how certain kinds of reports are not available to discourage punitive social behaviours like punishing developers who generate too many bugs. This is a strong hint that getting good information is all about managing people’s perception of the likelihood of punishment for telling the truth.
Sitting here typing this, I think the company who can do the best job of predicting the outcome of software development projects is
Inkling Markets. That’s because their entire business is about finding a way for people to communicate what they really think of something, not just what they think other people want them to say about something. I think
Todd Proebsting would agree.
This problem isn’t limited to dysfunctional environments where people cower in fear of saying anything except “Sir, Yes Sir” when told to
change the laws of physics.
Lemons, damned lemons, it’s always lemons
Project management suffers from a real lemon problem. I
quoted Bruce Schneier at length about this already, so this time I’ll explain things directly:
Most managers, especially those with limited experience shipping software on a predictable schedule, do not know how to correlate what they’re told about the project with the likelihood that the project will succeed.
Some information is valuable, some is junk. The problem is, managers “buy” information. They trade favours like letting you keep your job for information about how well you are doing your job. So there is a market for information just like a market for cars.
They also “sell” information, literally: they have to make a report or a presentation to their superiors, or to stake holders, or to their fellow founders at the
YCombinator dinner.
When a manager cannot tell the difference between information that is useful for predicting the outcome of a project and information that is not useful for predicting the outcome of a project, she thinks about the next best thing: the “resale value” of the information with people one step removed from the project, like her own manager. So she values things like pretty PowerPoints about the architecture higher than finished pieces of functionality.
(This is why I have always sweated my heart out to give good presentations. My teams have depended on me being able to take good information and sell it upstream just as if it were CMM Level Five Buzzword-Compliant Junk).
Do managers further removed from a project always value pretty junk more than good, solid information? Not always, but often. And that’s enough for people to be pressured to give the bad information that sells to their manager, while hiding the good information that doesn’t sell. Exactly like the owners of good cars taking their treasures off the market.
Lemon and bay leaf crème brûlée
Why does junk information outsell good information? Nice PowerPoint isn’t a good explanation by itself: there are nice PowerPoints explaining Agile, but managers still prefer waterfall.
Consider a
not so big design. Let’s call completing that design good information: we’ve done a good job finding out what’s really important for the project and making a design that emphasizes the way this project is unique, not the technology stack.
Now consider a typical technology design, emphasizing frameworks and technologies. Fully buzzword-compliant.
Which one sells better? The
technology stack does. Why? Well, for starters, managers have been exposed to seventeen billion dollars worth of advertising talking about the benefits of technology stacks. Nobody is advertising the specific ways the not so big design helps the project. How could they? Those are
specific to the project, that’s the whole point.
And managers are like anyone else, they compare what you are doing to successful projects they have seen in the past. Once again, the not so big design doesn’t have anything in common with other projects, but the technology stack does. (There are lots of failed projects with technology stacks, of course. But who cites those when bugging the team about whether they will use Hibernate as their ORM?)
How did this happen? How did things that have no correlation to the success of a project become more attractive than things that do?
Mmmmmm... Elegantly Easy Crème Brûlée... The title doesn’t lie, the recipes are easy: even I was able to make tasty desserts! For motivating a team or thanking your family for putting up with your devotion to your start up, dessert is always a good choice :-)
Quite simply, people have an incentive to look successful. So they imitate the outward appearances of successful projects. We have a really simple way of completing successful software projects:
we put successful people on them. But we have a broken way of thinking about it: we don’t like to think of the people as being special, we think that what the people do is successful.
And by that logic, we can take
anyone, have them do the same things as successful people, and our projects will succeed.
In a manager’s mind, the measure of whether information is good or not is,
Does it measure whether people are doing the same things that successful people have done on projects I’ve been told were successful?
This is not the same thing as measuring whether the project is on its way to success at all. This measures the outward appearance of a project. Things that can be measured easily are rarely the most significant things. Behaviours that can be “gamed,” like how many hours a team is working, will be gamed.
And as above, even if a manger knows better, does her manager know better? If not, good information will be difficult to sell and she will be under a lot of pressure to serve Lemon Pie.
Off balance sheet transactions
There’s another important reason that projects have bad information: the best information is
off balance sheet. That’s an expression meaning something a businessperson sweeps under the rug. Try Googling
Off Balance Sheet Transaction. It’s never pretty.
In essence, project plans and reports never include the most important information about the likelihood of project success. Never. (I mean never in the same sense that Joel Spolsky means “nobody,” as in “
fewer than 10,000,000 project plans”).
Let me give you a recent example. I was discussing a project with a certain manager in a client organization, and there was a rather linear series of releases. Her question was,
Can’t we shorten the plan by looking at the dependencies and starting some releases in parallel with others?
Of course we could, but in doing so, we increased the project risk. Just as a single developer has a problem keeping multiple tasks in one brain, a team has the same problem: when working on several unrelated pieces of software at the same time, the individual developers may only work on one thing at a time, but the managers and the testers and especially the stake holders who are thinking about functionality and exercising change control are overloaded, so they will make poorer decisions.
And worse, although you might
think that there are no dependencies, you only think that at the outset of a project because of assumptions you are making
now, before you fully understand what you are getting into. All in all, there’s a reason Agile projects have a practice of working on iterations with single themes, and the reason is to reduce risk.
Does this mean that manager wasn’t making a reasonable trade off between risk and time? Maybe she was making a very reasonable choice. But let me ask you this: where on the plan would you see the risk component of the decision? If you were comparing Plan “A,” with the linear releases that finish in August, and Plan “B” with some parallel releases finishing in July, how do you decide which plan is better?
Right, you can’t see anything except the difference in dates, so you choose Plan B. The risk component of the plan is off balance sheet. Enjoy your lemon.
(And it is very hard for a manager’s people to explain, for the twentieth time, that
it is a mistake to try to optimize a project by having everybody working at once, because that crystallizes certain architecture decisions too early, and haven’t you read any Eliyau Goldratt, whose novel The Goal explains what’s wrong with optimizing resource engagement when you should be managing project risk? The incentive is for them keep their own counsel and put their
résumés up on the web. Such is life.)
A more striking example comes from another project where, for various reasons, there was nearly 100% turnover of the senior staff, and finally an outside firm was brought in to clean it up. Do you think there was a notation on the plan for staff turnover? That has to have a huge risk implication, but the plan where you use the same team from start to finish looks exactly like the plan where new people are brought in for each phase or iteration.
Why is risk off balance sheet? I think it’s because people have a blind spot for risk in projects. After the fact of a project, you can always do a postmortem and say, “if we had done this, and this, and if we didn’t do that, we would have succeeded.” So you blame yourself for poor execution.
There’s a culture of
Boolean thinking about projects and plans. They worked or they didn’t. We’ll be done in July or August. Not “With Plan A, we’re 90% August, 8% September” and “With Plan B, we’re 12% July, 56% August, but 23% September and 7% October!”
Decisions that add risk to a project, such as excessive parallelization, or the use of unproved people, or the use of team who have never worked together in the past, or of forcing decisions prematurely, all of these things are not reflected in the plan.
(This is another reason there is pressure downwards on developer competence in many organizations. Consider two managers: the first staffs up quickly by selecting available developers, who may not be particularly good. In fact, they are less good then the existing team average. The second moves more cautiously, only adding to the team when the addition improves the average competency of the team. Do you think the second manager will keep their job long enough to finish a project? No, because the ticking time bomb of hiring certain types of developers is off balance sheet, but being understaffed is out there for everyone to criticize.)
I had this hope that through a kind of collaborative filtering I could have a piece of software notice that linear plans or plans involving hiring too quickly or whatever had less variance than parallel plans, and adjust the traffic light accordingly. Regardless of what people would say, it would shrug its shoulders and remorselessly remark on the actual historical averages.
Nice idea? No, it’s a dud. (Or at least, my execution was a dud!)
Somebody set up us the bomb
The second major reason I bombed is that I couldn’t find any buyers. Quite simply, the people who understand these principles are running successful projects. I know lots of people who agree that there is a problem and agree this would help, but
they don’t think they need help.
I couldn’t find anyone holding their head in their hands, saying, “If only I could get the
truth about our projects, I could open my budget and shower you with gold.” The people who were aware of the problems with project information were busy using decidedly low tech tools—like hiring good people and having daily meetings and estimating tasks in spreadsheets—to solve their problems.
And the people who could use some help quantifying the consequences of their choices—like managers who insist on calcifying decisions very early in a telephone book design document so that they can demonstrate progress in the form of an architecture—they don’t think there’s anything wrong with their approach.
Why is that? My conjecture is that people want help with things that fit their model of what’s important. Someone who uses skilled practitioners and XP thinks that limiting risk is important: that’s why they use two week iterations and merciless refactoring rather than up-front design.
Someone using a classical BDUF management approach thinks maximizing resource allocation and managing task dependencies is important, that’s why they spend all of their time looking at GANNT and PERT charts. When you try to sell either of them something, they want to know how it will fit the model they have in their head about how to succeed with software development, not why they should consider a new model.
And I’m not sure they’re wrong about what’s important to them, personally.
If one of those
BDUF projects fails due to the architecture being a poor fit for what the team discovers is really important about the projects, or from poor decisions being made in haste at the beginning of the project, well… managers will say that the problem was with the people making poor decisions. Such managers are not shopping for tools to help them understand the trade-offs, because they do not believe they are making trade-offs.
Critical Chain is an amazing book. The narrative form—a novella detailing a technical project team and their search for a way to manage an uncertain process—is a big win, it highlights the important ways that Critical Chain Project Management handles risks and uncertainty and makes it visible where everyone can manage it.
The section on estimating tasks alone is priceless. If you can’t afford a copy and your library doesn’t stock it, borrow mine. You must read this book if you participate in software development teams.
Something I learned from selling Macintoshes back in the day is this:
only make one sale. Convincing someone they have a problem is one sale. Convincing them you have the solution is another. And convincing them that today is the day to act is a third. If you have to do all three at the same time, you are doomed.
This is why experienced companies distinguish
sales from marketing. The first two steps are marketing, the third is selling. When you are a new company, you don’t have the resources to market and sell. You have to work with an established pain point (eliminating the first hurdle), then use PR and limited marketing funds to get the word out that you have solved the problem (the second hurdle). You only have time and energy for the third sale, separating customers from their money.
But trying to convince managers that they’re doing it wrong (tactfully), then convince them that they want your product rather than some big rigid waterfall thing or three by five cards and XP, and then convince them that they should write a cheque today…
Bad idea. I should know, I discovered that I was trying to do exactly that.
Now you understand why I have used so much of this post to rant about the state of project management in software development. When you understand what most companies are doing and why, you understand what will sell, and why. And when I understood that the things my project were measuring and reporting were of little interest to the people who were my market, well...
The not so big business plan
I think there are lots of things that will sell, that
do sell into this market. When you understand that this is a social problem, when you understand that most information is bad and that the incentives are to value bad information, and for managers to value activities that produce bad information over activities that produce good information, you can make something people
will buy.
Like books, lectures, methodologies, and video training. If people have a social problem, a social solution is a natural fit.
There were interesting possibilities like selling this to BigCo for “process improvement.” But whenever I thought about such ideas, I noticed that the software wasn’t in there. I could have written a book and lectured on these ideas. I could have done a “search and replace” of agile for certitude.
A good business plan is one that is really specifically about you and your software. If your plan could be executed by someone else, or with a different project, it simply isn’t the right plan. And all variations of turning this into ConsultingWare were excellent ways of monetizing Reg Braithwaite the brand, but not ways of spreading the adoption of certitude, the product.
And so… and so I looked the
sunken cost fallacy straight in they eye and said,
no more. As much as I
hate to lose, I have folded my project and I am sharing with you the important lessons I learned.
Lessons learned
Well, the good one is this: Paul Graham is right. If you phrase your venture as a question, you will walk away with something very valuable. I now know a lot more about project management and the culture of software development than I did when I started the project, and I wasn’t exactly a Spring Chicken. And set out to learn about Bayesian Networks and Supervised Classification, but I ended up learning about Unsupervised
Clustering and
Collaborative Filtering.
What a journey.
And there is the one I ignored for far too long: Always Be Selling. I can thank friends like
Leila Boujnane,
Sutha Kamal, and Erich Finkelstein for reminding me about this, incessantly. Asking people to “buy” your idea forces you to make something people want. That being said, I think there’s a right amount of “push.” You can’t invent by polling the market. Quickly now, jump in your time machine and go back to 1981 or so. How many people wanted a mouse for their computer? But yet… Always be selling!
(There’s a bunch of other stuff that is far more personal, and maybe some of it will appear one day in public, but I wanted to keep this post to essay length, so it’s almost entirely about products and markets.)
So, here I am. Older, wiser, and ready for life’s next adventure. It might be more consulting for BigCo. It might be joining a start up and helping someone else’s dream flower. I don’t know, but if you have an idea...
My brain is open.
—Pál Erdös
Labels: agile, passion, popular