raganwald
(This is a snapshot of my old weblog. New posts and selected republished essays can be found at raganwald.com.)

Saturday, September 30, 2006
  Wasabi cannot cure rotten fish


There is only one problem with development methodologies. Just one. It affects all methodologies, agile and otherwise: No methodology can 'fix' projects that are staffed with underperforming people. People are more important than process, period.

This is the underlying issue with the language 'wars' as well. No magical combination of JSON, static typing, design patterns, IDE whizzies, and frameworks will somehow help a million monkeys produce any of Shakespeare's works.

If you want to build a ship, don't drum up people to collect wood and don't assign them tasks and work, but rather teach them to long for the endless immensity of the sea.
Antoine de Saint-Exupery

Steve Yegge has suggested that if 90% of the projects adopting a methodology fail, you have to stop blaming the people. You have to stop saying "you're not doing it right." You have to say there's something wrong with the methodology itself if 90% of the projects fail to use it properly.

Sounds reasonable. But why would this be true for methodologies but not true for programming languages?

99.999% of the software written in almost any language (including Lisp, Python, and Ruby) is buggy. Yet we readily say that the problem is the programmer. We think that some languages are better than others, that some languages help eliminate certain classes of bugs, but we take as a given that good tools are not sufficient unto themselvesfor good results. We know that the quality of the result is almost entirely driven by the quality of the programmer.

I have the uncomfortable feeling that others are making a religion out of it, as if the conceptual problems of programming could be solved by a single trick, by a simple form of coding discipline!
Edsger Dijkstra

Home Depot might suggest that buying a new table saw will cause beautiful woodwork to appear in your home. Yet just because a duffer with a table saw cannot make built-in closets all by himself, we don't abandon table saws outright. We assume that good woodworking is better with good tools. And if we are blessed with the skill to make a built-in closet with a hand saw, we know that we can also make the closet with a table saw, and we judge the table saw on whether and how it can improve the results we could have obtained with another tool.

This is my criteria for judging a methodology. Can it improve a team that is already fundamentally qualified to write software?

If you want to stop reading here, my statement is that you can only judge a methodology on the basis of the results it delivers for a team that has already proven it can ship software. You judge it on the basis of incremental value. You don't compare its results to the results some other team gets, or to your fantasy of what results the team 'ought' to get.

The mythical marginal

What I've written seems obvious. Yet, billions of dollars are paid every year to people who are selling a different perspective. What is it that causes companies to buy silver bullet after silver bullet? Why do companies lurch from consultant to methodology to programming language like infomercial junkies looking for something that will flatten their tummies without sweaty exercise or unpleasant diets?

I bifurcate teams into those that are fundamentally qualified and those that aren't. And I believe that to fix an unqualified team, you start with the people, not the process or the tools. So what could the rest of the IT industry possibly believe?

I have observed a belief in a marginal team. A team that is somehow straddling the fence between incompetence and competence. They can ship software, but only with help from their process and tools. If 'marginal' is too perjorative a term, you could think of them as having unactualized potential.

In truth, I have worked with several such teams, so I believe they exist. However, I believe that such teams are quite rare, while proponents of silver bullets make a living convincing customers that marginal teams are commonplace.

The primary example I have observed of a marginal team is a team composed of individuals who are not working together well, yet they have achieved success on other teams in prior roles. Something like the Los Angeles Lakers before Phil Jackson arrived. There is a lot of latent championship potential in the individuals, but the team isn't performing.
Shipping software is not an emergant property of competent programming, nor is it a consequence of management technique.
Why does the IT industry believe that perhaps most teams that fail are marginal teams? They have to believe this, otherwise they wouldn't waste time and money trying new silver bullets ("everything should be a stored procedure," "tests are the only documentation that matter").

It always comes back to hiring

My broken-record assertion is that the industry as a whole embraces the above model of a marginal team: latent potential in the individuals. The difference between the industry and I is that I have an objective measure of latent potential: has the individual demonstrated success shipping software in the past.
True happiness comes from the joy of deeds well done, the zest of creating things new.
Antoine de Saint-Exupery

The industry doesn't apply this test rigorously, if at all. A typical interview with a developer focuses on patterns and archiecture, on talking about past achievements. Developer's don't juggle. Developer's don't design software. And especially, managers don't hire developers with super-strong references from strong employees that have worked with the candidate in the past.

Basically, managers ask developers if they can ship software and then take the candidate's word for it. If you take only one thing from this blog post, take this: shipping software is not an emergant property of competent programming, nor is it a consequence of management technique. Someone can demonstrate the ability to write software but lack the ability to actually ship software.

(Update: nothing in this post should be construed to suggest that people cannot become better developers through study, mentorships, or training. However, training is to hiring as tools are to skill. And they are not mutually exclusive, any more than good programmers and good tools are mutually exclusive.)

I believe that the reason why the IT industry believes that most failing teams are marginal rather than outright incompetent is that they believe that these teams are composed of individuals who can write software and managers who can direct work. All they need are the right tools to write software "better" and better software will emerge. All they need is a methodology to manage software development better and software will ship.

If you have a marginal team, a team composed of individuals with proven ability to ship working software, you might have something there. But if a team isn't even marginal, no methodology will help. And that's why a useful methodology can still fail to help 90% of the teams that try it.

For the same reason that Wasabi makes Sushi divine.

Follow-up: Three questions and three answers about Wasabi and Rotten Fish

Postscript: The difference between Sashimi and Nigiri

If we're judging success or failure of a development methodology, we have to judge the results on whether the development itself was successful. We can't judge whether the software made money, or whether the company's stock soared. Such things can be studied, however those are product management problems, not software development problems.

People have criticised Google, saying that althought it has a track record for shipping products, only one, AdWords, is financially successful. If you want to compare product management methodologies, that's fine, but you aren't talking about software development. You have to compare similar kinds of projects, for example you can't compare an in-house IT project with a known ROI (reduce the time spent entering a new Blort Ticket by 50%, saving $47.32 per week per Fizbang team member) to speculative product development like a new Web 2.0 calendar.

Who else is trying to create new products? Apple? Microsoft? Dan Bricklin? Compare them to Google, and if they are doing a better job of making money with their projects, post a critique of Google's product management on your blog.

-r.b.

Labels: ,

 

Comments on “Wasabi cannot cure rotten fish:
"People are more important than process, period."

So as a manager, how do you know -- in a specific situation -- if your problem is people or process?
 
as a manager, how do you know -- in a specific situation -- if your problem is people or process

The trite answer is that if you do not know whether people are the problem, then people are the problem.

By which I mean, when your people are all first class with an obvious talent for accomplishing things, but somehow they are not doing it in your specific situation, you are usually very well aware that there is a systemic issue such as bad process, bad morale, or bad management.

But life is not a series of trite sayings that fit onto a powerpoint slide. So quite honestly, the question of diagnosing a specific situation is too complex for a blog exchange.

Would you like to discuss this further? My contact information is on the weblog pages.
 
This is by far the best bit of writing I have ever seen from you.
 
It is rather meaningless to say that "99.999% of the software written in almost any language is buggy". It suggests that bugginess of programs is pretty much constant, while in fact it differs wildly.

Most programs only contain "bugs" that are rather insignificant (like bad localization/documentation, and things that could be better described as limitations or quirks or whatever), but some are just unbelievably buggy. The worst bugs are probably security bugs, and bugs that cause data loss. One bug that losses significant amount of data or that lets people break into your computer is much worse than 1000 minor quirks.

Security bugs are nicest here, because they are among the most serious bugs, and they are decently documented, so we can use some real numbers instead of just using anecdotal evidence.

What is the result ? Nobody have ever written a secure server program in C. Even the most talented and security-conscious people writing servers in C like the OpenBSD team and Daniel J. Bernstein (of qmail fame) did screw it up.

On the other hand Java servers, even written by very mediocre people, tend to be secure. And Java is not anything radical like Erlang or Ruby or Lisp.

Likewise Perl/Ruby/Python server programs tend to have a lot less security problems than PHP programs. And these languages aren't that different from each other.

So at least here technology is far better at reducing number of the most serious bugs than people. It can easily be verified empirically as most security-related bugs in popular software are documented in a few places.

I don't see any reason to think methodologies and languages cannot do the same thing to other kinds of bugs and productivity problems. It's just that it's very difficult to empirically verify such claims, as we don't really have statistics for different kinds of bugs.
 
Security bugs are nicest here, because they are among the most serious bugs, and they are decently documented

I am perplexed by this statement. The vast majority of vulnerable applications, such as on-line shops, do not disclose security vulnerabilities to the public. So what does this mean for the rest of your argument? Do you really have any idea how many have SQL Poisoning vulnerablities, for example?

On the other hand Java servers, even written by very mediocre people, tend to be secure.

If this is your belief, and you plan to act on it by employing mediocre programmers to write mission-critical applications, you are welcome to do so.
 
This is an interesting argument, but I don't buy it completely. I totally agree that there is no silver bullet, or moldy bullet, or that tools are the end-all, be-all.

However, I'm starting to get fatigued by all of these Joel-y assertions that most developers are worthless (rotten fish). There are bad apples in every field, sure, but in my experience the major obstacles to success occur at the top, not the bottom. An executive injects his own goals into a pre-existing project, or unrealistic deadlines are set by inexperienced managers, etc.

I rarely see a project fail to ship because the team was filled with stupid, incapable, lazy, sloppy programmers. Those guys exist, sure, but they usually hang out at the fringes and organize volleyball games that none of the rest of the staff attend. Or they work alone for tiny, little companies on one-off software, as is often the case in the examples on thedailywtf.com .

I'm not trying to assert that all developers are of equal strengths, but I am ready to say that project failures happen *all* the time regardless of the strength of the staff. The programming world is littered with the carcasses of brilliant people, who did everything right, & even used the "right," tools. See: Kiko.

So let's not pin the blame entirely on the developers. That's useless & lets management off the hook. It might be a great defense for your favorite methodology (e.g., my team makes it work, so your assertion that the methodology sucks must be rooted in the weakness of your team), but it ignores the complexity of any software project that requires more than 10 developers, and the countless ways that the folks with the most power can royally screw with any team's ability to ship.

ps - SQL injection vulnerabilities are indeed a big problem! Most of them are found in PHP code written by someone with little experience & no mentor. Even a brilliant CS major might be prone to that mistake, without a more experienced and knowledgable developer to take a look at his/her code. However, ANY framework (countless java, .Net, Rails, etc.) immediately removes that risk. It's gone, no matter how smart or dumb your developers are, they *can't* make that mistake. Something to think about, maybe.

Thanks for a thought-provoking post.
 
Let's not pin the blame entirely on the developers. That's useless & lets management off the hook.

I'm not blamestorming, I'm trying to describe the world as it is, not as it ought to be.

But if you ask me to find someone to blame, here's what I have to say: If a team of developers is fundamentally unqualified to ship software, I blame management for hiring them, for failing to train them, and for failing to gradually weed the worst offenders out and replace them with qualified people.

I will address your point about SQL Injection vulnerabilities elsewhere, the arms race between tools and bugs is very interesting and I'd like a little more room to spew.

p.s. I addressed the "failure of Kiko" in my postscript above. I do not consider Kiko a software development failure. I am only addressing the issue of building software, not turning it into money.

But since you ask my opinion... I do not consider Kiko a business failure. What kind of weird world do we inhabit where the conventional thinking is that a couple of undergraduates building something worth US$250,000 is a failure?

Unless they spent $250,001 doing it, they have done a fabulous thing with the last two years of their lives.

I come from a musical tradition. If you go out and record a CD and sell a few thousand copies of it to people who love your music, I think you've made the world a better place.
 
I am only addressing the issue of building software, not turning it into money.

If the goal is to make software, then you may have a point. If, however, the goal is to turn software into income then you might be a little off track. Most software companies have the later goal rather than the former. The methods used need to be analyzed in that light. Google might make great software and create better software than everybody else, but if that software doesn't generate revenue then the methodologies won't work at most other companies.

Steve Yegge has suggested that if 90% of the projects adopting a methodology fail, you have to stop blaming the people. You have to stop saying "you're not doing it right." You have to say there's something wrong with the methodology itself if 90% of the projects fail to use it properly.

Sounds reasonable. But why would this be true for methodologies but not true for programming languages?


A programming language is a formal system with mathematically provable properties. (Programs are proofs: http://goodmath.blogspot.com/2006/06/finally-modeling-lambda-calculus.html) A programming language's type system (especially a formally designed one) can prove (in the mathematical sense) a program has certain properties including, but not limited to, the absence of whole classes of errors (buffer overflow for example). No methodology can do this, because no methodology is as formal as mathematical logic.
 
Just to clarify: I'm not saying that you don't need good programmers, I'm simply saying that a programming language can make certain guarantees that no methodology can. So, you don't run into this sort of problem with programming languages because whole classes of problems are eliminated.

Which brings up a side point, could you site a study demonstrating that 90% of projects adapting agile methodologies fail (or that 90% of project adapting another programming language fail). I'm suspicious that somebody is making up some statistics somewhere, which makes me suspicious of the whole argument. In fact, I suspect that both you and Steve are wasting your breath until some more research is done (very little has been done so far).
 
Benjamin:

Good comments thanks.

Google might make great software and create better software than everybody else, but if that software doesn't generate revenue then the methodologies won't work at most other companies

That's a point to take up with people who suggest that Google's ways apply to other companies. I am not making that claim.

I was, however, pointing out that many people complain about Google's software not making money, but do not hold up an example of another company with a similar business model that has a different product management (not development) methodology.

a programming language can make certain guarantees that no methodology can

Are you saying that since a programming language may have some provable properties, that any shortcoming in the result must be a shortcoming in the programmer since the language itself is mathematically beyond reproach?

While I agree with the conclusion, I honestly don't think it follows from the premise. Modern software is sufficiently complex that eliminating whole classes of errors does not provide any wonderful guarantee of error-free or even useful results.

Finally, I know for a fact that Steve made up the 90% figure. I don't care what the number is for the prupose of my arguments: the important thing for my argument is that I don't believe that methodology has as significant an impact on results as people.

if you want to, take a felt marker and black out the number. The critical thing i was taking Steve to task for was suggesting that since "X+a" number of Agile projects fail, Agile is a failure. That statement is not useful unless you also know how many projects fail without Agile, "X-a".
 

While I agree with the conclusion, I honestly don't think it follows from the premise. Modern software is sufficiently complex that eliminating whole classes of errors does not provide any wonderful guarantee of error-free or even useful results.


Yes, you can't make software error free because you can't make people error free, which is of course your point.

In retrospect I think I misunderstood the meaning of "...why would this be true for methodologies but not true for programming languages?" The following paragraph explains it better. I guess that's what I get when reading a blog post: an unedited mind dump which can (on first reading) be hard to understand.

Sorry about the misunderstanding.
 
I don't care what the number is for the prupose of my arguments: the important thing for my argument is that I don't believe that methodology has as significant an impact on results as people.

One more thing: do you think that anybody really believes that methodologies have more impact than people? I always assumed the point of methodologies (which of course have varying degrees of success depending on the personalities and quality of all people involved) was to improve the success of a particular project with a particular team. In other words: you can’t always higher the best programmers, so how can you get the best out of what you have?
 
the point of methodologies [is] to improve the success of a particular project with a particular team

Thisis what I am suggesting.

I think most people accept that this is the case for teams that are already able to ship software and have proven that they can do so.

The interesting question is whether a methodology can help a team that has not proven it can ship software, for example a team that is currently failing or a team that is formed with inexeperienced programmers.

My (slight) experience suggests that there is a huge industry built around the premise that methodologies can help teams that are otherwise unable to help themselves.

I have been asked to turn failing projects around. Managers are quick to ask me to institute a new process. I am still waiting for the manager who calls to ask "Can you triage our people into must fire, must keep, and keep if they can keep up."
 
The talent myth seems relevant.
 
The Talent Myth is a must-read.

One thing I've noticed is that people get caught up in "hiring top talent" and treat it like a silver bullet.

Top talent is not a silver bullet!

My claim is that good people are neccessary, but not sufficient.

Remember, the statement "without good people you will almost never succeed" is not equivalent to "with good people, you will almost always succeed."
 
Per comment thread RSS feeds please, Reg. I keep forgetting to check back here.
 
This also doesn't seem to be about blame, but about thoughtfully attempting to think a bit outside the box. This isn't however a new argument; the mythical man month goes into much detail on this.

I am interested that commenters have immediately separated developers and managers. I note though, that management plays a role in a project. Specifically, Wasabi doesn't cure any kind of rotten fish, from Salmon and Tuna to Squid. Even a broken methodology must be laid at the feet of the fish who chose it!

My two cents.
 




<< Home
Reg Braithwaite


Recent Writing
Homoiconic Technical Writing / raganwald.posterous.com

Books
What I‘ve Learned From Failure / Kestrels, Quirky Birds, and Hopeless Egocentricity

Share
rewrite_rails / andand / unfold.rb / string_to_proc.rb / dsl_and_let.rb / comprehension.rb / lazy_lists.rb

Beauty
IS-STRICTLY-EQUIVALENT-TO-A / Spaghetti-Western Coding / Golf is a good program spoiled / Programming conventions as signals / Not all functions should be object methods

The Not So Big Software Design / Writing programs for people to read / Why Why Functional Programming Matters Matters / But Y would I want to do a thing like this?

Work
The single most important thing you must do to improve your programming career / The Naïve Approach to Hiring People / No Disrespect / Take control of your interview / Three tips for getting a job through a recruiter / My favourite interview question

Management
Exception Handling in Software Development / What if powerful languages and idioms only work for small teams? / Bricks / Which theory fits the evidence? / Still failing, still learning / What I’ve learned from failure

Notation
The unary ampersand in Ruby / (1..100).inject(&:+) / The challenge of teaching yourself a programming language / The significance of the meta-circular interpreter / Block-Structured Javascript / Haskell, Ruby and Infinity / Closures and Higher-Order Functions

Opinion
Why Apple is more expensive than Amazon / Why we are the biggest obstacles to our own growth / Is software the documentation of business process mistakes? / We have lost control of the apparatus / What I’ve Learned From Sales I, II, III

Whimsey
The Narcissism of Small Code Differences / Billy Martin’s Technique for Managing his Manager / Three stories about The Tao / Programming Language Stories / Why You Need a Degree to Work For BigCo

History
06/04 / 07/04 / 08/04 / 09/04 / 10/04 / 11/04 / 12/04 / 01/05 / 02/05 / 03/05 / 04/05 / 06/05 / 07/05 / 08/05 / 09/05 / 10/05 / 11/05 / 01/06 / 02/06 / 03/06 / 04/06 / 05/06 / 06/06 / 07/06 / 08/06 / 09/06 / 10/06 / 11/06 / 12/06 / 01/07 / 02/07 / 03/07 / 04/07 / 05/07 / 06/07 / 07/07 / 08/07 / 09/07 / 10/07 / 11/07 / 12/07 / 01/08 / 02/08 / 03/08 / 04/08 / 05/08 / 06/08 / 07/08 /