raganwald
(This is a snapshot of my old weblog. New posts and selected republished essays can be found at raganwald.com.)

Monday, September 24, 2007
  Ockham's razor as it applies to the big rewrite


Oh dear, the latest nine days wonder seems shocking on the face of it. To quote the protagonist in the tale:

Back in January 2005, I announced on the O’Reilly blog that I was going to completely scrap over 100,000 lines of messy PHP code in my existing CD Baby (cdbaby.com) website, and rewrite the entire thing in Rails, from scratch. I hired one of the best Rails programmers in the world (Jeremy Kemper aka bitsweat), and we set off on this huge task with intensity…

Two years (!) later, after various setbacks, we were less than halfway done… I said fuckit, and we abandoned the Rails rewrite… Then in a mere TWO MONTHS, by myself, not even telling anyone I was doing this, using nothing but vi, and no frameworks, I rewrote CD Baby from scratch in PHP. Done! Launched! And it works amazingly well.
Derek Sivers

There has been a running flamestorm over this, mostly centered on whether Rails is or isn’t the perfect framework for web applications. In other words, there is a prevailing sentiment that this rewrite failed for technical reasons.

I have my own experience with project failure, and I can’t blame any of it for technical problems. This is because, in my experience, the technical problems in projects are not root causes, they are symptoms of people and management problems.

Amazingly, Chad Fowler identified six reasons that big rewrites of production systems fail. Given that opinions are like noses (everybody has one), what’s so amazing about Chad’s list? What’s amazing is that Chad wrote his list a year ago, and yet he thoroughly nailed Derek’s experience.





Expecting a plug for a book about the latest miracle cure for project failure blues? Nope, sometimes the technical whizzies blind us to basic, fundamental wisdom that is as true today as it was twenty or even forty years ago. Don’t believe the hype.

For a solid grounding on how to successfully develop software, start with The Mythical Man-Month: Essays on Software Engineering. It is one of the most important books ever written about developing software, from the small to the large. Read the book that spawned the expression, “There is no silver bullet.”

I’ll bring up his first reason, Software as Spec. Re-read the condensed story above. Note, with horror, that a Rails expert took two years to fail what Derek accomplished in just two months. Well, if Jeremy couldn’t do it, obviously Rails is seriously broken, a complete toy. But this is like a sleight-of-hand trick. The paragraphs tell two stories, and we are so focused on one, “Two years for Rails, two months for PHP,” that we ignore the prestidigitator’s other story:

Two years for Jeremy, two months for Derek.

This is a complex, production system. When writing any complex, production system, what is the number one biggest risk? No, not technical risk. Requirements risk. Jeremy tried to create a copy of an existing, messy production system. Sure, he had Derek pulling the strings and calling the shots. But this is no guarantee of success, especially in an environment with heavy integration with other messy, chaotic systems.

Communicating requirements is hard enough in a green field situation. It’s nearly impossible when dealing with a real, production piece of code. Jeremy had almost no chance if his goal was to duplicate the original system’s functionality based on Derek’s direction and even assistance.

Derek, on the other hand, wrote the original system. He knows it intimately. If there is some weird, bogus code in one corner, Derek remembers why it’s there, what obscure bug it fixes, or perhaps what bug in some other piece of software’s code it mediates. If something doesn’t need to be there any more, Derek knows it can be dropped and he knows why it ought to go.

Of course Derek succeeded where Jeremy failed. Having Jeremy performing the rewrite with Derek’s assistance created an insurmountable barrier for understanding. Even with Derek trying to assist, the very fact that they were rewriting in Rails created a double whammy: Derek knew the system but had to try to explain its subtleties to Jeremy. Jeremy knew Rails but had to try to explain its subtleties to Derek.

Derek’s two-month rewrite with the hindsight of implementing the system in the first place and then living through the first rewrite failure is easy to understand: one person knew both the system and the implementation technology intimately. No communication barrier. No requirements risk.

If there is one lesson to draw here it is that “There is always more than one way to look at any issue.”
sikko


simplicity

So, was the problem choosing Rails? Or choosing to bring in an outsider to implement a technology the insider didn’t know intimately? This is a false dichotomy, of course. Failed projects may have many problems.

But when trying to simplify the situation and derive useful conclusions, my heuristic is to look for the simplest explanation, the one with the fewest moving parts. The technology explanation is very specific and with many dangling caveats. It only applies to projects involving rewrites from PHP to Rails, and even then only those projects that are messy and have legacy data models that aren’t particularly CRUD-dy.

The people explanation is much simpler. It explains what we observed here as well as what we observed in many other projects, and it does so with very few caveats and conditions.

My personal heuristic is to go with the simpler explanation that applies to the widest possible set of observations in the wild.

As several people have pointed out here and elsewhere, we are working with anecdotal evidence. And the plural of anecdote is not data. But we must soldier on in our industry. If we waited until we could rigorously “prove” anything about software development before forming conclusions, we wouldn’t have budged from toggling switches on front panels.
 

Comments on “Ockham's razor as it applies to the big rewrite:
See Reddit for a counterexample where completely rewriting a system in another language for technical reasons can turn out to be a net win and ends up getting you rich.

Maybe they should write an article or blog post about it and let other people advocate the "intelligence" of their decision through anectodal evidence and appeal to authority. Everyone's an expert.
 
If you mean the rewrite of reddit in Python instead of Lisp, I found the differences between the two scenarios interesting:

First, my understanding is that the original authors performd the reddit rewrite. They didn't bring in a "Snake Charmer."

Second, my understanding is that reddit was quite a bit smaller than CD baby at the time. Half of CD baby's challenge was the 100,000+ LOC. The other half is all of the systems integrated with it.

Reddit had a much smaller set of hard requirements at that stage of its maturity.

Third, the original system was badly broken for whatever reason (it could be that they were actually quite inexperienced with Lisp when they wrote version one of Reddit). They had a lot less to lose than Derek, who admits his system worked.

Like you say, everyone is an expert. Thanks to blogs, people like me can stop writing on bathroom walls.
 
I'd say that rails requiring you to restructure your entire database to fit its object model is a serious issue for all legacy apps, no matter who is doing the rewrite.
 
sean's right. For good or ill, Rails's opinion is that the right way to integrate a Rails app with other systems is through REST, not a shared database. If existing applications all interact with the same database, migrating one application to Rails means either working around the assumptions of ActiveRecord or updating all of the other applications to use the RESTful interface.

I don't envy people facing either one.
 
First, my understanding is that the original authors performd the reddit rewrite. They didn't bring in a "Snake Charmer."

Reddit did bring in a "snake charmer" named Aaron Swartz:

http://www.aaronsw.com/weblog/rewritingreddit
 
Cool, thanks, Spez!
 
Notice that the Reddit case is not a complete counterexample either. They first tried to use Django (it's like Python's version of Rails) and it did not work for them. They ended up using a different framework (webpy) that fit their needs better. Personally I think after people factors, the frameworks and libraries would make a bigger impact than the language. If it had been a rewrite from Fortran or C it would have been worth considering, but it is not like PHP, Python and Ruby are really that much different from each other.
 
Opinions are like noses -- everyone has one. And they all smell.

(Although I had always heard it about excuses: Excuses are like a******s -- everyone has one, and they all stink.)
 
The plural of anecdote is data, as long as you don't forget to include a confidence interval.

Similarly, data without a confidence interval are anecdotes.
 
Reg, the changes to your post have also changed the tone (from my perspective), such that my original critique is no longer relevant or useful. I've updated my post to reflect that reality.
 
It's Occam, not Ockham. Glad somebody else noticed this too, by the way. This whole hubbub hasn't got anything to do with languages in the first place.
 
It certainly is Occam. But perhaps it is also Ockham?

William of Ockham (c. 1288 - c. 1348) was an English Franciscan friar and scholastic philosopher, from Ockham, a small village in Surrey, near East Horsley.
 
"Occam" is a latinate transliteration -- the "k" of the English original would not have flown.
 
"This whole hubbub hasn't got anything to do with languages in the first place."

Well, but that's been Reg's point all along, hasn't it?
 
Yes, Mark. I was agreeing with him.
 
doh. my bad about the spelling.
 
Re Joel's 2000 posting: Netscape/mozilla did survive the rewrite, and now we are all grateful for a Firefox that seems pretty responsive to moving requirements e.g. HTML 5.
 




<< Home
Reg Braithwaite


Recent Writing
Homoiconic Technical Writing / raganwald.posterous.com

Books
What I‘ve Learned From Failure / Kestrels, Quirky Birds, and Hopeless Egocentricity

Share
rewrite_rails / andand / unfold.rb / string_to_proc.rb / dsl_and_let.rb / comprehension.rb / lazy_lists.rb

Beauty
IS-STRICTLY-EQUIVALENT-TO-A / Spaghetti-Western Coding / Golf is a good program spoiled / Programming conventions as signals / Not all functions should be object methods

The Not So Big Software Design / Writing programs for people to read / Why Why Functional Programming Matters Matters / But Y would I want to do a thing like this?

Work
The single most important thing you must do to improve your programming career / The Naïve Approach to Hiring People / No Disrespect / Take control of your interview / Three tips for getting a job through a recruiter / My favourite interview question

Management
Exception Handling in Software Development / What if powerful languages and idioms only work for small teams? / Bricks / Which theory fits the evidence? / Still failing, still learning / What I’ve learned from failure

Notation
The unary ampersand in Ruby / (1..100).inject(&:+) / The challenge of teaching yourself a programming language / The significance of the meta-circular interpreter / Block-Structured Javascript / Haskell, Ruby and Infinity / Closures and Higher-Order Functions

Opinion
Why Apple is more expensive than Amazon / Why we are the biggest obstacles to our own growth / Is software the documentation of business process mistakes? / We have lost control of the apparatus / What I’ve Learned From Sales I, II, III

Whimsey
The Narcissism of Small Code Differences / Billy Martin’s Technique for Managing his Manager / Three stories about The Tao / Programming Language Stories / Why You Need a Degree to Work For BigCo

History
06/04 / 07/04 / 08/04 / 09/04 / 10/04 / 11/04 / 12/04 / 01/05 / 02/05 / 03/05 / 04/05 / 06/05 / 07/05 / 08/05 / 09/05 / 10/05 / 11/05 / 01/06 / 02/06 / 03/06 / 04/06 / 05/06 / 06/06 / 07/06 / 08/06 / 09/06 / 10/06 / 11/06 / 12/06 / 01/07 / 02/07 / 03/07 / 04/07 / 05/07 / 06/07 / 07/07 / 08/07 / 09/07 / 10/07 / 11/07 / 12/07 / 01/08 / 02/08 / 03/08 / 04/08 / 05/08 / 06/08 / 07/08 /