Friday, March 24, 2006
I'll take Static Typing for $800, Alex.
There’s an argument that static typing prevents errors by detecting them at compile and/or edit time. In a trivial sense, this is absolutely true. You can, for example, write a Java program with such an error and watch Eclipse highlight the offence.
The interesting thing I’ve noticed is that many of the people in favour of static typing are arguing from a position of “Do what I say, not as I do.”
I don’t mean that they program in Python. I mean that when I ask them whether they would have typing troubles in a dynamic language, their answer is often “well, I wouldn’t, but every business needs hordes of monkey/offshore/intern/new graduate programmers who do make these errors.”
Let’s stop worrying about errors we don’t actually make. Let’s stop worrying about errors some hypothetical junior, error-prone programmer might make.
Here’s an interesting question: what sorts of typing errors do experienced, intelligent programmers actually make? And what sorts of typing errors sneak through unit tests and even QA and into production? And most especially, what sorts of typing errors have catastrophic consequences in production?
Now those are interesting errors. Those are worth worrying about. I’ll go further: those are worth static typing.
Here’s one from my actual, hands on experience. Distinguishing escaped from unescaped strings. I don’t know if I’m using the right words here: I’m thinking of a typical XML or XHTML application where some of the time a string is just a string, but some of the time it has a bunch of its characters replaced or escaped with special entities.
A Little Java, a Few Patterns: The authors of The Little Schemer and The Little MLer bring deep and important insights to the Java language. Every serious Java programmer should own a copy.How could static typing help? Well, imagine if you designate some strings as escaped and some as unescaped. So our type hierarchy is that there is String,
Now there are certain critical places where we would need to harden our application. The first is everywhere we get strings from users. These strings, just like
Then when we present strings, we type the methods as taking
To get around the errors, we need to escape our strings. We do that by writing a conversion method somewhere that, you guessed it, takes an
Naturally our application would be full of bookkeeping annotations as we keep track of which strings are escaped and which aren’t. But my gut feeling is that catching this kind of error at compile time would be worth it.
Here’s another error that I think is worth the effort of static typing. The bane of my existence when maintaining legacy Java code:
This is actually a solved problem in languages like Haskell. Static typing can easily distinguish between methods that might return a null (like getting a column from a database row) and variables that must not contain a null. The compiler can and should force you to write code that handles the null case.
The Little MLer introduces ML (and Ocaml) through a series of entertaining and straightforward exercises leading up to the construction of the Y Combinator.
With ML and Ocaml you can design rich types that fit the domain model and all types are checked at compile time through type inference.Here’s my question to my fellow Java programmers: why do we tolerate a compiler that forces us to type some things as
These are just two places where static typing could help experienced programmers solve problems that plague real, production code. I’m all for static typing, if it can help me with the errors I actually encounter.
That being said, there is a lot of work being done in this area, although obviously not by Sun or Microsoft (to be specific, not by their C# team). As mentioned, Haskell and several other languages provide static typing that is sophisticated enough to prevent errors like this.
I’m not even close to being the first person to notice the problem:
Here are my specific suggestions:
The interesting thing I’ve noticed is that many of the people in favour of static typing are arguing from a position of “Do what I say, not as I do.”
I don’t mean that they program in Python. I mean that when I ask them whether they would have typing troubles in a dynamic language, their answer is often “well, I wouldn’t, but every business needs hordes of monkey/offshore/intern/new graduate programmers who do make these errors.”
“People shouldn’t be able to open my classes.” “They need compile time type checking so the app doesn’t blow up.” “Extending system libraries is bad.”
Concerns like these all seem to boil down to one major theme: The people I work with are stupid.
—Java People Must Be Stupid, Pat Maddox
It’s not really stupid people, it’s people who’ve accepted a stupid idea. Java’s design is based on the idea that a language can prevent misuse by making bad things hard to do. It’s defensive thinking.
—A Favour, Giles Bowkett
Let’s stop worrying about errors we don’t actually make. Let’s stop worrying about errors some hypothetical junior, error-prone programmer might make.
Here’s an interesting question: what sorts of typing errors do experienced, intelligent programmers actually make? And what sorts of typing errors sneak through unit tests and even QA and into production? And most especially, what sorts of typing errors have catastrophic consequences in production?
Now those are interesting errors. Those are worth worrying about. I’ll go further: those are worth static typing.
Here’s one from my actual, hands on experience. Distinguishing escaped from unescaped strings. I don’t know if I’m using the right words here: I’m thinking of a typical XML or XHTML application where some of the time a string is just a string, but some of the time it has a bunch of its characters replaced or escaped with special entities.
Another case of escaped and unescaped strings concerns safely composing SQL queries and updates (another solved problem in other languages). The argument is always in favour of using library functions that do the conversion for you, likeI’m not alone in considering this a problem. Web applications that screw this up are vulnerable to cross site scripting (XSS) attacks. This is very bad, and if static typing could help I’d eagerly embrace it.PreparedStaement. If you think about it, that’s no damn good. What that does is treat everything like an unsafe String and only convert it at the very last second.
If you’re going to do any fancy SQL composition, you can only do it with stuff that isn’t a user value, so you have to keep track of escaped and unescaped strings anyways. And finally, your libraries still have all the unsafe APIs that don’t perform the conversion for you, so you are relying on your iron will and self-discipline to prevent errors, rather than having the compiler perform what is really a rather trivial check.
I have no problem with relying on iron will and self-discipline to eliminate errors. But if your argument is that iron will is appropriate for preventing SQL injection attacks, why isn’t iron will appropriate for preventing trivial type errors that would result in aMethodNotImplementedexception?
A Little Java, a Few Patterns: The authors of The Little Schemer and The Little MLer bring deep and important insights to the Java language. Every serious Java programmer should own a copy.
UnescapedString extends String, and EscapedString extends String. (Actually, I’d prefer interfaces if I were designing a Java-like language, but that’s by the by).Now there are certain critical places where we would need to harden our application. The first is everywhere we get strings from users. These strings, just like
tainted or unsafe variables in scripting languages like PHP, need to be UnescapedStrings. We would type our methods accordingly (in Java, this could be accomplished with annotations). For example, anything snarfed from the HttpRequest object is an UnescapedString.Then when we present strings, we type the methods as taking
EscapedStrings only. If we try to pass a POS (Plain Old String) or UnescapedString to a method parameterized by an EscapedString, we get a compile time error.To get around the errors, we need to escape our strings. We do that by writing a conversion method somewhere that, you guessed it, takes an
UnescapedString as a parameter and returns an EscapedString.Naturally our application would be full of bookkeeping annotations as we keep track of which strings are escaped and which aren’t. But my gut feeling is that catching this kind of error at compile time would be worth it.
Here’s another error that I think is worth the effort of static typing. The bane of my existence when maintaining legacy Java code:
NullPointerExceptions.This is actually a solved problem in languages like Haskell. Static typing can easily distinguish between methods that might return a null (like getting a column from a database row) and variables that must not contain a null. The compiler can and should force you to write code that handles the null case.
The Little MLer introduces ML (and Ocaml) through a series of entertaining and straightforward exercises leading up to the construction of the Y Combinator.
With ML and Ocaml you can design rich types that fit the domain model and all types are checked at compile time through type inference.
BigDecimals and some things as Integers, but we don’t insist that the compiler catch places where we aren’t checking for null?These are just two places where static typing could help experienced programmers solve problems that plague real, production code. I’m all for static typing, if it can help me with the errors I actually encounter.
That being said, there is a lot of work being done in this area, although obviously not by Sun or Microsoft (to be specific, not by their C# team). As mentioned, Haskell and several other languages provide static typing that is sophisticated enough to prevent errors like this.
I’m not even close to being the first person to notice the problem:
- Joel Spolsky described using naming conventions to highlight errors like this. It’s interesting that Hungarian Notation was invented for this kind of thing, but somehow a Cargo Cult has arisen around Systems Hungarian where programmers use it for things like marking integers, a fact that the compiler and IDE already know, but don’t use it for domain-specific things like whether a string is safe.
- Tim Sweeney of Epic Games wrote an incredibly lucid wish list for The Next Mainstream Programming Language, where he discusses in detail the exact issues his team has faced building and maintaining Unreal. His presentation is available in PowerPoint and PDF formats. He has given a lot of thought to how static typing could help build and maintain huge, complex, commercial applications.
Here are my specific suggestions:
- Stop worrying about the theoretical errors we don’t actually make. We unit test, we review our code. We’re not concerned with obvious, superficial problems.
- Agitate for language features that can help us solve these important problems. If the next version of javac can do escape analysis, it can identify potential null pointer exceptions, possibly through inference so we don’t even have to type more code.
- Educate ourselves about the bleeding edge of language development. No, that isn’t C# 3.0, Common Lisp, or Ruby 2.0. It’s ML, Haskell, Erlang, and a bunch of other things I need to learn. We may not be able to use Haskell to build Yet Another Boring Web Commerce Application, but we might learn enough to use a new naming convention or possible to write a string container that enforces escaped safety.
Labels: popular
Wednesday, March 22, 2006
An Encounter with a Programming Interview Problem
From the NexTag Java Development Jobs Page:
There's a name for this: The Lake Wobegon Effect. (This is related to the Blub Paradox, of course). How do we break out of this comfort zone? One way is by solving puzzles and problems where our solution is ranked against others. With that in mind, I decided to try this puzzle over lunch today.
My code runs in sixteen milliseconds or so on JDK 1.5/Eclipse in a garden-variety PC. After googling around, I discovered that the first version of my code could have been tighter and faster. That's terrific: I actually learned something by not being the best and the brightest today.
But worries me is this: how do I know whether my code could be even better? The Blub paradox dooms me to thinking I've got the answer, when perhaps I could be even better than I am. (If my code could be better, please comment or email. I want to hear from you!)
My tip is to seek out problems like this from time to time. Don't settle for thinking up the approach in your head and considering the actual coding to be an insignificant exercise in typing. Go ahead and write clean, debugged code, preferably with a time limit to put some pressure on yourself.
(By the way, some people argue that this type of problem has little to do with Enterprise stacks and all the other cruft of a daytime gig. This is a thinly veiled excuse for not trying, and roughly equivalent to an NFL linebacker refusing to do the bench press because an actual football game is more complicated than simply pushing an iron weight over his head while lying flat on his back.)
I saw mention of this over on reddit.com. It couldn't have come at a better time. I'd just read a provocative essay on programming's mediocracy: Beating the Average makes the point that programmers are terrible at judging their own skills. We all think we're good enough, and our environment does not promote the kind of objective competition and ranking that would shake us out of our complacency.Given a deck of nCards unique cards, cut the deck iCut cards from top and perform a perfect shuffle. A perfect shuffle begins by putting down the bottom card from the top portion of the deck followed by the bottom card from the bottom portion of the deck followed by the next card from the top portion, etc., alternating cards until one portion is used up. The remaining cards go on top. The problem is to find the number of perfect shuffles required to return the deck to its original order. Your function should be declared as:
static long shuffles(int nCards,int iCut);
Please send the result of shuffles(1002,101) along with your program and your resume to 'resume' at nextag.com.
There's a name for this: The Lake Wobegon Effect. (This is related to the Blub Paradox, of course). How do we break out of this comfort zone? One way is by solving puzzles and problems where our solution is ranked against others. With that in mind, I decided to try this puzzle over lunch today.
My code runs in sixteen milliseconds or so on JDK 1.5/Eclipse in a garden-variety PC. After googling around, I discovered that the first version of my code could have been tighter and faster. That's terrific: I actually learned something by not being the best and the brightest today.
But worries me is this: how do I know whether my code could be even better? The Blub paradox dooms me to thinking I've got the answer, when perhaps I could be even better than I am. (If my code could be better, please comment or email. I want to hear from you!)
My tip is to seek out problems like this from time to time. Don't settle for thinking up the approach in your head and considering the actual coding to be an insignificant exercise in typing. Go ahead and write clean, debugged code, preferably with a time limit to put some pressure on yourself.
(By the way, some people argue that this type of problem has little to do with Enterprise stacks and all the other cruft of a daytime gig. This is a thinly veiled excuse for not trying, and roughly equivalent to an NFL linebacker refusing to do the bench press because an actual football game is more complicated than simply pushing an iron weight over his head while lying flat on his back.)
Labels: jobs
Wednesday, March 15, 2006
Hunh? Could you say that again, but slowly, please?
I just read this Innitech-style elevator pitch for a product:
Tuesday, March 14, 2006
A "fair and balanced" look at the static vs. dynamic typing schism
A long time ago in an industry far, far away, everyone programmed in C.
The second was the almost complete lack of type safety. That's a big hand wave. I realize that if you're very careful you can make a C compiler, especially an ANSI C compiler, do a little static type checking for you. That's the theory. But in practice, it had this one little thing that made type safety moot: the unconditional type cast.
How many readers know what I'm talking about? Put your hand up... Good! Talk amongst yourselves for the next paragraph while I explain this to the readers who are under the age of forty.
This problem was so bad that it was one of the things C++ tried to fix with its casting constructs. One of them, the dynamic cast, specifically checks the situation at run time to make sure that the cast is safe. The only trouble was, C++ let you keep the dangerous C cast and then threw in a variation called static cast that was nicer looking but still unsafe some of the time.
I want to underline the consequences of bad casts here. When discussing risks, you always have to thing along two axes: the likelihood of disaster and the magnitude of the consequences. C programs with bad casts have extremely bad consequences. The might crash. They might corrupt all of their data. They could cause other things to crash. These are all terrible.
When we discuss "type safety" we have to think along the same two axes of risk: there is safety that protects us from something going wrong, and there is safety that limits the consequences when something does go wrong.
Well, one day Java came along. You may deride it as "Cobol Lite," but it did two things an awful lot better than C++, and if you look back at history it succeeded by converting people away from C++. The first thing it did better was to make automatic memory management mandatory.
The second was to throw out all of the dangerous casts and replace them with a single casting operation that always checks types at run time and throws a nice exception right away if you get it wrong. Java, in effect, provides two kinds of type safety: its static type checking reduces the likelihood of a typing error, and its runtime cast checking sharply limits the consequences of a bad cast.
Before we argue about static vs. dynamic type checking, let's remember where this fanaticism for so-called strong typing came from: it came from an entire industry that had been burned by having a trap door that led to having no typing. Back when it was C/C++ vs. Java, the debate was between having no type checking at run time vs. type checking at run time.
(Attention pedants: I'm very aware of how much static type checking ANSI C and C++ can do. I'm judging these languages by the bad news edge case bugs, not by the billions of perfectly fine lines of code that eschewed unsafe casts.)
For the most part, today we see people arguing about static vs. dynamic typing. They argue about the effort involved in telling the compiler what to check, and whether having the compiler find some errors for them does or doesn't make up for the extra code.
The static folks say "hell, yeah, you're headed for big trouble if a type checking bug slips through the compiler into production." Okay, let's look at the history. Back in the day when Reg was young and user interface design consisted of colouring the punch cards, a type error slipping into a C program represented a real risk of a catastrophic problem. True.
But let's say you want to use one of these "new-fangled" dynamic languages. Are you exposed to the same risk? The answer is no. The reason is that these dynamic languages have types and check them for you, just like Java's checked type casts. The consequences of a type error are trivial compared to C/C++ errors.
So you might try dereferencing the handle twice, looking up the offset of the pointer to the array, then looking up the offset of the function you want, dereference that pointer, and call the function. If any of this is wrong, boom. Actually, it's worse than boom. You might not find out it was wrong for a very long time. Silent but deadly!
In a dynamic language, you try sending a message to the object and if the object doesn't handle that message, there's a very explicit behaviour for managing the error. It calls a special missing method method. Or it throws a specific exception. Dynamic languages address the issue just like C++ did with dynamic casts and Java does all the time with its casts.
Dynamic language advocates argue for more automated unit tests instead of compiler checking. Well, I'm in favour of more testing whenever you can get it. But to my eye, that isn't the point. The point to me is that the potential for catastrophe with both Java and dynamic languages is so much smaller than the potential for catastrophe with C/C++ that the debate about type safety is almost moot.
I really don't want to get into a shrill "do so! do not!!" debate. But if you're reading this and you're still planted 100% in the strong, static type checking camp, let me ask you one question:
Have you actually worked on a project where casting errors caused the failure of the project? I mean the product failed in the marketplace, or you spent so much time trying to find and squash critical bugs that the project was cancelled?
I have a funny feeling that most or all of the people who answer "yes" were working with C/C++ and unconditional casts. I have a feeling that as an industry we're so scarred with those problems that we don't realize that moving to run time type checking solves 75% of our problems and makes the errors 99% less dangerous.
Like Joel says, this is a broad stroke observation. I happened to use Pascal an awful lot at the time, so I realize that "everyone" doesn't mean "every last one of us." But it does mean "so many people that no other language mattered to the industry as a whole."C has many delightful advantages. It also has two, umm, features that programmers routinely screwed up. The first, as everyone knows, is the lack of memory management. Programmers had to do everything themselves and do it properly. If you made a mistake, your program would suffer from dangling pointers, or have a memory leak.
The second was the almost complete lack of type safety. That's a big hand wave. I realize that if you're very careful you can make a C compiler, especially an ANSI C compiler, do a little static type checking for you. That's the theory. But in practice, it had this one little thing that made type safety moot: the unconditional type cast.
How many readers know what I'm talking about? Put your hand up... Good! Talk amongst yourselves for the next paragraph while I explain this to the readers who are under the age of forty.
Why C casts are dangerousOkay, everyone back? This was phenomenally bad, especially because we do this Von Neumann thing and have bytes of memory that are sometimes code to be executed and sometimes data to be shuffled around. If you have a pointer that is pointing to the wrong thing, the program would sometimes crash right away, or it would sometimes chug along for a while silently corrupting memory until everything failed. There's an entire industry of malicious people taking advantage of this possibility to craft buffer overflow attacks.
With an unconditional type cast, the C compiler would let you treat an address of something in memory as whatever you want. You could treat a byte as the start of a string. Or you could treat a long word as the address of a function. Or you could treat it as an array of pointers to functions. So if you had a pointer to the address of an array of pointers to functions and you mistakenly cast it to the address of an array of pointers to functions, and then you tried to call one of those functions, bad things would happen.
This problem was so bad that it was one of the things C++ tried to fix with its casting constructs. One of them, the dynamic cast, specifically checks the situation at run time to make sure that the cast is safe. The only trouble was, C++ let you keep the dangerous C cast and then threw in a variation called static cast that was nicer looking but still unsafe some of the time.
I want to underline the consequences of bad casts here. When discussing risks, you always have to thing along two axes: the likelihood of disaster and the magnitude of the consequences. C programs with bad casts have extremely bad consequences. The might crash. They might corrupt all of their data. They could cause other things to crash. These are all terrible.
When we discuss "type safety" we have to think along the same two axes of risk: there is safety that protects us from something going wrong, and there is safety that limits the consequences when something does go wrong.
Well, one day Java came along. You may deride it as "Cobol Lite," but it did two things an awful lot better than C++, and if you look back at history it succeeded by converting people away from C++. The first thing it did better was to make automatic memory management mandatory.
The second was to throw out all of the dangerous casts and replace them with a single casting operation that always checks types at run time and throws a nice exception right away if you get it wrong. Java, in effect, provides two kinds of type safety: its static type checking reduces the likelihood of a typing error, and its runtime cast checking sharply limits the consequences of a bad cast.
Before we argue about static vs. dynamic type checking, let's remember where this fanaticism for so-called strong typing came from: it came from an entire industry that had been burned by having a trap door that led to having no typing. Back when it was C/C++ vs. Java, the debate was between having no type checking at run time vs. type checking at run time.
(Attention pedants: I'm very aware of how much static type checking ANSI C and C++ can do. I'm judging these languages by the bad news edge case bugs, not by the billions of perfectly fine lines of code that eschewed unsafe casts.)
For the most part, today we see people arguing about static vs. dynamic typing. They argue about the effort involved in telling the compiler what to check, and whether having the compiler find some errors for them does or doesn't make up for the extra code.
The static folks say "hell, yeah, you're headed for big trouble if a type checking bug slips through the compiler into production." Okay, let's look at the history. Back in the day when Reg was young and user interface design consisted of colouring the punch cards, a type error slipping into a C program represented a real risk of a catastrophic problem. True.
But let's say you want to use one of these "new-fangled" dynamic languages. Are you exposed to the same risk? The answer is no. The reason is that these dynamic languages have types and check them for you, just like Java's checked type casts. The consequences of a type error are trivial compared to C/C++ errors.
How trivial can a type error be?Let's consider sending a message to an object (not all dynamic languages have objects, but whatever). How would this happen in C? Well, you'd have a pointer or a handle to an object, and it would have a pointer to an array of pointers to functions representing its methods. (It could be more complex if there's inheritance or aggregation).
Consider a web application. If a type error takes place, users will get a server error 500 response. The server will not crash, the database will not be infested with corrupt data, and users will not find themselves charging their purchases to someone else's credit cards. The consequences of a type error are relatively mild when the language checks types at run time.
So you might try dereferencing the handle twice, looking up the offset of the pointer to the array, then looking up the offset of the function you want, dereference that pointer, and call the function. If any of this is wrong, boom. Actually, it's worse than boom. You might not find out it was wrong for a very long time. Silent but deadly!
In a dynamic language, you try sending a message to the object and if the object doesn't handle that message, there's a very explicit behaviour for managing the error. It calls a special missing method method. Or it throws a specific exception. Dynamic languages address the issue just like C++ did with dynamic casts and Java does all the time with its casts.
Dynamic language advocates argue for more automated unit tests instead of compiler checking. Well, I'm in favour of more testing whenever you can get it. But to my eye, that isn't the point. The point to me is that the potential for catastrophe with both Java and dynamic languages is so much smaller than the potential for catastrophe with C/C++ that the debate about type safety is almost moot.
I really don't want to get into a shrill "do so! do not!!" debate. But if you're reading this and you're still planted 100% in the strong, static type checking camp, let me ask you one question:
Have you actually worked on a project where casting errors caused the failure of the project? I mean the product failed in the marketplace, or you spent so much time trying to find and squash critical bugs that the project was cancelled?
I have a funny feeling that most or all of the people who answer "yes" were working with C/C++ and unconditional casts. I have a feeling that as an industry we're so scarred with those problems that we don't realize that moving to run time type checking solves 75% of our problems and makes the errors 99% less dangerous.
Labels: popular
Friday, March 10, 2006
Roadmaps: Don't go there!
In the Wikipedia entry for Windows Vista, I found this gem:
Windows Vista was originally expected to ship sometime late in 2003 as a minor step between Windows XP and Windows "Vienna", formerly known as "Blackcomb". Gradually, Vista assimilated many important new features and technologies of "Vienna", and so the date of release was pushed back to first quarter of 2006.
That sounds familiar. How about the entry for Apple's Pink:
The OS developers had a meeting in which they decided what they should be doing in the future, and started writing down their ideas on index cards. Ideas that were simple and could be included in a new version of the existing software were written on blue colored cards, those that were more "far out" were written on pink cards.
And lo and behold, "Blue" grew and grew as it assimilated features and engineers from "Pink." Blue eventually shipped, late and bloated, as System 7. Pink was spun out and died as Taligent.
Both OS efforts started with a reasonable idea: focus on an achievable medium-term goal while simultaneously building towards a more aggressive long-term vision to follow.
The trouble with this strategy is that business pressures conspire to move features from the 'long-term' plan into the medium-term plan. People are inspired and want to ship good stuff as soon as possible. Competitors won't wait for your schedule and announce or even ship products with the features you planned to delay.
And customers want all that good stuff immediately. Why shouldn't they? By announcing your roadmap for the future you told them that they need those features!
My observation is that whenever a team or company announces a multi-version roadmap for the future, they end up pushing everything of substance into the immediate release, bloating what could have been an agile, achievable objective into a death march.
Joel Spolsky wrote an interesting essay about forcing competitors to react. He called this strategy "fire and motion." When you announce your own roadmap, you're firing at yourself! You've taken the decision for what to do and when to do it out of your own hands and given it to the marketplace.
Perhaps you didn't intend to aim at your own foot. But you certainly drew a bull's eye on your forehead, then walked down the street giving away ammunition, dressed as a target. With the deepest respect to my colleagues in Product Management, I suggest you do not announce a long term roadmap. It will bite you where you usually sit.
Some companies routinely get away with saying nothing or very little about the future. Right now, I'm thinking about Apple and Google.
Roadmaps. Don't go there.
post scriptum:
Another guy blathering about how to run a billion dollar company. But we need to ship software, how will this help?
Good point, here's what to do: scale this whole thing down to your world. Are you in the habit of promising stuff to your boss or colleagues? Focus on promising what your team can deliver in the next two-four weeks, tops. Everything else can be 'prioritized later'. Because if you say you can deliver A this week, B this month, and C this quarter, do not be surprised if you wind up trying to deliver A, B, and C this month.
Windows Vista was originally expected to ship sometime late in 2003 as a minor step between Windows XP and Windows "Vienna", formerly known as "Blackcomb". Gradually, Vista assimilated many important new features and technologies of "Vienna", and so the date of release was pushed back to first quarter of 2006.
That sounds familiar. How about the entry for Apple's Pink:
The OS developers had a meeting in which they decided what they should be doing in the future, and started writing down their ideas on index cards. Ideas that were simple and could be included in a new version of the existing software were written on blue colored cards, those that were more "far out" were written on pink cards.
And lo and behold, "Blue" grew and grew as it assimilated features and engineers from "Pink." Blue eventually shipped, late and bloated, as System 7. Pink was spun out and died as Taligent.
Both OS efforts started with a reasonable idea: focus on an achievable medium-term goal while simultaneously building towards a more aggressive long-term vision to follow.
The trouble with this strategy is that business pressures conspire to move features from the 'long-term' plan into the medium-term plan. People are inspired and want to ship good stuff as soon as possible. Competitors won't wait for your schedule and announce or even ship products with the features you planned to delay.
And customers want all that good stuff immediately. Why shouldn't they? By announcing your roadmap for the future you told them that they need those features!
My observation is that whenever a team or company announces a multi-version roadmap for the future, they end up pushing everything of substance into the immediate release, bloating what could have been an agile, achievable objective into a death march.
Joel Spolsky wrote an interesting essay about forcing competitors to react. He called this strategy "fire and motion." When you announce your own roadmap, you're firing at yourself! You've taken the decision for what to do and when to do it out of your own hands and given it to the marketplace.
Perhaps you didn't intend to aim at your own foot. But you certainly drew a bull's eye on your forehead, then walked down the street giving away ammunition, dressed as a target. With the deepest respect to my colleagues in Product Management, I suggest you do not announce a long term roadmap. It will bite you where you usually sit.
Some companies routinely get away with saying nothing or very little about the future. Right now, I'm thinking about Apple and Google.
Apple is an interesting case. They had to reveal their plans for Intel to get developers to port their binaries, so they had to open the kimono. But notice how they only really talked about what was next, not a multi-year schedule? And then they delivered way ahead of schedule, proving that they were actually discussing something they had already finished.If you cannot get away with saying nothing about your future plans, I recommend obfuscating as much as possible. Talk in generalities, talk about "spaces" and "partnerships," learn how to say absolutely nothing with flair. Do not show a mock up, do not leak a code name.
That is very different from discussing something that won't ship for a year or more. No matter how much customers bark about wanting stuff now, you can always hold them off for 60-90 days. Or release what you have now and start working on what they want.
Roadmaps. Don't go there.
post scriptum:
Another guy blathering about how to run a billion dollar company. But we need to ship software, how will this help?
Good point, here's what to do: scale this whole thing down to your world. Are you in the habit of promising stuff to your boss or colleagues? Focus on promising what your team can deliver in the next two-four weeks, tops. Everything else can be 'prioritized later'. Because if you say you can deliver A this week, B this month, and C this quarter, do not be surprised if you wind up trying to deliver A, B, and C this month.
Monday, March 06, 2006
I heard you twice the first time
Dear programmer:
Most people have at least one manager. They may not have eight managers, but they have at least one. The way the software business is usually structured, most programmers have two managers.
The first manager is often called a team leader. Programmers report directly to the team leader and work with her on a daily basis. Team leaders direct the work of programmers, mentor them, and evaluate their contribution.
Team leaders do not have true 'hire/fire' authority. They also usually don't have final say over things like deadlines and requirements (although this may not be the most effective way to do things, that's how it usually works). In other industries, team leaders often have titles like 'supervisor,' reflecting the fact that they direct the work but have little tangible authority.
Programmers also usually have a true manager, often the team leader's direct manager. This manager has some real authority, usually including the authority to hire, fire, set pay or bonus, and negotiate what can be done by when.
Most programmers have no trouble with this setup. They work well with their team leader, have a reasonable (but not awestruck) respect for their team leader's manager, and in the fullness of time their contributions are recognized in the form of a promotion to team leader or something more technical like 'architect' (whatever that means).
However... From time to time a programmer chafes. He smarts at the setup. He agitates.
In my experience, there are several reasons for this. Many programmers don't respect their team leader's technical competence. Some programmers combine an adversarial personality with a short term horizon, and they have trouble reporting to someone who can't impose immediate consequences or rewards. Some feel unfairly treated because they are not the anointed team leader and feel the need to express this feeling at every opportunity.
Obviously, the programmer, the team leader, and the team as a whole suffers when one of the members of the team begins to express their frustration with the situation. I'd like to share the manager's perspective. What do you think is running through my mind when the feces begin to flow uphill and land in my in box?
The first thing is, obviously the programmer is sending me a message that he feels the setup is wrong. He ought to be the team leader, or we shouldn't have a team leader, or maybe we should have a team and a team leader but he's an exception and ought to report to me, or we should have a separate technical structure and he should report to the Exalted Grand Visionary Poobah of Architecture Astronautics, or some other scheme.
Well, my door is open and I want to listen, and it's ok to tell me that. But do me a favour, okay? Tell me directly and tell me once.
Quite often people walk around with this dysfunctional idea that there's a black and white, right and wrong for everything. When things aren't exactly so, they say things "ought to" be another way. By strange coincidence, the way things "ought to be" happens to be something that favours them. Go figure.
Anyways, I have found that this "ought to" mentality invariably walks in tandem with a belief that everyone should know how things ought to be. This means, in practice, that the "ought to" believer shouldn't have to explain how things ought to be, the rest of us should just get it.
Please, please, please don't let this pathology take over your career. If you have an issue, the very best thing to do is pretend that I'm a smart guy, that I get it, that I understand and support you, and that perhaps I've been a little busy with a ton of other things that need my attention. So just walk up to me and tell me flat out what you want and why you want it.
Don't drop hints. Don't undermine the team leader. I may not act like I get why you shouldn't have a team leader, but I totally get that something's wrong when you constantly come to me for decisions and information that you should be getting from the team leader. Sure, I'm pissed off at the team leader for not keeping you happy.
And if that's the reaction you wanted, congratulations. But don't think that somehow you're smelling like a rose. After all, everyone else seems to be getting along ok. Am I supposed to believe that you are the sole voice of reason in the building?
I also get that something's wrong when you undermine the team leader by somehow forgetting to do or being to busy to do a zillion things she asks you to do, like the administrivia of project management. But you know I don't get? I don't get the reports and statistics and numbers I need to do my job, which is convincing the people that write the cheques to keep the money flowing.
You know, I'm somewhat aware of the team leader's strengths and weaknesses. And thanks for double underlining the weaknesses in red pencil. And maybe she won't have my job one day, or my boss' job. Maybe I ought to find someone else to lead your team.
But you know what? I may be busy. I may have other plans for her and for you. I may have other priorities. That doesn't mean I will or won't act on your initiative. But it does mean that dropping hint after hint after hint very quickly goes from irritating to full blown obstruction to success.
Right now, I need software. Are your actions helping us ship higher quality software, with less risk, in less time? So...
Do yourself a favour. Come to see me. Talk to me. Make sure I understand by speaking plainly. And then listen to what I have to say. Be sure you understand. And then stop dropping hints. One way or the other, let's ship software.
Yours very truly,
Reginald Braithwaite
post scriptum
I've run into this several times in my career. From my experience, once it reaches the point of sabotaging the man in the middle, both parties become very tarnished, very quickly.
I'll share one story where I was the hapless man in the middle. I was managing the development activities for one company, where I had several team leaders reporting to me. The company founder also had an R&D team cooking up some revolutionary software.
This team had been whiling away the years on this project, and to all appearances they had an unlimited budget and schedule. The project required a very deep domain knowledge, and the team tended to hire people with high domain knowledge and little or even no experience writing software, much less developing products.
The founder was exceptionally interested in this project and loved to brainstorm with the team. One day I got the news: he wanted me to manage this team, to teach them how to ship software. He saw their brains and my experience as a match made in heaven. The team's leader was told to stop reporting to the founder and start reporting to me.
Well, managing software development isn't that hard, is it? I asked the team leader for a list of what he thought the team could accomplish, by when. I simultaneously asked the company's business development folks for a list of what we needed to have in order to close deals. I proposed that we put the two lists together and create a development schedule.
Apparently this was not to the R&D team leader's liking. He questioned whether someone lacking a doctoral degree in his field could understand the software, much less participate in its creation.
He flat out refused to participate in any management exercise I cared to propose, letting me know that the software would be ready when his team shipped it, and that it would contain whatever features his team deemed necessary to include, and that the team was not going to commit to those features in advance, as research was still underway.
And all the while, he continued to deal directly with the company founder, who remained involved as a 'product manager' while we pretended that I was the actual manager.
Needless to say, that didn't last very long. The company founder realized that the team leader was forcing his hand and reluctantly took action.
Most people have at least one manager. They may not have eight managers, but they have at least one. The way the software business is usually structured, most programmers have two managers.
The first manager is often called a team leader. Programmers report directly to the team leader and work with her on a daily basis. Team leaders direct the work of programmers, mentor them, and evaluate their contribution.
Team leaders do not have true 'hire/fire' authority. They also usually don't have final say over things like deadlines and requirements (although this may not be the most effective way to do things, that's how it usually works). In other industries, team leaders often have titles like 'supervisor,' reflecting the fact that they direct the work but have little tangible authority.
Programmers also usually have a true manager, often the team leader's direct manager. This manager has some real authority, usually including the authority to hire, fire, set pay or bonus, and negotiate what can be done by when.
Most programmers have no trouble with this setup. They work well with their team leader, have a reasonable (but not awestruck) respect for their team leader's manager, and in the fullness of time their contributions are recognized in the form of a promotion to team leader or something more technical like 'architect' (whatever that means).
However... From time to time a programmer chafes. He smarts at the setup. He agitates.
In my experience, there are several reasons for this. Many programmers don't respect their team leader's technical competence. Some programmers combine an adversarial personality with a short term horizon, and they have trouble reporting to someone who can't impose immediate consequences or rewards. Some feel unfairly treated because they are not the anointed team leader and feel the need to express this feeling at every opportunity.
Obviously, the programmer, the team leader, and the team as a whole suffers when one of the members of the team begins to express their frustration with the situation. I'd like to share the manager's perspective. What do you think is running through my mind when the feces begin to flow uphill and land in my in box?
The first thing is, obviously the programmer is sending me a message that he feels the setup is wrong. He ought to be the team leader, or we shouldn't have a team leader, or maybe we should have a team and a team leader but he's an exception and ought to report to me, or we should have a separate technical structure and he should report to the Exalted Grand Visionary Poobah of Architecture Astronautics, or some other scheme.
Well, my door is open and I want to listen, and it's ok to tell me that. But do me a favour, okay? Tell me directly and tell me once.
Quite often people walk around with this dysfunctional idea that there's a black and white, right and wrong for everything. When things aren't exactly so, they say things "ought to" be another way. By strange coincidence, the way things "ought to be" happens to be something that favours them. Go figure.
Anyways, I have found that this "ought to" mentality invariably walks in tandem with a belief that everyone should know how things ought to be. This means, in practice, that the "ought to" believer shouldn't have to explain how things ought to be, the rest of us should just get it.
Please, please, please don't let this pathology take over your career. If you have an issue, the very best thing to do is pretend that I'm a smart guy, that I get it, that I understand and support you, and that perhaps I've been a little busy with a ton of other things that need my attention. So just walk up to me and tell me flat out what you want and why you want it.
Don't drop hints. Don't undermine the team leader. I may not act like I get why you shouldn't have a team leader, but I totally get that something's wrong when you constantly come to me for decisions and information that you should be getting from the team leader. Sure, I'm pissed off at the team leader for not keeping you happy.
And if that's the reaction you wanted, congratulations. But don't think that somehow you're smelling like a rose. After all, everyone else seems to be getting along ok. Am I supposed to believe that you are the sole voice of reason in the building?
I also get that something's wrong when you undermine the team leader by somehow forgetting to do or being to busy to do a zillion things she asks you to do, like the administrivia of project management. But you know I don't get? I don't get the reports and statistics and numbers I need to do my job, which is convincing the people that write the cheques to keep the money flowing.
You know, I'm somewhat aware of the team leader's strengths and weaknesses. And thanks for double underlining the weaknesses in red pencil. And maybe she won't have my job one day, or my boss' job. Maybe I ought to find someone else to lead your team.
But you know what? I may be busy. I may have other plans for her and for you. I may have other priorities. That doesn't mean I will or won't act on your initiative. But it does mean that dropping hint after hint after hint very quickly goes from irritating to full blown obstruction to success.
Right now, I need software. Are your actions helping us ship higher quality software, with less risk, in less time? So...
Do yourself a favour. Come to see me. Talk to me. Make sure I understand by speaking plainly. And then listen to what I have to say. Be sure you understand. And then stop dropping hints. One way or the other, let's ship software.
Yours very truly,
Reginald Braithwaite
post scriptum
I've run into this several times in my career. From my experience, once it reaches the point of sabotaging the man in the middle, both parties become very tarnished, very quickly.
I'll share one story where I was the hapless man in the middle. I was managing the development activities for one company, where I had several team leaders reporting to me. The company founder also had an R&D team cooking up some revolutionary software.
This team had been whiling away the years on this project, and to all appearances they had an unlimited budget and schedule. The project required a very deep domain knowledge, and the team tended to hire people with high domain knowledge and little or even no experience writing software, much less developing products.
The founder was exceptionally interested in this project and loved to brainstorm with the team. One day I got the news: he wanted me to manage this team, to teach them how to ship software. He saw their brains and my experience as a match made in heaven. The team's leader was told to stop reporting to the founder and start reporting to me.
Well, managing software development isn't that hard, is it? I asked the team leader for a list of what he thought the team could accomplish, by when. I simultaneously asked the company's business development folks for a list of what we needed to have in order to close deals. I proposed that we put the two lists together and create a development schedule.
Apparently this was not to the R&D team leader's liking. He questioned whether someone lacking a doctoral degree in his field could understand the software, much less participate in its creation.
He flat out refused to participate in any management exercise I cared to propose, letting me know that the software would be ready when his team shipped it, and that it would contain whatever features his team deemed necessary to include, and that the team was not going to commit to those features in advance, as research was still underway.
And all the while, he continued to deal directly with the company founder, who remained involved as a 'product manager' while we pretended that I was the actual manager.
Needless to say, that didn't last very long. The company founder realized that the team leader was forcing his hand and reluctantly took action.
Labels: jobs
