Hubris Sun’s CEO Jonathan Schwartz opines on Java:

It's one of the few technologies that may outlast the century.

  ¶ 3:56 PM

  SOA for Dummies

Suppose you are a programmer and you have to build a large, complicated application. You have to break up the code into pieces so you can keep everything straight and not get lost. For years, programmers have divided code into interconnected hierarchies of objects, a practice called object-oriented programming. Service Oriented Architecture is an alternative code organization, based on services. For example, you might have a weather service that returns weather conditions, given a city. A service is a chunk of code that has several properties:

1. A service can take requests


2. Given a well-formed request, a service will send a response


3. A service can use other services

Service Oriented Architecture has become a major buzzword in the software design community, and everyone and his code monkey is claiming to use it. So if it doesn’t do what’s promised, a lot of people are going to lose a lot of money.

Many people have noted my interest in software architecture and steered me in the direction of the big new thing, Service Oriented Architecture. I am always bemused by SOA, not least of which is that I can’t help but think that the acronym is unfinished and really needs a “B” on the end.

I am pleased to tell you that SOA really does work, and has been proven to work in the field, under rigourous conditions, by real programmers and not just consultants touting books or neophyte programmers trying to inflate their resumés.

Those of you born after 1980 will be surprised to hear that SOA was first invented in the 1960s. They called in Procedural Programming in those days, and it featured this nifty abstraction where:

1. A procedure can take requests called “Procedure Calls”


2. Given a well-formed request, a procedure will provide a response


3. A procedure can use other procedures

It worked remarkably well, so well that in 1976 or thereabouts, somebody got the bright idea that you should be able to call procedures on other systems, so that a program talking to the user on a system in their office might invoke a procedure on a different computer running a database.

This was called a remote procedure call, a remarkably simple name: you take the name of the thing you already understand and merely explain how it is slightly different.

The industry sliced and diced this idea, reinventing it many times. For example, one company was not satisfied with calling things procedures and remote procedure calls, so it invented the new acronyms COM and DCOM.

I am told that academics have proven that a computer program which prints all of the variations of this idea cannot be proven to halt, possibly because we invent new ones faster than we can list the old ones. But I am quite sure that SOA is nothing more or less than that old idea, wearing new clothes worthy of an emperor.

And when I see criticism of SOA, I am quite sure that it is fully congruent to valid criticism of Procedural Programming. For example, when told that SOA can cause Strange Loops, I think of John Lakos’excellent book Large-Scale C++ Software Design, where he spends 896 pages explaining how to eliminate the loops from large C++ programs for speed of compilation and testability.

Or when told that SOA hides the implementation of services, so that clients cannot be sure when the behaviour of services change, I am reminded of a very old debate about Leaky Abstractions. I do not expect this debate to go away any time soon, even if we discard SOA in favour of the next shiny consulting toy.

Of course, Procedural Programming was supplanted in the mainstream “Blub” community by Object-Oriented Programming. I am not surprised to see that the main opponents of SOA advocate something called “REST.” REST’s chief feature seems to be that it favours decomposing large architectures into a taxonomy of entities with well-defined interfaces (PUT, GET, DELETE, and so forth).

Alphonse Karr said it best:

Plus ça change, plus c’est la même chose.

  ¶ 3:22 PM

  Off Topic: Blaming the Victims

Frankly, the world wasn’t 100 percent ready for Windows Vista.

Why no, we weren’t ready. We were “ready” for a rock-solid, innovative operating system with 50% or more managed code, the transactional WinFS file system, and other compelling features. And we were ready for it sometime around when you told us to expect it, not two years later. What were we thinking?

The lukewarm, apathetic response to the Vista you actually shipped is clearly our fault. We were not ready. Thanks for reminding us.  ¶ 1:40 PM

  Off Topic: Meanies! This exchange on comp.compression is like one of those depressing movies where you suddenly realize you don’t like any of the characters.

There’s this guy, you think maybe he’s misguided and needs a few pointers to realize he’s trying to create a perpetual motion machine. But no, he turns out to be a complete troll with sock puppets flying in formation.

And his interlocutors? Are they patient, nice people pushed over the edge by his behaviour? No, they come across as just plain mean, like a group of nerds in the high school cafeteria laughing at someone because he isn’t as smart as they think they are. Yes, the guy is a troll, but there’s absolutely no need to be so rude to him. Especially when plonk is a key press away.

My experience is that rude behaviour is never forced upon people by circumstance. Some people are rude by nature, some people aren’t. Don’t tell me you have to be rude to people like him because of such and such. That’s just rationalization.

Argue with idiots, and you become an idiot.

The moral of the story is this: do not be lured into taunting or insulting someone on the premise that they are wrong, stupid, or rude themselves. They’re an idiot. Fine. Keep that opinion to yourself. Sharing it with the whole world literally drags you into the muck and reflects very poorly on your character.

I am not giving you advice here, I am sharing a painful realization. Looking back on many of my own exchanges, once the heat of the moment passes, I am often dismayed at what I have written, at the tone I have taken, at the sheer meanness of my flames.

Learn from my experience: if there is no intellectual benefit to be had from a discussion, walk away.

---

p.s. My comment policy on my blog, as always, is that rudeness to me is always tolerated. Well, I prefer that you not be profane, as a courtesy to other readers, but being mean to me is always acceptable.  ¶ 9:49 AM

  Ruminations about the performance of anonymous functions in naive Javascript implementations Block-Structured Javascript (better known as the Module Idiom) looks like this:

 (function () {
    var something_or_other;
    // code elided
    return something_or_other;
 })()

This creates a new, anonymous function with its own local scope. Whenever this code is execututed, the interpreter creates a function record in its memory. The exact same thing happens if you create a function and bind it to a variable with var foo = function (...) { ... };.

Now let’s consider another common pattern, the Inner Function: we have a function, and the function needs a helper function. We define the helper function inside our function to make our code more encapsulated:

 var factorial = function (n) {
     var factorial_acc = function (acc, m) {
         if (0 == m) {
             return acc;
         } else {
             return factorial_acc(m * acc, m - 1);
         }
     };
     return factorial_acc(1, n);
 };

What happens when we invoke factorial it six times?

When the interpreter first encounters the code defining factorial, it creates a function and assigns it to the variable factorial. Then each time we invoke the factorial function, the interpreter creates a new function record for factorial_acc. So in total, the interpreter creates seven functions in memory, not two.

hand-rolling

If this code needed hand optimization, you might want to consider ‘lifting’ the definition of factorial_acc outside of factorial, so it doesn’t get recreated with every invocation:

 var factorial_acc = function (acc, m) {
     if (0 == m) {
         return acc;
     } else {
         return factorial_acc(m * acc, m - 1);
     }
 };
 var factorial = function (n) {
     return factorial_acc(1, n);
 };

This produces exactly the same result as our Inner Function version. factorial_acc doesn’t use any of factorial’s parameters or variables, so it does not really need to be inside its scope to produce the correct result.

Now you only need two function records, not seven. Two is cheaper than seven. The problem with this approach is that you are proliferating names. If you are binding functions to names in the global environment, it quickly becomes crowded. And you also have a readability issue. Does anything else need to use factorial_acc? The original code made it very obvious that factorial_acc is only ever used by factorial.

A block can help. Yes, the cause of our performance consideration—dynamically creating functions—can actually be part of the solution:

 var factorial = (function () {
     var factorial_acc = function (acc, m) {
         if (0 == m) {
             return acc;
         } else {
             return factorial_acc(m * acc, m - 1);
         }
     };
     return function (n) {
         return factorial_acc(1, n);
     }
 })();

Now what happens? Well, we create an anonymous function for our block. One function record. Within that block’s execution, we create two more functions, one assigned to the variable factorial_acc, and one returned from the block (and then assigned to the variable factorial). This code creates three function records, which is still much better than seven.

As a correspondent summarized in email, “we’ve shown how to replace a simple function containing an inner function with a block call that returns a closure referencing the inner function so as to avoid re-defining it on each call. That’s all there is to it.”

(By the way, Douglas Crockford has done a very good job of explaining this idiom in Javascript, and named it the Module Pattern. Here’s a discussion with particular emphasis on OO-style programming. And here’s a really detailed examination from the YUI team.)

So should you always rewrite inner functions to use a block like this?

I don’t personally fool around with this kind of hand optimization willy-nilly (Of course, you may find the block version more readable than the inner function version. If you do, it’s a win to write it that way). It has a cost: in a more complex function, defining helpers outside of the function may be moving them further away from where they are used, which is a loss for readibility. If you prefer the inner function version, you should be very sure you have a performance problem before you leap to the conclusion that you should rewrite it.

a heuristic for automatic optimization of inner functions and blocks

Lisp implementations have been optimizing this kind of code, automatically, for decades. That’s because Lisp programmers have been writing programs in this style for decades, either directly or using macros like let. Here’s the basic heuristic:

• Find every function invocation that is nested inside another function.


• Analyze all of its variable references lexically. If there are no references to a parameter or local variable of its immediate parent,¹ promote the function by defining it in its grand-parent and assigning the definition to a new variable in the grand-parent.


• Replace the original definition in the parent with a reference to the variable you just created in the grand-parent.


• Repeat until all functions are either global or refer to variables in their immediate parent functions.

There’s also a well-know optimization for making blocks themselves free or nearly free: lambda lifting. So before optimizing things prematurely, test your implementation and see if it is already fast enough for your purposes.

You may discover that you don’t save anything by rewriting things yourself. (You may make your code slower: some optimizations rely on knowing the exact scope of the code being optimized. If you proliferate names by lifting things yourself, the optimizer may not be able to use all of its tricks.)

These techniques have been known for twenty-five years. If a Javascript implementation that you are forced to target doesn’t include it, why not demand that the implementers get with the program and, you know, use some of the stuff we’ve known about programming for almost as long as they’ve been alive? Especially if they brag about their prowess at creating programming languages?

conclusion: nice-to-know, but not essential

My personal conclusion is that the behaviour of a naïve implementation is a “nice-to-know.” I don’t personally worry about optimizing it until I have a known performance issue, at which point it is essential to test to see whether some of the hand-optimizations will actually help.

YMMV.

---

1. Full closures make things tricky:

Functions that refer to variables in their immediate parent scope are much trickier to optimize away. Sometimes, such a function is supposed to be created anew for each invocation of its parent. For example, if you want to construct a bank balance thingy without using objects, you might write:

    function (balance) {
      return function (amount) {
          balance = balance + amount;
          return balance;
        };
    };

You pass in an initial balance, and it gives you a single function that you can use to deposit (pass a positive number), withdraw (pass a negative number), or check the balance (pass zero). It returns the updated balance in each case.

The inner anonymous function cannot be lifted or optimized away because of its reference to balance in its parent and because the function can be shown to “escape” its parent.

Whereas in this contrived example:

    function (balance, owner) {
      return function (amount) {
          return {
              new_balance: (function () {
                  if (balance + amount >= 0)
                      return balance + amount;
                  else return balance;
              })(),
              account_owner: owner
          }
        };
    };

Although the inner function still cannot be optimized away, the block within it can be lifted into the inner function and removed, producing:

    function (balance, owner) {
      return function (amount) {
          var __temp;
          if (balance + amount >= 0)
              __temp = balance + amount;
          else __temp = balance; 
          return {
              new_balance: __temp,
              account_owner: owner
          }
        };
    };

Labels: lispy, popular

  ¶ 11:44 AM

  How many products make up a project? Software applications are often defined in advance as masses of features, loosely organized into modules and components. One of the things I do as a matter of course when starting a project is to count the number of products. A “product” is a piece of software that stands alone and delivers value. In the context of a big project, products are parts of the project that could, all by themselves, be projects.

For example, I was once working with a company that wanted to move their customer service operations from an old proprietary minicomputer system to a web system with an equally proprietary but much more current database back-end. They also wanted “workflow” while we were at it.

This is not about maximizing programmer productivity. In fact, it has nothing to do with programmers. It’s about a very scarce resource: management attention.

I pointed out that the web “port” was a product, and so was the workflow. You could build a workflow system that worked alongside the legacy software, and it would deliver business value. Or you could port the legacy application, and again deliver business value. So this project was really two separate products.

The value of drawing this distinction lies in understanding that the capacity of a team to effectively develop simultaneous products is nearly always one. That’s right, no matter what the size, teams are most effective when they work on one product at a time. And by teams, I mean the full team, including stake-holders, business analysts, everyone, not just the programmers.

When you work on one product, you have a clear vision, priorities are straightforward, you know when you’re “done”… It all works smoothly. When you work on two or more products simultaneously, all this falls apart in a hurry.

This is not about maximizing programmer productivity. In fact, it has nothing to do with programmers. It’s about a very scarce resource: management attention. Whether you are building a commercial application for an ISV or a client project for a government ministry, there is a very limited ability of the stake-holders to digest the project’s progress and contribute with feedback and priorities.

Developing multiple products simultaneously hits the stake-holders hard, forcing them to try to manage the relationships between them in their head, understand the impact of changes from one product onto another, and juggle twice as many documents and meetings.

The results are invariably poor.

So what to do? The answer is surprisingly easy. Divide large projects up into their natural products, then develop the products serially. Work out which product can be developed first, and then start with it, religiously rejecting attempts to extend it horizontally with features from the other products.

When it is done and delivering value, build on it by adding another product. Repeat until done.

As mentioned, this has nothing to do with programmers. It’s about maximizing the business’s benefit. This is the theory behind incremental delivery practices such as SCRUM, implemented at a coarse level.

And that project I mentioned? They ported the application, and in the process discovered many ways to extend it to add much more value than the workflow would have provided.  ¶ 4:13 PM

  PG Nails It Again Paul Graham explains in one paragraph what I have struggled to explain in three years of blogging:

There is a contradiction in the very phrase “software company.” The two words are pulling in opposite directions. Any good programmer in a large organization is going to be at odds with it, because organizations are designed to prevent what programmers strive for.

  ¶ 3:29 PM

  Here and Now

We can either live in a “small here” (a great apartment in a crappy part of town) or a “big here” (a beautiful city in a great location with perfect weather and amazing vistas), and we can live in a “short now” (my deadline is my life) or a “long now” (how does this project change the company, the industry or the planet?).

  ¶ 10:37 PM

  Words I will take to heart

Be discreet with constructive criticism. A developer is much more likely to be accept casual suggestions and quiet leading questions than they are if the same is emailed to the entire group. Widening the audience is more likely to yield defensiveness and retribution. The team is always considering what your motives are, and you will be called on it and exiled if you degrade the work of others for self-promotion.

I hearby name this Forbes’ Seventh Law:

The constructiveness of your criticism is in inverse exponential proportion to the size of the audience.

  ¶ 10:53 AM

  Block-Structured Javascript Javascript provides closures with first-class access to variables declared in the enclosing environment. Besides being a handy piece of trivia if you are ever playing Programming Jeopardy, what use is this to the actual working programmer?

There are a lot of ways to take advantage of Javascript’s closures, I am going to describe just one, replicating Algol’s block structure (or Lisp’s let and begin macros, if you prefer). When we’re all done you’ll have a handy tool for making your code more readable, separating its concerns, and generally making life easier for programmers who have to read what you’ve written.

From Bricks to Blocks

A block is a chunk of code inside a function. Blocks have well-defined entry and exit points. Blocks have their own local variables and functions, and they also have first-class access to variables and functions defined around them. Blocks may nest.

Structuring code into blocks makes large functions more readable and easier to refactor. All of the variables and logic needed for one thing are encapsulated together in the blocks where they are needed, not scattered about in functions everywhere.

when a better design—if unfamiliar—is shown to developers or experienced users, they tend to reject it. Often, it takes careful explanation and having them gain experience with it before the improvement is understood.

While the idiom may be slightly unfamiliar to some, I think you’ll agree that it is highly accessible and not some sort of über-contruct of interest only to the self-professed hacker. And when a programmer encounters it in your code, she will have no trouble figuring out what it does and how it does it.

block structured code

I think everyone can agree that structured programming is a good thing. Functions should be composed of blocks, with the blocks linked together by constructs such as if statements:

if (some_condition) {
    // a block
}
else {
    // another block
}

Back before structured programming was introduced, there were gotos everywhere. Looking at a piece of code with labels, it was hard to know what the flow of control might be at run time. In short, you never had the confidence that everything you needed to know about that code was right there next to that code.

Like almost all modern languages, Javascript’s blocks do structure the flow of control so that you have nice clean entries and exits from each block. And also like most other modern languages, Javascript does nothing to structure access to variables inside blocks. For example:

var j = 0;
if (some_condition) {
    // a block with some code elided
}
alert(j);

What will be shown? You can’t guarantee that zero will pop up, because the blocks might modify j, like this:

var j = 0;
if (true) {
    var j = 42;
}
alert(j);

This looks like a programmer error. After all, the var keyword should be used to declare a new variable. The code between the braces isn’t really a new block of code with its own variables. This is hardly a problem with trivial examples. But if you start building some larger functions, the possibility of accidentally overwriting some code looms larger. Especially once you start refactoring and moving blocks of code around.

What can we do? How about this: the “block” idiom (you can also call it let or begin if you want to sound like a Schemer). Here’s the code:

var j = 0;
if (true) 
    (function () {
        var j = 42;
    })();
alert(j);

There’s a lot of syntactic noise, what does it mean? In short, we said: “Create a new function. In the body of the new function define a variable called j and assign 42 to it. Then call that function without any parameters.” Because our new instance of j is inside a function, it is not the same variable as the j outside of the function. That can be handy.

Are there any other benefits of this idiom? Yes indeed. Sometimes you have an assignment and you need some logic on the right hand side. How do you write:

var proven = {
    var n = Math.round(100*Math.random());
    var total = 0;
    for (var i = 0; i <= n; ++i) {
        total = total + (2*i) + 1
    }
    return total == ((n + 1) * (n+1));
};
alert(proven);

You can’t, of course. There are two problems with trying to use braces in this case. First, Javascript only allows braces to form code blocks in conjunction with specific keywords like if and function. Second, Javascript code blocks are not expressions—they do not produce values. This is why languages like Javascript need an if statement and a ternary operator: if blocks produced values, you would only need if expressions.

So in traditional Javascript style, you have to define a function somewhere else and call it… You’ll notice our block idiom includes defining and calling a function. What if our function returns a value? In that case, we can use a block anywhere we want a value, for example:

var proven = (function () {
    var n = Math.round(100*Math.random());
    var total = 0;
    for (var i = 0; i <= n; ++i) {
        total = total + (2*i) + 1
    }
    return total == ((n + 1) * (n+1));
})();
alert(proven);

This new idiom allows us to make first-class blocks anywhere we like. Our blocks are expressions, and we can use them anywhere we need a value. And as above, Our variables are fully encapsulated, they do not overwrite variables defined elsewhere.

blocks vs. named functions

You may be wondering, "Why can’t we use a named function?" This is the style in languages like Python, where the Benevolent Dictator does not permit constructions like this. Here is the above code using a named function:

var proven_helper = function () {
    var n = Math.round(100*Math.random());
    var n_plus_1_squared = function (n) {
        return (n + 1) * (n+1); 
    };
    var sum = function (n) {
        var total = 0;
        for (var i = 0; i <= n; ++i) {
            total = total + (2*i) + 1
        }
        return total;
    };
    return sum(n) == n_plus_1_squared(n);
};
var proven = proven_helper();
alert(proven);

I find this almost as good as the block. Since you only use it in one place, it is defined where you use it. That is good. And the name might be helpful documentation, just like a one or two-word comment. Balanced against this is the fact that you have added a new function to the outer scope. Reading it later, you might have to scan the rest of the code to see if it is used elsewhere.

There's also a very small advantage of the block over the named function: since you need two statements (one to name a function, another to use it), you can only use a named function in normal code blocks. You cannot use a named function when you need an expression, unless you resign yourself to naming the function in one place and using it somewhere else.

For example, when constructing array or hash literals, you can use expressions. A block is an expression, while two statements (one to create a named function and one to call it) are not an expression. So a named function would need to be defined outside of an array or hash literal, while a block can be used inside it, placing the code closer to where it is used.

block structure and cleaner code

Structuring code into blocks makes large functions more readable and easier to refactor. All of the variables and logic needed for one thing are encapsulated together in the blocks where they are needed, not scattered about in functions everywhere. If you see a variable declared inside a block, you know it is only used inside the block. If you see a variable with the same name outside the block—a regrettable occurrence—you know that moving or changing the block will not affect the code working with variables outside of the block.

You probably know that you can put a function inside of a function in Javascript:

var factorial = function (n) {
    var factorial_acc = function (acc, n) {
        if (0 == n) {
            return acc;
        } else {
            return factorial_acc(n * acc, n - 1);
        }
    };
    return factorial_acc(1, n);
}
alert(factorial(6));

And this is a good thing, it keeps the function factorial_acc inside of factorial. Since that’s the only place you need it, why declare it anywhere else? The fact that you can put a function inside of a function implies that you can put a function inside of a block as well:

var proven = (function () {
    var n = Math.round(100*Math.random());
    var n_plus_1_squared = function (n) {
        return (n + 1) * (n+1); 
    };
    var sum = function (n) {
        var total = 0;
        for (var i = 0; i <= n; ++i) {
            total = total + (2*i) + 1
        }
        return total;
    };
    return sum(n) == n_plus_1_squared(n);
})();
alert(proven);

If you only need the functions n_plus_1_squared and sum to do this one job, in this one place, why should they be defined at top level cluttering up your code? Why force other programmers to search through your code figuring out where they are used before making changes?

Block structure may seem unfamilar at first, but give blocks a try and see whether you start finding the code even easier to read and refactor with blocks. Like me, you will find that structuring your code with blocks puts the things you use right where you use them.

update: Ruminations about the performance of anonymous functions in naive Javascript implementations.

Labels: lispy, popular

  ¶ 10:08 PM

  Bricks I’m just finishing off some work with a corporate client before moving back to my natural position in product development (with a really exciting company!).

It’s a good time to reflect back over what was straightforward and what was difficult, what worked and what didn’t. It has been a very positive experience overall, and I have learned a few more things. Here are a hotch-potch of thoughts about corporate projects, clumsily organized around a single metaphor.

software is not made of bricks

Although very few managers ever express it directly this way, many behave as if developing a piece of software is like building something fairly simple out of bricks. It might be something large. But it’s still fairly simple.

This is tempting. The inexperienced person thinks that bricks are easy to understand: they’re all the same: if you know how to work with one, you know how to work with them all. You can measure the progress of building something out of bricks by counting the bricks in place. And anyone can contribute, even if only by moving bricks from one place to another using brute force.


When you have a brick by brick mentality, deep in your soul you believe that a project contains a fixed amount of work. It’s just a pile of bricks. There are so many screens, so many lines of code. You think this to be true because when you examine finished applications, you can count these things. So you engage in a discovery step up front where you estimate how many screens and how much code will be needed, then you play some games with numbers of people and the amount of work they can do per day, and out comes an estimated ship date.

You believe that since the finished work contains a fixed number of bricks, it is possible to know in advance how many bricks will be needed, and where they belong in the finished software.

This model of software development leads to several natural assumptions about how to organize a project. These assumptions are logical consequences of the belief that software is made of bricks:

assumption: it’s all abut moving bricks

The brick by brick mentality thinks of software development as making a pile of bricks. Think of the stereotypical Egyptian Pyramid as an example. There are so many bricks to pile and then you’re done. If it’s all about moving bricks, any work that moves bricks contributes to the success of the project.

That’s a comforting thought. Just keep those bricks moving. This helps us with all sorts of problems. Some people debate whether star programmers really are twenty times more productive than doofuses. Who cares? As long as the doofus can move bricks, eventually the work will get done.

So if you have a poor performer, someone who is slow and not very careful, you can use them on a project. Just find the right place for them where they can’t accidentally wreck the whole pyramid, and they can help. Ok, they are not good with the tricky booby traps or aligning the windows to allow light to strike the altar at the solstice. Fine. But what about ferrying bricks from the dhow to the base of the pyramid? Doesn’t that move the project forward?

Can’t you hire almost any warm body with ten working fingers and put them to work somewhere? Perhaps they can fiddle with page layouts, or copy the work of more experienced developers when implementing new features that are similar to existing features. But an extra pair of hands is always helpful, right?

software is more complicated than bricks

This assumption is wrong. The reason it is wrong is that software is deep. It is not a simple pile of bricks. Examining a finished piece of software, it is easy to discern surface forms like patterns, variable names, or rough organization. But the motivations for these choices are often subtle and opaque to the journeyman.

You can observe this the next time you are interviewing developer candidates. Ask them to name a design pattern: perhaps they respond, “Singleton.” Design patterns are surface forms. Now ask them to explain what problem the pattern solves. They respond, “Ensuring there is exactly one of something.” We are still working with the surface form.

Ask why we just want just one of something like a database connection pool. What problem are we solving? Why can’t we use class or static methods to solve this problem? What are the real-world issues with having 1,000 threads sharing a single database connection pool? How would you build ten pools? Or share connections without a single pool?

All of these questions drive at the deeper issues underlying development choices. A developer who treats their work as moving bricks, who simply copies the surface form of code they encounter, is oblivious to the motivations behind the code. They do not know when to use it and when to forgo it. They do not understand alternative ways of solving the same problem. They reproduce it blindly (and often incorrectly).

The result is software that superficially appears to be of acceptable quality because its surface form has things in common with good software. However, just because good software may be constructed out of commands and strategies, this does not mean that software constructed of commands and strategies is good.

What is needed on a software development project are people who understand the nuances, the requirements, the underlying problems. If you think that you are building a pyramid, what you want are architects, not slaves.

When you add people to a project who do not deeply understand their work or the problems the project faces, you create the superficial appearance of progress (look at all the bricks!), but you are slowly building up a mass of unworkable code. You may not see the problems immediately, but in time you will discover that everything they have touched needs to be re-written.

determine the baseline competence required for a project and don’t violate it

Once you understand that software is not a simple pile of bricks, you understand that the minimum level of competence required to contribute positively to a project is non-trivial. You can decide for yourself whether you need the mythical great hackers or not. But there is a minimum level of competence, and if you do not allow persons below that level onto your project, you will succeed.

If fact, you are far better off with a core of competent people and no sub-competent help than you are with the same group of people and the assistance of “helpers” on the side. Those “helpers” require three or four times as much management attention as the core contributors if you are to keep them from breaking things. And as we’ll see below, re-organizing your project such that there are tasks to keep them busy is usually harmful in and of itself.

Protecting yourself from people unlikely to make a positive contribution may require adroit maneuvering on your part. On one project, I explained that we could not complete the work in the time requested by the client. The response was to offer us some part-time assistance by employees of the client. Those particular employees may have been talented, but their experience was not a direct fit for the technical work of the project, and they did not have a full-time commitment to the success of the project.

Rejecting such “assistance” is tricky: other managers may have trouble with the idea that the project will move more slowly with the extra help, rather than move more quickly. Those managers see your project as a pile of bricks, and you’ll need to educate them if you are to avoid disaster.

software development is difficult to parallelize

The metaphor of a pyramid being built, brick by laborious brick is useful for illustrating another principle. When you assume that an application is a pile of a million bricks, you assume that you can move bricks in parallel. You can have one thousand people on the project, and if each places one brick per hour you will move forward at a constant rate of one thousand bricks per hour.

Software is not like this. Parallelizing development has serious and sometimes fatal consequences. The main problem is that the pieces are usually coupled in some way. There are techniques for lowering coupling between “bricks,” but when you set out to place two related bricks simultaneously, you must, perforce, do some kind of design or thinking ahead of time as to how they relate so that you can place them properly.

Consider two pieces, A and B. The natural dependency between them is that B depends on A. The right thing to do is to build A, and then B when you are happy with A. But the zealous manager with bricks on her mind asks, “why can’t we decide on an interface, I, between A and B, then build both at once?” They want to build I, then A and B simultaneously.

Of course, this constrains A and B tremendously. As you are building them, any flaw or shortcoming with I you discover as you build the pieces will result in rewriting both A and B. Only you are under time constraints so, you just patch and kludge, because the schedule does not have time allocated for redoing things: your motivation in parallelizing A and B was to save time, so the schedule has no room for the possibility that it will take longer to write A and B in parallel than in series.

This makes no sense to the person who thinks software is made of bricks! Looking at the finished brick, what’s the problem, it takes x hours to make A, y hours to make B, why would making them in parallel take longer than x + y instead of roughly max(x, y)?

Try the following: give piece A to one person, wait for it to be done, and then give piece B to another. Whoops, when the person working on B has a question about how A works, they have to track down the author and interrupt her. And if working on B teaches you something about A, is the person working on B supposed to change A? Or is the original developer supposed to backtrack and change it?

This explains a well-known nugget of wisdom: One reason adding people to a late project will cause it to slip further is that you are increasing parallelism. If the project was originally at or beyond its natural limit, further parallelism lowers productivity.

Or another example. You have 100 reported bugs to fix. You have 100 people. Do you assign one bug to each person? No way! Experience shows that bugs are rarely fully de-coupled form each other. You have to analyze the bugs as a team and try to guess their causes and relationships. If bug forty-nine is a simple text change on a page, anyone can fix it. But if bugs one, four and nine are all related, you need one contributor to address them simultaneously. Sending three people in to fix them in parallel thing would be a disaster.

Any time two or more pieces are strongly related either by design or by coupling in the application, it is a mistake to give each one to different people to build or fix.

In software, you want to minimize dependencies between pieces, which in turn means being very, very careful to minimize parallelism. Obviously, there must be some parallelism on any project with more than one contributor. But every project has a natural maximum amount of parallelism. Gratuitously chopping tasks into bricks to increase parallelism beyond this natural limit lowers productivity rather than increases it.

how to make the team twice as productive without parallelizing everything

What if you need two pieces, A and B, and you can’t wait for the normal amount of time to develop A and then B? Here’s an idea: instead of treating them like bricks and trying to develop A and B in parallel, why not simply hire one person who works twice as quickly? And have them develop A and B in series?

Think about this for a moment. There are a lot of claims out there that good people are three, five, ten, even twenty times as productive as the average. This seems intuitively wrong: when you look at their finished work, it rarely looks that much different from the work of the average person. So you figure the claims can’t be correct.

The finished work of the allegedly great person doesn’t look too outlandish. Ok, it has map and reduce instead of loops, and now that we look at it, the so-called great person seems to deliver fewer bricks, not more. What’s going on?

Let’s think about bricks for a moment. What if this essay right, and many times building bricks in parallel takes more time than building bricks serially? What if it’s very hard to coördinate the interfaces and contracts between pieces that are built by different people?

If most projects assign related bricks to different people, and most projects further compound this error by trying to “exploit parallelism,” you can get a big productivity win just by bucking the popular choice and asking one person to do all of the related work themselves. They’ll be as productive as a team of other people simply because they aren’t burdened with the heavy cost of parallelism and from the wrong people working on pieces.

Of course, you need someone who is able to keep two pieces in their head at one time. That’s one of the advantages of hiring good people: they don’t necessarily need to build things that are twice as complicated: if they can keep twice as much in their head at one time, they can build related things without incurring the costs of splitting development between people.

software is transfinite

The other wrong assumption about software being like bricks is that you can measure progress on a software development project by examining physical features of the software, by counting bricks.

The underlying thought is that you imagine the finished software as a pyramid of bricks, a pile of them. You count how many bricks will be in the finished application. Now you can measure your progress by counting how many bricks are “done.”

This is very wrong, and it leads to troubled projects. The first problem with this assumption has been given above: if you need a million bricks for your application, you ought to be able to make use of absolutely anyone to move the project forward in some small capacity. As long as they move a brick an hour, they are helping. So, a brick a day, a million bricks… let’s employ 1,000 sweating slaves for ten hours a day for one hundred days and we’ll have our pyramid. All we need are an architect and a team of overseers with sharp whips to see to it that they work without flagging.

But what happens when the millionth brick is placed and we are nowhere near completion? It turns out that software’s requirements are fluid, so fluid that you could place as many bricks as you like and still not be finished.

Measuring software productivity by lines of code is like measuring progress on an airplane by how much it weighs.

In fact, moving a lot of bricks is counterproductive: the physical manifestation of software, like written code, design specifications, unit tests, and so forth, have mass just like bricks. And if you want to redo things, the more mass you have, the harder it is to move and reshape it. In that sense, software is exactly like bricks.

Only, what you want to is to move the minimum number of bricks required to test your assumptions about whether the software is complete enough. It will never be complete, but trying to measure completeness by bricks is wrong.

There are only two meaningful ways to measure progress on a software development project: the first is to ask the team to estimate how much work remains, given the most up-to-date expectation for the form of the finished application. The second is to measure customer satisfaction.

how to measure progress on software development projects with estimated work remaining

Given the most current understanding of what is to be done to complete the application, it is meaningful to ask the team to estimate how much time will be required to complete the work.

This sounds conservative, even traditional. Doesn’t every project do this when they prepare the plan? What’s the catch? The catch is, if you only do it once, you only know your progress once. This differs markedly from the traditional model, where you plan once, estimate once, and thereafter you measure progress against your plan, rather than estimating again.

As the project progresses, the client’s requirements change. This is especially true if the client is given the opportunity to interact with the team and engage in the learning process: Agile’s claim that requirements are subject to change is a self-fulfilling prophesy.

To measure meaningful progress, you must re-estimate on a regular basis. If you wish to give a meaningful progress report every two weeks, you must ask the team to estimate the work remaining every two weeks. if, instead, you simply take how many bricks you thought you needed a few months ago, count how many have been moved to date, and calculate the work remaining through simple subtraction, your reports are drifting further and further away from reality every fortnight.

We know that this does not work, both from experience in the field and from critical thinking: as the project progresses, the client’s requirements change. This is especially true if the client is given the opportunity to interact with the team and engage in the learning process: Agile’s claim that requirements are subject to change is a self-fulfilling prophesy.

We also know that as we work with the software itself, we learn more about how much work is required to complete the software. For example, if you load a project’s team up with in appropriate contributors, maximize parallelism, and perhaps go three for three by minimizing testing and bug fixing, you will compound a tremendous “technical debt” over time.

Measuring “progress” against the original plan does not include the technical debt in your estimate of the work to be completed. Asking the team to estimate the amount of work to be done gives them an opportunity to factor the consequences of technical debt into their estimates. Or of any other factor that reflects what the team is learning over time.

In essence, you have an opportunity to include off-balance sheet items in your measurement of progress, whereas measuring against bricks would have excluded those factors.

how to measure progress on software development projects with customer satisfaction

Measuring customer satisfaction is easy. All you have to do is ask the customer. A successful project increases satisfaction over time. An unsuccessful project does not. I boldly posit that any project that increases customer satisfaction over time is a successful project, regardless of what was originally written in a specification.

There is no simpler or surer way to increase customer satisfaction over the long term than to let them experience the application as it grows and to rate your progress by how much their satisfaction increases with the software itself.

Customer satisfaction is a key metric because software is not a pile of bricks. It is impossible to predict with certainty the set of requirements that will result in maximum customer satisfaction at the end of the project, so you must measure satisfaction as you go. That being said, there is a pitfall looming when you ask the customer to judge their own satisfaction.

Some customers have difficulty understanding the features and characteristics of software that will meet their needs in a cost-effective way. They have trouble distinguishing good software from bad, good applications from lemons.

Although such a customer may need a not-so-big application, they may demand an Enterprise Solution.

This manifests itself in the customer demanding proof of progress in the form of elaborate documents, plans, and diagrams instead of working software that solves a portion of their problem. It manifests in the customer demanding proof that you’ve “hired up” to meet their needs. Although these things have their place, none of them are working software. They are promises to develop software, and subprime promises at that.

It is easy to obtain short-term success with such a client: deliver what they ask for. This is the exact business model followed by real estate developers specializing in inexperienced, first-time buyers: offer the superficial features that provide short-term excitement at the expense of long-term satisfaction. In the case of software, you can dazzle the inexperienced customer with head counts, power points, and diagrams showing Jenga-piles of technology.

Should you do this? That is up to you, it depends on whether you wish to build short-term customer satisfaction at the expense of long-term satisfaction with the software itself. If you wish to deliver long-term satisfaction with the software, you may need to educate the customer to focus on the software itself.

And that means delivering increments of a functioning application for the client to experience. There is no simpler or surer way to increase customer satisfaction over the long term than to let them experience the application as it grows and to rate your progress by how much their satisfaction increases with the software itself.

building software without treating it like a pile of bricks

Sometimes, it really does boil down to a few simple ideas, working in concert:

1. Hire people with a minimal competency. Do not be seduced into accepting “help” from people who are not able to contribute to the team at this level.


2. Minimize parallelism. Exploit the talent of your best developers by giving them chunks of related work.


3. Measure progress by continually re-estimating the work to be done and by customer satisfaction. Educate the customer to prefer completed work over documentation and promises.

Labels: popular

  ¶ 11:56 AM

  The people you have to please in the enterprise market are the ones purchasing and supporting the products, not the poor schmucks who use them

One of my favorite dumb questions from the iPhone launch [is]: “Why hasn’t anyone made a phone like this before?” The answer is that every other phone manufacturer is tied up in the enterprise market in one way or another.

The “dream phone” for the enterprise looks quite different than the iPhone. It works with the corporate VPN. It does Exchange. It supports device-wide encryption and remote deletion of data on lost devices. It’s available in several compatible forms from multiple manufacturers. It has a well-defined public roadmap for hardware and software. It can be backed up and restored en masse, preferably over the network. If it has a camera, it can be disabled. The battery can be removed and replaced. And on and on.

Maybe around item two hundred in this list there might be a bit about the people who will actually use these enterprise dream phones tolerating the things. Really, as long as they don’t openly revolt, it’s fine. The people you have to please in the enterprise market are the ones purchasing and supporting the products, not the poor schmucks who actually have to use them.

  ¶ 4:41 PM