Which theory fits the evidence? There are two schools of thought about the practice of managing software development (the theory of managing software development is of little use to us because “the gap between theory and practice is narrower in theory than it is in practice”).

One school is that everything is fully deterministic in practice (“Theory D”). If development appears, from the outside, to be probabilistic, it is only because we haven’t discovered the “hidden variables” that fully determine the outcome of software development projects. And, since we are talking about development in practice, it is practical to measure the variables that determine the outcome such that we can predict that outcome in advance.

The other school of thought is that development is fully probabilistic in practice (“Theory P”), that there are no hidden variables that could be used to predict with certainty the outcome of a software development project. Theory P states that the time and effort required to measure all of the variables influencing a software development project precisely enough to predict the outcome with certainty and in advance exceeds the time and effort required develop the software.

Theory P does not mean that software development cannot be managed in such a way that the desired outcome is nearly certain: the flight of an airplane is fully probabilistic as it encounters atmospheric conditions, yet we have a huge industry built around the idea that airplanes arrive at their destinations and land on the runway as planned.

why do theory p and theory d matter?

Understanding whether software development follows the Theory D (fully deterministic) model or the Theory P (probabilistic) model helps us set our expectation for the relationship between what we plan and what transpires.

If we believe Theory D, we believe that it is possible and practical to plan software development entirely in advance. Therefore, when things do not go as planned, our first reaction is to either blame the planners for faulty planning or to blame the implementers for failing to carry out a reasonable plan. Believing in Theory D, we believe that we ought to have a plan that can be carried out to perfection.

Programming is not complicated because computers are complicated—it’s complicated because your requirements are complicated (even if you don’t know it yet).

If we believe Theory P, we believe that it is only possible and practical to plan some part of software development in advance. Therefore, when things do not go as planned, our first reaction is to embrace the new information and update our expectations. Believing in Theory P, we believe we ought to have a process for continually updating a plan that asymptotically approaches a description of reality as the project nears its conclusion.

belief drives behaviour

Our belief about which theory is true drives the way we manage software development projects in almost every way. Here are three examples: the way we manage software design, the way we manage time estimates, and the way we manage selecting people.

If extra time is required, people on Theory D projects work nights or weekends, or they cut testing time. They do this because their belief is that if a task takes too long, the fault lies with the estimate or with the worker carrying out the task, and by working overtime they can “make up for their fault.”

Theory D adherents believe you can design software in advance. They believe it is possible to collect all of the information needed about software’s requirements and the technical elements of its construction, such that you can fully specify how to build it before you start. In short, Theory D adherents believe in Big Design Up Front.

Theory P adherents believe that software can only partially be designed in advance. They believe that requirements suffer from observation, that the act of building software causes the requirements to change. Theory P adherents also believe that technical factors cannot be perfectly understood, that only the act of trying to build something with specific components will reveal all of the gotchas and who-knews associated with a chosen technology strategy. They believe that software design is an iterative process, starting with a best guess that is continually refined with experience.

Theory D adherents believe it is possible to estimate the amount of time required to develop software (in both the large and the small) with precision. This is partly a consequence of their belief that you can know the requirements and design in advance, and therefore you can plan the activities required without uncertainty. Theory D adherents do not plan to miss milestones. Theory D adherents do not, in fact, have a process around re-estimating tasks; instead, they have a mechanism for raising exceptions when something goes wrong.

Theory D adherents believe that the normal case for software projects is that tasks are completed within the time estimated. (If extra time is required, people on Theory D projects work nights or weekends, or they cut testing time. They do this because their belief is that if a task takes too long, the fault lies with the estimate or with the worker carrying out the task, and by working overtime they can “make up for their fault.” Theory D managers often “game” their workers by “negotiating” estimates downward in a cruel game of “guess the estimate I’m think of.”)

Theory P adherents believe that there are lies, damned lies, and software development estimates. This is partly a consequence of their lack of faith that the requirements are truly fixed and that the technology is fully understood. If you don’t know what you’re doing and how you’ll do it with precision, how can you know when it will be done? Theory P adherents build processes around re-estimating estimates, such as burndown charts and time-boxed iterations.

Theory P adherents are always fussing with an updated view of how long things will take. They talk about “velocity” or “effective vs. actual programmer-hours.” Theory P adherents believe that the normal case for software projects is that tasks are rarely completed exactly as estimated, but that as a project progresses, the aggregate variance from estimates falls.

Theory D adherents believe that the most important element of successful software development is planning. If a plan is properly constructed for the design and development of a software project, the actual implementation is virtually guaranteed. Theory D adherents invest most of their human capital in “architects” and “managers,” leaving little for “programmers.” They often have architects, senior developers, and other “valuable resources” involved in the early stages of projects and then moved to the early stage of other projects, leaving the team to implement their “vision.” They likewise believe that you can “parachute” rescuers into a troubled project. Since the plan is perfect, it is easy to jump in and be productive.

Theory D adherents believe in “architecture by proxy,” the belief that using frameworks, components, programming languages, libraries, or other golden bullets makes it possible to employ lesser talents to perform the implementation of software, since the difficult decisions have been made by the creators of the pre-packaged software. Theory D adherents also believe in “success by proxy,” the belief that using methodologies, practices, SDLCs, or other buzzwords makes it possible to employ lesser talents to perform the management of software development, since the difficult project management decisions have been made by the “thought leaders” who coined the buzzwords.

Theory P adherents believe that the most important element of successful software development is learning. They invest their human capital more evenly between implementers and architects, often blurring the lines to create a flatter technical structure and a more egalitarian decision-making environment. This is a consequence of the belief that learning is important: if you invest heavily in a few “smart” people, you have a very small learning surface exposed: there is only so much even very bright people can learn at one time. Whereas when the entire team meets a certain standard for competence, there is a very large learning surface exposed and the team is able to absorb more information.

Theory P adherents believe that there are lies, damned lies, and software development estimates.

They strongly prefer to have the same team work a single project from start to finish, believing that when a member moves on to another project, crucial knowledge moves on with them. They likewise abhor bringing new members onto a team late in a project, believing that the new people will need experience with the project to “get up to speed.”

Theory P adherents use frameworks (especially testing frameworks), but are skeptical of claims that the framework eliminates technical risk or the need for talented contributors. Theory P adherents, even Agilists, are skeptical of methodology claims as well. They do not believe that a deck of slides and a nicely bound book can capture the work required to learn how to develop software for a particular user community in a particular environment.

Theory D and Theory P adherents are easy to distinguish by their behaviour.

so which theory fits the evidence?

Which theory fits the evidence collected in sixty years of software development?

To date, Theory P is the clear winner on the evidence, and it’s not even close. Like any reasonable theory, it explains what we have observed to date and makes predictions that are tested empirically every day.

Theory D, on the other hand, is the overwhelming winner in the marketplace, and again it’s not even close. The vast majority of software development projects are managed according to Theory D, with large, heavyweight investments in design and planning in advance, very little tolerance for deviation from the plan, and a belief that good planning can make up for poor execution by contributors.

Does Theory D reflect reality? From the perspective of effective software development, I do not believe so. However, from the perspective of organizational culture, theory D is reality, and you ignore it at your peril.

Do not confuse Computer Science—the study of the properties of computing machines—with Software Development, the employment of humans to build computing machines. The relationship between Computer Science and Software Development parallels the relationship between Engineering, the hard science of the behaviour of constructions, and Project Management, the employment of humans to construct engineered artefacts.

---

(A portion of this essay originally appeared in What I’ve Learned from Sales, Part II: Wanna Bet?.

Update: D is for “D’oh! We should have gone with P!” and The Myth of Project Management, a SFW retelling of Project Management is Bollocks!.

Labels: agile, popular

  ¶ 11:38 AM

  Does your team have well-defined boundaries you can articulate?

One of the (hundreds of) cool things about working for Google is that they let teams experiment, as long as it's done within certain broad and well-defined boundaries. One of the fences in this big playground is your choice of programming language. You have to play inside the fence defined by C++, Java, Python, and JavaScript.

My project's playground is actually a bit smaller than that, since it has to run on the JVM. But that's still a pretty big playground, and trying out (or creating) new web frameworks is totally fair game for experimentation, even if you're in a relatively high-profile domain.

Another boundary we have to play within is software engineering, including unit testing, documentation, code refactoring, security, scalability, internationalization, and a host of other non-negotiable criteria.

I hope you're beginning to see, at least faintly, why I love working at Google. It's because the code base is clean. And anything that takes more than a week of effort requires a design document, with specific sections that have to be filled out, and with feedback from primary and secondary reviewers of your choice. The net result is that for any significant piece of code at Google, you can find almost a whole book about it internally, and a well-written one at that.

I've never seen anything like it before, to be honest. I don't think you can get that kind of software-engineering discipline without doing it right from the start, and creating a culture that enforces and reinforces that discipline as the organization grows.

  ¶ 9:15 AM

  R-E-S-P-E-C-T In My “Working” Day, I described an office with an “interrupt-driven” culture. And in the comments, someone (probably from the same office) asked:

Do you really think there’s anyone in our office that doesn’t view you, your time and your privacy with the utmost respect?

This is an important point. Of course the people I choose as my colleagues view my time with respect. Naturally. But interruptions can still cause a disconnect even when people consider your time a precious commodity.

They aren’t interrupting me because they think my time is worthless. They’re interrupting me because with the information they have at the moment, a conversation with me is the most valuable activity the two of us can conduct for the company.

That isn’t so odd. They have a different set of information in front of them, so while I look at the situation and think: The best use of my time is making sure that deliverable X is complete in this iteration, they look at the situation and think: The best use of Reg’s time is giving me a status update on deliverable X right now, so I can provide top-notch customer service when I call the client in five minutes.

Am I right overall? Are they right overall? What is the most important thing for me to be working on at this moment?

important vs. urgent

One reason people who respect your time interrupt you is that they believe that the activity requiring your participation is urgent. And if the activity they are interrupting is actual programming, you probably view them as impeding your ability to accomplish something important.

One model for understanding the conflict is to judge activities on two orthogonal scales: importance, which measures long-term value and assumes that you can perform the task now or later, and urgency, which measures the degree to which the value of a task declines if you execute it later.

(Of course, both scales go into the negative as well as the positive. So the value of writing a rent cheque is that you avoid an incredibly negative outcome in the future).

If the quasi-economic explanation of the difference doesn’t strike a chord, try the exercise Steven Covey suggests in his book The 7 Habits of Highly Effective People: Look back on last year. The whole year. Now ask yourself: of all the things you wanted to accomplish, what would you like to have done if you had more time? Those things are the things that were important but not urgent.

Making time to do things that are important but not urgent is one of life’s greatest challenges. Do you really want to learn a new programming language? Network with more people? Read more books? Visit your mother more often? You need to make the time to make it happen. You need to find a way to take time away from urgent activities and give it to important activities.

A reasonable explanation for the conflict between the boss who wants status now and the programmer who wants to get into the flow and complete the work is that the programmer is concentrating on the important while the boss is concentrating on the urgent.

After all, if it wasn’t urgent, the boss would request status by email, or wait for the next daily meeting we have on that particular project. She’s no dummy, she wouldn’t be asking now if she didn’t need an answer now.

And if it wasn’t important, the programmer wouldn’t be so obsessed with trying to focus and do it right, trying to think all of the consequences and cases through so as to produce a really solid piece of code.

To summarize, one reason people who respect your time interrupt you is that they believe that the activity requiring your participation is urgent. And if the activity they are interrupting is actual programming, you probably view them as impeding your ability to accomplish something important.

optimization

There’s another important factor to consider. Let’s say that you and your boss synchronize on importance and urgency. Let’s say you see eye to eye on how to evaluate what you ought to be doing at any one moment in time. Can you now allow your boss to interrupt you at will and still maximize the value you provide to her?

This is a trick question. What does it mean “to evaluate what you ought to be doing at any one moment in time?” Does it mean that you can look at all the data in front of you, stuff it into some kind of choice function, and out pops the activity you ought to be doing?

The answer for my fellow programmers is, Yes, but the function is a closure.

In other words, local maxima are not global maxima. Or if you like compilers, peephole optimization is most pessimum.

Let’s take an example. What’s the best use of your time right now? Programming? Ok, why isn’t that the best use of your time 24/7/365? Forget sleep and social interaction and other non-work things if that seems complicated. Let’s just think about work. Why do anything else while you’re at work? The answer, of course, is that the optimal use of your time involves a balance between different ways of producing value.

Code is more important than documents. But code without any documents, ever, is less valuable than code with a certain amount of well-written documentation. And a lead programmer needs to write code. But they also need to coach their colleagues. So why don’t they just code? Or just coach?

It turns out that people who just coach, consultants… oh well, you can fill that one in. And more than a few electrons have zipped through the pipes about programmers who only produce code, but never interact with their colleagues. Most teams need leads who have a balance between producing code and raising the value of their colleagues.

Another classic example is a bug. Stuart reports a bug in Sally’s code. It doesn’t erase user data or install a root kit. Should Sally fix it? Stupid question. Once you leave the comfort of a weblog, you know that you cannot evaluate any but the most catastrophic show-stoppers individually. If you try to make these decisions one at a time, you invariably will work on the wrong things. You need to establish a rhythm of bug triage meetings, so that at any one time you are looking at enough bugs that you can choose those with the highest value and skip those that are not relatively important.

Balancing activities to produce the overall maximum value for your team (or yourself if you are a team of one) requires a broad view of your priorities and activities, just like triaging bugs requires a broad view of the outstanding issues with a release. You can’t make those decisions on a report-by-report basis, you need to optimize globally.

Which leads us back to interruptions. The primary issue with an interrupt-driven culture is not that people don’t respect each other’s time. It’s that making decisions on a moment-by-moment basis means you never have the perspective with respect to what’s important and to the overall balance of what you need to be doing to maximize the value you deliver to your team.

That’s why time management systems like Getting Things Done place such a premium on daily and weekly reviews: you can’t be effective if you are a slave to the to do list and never step back to optimize the value you create in your life.

Interruptions may maximize response to urgent issues. Interruptions may appear to be the best use of your time given the information the interrupter has at the time. But other people almost never have all the information you have to know what you need to be doing right now, so for this reason regular interruptions sacrifice your global maximum value for the appearance of local maximum.

And for that reason, it is easy for other people to have the utmost respect for the value of my time but still impeded me from providing the maximum value to our team if they chose to interrupt me rather than using asynchronous channels like email.

so it’s wrong to interrupt programmers?

In general, if what you want from programmers are working programs, don’t interrupt them. Maximize their programming productivity in every way you know how.

But don’t you think the managers running an “interrupt-driven” workplace know that? Of course they do. They know how to maximize the production of programs, starting with hiring great programmers and working from there. If they aren’t doing those kinds of things, it may be that they aren’t trying to maximize programming productivity.

There could be many reasons for this. Perhaps they run their business on a time and materials basis, so they are trying to maximize customer service while optimizing for the cost of labour. A programmer who wishes to be successful in a particular culture should be careful that programming is really their boss’ number one priority before trying to optimize their personal programming productivity.

The bottom line, again, is that their colleagues may respect their time to the utmost, but not believe that it is best spent with their head down programming.

In that case, the only problem I have with interruptions is the realization that what I think is important is out of sync with what the team thinks is important.  ¶ 5:21 PM

  My "Working" Day Someone asked how I seem to get so much done. I had to answer honestly:

The unfortunate truth about my personality and my day is that I lurch back and forth between being intensely focused and very distractible. There is some daily rhythm to these swings in attention capacity, some random X factor, and the undeniable fact that if I am “in the zone” and you interrupt me enough, I lose focus and I can’t get back into it.

When I am working from home (either for clients or on my own projects), the key to productivity is not letting those focus hours slip through my fingers. When I’m “on,” that’s the time to put TextMate front and centre. When I’m “off,” I try not to force it too much, there are plenty of things to do that can be done in short dribs and drabs.

My current day job involves working in an office where the prevailing culture is “interrupt-driven,” to the point where it is mandatory (yes, a company rule) to have IM on so that people can interrupt your coding without even bothering to invest the five minutes to walk over to your desk. Needless to say, the day job environment does not feature offices for programmers. Nor do people respect the fact that there is code on your screen and headphones over your ears as signs you may be performing a billable activity.

This would be a complete block to productivity, except for the fact that our clientele impose so much bureaucracy on getting anything done that my ratio of virtual paperwork to code approaches one.¹

So:

When I have some serious code to write, I do it at home before I go to the office, or after five when everyone leaves (you can tell a lot about the company culture by observing the stampede for the exits every afternoon). I can debug, embellish, and tinker during the day, but I can’t actually concentrate enough to write anything serious, so I no longer bother trying.

That’s when I usually lurk on reddit, talk to colleagues, talk to clients, and answer emails. It’s a good time for lightweight research, like figuring out how to get JRuby working so it can be called as a scripting language from a Java application.

I use Actiontastic to manage my GTD system on my Apple MacBook Pro. That way, although I’m not getting hard coding done, I am advancing my projects and my clients love my high-bandwidth, real-time communication.

Now, from time to time there is a crunch of some kind where I absolutely must get something done and for whatever reason I can’t just leave. I usually hide in the board room and crank. I have to slap myself if the mouse hovers over the Firefox icon. In extreme cases, I turn Internet Access off entirely.

I also get some night hacking time, and if I’m not too exhausted, that’s a good time for coding. I like to save several evenings each week for family time: from dinner with my family to putting my son to bed is a time slot where work, blogs, and even start ups do not intrude.

---

1. Update: R-E-S-P-E-C-T.

Do you have a better idea for how I could be applying my talents to helping a software company grow?  ¶ 2:55 PM

  Lemonade I place a table and chair at the front of my property, just in from the street. I cover the table with a table cloth, then place a stack of paper cups and a pitcher of iced lemonade on the table.

A man comes along, mops his brow in the heat, and pours himself a drink. Hmmm, refreshing, he says to himself, and passes the word to his friends that walk in the same neighbourhood.

Others stop by from time to time, have a drink of lemonade. Sometimes a few people stand around, discussing a topic of the day. I often join in, it’s a great way to meet new friends. From time to time people argue with me about lemonade, or the chairs I put out. That’s great, I can always use help, and the kinds of people walking down my street usually have really valuable experience to share.

I add a small book stand, where I sell books I enjoy reading to a few passers by. I’m not making enough to pay for the lemonade yet, but I like making lemonade, so extra pocket change is handy when I’m shopping for books myself. Everything is great, I’m glad I started giving lemonade away.

But every once in a while, someone stops, has a drink, and spits it out in disgust. “Lemons!” they often say, and hurry away. They never leave their name, or tell me what is bothering them. Too sweet? Too sour? Would they like to taste my Lemon and Bay Leaf Crème Brûlée? or my Sweet Lemon Risotto?

No, they pass out of my life, leaving a crumpled cup on the table and a puddle of lemonade on the sidewalk as the only evidence they were ever there. In time, I toss the cup away and the puddle evaporates. And when I pour another jug, people stop by and the cycle begins anew.

The lemonade is a very rewarding hobby, I meet some great people, I learn new things, and there’s a small feeling of accomplishment when I see people enjoying another pitcher (I have to remember, it’s only lemonade, this isn’t Haute Cuisine).

It’s a good life.  ¶ 8:19 AM

  The Show Must Go On: Query Languages This is my first really technical post since coming to grips with the end of my side project. It isn’t my best writing about programming by a long shot, but I wanted to remind myself—and you if you need reminding—that no matter what happens with the business of software, the programming of software is still damn interesting.

Here is some JavaScript flavoured with Prototype goodness:

$$('a').select(
    function (anchor) {
        return anchor.href && !anchor.onclick &&
            anchor.href.match(/^https?:\/\/(?!(.*bigco\.com.*)|(localhost.*)$).*$/i);
    }
    ).each (
        function (anchor) {
            external_url = anchor.href;
            anchor.onclick = function () {
                doSomethingWhenUserClicksOffSiteLink(external_url);
            };
            anchor.href='#';
        }
        );

This code finds all of the anchors (<a href="...">...</a>) that lead off the BigCo site and changes them to have an onclick handler. You might use something like this to log outgoing clicks so you can judge the popularity of links, for example.

The first thing you might notice is the Smalltalk-style methods—select and each—for handling collections. If you like Smalltalk (or Ruby) style collection methods, Prototype gives them to you. If you haven’t seen this kind of JavaScript before, the thing that will probably stand out is the use of anonymous functions as parameters for select and each.

The thing about this code that grabbed my attention is the use of two different embedded languages. If you aren’t familiar with Prototype, $$ is a function that takes a CSS Selector as an argument and returns an array of elements matching the selector. In this case, we’re simply asking for every anchor (“a”). But we could ask for every anchor inside of a specific div, or every anchor of a specific class, or almost anything else you might want to use.

CSS selectors are a kind of query language specifically optimized for applying styles to DOM elements in web pages. But Prototype (and several other JavaScript libraries like Mootools) make CSS Selectors a general purpose tool. The Groovy Language provides a similar feature called GPath, a means of querying object networks using a language very similar to XPath.

(I personally think GPath is the most interesting thing about Groovy—having an object query language “baked in” changes the way you think about writing programs, much as SQL changes the way you think about using databases. Or it would if you were brought up on non-relational databases, or if you have been eating of the ORM fruit that leads to madness.)

Any ways, $$('a') really changes everything. It’s not such a big whup in this particular snippet of code, but think about the separation of concerns here: if you do want to change the elements you are manipulating you change one place. You don’t have to restructure a pile of loops and conditionals.

Which leads to the other embedded language, a rather popular one known as a Regular Expression: /^https?:\/\/(?!(.*bigco\.com.*)|(localhost.*)$).*$/i. Regular expressions are inscrutable to most programmers: if you don’t use them on a daily basis, you are in danger of losing the knack of writing them. And I’m not sure if anyone has the knack of reading them. What does /^1?$|^(11+?)\1+$/' match, and why is it famous?

That being said, as inscrutable as they may be, it is very powerful to wrap all your string matching up into a single blob where you can put it in one place. Having loops and scanners and parsers breaking URLs up and looking at individual pieces obscures the intent of your code.

He had a hat! (John Gruber)

Having had a taste of embedded query languages, I’m left with a hunger for more. Quite honestly, I may be thinking of solutions in want of problems, or perhaps it is effete aesthetics, but the more I think about it, the more the following questions bug me:

Why aren’t patterns first class values? Obviously, you can assign them to variables and return them from functions. But can I take them apart? Can I compose them? Is there a query language for extracting pieces of a query? Queries (be they regular expressions, XPath, or CSS Selectors) ought to be structures that can be manipulated just like the DOM or like object networks.

Why don’t we have better support for transforming structures with embedded languages? Regular expressions lead the way here: there’s a powerful feature for using a regular expression to take a string apart and put it back together in a different order, or with new bits added to it. And SQL is fully integrated, there’s a natural syntax for updates integrated with queries.

Where’s the same facility for object networks? Right now I can use CSS selectors from Prototype to find elements and builders from Scriptaculous to modify the DOM. Or I can use JQuery (thanks, David) to do all my DOM manipulation in one go. That’s terrific, it’s even better than the snippet above. But...

Why can’t I use the same power for transforming JSON or my own objects? It’s like regular expressions: once you taste the power with strings, you want to use them with arrays. Once you use XPath with XML, you want to use GPath with graphs of objects.

Once I get going, my mind immediately jumps a level from things that would be useful to things that would be cool for the sake of being cool (and nothing else): With OO, we’re hung up on messages. But those messages are ridiculously primitive! A verb and a bunch of parameters that are usually nouns. Imagine a meta-language where each receiver could interpret its own language. So strings would interpret regular expressions, and DOMs would interpret XPath and/or CSS Selectors. What we call a type or an interface today—a set of verbs with rules for the accompanying parameters—would be replaced by a set of languages receivers understand.

I have some vacation time coming in a few months. Now I have something to think about!

---

Now, I have asked a lot of “why can’t I…” questions. I hope I get a lot of comments saying “You can, check out the KulTulz library for JavaScript, or the Mazenblitz Macro Package for Common Lisp, or even RTFM about Ruby…” What libraries, languages, and packages provide the kind of features I’m daydreaming about?  ¶ 9:24 AM

  Is your door open or closed?

I notice that if you have the door to your office closed, you get more work done today and tomorrow, and you are more productive than most. But 10 years later somehow you don’t know quite know what problems are worth working on; all the hard work you do is sort of tangential in importance.

This quote came via Byrne Hobart. His weblog, Byrne’s Eye View, is a fascinating read, especially if you’re interested in finance, math and economics. And if you’re the sort of programmer who likes this weblog and also finds those subjects fascinating, you might want to talk to Byrne: He is a recruiter with connections into the world of Hedge Funds and the programmers who work for them.  ¶ 1:14 PM

  Why Buzzword Compliant Software is Less Valuable than a McMansion A very nice and insightful criticism of The Not So Big Software Design:

It’s a good discussion of the realities of custom software. The metaphor does tragically fall down though. At least buyers of used cars and tract housing can resell them to the next ignorant buyer. Companies are generally stuck with their custom software.

  ¶ 10:52 AM

  Portfolios Shanti Braford asked:

While Google, Microsoft, Facebook, etc have an ongoing War for Talent, I wonder if another company could develop a system for developers who most Get Stuff Done, ship code, etc. (as in Joel’s “Smart & Gets Stuff Done” goal of hiring)

Looking at resumes only measures this trait so much. 2-minute academic coding problems usually measure IQ as opposed to brute coding that is more likely to be done in the wild.

Any ideas?

Well yes, as a matter of fact, I do have an idea:

Portfolios.

For example, this is my teaser portfolio. I also carry a physical portfolio to meetings with employers (the photos aren’t that impressive, but this is something I show in a face to face meeting, not something I use to justify meeting with me).

Quite simply, hire people who can show you evidence of their Smarts and ability to Get Stuff Done™. Of course, the big companies will seize everyone with IQs off the charts, GPAs through the roof, and all the other obvious traits. So you’re looking for people who are “false negatives” by the standard tests, people who don’t get an offer from Google, maybe they don’t even get an interview, but they’re great anyway.

Of course, a portfolio is not the only thing. Readers, please don’t write and say, “but you also need to ask them to write FizzBuzz,” or “what about Monopoly,” or “a degree from a good school shows they can handle doing the stuff they hate as well as the stuff they love.” I believe you! Test those things as well!!

I’m just saying, look for portfolios. Ask for them. Right in jour job ad, if you like.

p.s. I regularly receive requests for my résumé. I expect more requests now that I’ve officially announced that I am no longer working on my start up and am ready for the next chapter in my career. But it’s amazing how few people ask me to send a portfolio, or links to recent work, or source code samples.

Labels: jobs

  ¶ 9:12 AM

  Still failing, still learning The good news: I’m still learning.
The bad news: …from failure.

This post officially announces that my side project (originally named cause & effect and later named certitude) is over.

What I wanted to achieve

For those of you who weren’t subjected to one of my enthusiastic rants, here was my Graham Question: Can we predict the outcome of a software development project with objective observation?

I have always believed that the answer is Yes. And I manage projects that way. But I also strongly believe that the only way to prove that we have an objective understanding of an algorithm is to mechanize it, to write a piece of software that executes your algorithm.

So I set out to write a piece of software that, pure and simple, would look at a software development project and show you a traffic light: a green light would mean that the project looks like it’s on track, a yellow light would mean that the project needed help, and a red light would mean that there is no hope.

I won’t go into a lot more detail, mostly because this isn’t a story about teeming hordes of customers and me not being able to finish by deadline. It’s a story of inventing a great solution to a problem nobody cares about. But if you absolutely must visualize the application, think of something that gobbles up your MS project files, your bug tracking data, even your burn down spreadsheets, and belches out that traffic light when it’s done. That’s it.

So how and why did I fail?

Project management software is social software

Reason zero, just as Paul Graham warned, my age. Really. I remember how much code I could crank at age 22, and now that I am double the age, I write an order or magnitude less. I’d like to say that my code is that much better that it makes up for the lack of volume. That might be true if I start a project with a specific end in mind, but the act of experimenting, tinkering, and exploring benefits from being able to turn large amounts of code around in short amounts of time.

This is reason zero because I wanted to get it out of the way before explaining why I think I still would have failed if I were 22. But it’s still important to understand, because I might have known that I was going to fail two years ago, instead of today.

Reason one that I failed—and this is the most important reason I failed—is that I was trying to tackle a social problem with technology. This can work—ask any dating site zillionaire—but it is a very high-risk strategy for software. Not just for making money, but for what really matters, adoption.

Project management is a social problem.

Project management is a social problem. It is 99.5% about getting everyone who knows something about the state of the project to share what they know with everyone else. Getting all the relevant information is 99.5% of the problem, analyzing the information is 0.5% of the problem.

Joel likes to brag about how much trouble FogBugz goes to to make it easy to enter bugs, about how certain kinds of reports are not available to discourage punitive social behaviours like punishing developers who generate too many bugs. This is a strong hint that getting good information is all about managing people’s perception of the likelihood of punishment for telling the truth.

Sitting here typing this, I think the company who can do the best job of predicting the outcome of software development projects is Inkling Markets. That’s because their entire business is about finding a way for people to communicate what they really think of something, not just what they think other people want them to say about something. I think Todd Proebsting would agree.

This problem isn’t limited to dysfunctional environments where people cower in fear of saying anything except “Sir, Yes Sir” when told to change the laws of physics.

Lemons, damned lemons, it’s always lemons

Project management suffers from a real lemon problem. I quoted Bruce Schneier at length about this already, so this time I’ll explain things directly:

Most managers, especially those with limited experience shipping software on a predictable schedule, do not know how to correlate what they’re told about the project with the likelihood that the project will succeed.

Some information is valuable, some is junk. The problem is, managers “buy” information. They trade favours like letting you keep your job for information about how well you are doing your job. So there is a market for information just like a market for cars.

They also “sell” information, literally: they have to make a report or a presentation to their superiors, or to stake holders, or to their fellow founders at the YCombinator dinner.

When a manager cannot tell the difference between information that is useful for predicting the outcome of a project and information that is not useful for predicting the outcome of a project, she thinks about the next best thing: the “resale value” of the information with people one step removed from the project, like her own manager. So she values things like pretty PowerPoints about the architecture higher than finished pieces of functionality.

(This is why I have always sweated my heart out to give good presentations. My teams have depended on me being able to take good information and sell it upstream just as if it were CMM Level Five Buzzword-Compliant Junk).

Do managers further removed from a project always value pretty junk more than good, solid information? Not always, but often. And that’s enough for people to be pressured to give the bad information that sells to their manager, while hiding the good information that doesn’t sell. Exactly like the owners of good cars taking their treasures off the market.

Lemon and bay leaf crème brûlée

Why does junk information outsell good information? Nice PowerPoint isn’t a good explanation by itself: there are nice PowerPoints explaining Agile, but managers still prefer waterfall.

Consider a not so big design. Let’s call completing that design good information: we’ve done a good job finding out what’s really important for the project and making a design that emphasizes the way this project is unique, not the technology stack.

Now consider a typical technology design, emphasizing frameworks and technologies. Fully buzzword-compliant.

Which one sells better? The technology stack does. Why? Well, for starters, managers have been exposed to seventeen billion dollars worth of advertising talking about the benefits of technology stacks. Nobody is advertising the specific ways the not so big design helps the project. How could they? Those are specific to the project, that’s the whole point.

And managers are like anyone else, they compare what you are doing to successful projects they have seen in the past. Once again, the not so big design doesn’t have anything in common with other projects, but the technology stack does. (There are lots of failed projects with technology stacks, of course. But who cites those when bugging the team about whether they will use Hibernate as their ORM?)

How did this happen? How did things that have no correlation to the success of a project become more attractive than things that do?

Quite simply, people have an incentive to look successful. So they imitate the outward appearances of successful projects. We have a really simple way of completing successful software projects: we put successful people on them. But we have a broken way of thinking about it: we don’t like to think of the people as being special, we think that what the people do is successful.

And by that logic, we can take anyone, have them do the same things as successful people, and our projects will succeed.

In a manager’s mind, the measure of whether information is good or not is, Does it measure whether people are doing the same things that successful people have done on projects I’ve been told were successful?

This is not the same thing as measuring whether the project is on its way to success at all. This measures the outward appearance of a project. Things that can be measured easily are rarely the most significant things. Behaviours that can be “gamed,” like how many hours a team is working, will be gamed.

And as above, even if a manger knows better, does her manager know better? If not, good information will be difficult to sell and she will be under a lot of pressure to serve Lemon Pie.

Off balance sheet transactions

There’s another important reason that projects have bad information: the best information is off balance sheet. That’s an expression meaning something a businessperson sweeps under the rug. Try Googling Off Balance Sheet Transaction. It’s never pretty.

In essence, project plans and reports never include the most important information about the likelihood of project success. Never. (I mean never in the same sense that Joel Spolsky means “nobody,” as in “fewer than 10,000,000 project plans”).

Let me give you a recent example. I was discussing a project with a certain manager in a client organization, and there was a rather linear series of releases. Her question was, Can’t we shorten the plan by looking at the dependencies and starting some releases in parallel with others?

Of course we could, but in doing so, we increased the project risk. Just as a single developer has a problem keeping multiple tasks in one brain, a team has the same problem: when working on several unrelated pieces of software at the same time, the individual developers may only work on one thing at a time, but the managers and the testers and especially the stake holders who are thinking about functionality and exercising change control are overloaded, so they will make poorer decisions.

And worse, although you might think that there are no dependencies, you only think that at the outset of a project because of assumptions you are making now, before you fully understand what you are getting into. All in all, there’s a reason Agile projects have a practice of working on iterations with single themes, and the reason is to reduce risk.

Does this mean that manager wasn’t making a reasonable trade off between risk and time? Maybe she was making a very reasonable choice. But let me ask you this: where on the plan would you see the risk component of the decision? If you were comparing Plan “A,” with the linear releases that finish in August, and Plan “B” with some parallel releases finishing in July, how do you decide which plan is better?

Right, you can’t see anything except the difference in dates, so you choose Plan B. The risk component of the plan is off balance sheet. Enjoy your lemon.

(And it is very hard for a manager’s people to explain, for the twentieth time, that it is a mistake to try to optimize a project by having everybody working at once, because that crystallizes certain architecture decisions too early, and haven’t you read any Eliyau Goldratt, whose novel The Goal explains what’s wrong with optimizing resource engagement when you should be managing project risk? The incentive is for them keep their own counsel and put their résumés up on the web. Such is life.)

A more striking example comes from another project where, for various reasons, there was nearly 100% turnover of the senior staff, and finally an outside firm was brought in to clean it up. Do you think there was a notation on the plan for staff turnover? That has to have a huge risk implication, but the plan where you use the same team from start to finish looks exactly like the plan where new people are brought in for each phase or iteration.

Why is risk off balance sheet? I think it’s because people have a blind spot for risk in projects. After the fact of a project, you can always do a postmortem and say, “if we had done this, and this, and if we didn’t do that, we would have succeeded.” So you blame yourself for poor execution.

There’s a culture of Boolean thinking about projects and plans. They worked or they didn’t. We’ll be done in July or August. Not “With Plan A, we’re 90% August, 8% September” and “With Plan B, we’re 12% July, 56% August, but 23% September and 7% October!”

Decisions that add risk to a project, such as excessive parallelization, or the use of unproved people, or the use of team who have never worked together in the past, or of forcing decisions prematurely, all of these things are not reflected in the plan.

(This is another reason there is pressure downwards on developer competence in many organizations. Consider two managers: the first staffs up quickly by selecting available developers, who may not be particularly good. In fact, they are less good then the existing team average. The second moves more cautiously, only adding to the team when the addition improves the average competency of the team. Do you think the second manager will keep their job long enough to finish a project? No, because the ticking time bomb of hiring certain types of developers is off balance sheet, but being understaffed is out there for everyone to criticize.)

I had this hope that through a kind of collaborative filtering I could have a piece of software notice that linear plans or plans involving hiring too quickly or whatever had less variance than parallel plans, and adjust the traffic light accordingly. Regardless of what people would say, it would shrug its shoulders and remorselessly remark on the actual historical averages.

Nice idea? No, it’s a dud. (Or at least, my execution was a dud!)

Somebody set up us the bomb

The second major reason I bombed is that I couldn’t find any buyers. Quite simply, the people who understand these principles are running successful projects. I know lots of people who agree that there is a problem and agree this would help, but they don’t think they need help.

I couldn’t find anyone holding their head in their hands, saying, “If only I could get the truth about our projects, I could open my budget and shower you with gold.” The people who were aware of the problems with project information were busy using decidedly low tech tools—like hiring good people and having daily meetings and estimating tasks in spreadsheets—to solve their problems.

And the people who could use some help quantifying the consequences of their choices—like managers who insist on calcifying decisions very early in a telephone book design document so that they can demonstrate progress in the form of an architecture—they don’t think there’s anything wrong with their approach.

Why is that? My conjecture is that people want help with things that fit their model of what’s important. Someone who uses skilled practitioners and XP thinks that limiting risk is important: that’s why they use two week iterations and merciless refactoring rather than up-front design.

Someone using a classical BDUF management approach thinks maximizing resource allocation and managing task dependencies is important, that’s why they spend all of their time looking at GANNT and PERT charts. When you try to sell either of them something, they want to know how it will fit the model they have in their head about how to succeed with software development, not why they should consider a new model.

And I’m not sure they’re wrong about what’s important to them, personally.

If one of those BDUF projects fails due to the architecture being a poor fit for what the team discovers is really important about the projects, or from poor decisions being made in haste at the beginning of the project, well… managers will say that the problem was with the people making poor decisions. Such managers are not shopping for tools to help them understand the trade-offs, because they do not believe they are making trade-offs.

Something I learned from selling Macintoshes back in the day is this: only make one sale. Convincing someone they have a problem is one sale. Convincing them you have the solution is another. And convincing them that today is the day to act is a third. If you have to do all three at the same time, you are doomed.

This is why experienced companies distinguish sales from marketing. The first two steps are marketing, the third is selling. When you are a new company, you don’t have the resources to market and sell. You have to work with an established pain point (eliminating the first hurdle), then use PR and limited marketing funds to get the word out that you have solved the problem (the second hurdle). You only have time and energy for the third sale, separating customers from their money.

But trying to convince managers that they’re doing it wrong (tactfully), then convince them that they want your product rather than some big rigid waterfall thing or three by five cards and XP, and then convince them that they should write a cheque today…

Bad idea. I should know, I discovered that I was trying to do exactly that.

Now you understand why I have used so much of this post to rant about the state of project management in software development. When you understand what most companies are doing and why, you understand what will sell, and why. And when I understood that the things my project were measuring and reporting were of little interest to the people who were my market, well...

The not so big business plan

I think there are lots of things that will sell, that do sell into this market. When you understand that this is a social problem, when you understand that most information is bad and that the incentives are to value bad information, and for managers to value activities that produce bad information over activities that produce good information, you can make something people will buy.

Like books, lectures, methodologies, and video training. If people have a social problem, a social solution is a natural fit.

There were interesting possibilities like selling this to BigCo for “process improvement.” But whenever I thought about such ideas, I noticed that the software wasn’t in there. I could have written a book and lectured on these ideas. I could have done a “search and replace” of agile for certitude.

A good business plan is one that is really specifically about you and your software. If your plan could be executed by someone else, or with a different project, it simply isn’t the right plan. And all variations of turning this into ConsultingWare were excellent ways of monetizing Reg Braithwaite the brand, but not ways of spreading the adoption of certitude, the product.

And so… and so I looked the sunken cost fallacy straight in they eye and said, no more. As much as I hate to lose, I have folded my project and I am sharing with you the important lessons I learned.

Lessons learned

Well, the good one is this: Paul Graham is right. If you phrase your venture as a question, you will walk away with something very valuable. I now know a lot more about project management and the culture of software development than I did when I started the project, and I wasn’t exactly a Spring Chicken. And set out to learn about Bayesian Networks and Supervised Classification, but I ended up learning about Unsupervised Clustering and Collaborative Filtering.

What a journey.

And there is the one I ignored for far too long: Always Be Selling. I can thank friends like Leila Boujnane, Sutha Kamal, and Erich Finkelstein for reminding me about this, incessantly. Asking people to “buy” your idea forces you to make something people want. That being said, I think there’s a right amount of “push.” You can’t invent by polling the market. Quickly now, jump in your time machine and go back to 1981 or so. How many people wanted a mouse for their computer? But yet… Always be selling!

(There’s a bunch of other stuff that is far more personal, and maybe some of it will appear one day in public, but I wanted to keep this post to essay length, so it’s almost entirely about products and markets.)

So, here I am. Older, wiser, and ready for life’s next adventure. It might be more consulting for BigCo. It might be joining a start up and helping someone else’s dream flower. I don’t know, but if you have an idea...

My brain is open.

Labels: agile, passion, popular

  ¶ 6:49 PM

  Three men and a lady McBride

A friend of mine is the executive assistant for a very active investor, McBride. His angle is “workouts:” he likes investing in tech companies that have fallen on hard times. Not start ups that have flamed out, but solid companies with revenues that have fallen behind the curve or have imploded for one reason or another.

He’s like one of those Real Estate people that specialize in fixer-uppers. He looks for houses with good bones in good locations that are undervalued because nobody wants the hassle and the risk of fixing the problems. He says Tech is a great area for this because everyone is so obsessed with shiny new toys that you can pick companies up for a song just because they are making a lot of money with client-server desktop applications, or some such violation of the laws of Silicon Valley optics.

My friend was flying with him on the corporate jet the other day (fractional ownership, one of the reasons he does well is that he has a good feel for the best trade-offs between time and money). He had one of his buddies on the flight, a guy running a web and database contracting shop.

She calls the buddy “House,” because she says he looks and acts like the fictional Dr. Greg House. Not the brilliant, diagnose any problem thing, but the witty but cutting manner and he likes to wear jeans and sneakers with sports jackets. House is hitching a ride to Chicago and has brought along “Wilson,” one of his long-time employees.

House and Wilson

McBride and House know each other from the golf course. When McBride was serving as the Foundation Chair of one of the country’s most prestigious Modern Art Museums, House’s company built and maintained a lot of the task and contact management systems you need to raise serious money for its newest wing. House’s dad knew the Museum’s managing director, one thing led to another, and now House is on the jet securing an important contract: McBride has analysed one of his portfolio companies, Vogler Technologies, and he isn’t happy with their IT department. He wants House to go in there and build some new systems for them.

After they get those details out of the way, they carouse a bit, and then McBride and my friend buckle down to work on valuations for a potential acquisition. House and Wilson are having a few drinks and arguing about movies.

House goes off about The Good Shepherd. How, he wants to know, is anyone supposed to buy the son’s character: this man, we are supposed to believe, follows his father into the CIA, is a member of the mysterious Skull and Bones fraternity that is loaded with CIA people and politicians, and yet trustingly tells crucial state secrets to his lover in Africa just because she tells him that people in love don’t have secrets from each other?

House is adamant, this makes no sense, especially when Matt Damon’s character, the father, warns his son that she is a spy. Even if he didn’t believe it until that moment, why isn’t there a sudden Sixth Sense moment when everything clicks into place and he realizes he’s been had?

Wilson laughs at House’s naïvité. Of course the son is a dunderhead, Wilson explains. Not in spite of his roots in the agency and the fraternity, but because of them: the point of the movie is that the CIA screw everything up because instead of hiring people based on their competence, they recruit their fraternity buddies and the sons of their fraternity buddies.

This naturally leads to “The Company” being riddled with people who put their own political games ahead of their country’s interests, and those are the ones with any talent for the job. The rest are there because they’re buddies or buddies of buddies. naturally they screwed up the Bay of Pigs, and maybe the whole movie is a commentary on the last decade of US foreign policy.

Wilson laughs loudly, makes a few comments about Angelina Jolie’s attractiveness, and then says they should have put her in charge of the CIA. She raised a son single-handedly, she’s tough, and she would never put a college buddy’s career ahead of her country’s safety.

McBride and my friend keep working while House and Wilson throw back doubles of Scotch. House and Wilson get along well, they both race motorcycles out of the same club and have been trading insults with each other back to their BMX days.

A mission suitable for a lady

The flight lands uneventfully in Chicago, and there is a car waiting for McBride and my friend. They exchange good byes with House and Wilson. My friend knows that McBride has had enough of work for now and is turning his mind to other things. She reads a book quietly.

A few moments before their ride ends, McBride leans forward. He interrupts her reading politely. Did you see The Good Shepherd? My friend has seen it. McBride purses his lips. House, he’s a Good Man. And I trust him. But does House really know his stuff?

My friend hesitates, knowing how close House is to McBride. McBride makes a living reading people’s tells, he waves his hand apologetically.

Sorry, I didn’t mean to put you on the spot. McBride is reassuring. But you know the Vogler deal? Of course my friend knows the Vogler deal, they worked ninety hours a week together making it happen.

This work is critical to our success. Find out who is The Best, and make sure I talk to them before we decide anything.

That’s all he has to say.  ¶ 12:13 PM

  Chicken Soup for the Ruby Soul As snarfed from Kevin Greer’s interesting space:

T.Rex and the Chicken:


+ (65 million years) =



Think about that with respect to programming languages. Java is still a chicken in comparison to C++, even though it has gained a little weight over the last decade.

And Ruby irritates me at times, driving me to great lengths to work around it or greenspun other language features into it. It is definitely a chicken in comparison to Lisp.

Sometimes smaller and weaker really will replace bigger and more powerful.

p.s. As requested, JavaScript showing off both prototype and script.aculo.us:

  ¶ 4:42 PM