Haggling about the price A few minutes ago I saw this interesting post: Ajax Web Developer: $240k per year… with only one catch. If you want the executive summary, the job location is Iraq, a battle zone.

What I find interesting about jobs like this are the people who debate whether $240,000 is enough to compensate for the very real possibility of being killed:

Unfortunately, the whole 12/hr/day, 7/day/week deal works out to only about $52/hour…

I think that for $240k a year, I could shift my sleep schedule to iRaq time…

I think it would be a cool opportunity, but the 12 hour days, 7 days a week drops the money down to just over $50 an hour. If there is the possibility of being shot at, I need at least $80 per hour :-P

I would take 300k, though probably (?) not 200 to work in Iraq…

And lots of people were debating the tax-free status of this position, as if that’s the only deciding factor between staying or going. I’m sure many of the people commenting were being facetious, however Winston Churchill really put things into perspective in this apocryphal exchange with a socialite:

Churchill: “Madam, would you sleep with me for five million pounds?”

Socialite: “My goodness, Mr. Churchill… Well, I suppose… we would have to discuss terms, of course…”

Churchill: “Would you sleep with me for five pounds?”

Socialite: “Mr. Churchill, what kind of woman do you think I am?!”

Churchill: “Madam, we’ve already established that. Now we are haggling about the price.”

If you’re going to sacrifice your life on principle, I commend you for your bravery.

But when it comes to money, either you incorruptible, or you have decided that your life is a commodity, something that can be bought or sold, traded or bartered, played as a pawn and sacrificed so that some contracting company somewhere makes another billion dollars for its shareholders.

but I’m not working in a battle zone…

This all seems very surreal, I’m quite sure few or none of my blog readers are wearing combat fatigues and body armour right now.

But the principle applies elsewhere. If you ever find yourself being subjected to personal abuse, or manipulative behaviour by your manager, or asked to lie to clients, or outright cheat such as double billing, you are in the exact same position as someone asked to sacrifice their life. You have to decide whether you are sacrificing your self-esteem for a noble cause, or whether it’s about the money.

And trust me on this one, if it’s about the money, you are on the road of perpetual unhappiness, where every well is dry and every inn is full, and your journey never ends.

If you have a dream, and making that dream a reality involves tough business decisions, good for you. I commend you for having the courage to duke it out with VCs, lawyers, and whomever else will try to get you to mortgage your honesty to fill their coffers.

But if you are dragging yourself to work every day, if you hate what your boss makes you do, if it’s about the money and the vacations and the toys, but not about fundamental satisfaction with your job, you need to stop right now, sit down with the people you love and trust, and reëvaluate your choices.

Trust me on this one. If $100,000 isn’t enough, neither is $200,000, and $300,000 won’t do it either. We’ve established what they want you to become. All you’re doing is haggling about the price.

the sun also rises

The good news is that unless you have signed up for a two year tour of duty with the army, you can stop any time. You can get off the treadmill. You can just stop doing things that aren’t consistent with your values.

I’m not saying you shouldn’t work long hours, or compromise your technical integrity by using a for loop instead of a map function. Making tough choices is part of growing up.

But you do not need to compromise your integrity, ever. And the good news is, you can just say no. Tell your boss to handle all communications with the client if he doesn’t want you to mention that you only put in twenty hours on their project last week because he has you working for two clients at once.

Tell your client that no, you cannot submit an invoice for $2,500 but accept only $2,000 in payment, with her skimming the extra $500 as a secret commission.

It’s incredible, but this little word, no, it really works. The world does not stop. You don’t get dragged from the room and imprisoned without trial. If you are let go, you will find another job, a better one, a job with people who like you and respect you and want to make money in an honourable fashion.

We have a lot of freedoms. The point of those freedoms is to have an almost completely unrestrained ability to pursue happiness. All you have to do is make the right choices. And the first choice is to simply say no to anything you cannot abide.

That is the road that leads us into the sunshine.

Labels: passion

  ¶ 3:37 PM

  Still failing, still learning The good news: I’m still learning.
The bad news: …from failure.

This post officially announces that my side project (originally named cause & effect and later named certitude) is over.

What I wanted to achieve

For those of you who weren’t subjected to one of my enthusiastic rants, here was my Graham Question: Can we predict the outcome of a software development project with objective observation?

I have always believed that the answer is Yes. And I manage projects that way. But I also strongly believe that the only way to prove that we have an objective understanding of an algorithm is to mechanize it, to write a piece of software that executes your algorithm.

So I set out to write a piece of software that, pure and simple, would look at a software development project and show you a traffic light: a green light would mean that the project looks like it’s on track, a yellow light would mean that the project needed help, and a red light would mean that there is no hope.

I won’t go into a lot more detail, mostly because this isn’t a story about teeming hordes of customers and me not being able to finish by deadline. It’s a story of inventing a great solution to a problem nobody cares about. But if you absolutely must visualize the application, think of something that gobbles up your MS project files, your bug tracking data, even your burn down spreadsheets, and belches out that traffic light when it’s done. That’s it.

So how and why did I fail?

Project management software is social software

Reason zero, just as Paul Graham warned, my age. Really. I remember how much code I could crank at age 22, and now that I am double the age, I write an order or magnitude less. I’d like to say that my code is that much better that it makes up for the lack of volume. That might be true if I start a project with a specific end in mind, but the act of experimenting, tinkering, and exploring benefits from being able to turn large amounts of code around in short amounts of time.

This is reason zero because I wanted to get it out of the way before explaining why I think I still would have failed if I were 22. But it’s still important to understand, because I might have known that I was going to fail two years ago, instead of today.

Reason one that I failed—and this is the most important reason I failed—is that I was trying to tackle a social problem with technology. This can work—ask any dating site zillionaire—but it is a very high-risk strategy for software. Not just for making money, but for what really matters, adoption.

Project management is a social problem.

Project management is a social problem. It is 99.5% about getting everyone who knows something about the state of the project to share what they know with everyone else. Getting all the relevant information is 99.5% of the problem, analyzing the information is 0.5% of the problem.

Joel likes to brag about how much trouble FogBugz goes to to make it easy to enter bugs, about how certain kinds of reports are not available to discourage punitive social behaviours like punishing developers who generate too many bugs. This is a strong hint that getting good information is all about managing people’s perception of the likelihood of punishment for telling the truth.

Sitting here typing this, I think the company who can do the best job of predicting the outcome of software development projects is Inkling Markets. That’s because their entire business is about finding a way for people to communicate what they really think of something, not just what they think other people want them to say about something. I think Todd Proebsting would agree.

This problem isn’t limited to dysfunctional environments where people cower in fear of saying anything except “Sir, Yes Sir” when told to change the laws of physics.

Lemons, damned lemons, it’s always lemons

Project management suffers from a real lemon problem. I quoted Bruce Schneier at length about this already, so this time I’ll explain things directly:

Most managers, especially those with limited experience shipping software on a predictable schedule, do not know how to correlate what they’re told about the project with the likelihood that the project will succeed.

Some information is valuable, some is junk. The problem is, managers “buy” information. They trade favours like letting you keep your job for information about how well you are doing your job. So there is a market for information just like a market for cars.

They also “sell” information, literally: they have to make a report or a presentation to their superiors, or to stake holders, or to their fellow founders at the YCombinator dinner.

When a manager cannot tell the difference between information that is useful for predicting the outcome of a project and information that is not useful for predicting the outcome of a project, she thinks about the next best thing: the “resale value” of the information with people one step removed from the project, like her own manager. So she values things like pretty PowerPoints about the architecture higher than finished pieces of functionality.

(This is why I have always sweated my heart out to give good presentations. My teams have depended on me being able to take good information and sell it upstream just as if it were CMM Level Five Buzzword-Compliant Junk).

Do managers further removed from a project always value pretty junk more than good, solid information? Not always, but often. And that’s enough for people to be pressured to give the bad information that sells to their manager, while hiding the good information that doesn’t sell. Exactly like the owners of good cars taking their treasures off the market.

Lemon and bay leaf crème brûlée

Why does junk information outsell good information? Nice PowerPoint isn’t a good explanation by itself: there are nice PowerPoints explaining Agile, but managers still prefer waterfall.

Consider a not so big design. Let’s call completing that design good information: we’ve done a good job finding out what’s really important for the project and making a design that emphasizes the way this project is unique, not the technology stack.

Now consider a typical technology design, emphasizing frameworks and technologies. Fully buzzword-compliant.

Which one sells better? The technology stack does. Why? Well, for starters, managers have been exposed to seventeen billion dollars worth of advertising talking about the benefits of technology stacks. Nobody is advertising the specific ways the not so big design helps the project. How could they? Those are specific to the project, that’s the whole point.

And managers are like anyone else, they compare what you are doing to successful projects they have seen in the past. Once again, the not so big design doesn’t have anything in common with other projects, but the technology stack does. (There are lots of failed projects with technology stacks, of course. But who cites those when bugging the team about whether they will use Hibernate as their ORM?)

How did this happen? How did things that have no correlation to the success of a project become more attractive than things that do?

Quite simply, people have an incentive to look successful. So they imitate the outward appearances of successful projects. We have a really simple way of completing successful software projects: we put successful people on them. But we have a broken way of thinking about it: we don’t like to think of the people as being special, we think that what the people do is successful.

And by that logic, we can take anyone, have them do the same things as successful people, and our projects will succeed.

In a manager’s mind, the measure of whether information is good or not is, Does it measure whether people are doing the same things that successful people have done on projects I’ve been told were successful?

This is not the same thing as measuring whether the project is on its way to success at all. This measures the outward appearance of a project. Things that can be measured easily are rarely the most significant things. Behaviours that can be “gamed,” like how many hours a team is working, will be gamed.

And as above, even if a manger knows better, does her manager know better? If not, good information will be difficult to sell and she will be under a lot of pressure to serve Lemon Pie.

Off balance sheet transactions

There’s another important reason that projects have bad information: the best information is off balance sheet. That’s an expression meaning something a businessperson sweeps under the rug. Try Googling Off Balance Sheet Transaction. It’s never pretty.

In essence, project plans and reports never include the most important information about the likelihood of project success. Never. (I mean never in the same sense that Joel Spolsky means “nobody,” as in “fewer than 10,000,000 project plans”).

Let me give you a recent example. I was discussing a project with a certain manager in a client organization, and there was a rather linear series of releases. Her question was, Can’t we shorten the plan by looking at the dependencies and starting some releases in parallel with others?

Of course we could, but in doing so, we increased the project risk. Just as a single developer has a problem keeping multiple tasks in one brain, a team has the same problem: when working on several unrelated pieces of software at the same time, the individual developers may only work on one thing at a time, but the managers and the testers and especially the stake holders who are thinking about functionality and exercising change control are overloaded, so they will make poorer decisions.

And worse, although you might think that there are no dependencies, you only think that at the outset of a project because of assumptions you are making now, before you fully understand what you are getting into. All in all, there’s a reason Agile projects have a practice of working on iterations with single themes, and the reason is to reduce risk.

Does this mean that manager wasn’t making a reasonable trade off between risk and time? Maybe she was making a very reasonable choice. But let me ask you this: where on the plan would you see the risk component of the decision? If you were comparing Plan “A,” with the linear releases that finish in August, and Plan “B” with some parallel releases finishing in July, how do you decide which plan is better?

Right, you can’t see anything except the difference in dates, so you choose Plan B. The risk component of the plan is off balance sheet. Enjoy your lemon.

(And it is very hard for a manager’s people to explain, for the twentieth time, that it is a mistake to try to optimize a project by having everybody working at once, because that crystallizes certain architecture decisions too early, and haven’t you read any Eliyau Goldratt, whose novel The Goal explains what’s wrong with optimizing resource engagement when you should be managing project risk? The incentive is for them keep their own counsel and put their résumés up on the web. Such is life.)

A more striking example comes from another project where, for various reasons, there was nearly 100% turnover of the senior staff, and finally an outside firm was brought in to clean it up. Do you think there was a notation on the plan for staff turnover? That has to have a huge risk implication, but the plan where you use the same team from start to finish looks exactly like the plan where new people are brought in for each phase or iteration.

Why is risk off balance sheet? I think it’s because people have a blind spot for risk in projects. After the fact of a project, you can always do a postmortem and say, “if we had done this, and this, and if we didn’t do that, we would have succeeded.” So you blame yourself for poor execution.

There’s a culture of Boolean thinking about projects and plans. They worked or they didn’t. We’ll be done in July or August. Not “With Plan A, we’re 90% August, 8% September” and “With Plan B, we’re 12% July, 56% August, but 23% September and 7% October!”

Decisions that add risk to a project, such as excessive parallelization, or the use of unproved people, or the use of team who have never worked together in the past, or of forcing decisions prematurely, all of these things are not reflected in the plan.

(This is another reason there is pressure downwards on developer competence in many organizations. Consider two managers: the first staffs up quickly by selecting available developers, who may not be particularly good. In fact, they are less good then the existing team average. The second moves more cautiously, only adding to the team when the addition improves the average competency of the team. Do you think the second manager will keep their job long enough to finish a project? No, because the ticking time bomb of hiring certain types of developers is off balance sheet, but being understaffed is out there for everyone to criticize.)

I had this hope that through a kind of collaborative filtering I could have a piece of software notice that linear plans or plans involving hiring too quickly or whatever had less variance than parallel plans, and adjust the traffic light accordingly. Regardless of what people would say, it would shrug its shoulders and remorselessly remark on the actual historical averages.

Nice idea? No, it’s a dud. (Or at least, my execution was a dud!)

Somebody set up us the bomb

The second major reason I bombed is that I couldn’t find any buyers. Quite simply, the people who understand these principles are running successful projects. I know lots of people who agree that there is a problem and agree this would help, but they don’t think they need help.

I couldn’t find anyone holding their head in their hands, saying, “If only I could get the truth about our projects, I could open my budget and shower you with gold.” The people who were aware of the problems with project information were busy using decidedly low tech tools—like hiring good people and having daily meetings and estimating tasks in spreadsheets—to solve their problems.

And the people who could use some help quantifying the consequences of their choices—like managers who insist on calcifying decisions very early in a telephone book design document so that they can demonstrate progress in the form of an architecture—they don’t think there’s anything wrong with their approach.

Why is that? My conjecture is that people want help with things that fit their model of what’s important. Someone who uses skilled practitioners and XP thinks that limiting risk is important: that’s why they use two week iterations and merciless refactoring rather than up-front design.

Someone using a classical BDUF management approach thinks maximizing resource allocation and managing task dependencies is important, that’s why they spend all of their time looking at GANNT and PERT charts. When you try to sell either of them something, they want to know how it will fit the model they have in their head about how to succeed with software development, not why they should consider a new model.

And I’m not sure they’re wrong about what’s important to them, personally.

If one of those BDUF projects fails due to the architecture being a poor fit for what the team discovers is really important about the projects, or from poor decisions being made in haste at the beginning of the project, well… managers will say that the problem was with the people making poor decisions. Such managers are not shopping for tools to help them understand the trade-offs, because they do not believe they are making trade-offs.

Something I learned from selling Macintoshes back in the day is this: only make one sale. Convincing someone they have a problem is one sale. Convincing them you have the solution is another. And convincing them that today is the day to act is a third. If you have to do all three at the same time, you are doomed.

This is why experienced companies distinguish sales from marketing. The first two steps are marketing, the third is selling. When you are a new company, you don’t have the resources to market and sell. You have to work with an established pain point (eliminating the first hurdle), then use PR and limited marketing funds to get the word out that you have solved the problem (the second hurdle). You only have time and energy for the third sale, separating customers from their money.

But trying to convince managers that they’re doing it wrong (tactfully), then convince them that they want your product rather than some big rigid waterfall thing or three by five cards and XP, and then convince them that they should write a cheque today…

Bad idea. I should know, I discovered that I was trying to do exactly that.

Now you understand why I have used so much of this post to rant about the state of project management in software development. When you understand what most companies are doing and why, you understand what will sell, and why. And when I understood that the things my project were measuring and reporting were of little interest to the people who were my market, well...

The not so big business plan

I think there are lots of things that will sell, that do sell into this market. When you understand that this is a social problem, when you understand that most information is bad and that the incentives are to value bad information, and for managers to value activities that produce bad information over activities that produce good information, you can make something people will buy.

Like books, lectures, methodologies, and video training. If people have a social problem, a social solution is a natural fit.

There were interesting possibilities like selling this to BigCo for “process improvement.” But whenever I thought about such ideas, I noticed that the software wasn’t in there. I could have written a book and lectured on these ideas. I could have done a “search and replace” of agile for certitude.

A good business plan is one that is really specifically about you and your software. If your plan could be executed by someone else, or with a different project, it simply isn’t the right plan. And all variations of turning this into ConsultingWare were excellent ways of monetizing Reg Braithwaite the brand, but not ways of spreading the adoption of certitude, the product.

And so… and so I looked the sunken cost fallacy straight in they eye and said, no more. As much as I hate to lose, I have folded my project and I am sharing with you the important lessons I learned.

Lessons learned

Well, the good one is this: Paul Graham is right. If you phrase your venture as a question, you will walk away with something very valuable. I now know a lot more about project management and the culture of software development than I did when I started the project, and I wasn’t exactly a Spring Chicken. And set out to learn about Bayesian Networks and Supervised Classification, but I ended up learning about Unsupervised Clustering and Collaborative Filtering.

What a journey.

And there is the one I ignored for far too long: Always Be Selling. I can thank friends like Leila Boujnane, Sutha Kamal, and Erich Finkelstein for reminding me about this, incessantly. Asking people to “buy” your idea forces you to make something people want. That being said, I think there’s a right amount of “push.” You can’t invent by polling the market. Quickly now, jump in your time machine and go back to 1981 or so. How many people wanted a mouse for their computer? But yet… Always be selling!

(There’s a bunch of other stuff that is far more personal, and maybe some of it will appear one day in public, but I wanted to keep this post to essay length, so it’s almost entirely about products and markets.)

So, here I am. Older, wiser, and ready for life’s next adventure. It might be more consulting for BigCo. It might be joining a start up and helping someone else’s dream flower. I don’t know, but if you have an idea...

My brain is open.

Labels: agile, passion, popular

  ¶ 6:49 PM

  Hard-Core Concurrency Considerations Kevin Greer responded to What Haskell teaches us about writing Enterprise-scale software with some insightful emails about the pros and cons of using purely functional data structures for managing concurrency in a multi-processor environment. I’ve reproduced them (with his permission) below.

Now, your first reaction might be, “Ah, Reg is not nearly as smart as he thinks he is.”

If you feel like giving me some credit, you can keep in mind that I was not writing the definitive essay on designing concurrent software, just pointing out some interesting overlaps between what I consider to be the most academic programming language around and the most practical requirements in business applications.

But there’s something more important to take from this.

The original post was a response to someone asking whether there was any value to the stuff you read on programming.reddit.com. His query was whether reading about Haskell was practical. My response was, yes it is, and here are some examples of where functional data structures have an analogue in concurrent data structures. My thesis (that’s a two dollar word for “point”) was that many “impractical” things have a lot to teach us about things we will encounter in the pragmatic “real world.”

Of course, most people read the post and thought, “Cool, some neat stuff about concurrency.” Nothing wrong with that. If you value tips on concurrent programming (and I certainly do), you’ll find Kevin’s emails very insightful.

And if you are still wondering whether you should look at “impractical” and “academic” material like Haskell, Erlang, or other things not being promoted by MegloSoft, consider that one of the papers Kevin cites describes a data structure running on a 768 CPU system. Is this in a University lab somewhere? No, it is in a company that promotes itself as an Enterprise-Scale Java company.

You simply can’t assume that everything the industry tells you to learn is everything you need. Or that any one source (Cool! Raganwald has a new post on Lazy Evaluation) has the definitive answer.

You need to commit to life-long learning to be a software developer. Some of that learning is straightforward MSDN Magazine stuff, simple updates to things you already know. And some of that learning is a little further out on the curve.

Without further ado, here is one of the most comprehensive technical replies to a post on my blog I’ve received to date.

Concurrency Considerations

Hi Reg,

I was just reading your article on concurrent collections and have a few comments:

1. Just because readers do not update the structure doesn’t mean that they don’t need to synchronize. This belief is a common source of concurrency bugs on multi-processor systems.
   
   Unless you synchronize on the root of your data-structure (or declare it as volatile), you can’t be sure that your cache doesn’t have a stale version of the data (which may have been updated by another processor). You don’t need to synchronize for the entire life of the method, as you would by declaring the method synchronized, but you still need to synchronize on the memory access. You hold the lock for a shorter period of time, thus allowing for better concurrency, but you still have to sync.
   
   If you fail to synchronize your memory (or declare it as volatile), then your code will work correctly on a single CPU machine but will fail when you add more CPU’s (it will work on multi-core machines provided that the cores share the same cache). If you have a recursive datastructure (like a tree) then you will need to actually synchronize on each level/node, unless of course you use a purely functional data-structure, in which case, you’ll only need to synchronize on the root.
   
   Given that you need to make a quick sync anyway, this approach is not much better than just using a ReadWrite-lock (it is slightly better because you can start updating before the readers finish reading (not a big deal for get()’s but potentially a big deal for iterators()), but then again updates are more expensive because of the copying).


2. I don’t think that you should be using Haskell as a model of “Enterprise-scale” anything. “Enterprise-scale software” usually entails “Enterprise-scale hardware” which means, among other things, multiple-CPU’s. The problem is that Haskell purely-functional model doesn’t support multiple-CPU’s (it’s true, check the literature (except for specialized pipelined architectures, but not in the general case)).
   
   The whole processing “state” is essentially one large purely-functional data-structure. The problem of synchronizing your data-structure appears to be eliminated, but it has really just been moved up to the top-level “state” structure (monad), where the problem is actually compounded. This is worse, because not only would you need to lock your structure in order to make an update, but you would in fact need to lock “the whole world”.
   
   Some people will advocate launching one process per CPU and then using message passing to communicate between them. This is just a very inefficient (many orders of magnitude slower) way of pushing the synchronization issue off onto the OS/Network-stack. (Q: What do multi-core systems mean for the future viability of Haskell?)


3. You forgot to mention the common technique of using “striping” to improve concurrency. Basically, what you do is create multiple sub-datastructures which each contain only a sub-set of the data. You can partition the data based on the hash of the key being stored. You then wrap the whole thing up so that it still has the same interface as the single data-structure.
   
   The advantage of this approach is that when you use a ReadWrite lock you only need to lock a small portion of the data-structure, rather than the whole thing. This allows multiple updates to be performed in parallel. This is how Java’s concurrent collections work. See: ConcurrentHashMap: Java creates 16 sub-structures by default but you can increase the number when even more concurrency is required.


4. Have a look at Azul’s non-blocking HashMap implementation. They can do 1.2 billion hashMap operations per second (with 1% of 12 million/sec of those being updates) on a 768 cpu machine (the standard Java ConcurrentHashMap still does half a billion ops/sec which isn’t bad either) . This shows how scalable non-functional methods can be.
   
   I’ve never read of any Haskell or other purely-functional implementation being able to scale to anywhere near these numbers.

Summary: Using a purely-functional data-structure does make it easier to support multiple readers, but you still need to sync briefly at the “top” of the structure. Striping has the added advantage of also supporting multiple-writers as well, and in practice, this is a much bigger win. Haskell is inherently limited to a single CPU, which doesn’t match modern hardware; especially of the “enterprise” class. Shared-memory models of concurrency have demonstrated much better performance than purely-functional models.

Best Regards,

Kevin Greer

p.s. I've actually implemented a b-tree structure for very large data-sets and chose to use a purely-functional data-structure myself. My reason for doing so was that I have some very long-running read operations (~8 hours) and I obviously can't afford a ReadWrite lock that's going to starve writer threads for that long.

Another nice advantage of using purely-functional data-structures is that they make it easy to implement temporal-databases that let you perform time-dependent queries.

I just wanted to point out that if all you have is quick-reads then purely-functional is no different than a simple ReadWrite lock and that a super-pure implementation, as with Haskell, doesn't scale to multiple-CPU's or to highly concurrent updates. However, it can be used to good effect in combination with striping or other techniques.

(A tangential note: Java's GC is getting so good in recent releases that P-F data-structures are becoming much more efficient (given that P-F generates more garbage).)

p.p.s. One more advantage of functional data-structures (the largest advantage for me actually):

They map well to log(or journal)-based storage. Functional data-structures never update old data, but instead, just create new versions. If your data-structure uses disk-based storage then this means that you never need to go back and overwrite your file; you only need to append data to the end of the file. This has two advantages:

1. This works well with the physical characteristics of hard-disks. They have incredibly high transfer rates (10’s of Megs/sec) but very slow seek times (~ 200 seeks/sec if 5ms access time). If you are adding say 1k objects to a data-structure which requires 2 updates on average to a file then you’re looking at about 100 updates per second. If on the other-hand you write all of these updates to the end of the file then you’re looking at say 20,000 updates per second!


2. You can’t corrupt old data with interrupted writes. The old data is always still there. The only place that a corruption occur is at the end of the file, in which case you just scan backwards until you find the previous good head of your data-structure.

---

This is fantastic stuff to share, thanks, Kevin!

What other tips can readers contribute for people building highly concurrent software (besides the frequent use of JProbe Threadalyzer, of course)? What online resources (papers, articles, books) do readers recommend for the professional developer?

Labels: java, passion, popular

  ¶ 2:17 PM

  Having fun

I’m just going to keep trying to find interesting problems to work on, and if one of them happens to change the world, then great. And if not, at least I’ll have had fun trying to solve some hard problems.

Labels: passion

  ¶ 9:32 AM

  Quote. Unquote.

I admit I live in a bubble. The thing is, my bubble is the one where they develop the technologies that you’ll be using in your bubble in ten years.

Basically, I hang around with people who are good programmers. I realize this is not a random cross section of computer users. But it is a disproportionately important subset, because what they use, other people will be using later. These were the people who were using microcomputers in 1980; now everyone is; the people who were using email in 1988; now everyone is; the people who were using the Web in 1995; now everyone is; etc etc.

Ok, take a deep breath. Forget about whether you find these statements arrogant or elitist. Forget about whether you think they are true or not today. Forget about whether they were once true (I know that on Saturday, November 9th, 2002, they were absolutely true).

The question I ask myself, and that I think you might want to try asking yourself, is this:

Am I developing the ideas, technologies, and products today that the rest of the world will be using in ten years?

If not, what’s stopping me?

p.s. Don’t miss the comments!

p.p.s. And of course, this is not an orginal thought on my part.

Labels: passion

  ¶ 2:59 PM

  I wanted to be on the side that was right, even if I didn’t win

This is where I am great. I am great at being very, very stubborn and ignoring all sorts of reasons why you should change your goal, reasons that many other people will be susceptible to. Many people want to be on the winning side. I didn’t give a damn about that. I wanted to be on the side that was right, and even if I didn’t win, at least I was going to give it a good try.

Labels: passion

  ¶ 9:46 PM

  Programmers are Sages Raymond Smullyan does much more than write about Infinity, Combinatory Logic, Recursion Theory, and Gödel’s Incompleteness Theorems. And he does much more than teach these concepts delightfully by turning them into stories and puzzles to solve. He also teaches philosophy by way of insightful essays and anecdotes.

I’m going to paraphrase one of his down-home insights. This particular little story has helped me through many dark periods in my life. Raymond was talking about Taoism, and he explained that his two pet dogs embrace the Tao. I think he described them as “Sages.”



Raymond explained that like many dogs, they preferred rich chunks of meat and gravy to dried kibble. And like many people, Raymond had tried making the kibble more palatable by mixing in some meat with the dry dog food and pouring gravy over the whole thing.

The dogs, of course, would carefully pick the meat out and lick up all the gravy, leaving nothing but the dried kibble behind. But did they resent this? Did they howl and whine about the kibble ruining the meat? Of course not! They would wag their tails and frisk about, joyous in finding all that good meat and tasty gravy in their bowls. It was like the kibble existed in some alternate dimension that the dogs cold not perceive.

Aaah… Dogs truly are Sages.

Kibble

Every time someone writes about interviewing and hiring, they are writing about judging people. That’s really deep and meaningful to us as people: being picked or not picked, picking or not picking is pretty much all of our social structure. So when you discuss it, you are reaching deep into people’s souls.

I know this, because it hurts me when I interview for a job and the response is “thanks, but no thanks.” It hurts when people say, “you don’t have the right kind of degree” or when they tell me that I forgot to use GROUP BY when writing a SQL query to sum the sales of automobiles by brand, or that they don’t think a Black engineer would go over well with Southern clients (all true stories).

There’s almost no way¹ to write about picking people without getting some heartfelt “thank you for recognizing my worth” responses. And some invective from people who feel you’re saying that you would pass them over. The fact is, there’s some unpleasant dried dog food in every post about interviews and hiring. Even if you pick something you think is obvious and unassailable.

Delicious chunks of meat!

Of course, some programmers just eat the delicious meat and ignore the kibble. Look! A tasty programming problem!

I’m one of these. What was my first thought when I saw a certain notorious screening question? I thought, “Hmmm… Instead of a loop with three conditions, this could be expressed as the composition of three functions, let’s try writing it that way.”² A tasty chunk of thinking about programming meat!

You know, it never even occurred to me to wonder whether I fail the test and be rejected and be cast out of the herd.³ I mean, honestly, was the original author interviewing me for real? No! So why eat the dry dog food? Honestly??

But in even the most trivial problem there’s some meaty thinking to enjoy. Composing functions. Yummy.

Covered in Gravy

Based on responses, the overwhelming majority of readers of this blog are Sages.

Most people respond to a mixture of dried kibble and meaty chunks friskily, wagging their tails and leaping about with programming solutions and discussion about how best to solve the problem. Yummy stuff indeed.

Some people might wonder if we’re playing Stairway to Heaven. Of course if you walk into a music shop and yell “nobody can play Portrait of Tracy,” everyone with a bass is going to start hitting harmonics.

That isn’t because we’re insecure and need to prove ourselves. It’s because we love playing and you just named a tune.

What can I say, I’m delighted with how many people love to play music. And I’m delighted with how many programmers love to program. For its own sake.

So thank you all for reminding me that life is all about the tasty chunks and delicious gravy.

I really appreciate it. Thank you. Especial thanks to Tom and Eric for constantly telling us all about the tender and sublime taste of Haskell. It’ll be on the menu one of these days.

Wonderful, the universe is covered in gravy!

---

1. Well, there is one way to write about picking people without offending anyone: if you really say nothing at all. For example: At NiceCo, we conduct our interviews as conversations. We discuss each candidate’s experience and accomplishments, we canvas their views and share our objectives with them, working collaboratively to identify the most appropriate contribution they can make to the team’s success.
   
   Now see, you can’t criticize that because it didn’t actually say anything. Maybe it’s a wonderful process involving nice people, or maybe what really happens is that they decide whether to hire you in the first thirty seconds based on some triviality like whether you have body odour. If you don’t say anything, there’s no kibble and no meat.
   
   It’s an empty dish.
   
   This does not mean it’s a bad process or a bad statement. But the process isn’t reproducible from the statement alone. That’s the exact sense in which I say that the statement is empty.


2. This separates the concern of how to generate the list from the concern of how to transform the list. Someone posted a variation with different conditions: you can simply generate a new transformation function by composing two of the exiting functions with two new functions, more or less. And if someone asked for the same thing in Ancient Rome, you could write a different generator that generated the numbers from I to C.


3. Don’t bother writing to say that my confidence is the reason why: I recall failing a technical interview when I brain cramped and was unable to write a program to print the first 100 prime numbers. I’m not that smart.

Labels: passion

  ¶ 1:23 PM

  But Y would I want to do a thing like this?

Choose life. Choose a job. Choose a starter home. Choose dental insurance, leisure wear and matching luggage. Choose your future. But Y would I want to do a thing like that?

Writing about first-class functions and their compatibility with object-oriented programming naturally leads to the Y combinator. And that is the point where eyes glaze over and soft, snoring sounds rise from RSS readers everywhere.

But please bear with me, this essay is not really about the Y combinator, it’s about learning new things and expanding our capacity to think.

Sharpening the saw

Years ago I picked up Steven Covey’s book The 7 Habits of Highly Effective People. If the book is a test out of seven, I really wasn’t doing very well.

If you’ve read the book, you probably remember that he talked about “Sharpening the Saw,” investing in your own abilities. That’s incredibly important, but I don’t need to tell you that. If you exercise with programming katas, or learn a new programming language once a year, or pick up a book like The Reasoned Schemer and actually go through the exercises, then you are already in the top 1% of software developers for personal skills improvement. (Sorry, certifications don’t count. They are the classic case of doing the wrong thing for the wrong reason!)

New ideas—by which I mean, new to you—are an important way to sharpen your saw. If for no other reason than this: the brain needs regular exercise to perform at or near its potential. Learning new things keeps you sharp, even if you don’t directly use the things you learned.

Others have suggested that learning Lisp is beneficial to your programming skills in its own right. That’s one good way to sharpen your saw. But I add to that an important caveat: to obtain the deepest benefit from learning a new language, you must learn to think in the new language, not just learn to translate your favourite programming language syntax and idioms into it.

Think different

The interesting thing about that is that almost by definition, if you see something in, say, Lisp that solves a problem you already have, you won’t learn much from the Lisp code. It is tempting to think that Lisp (or any other language) will somehow do what you’re already doing in some wonderfully magic way that is obviously better. But no, that isn’t how it really works.

For your problems are tuned to your existing tools. You simply can’t imagine problems that your tools can’t solve well, much less can’t solve at all. That’s why there are so few continuation-based web servers. Who’s going to invent one unless they have a programming paradigm with continuations?

And worse, when a new tool is applied to a problem you think you know well, you will probably dismiss the things the new tool does well. Look at how many people dismiss brevity of code. Note that all of the people ignore the statistics about the constant ratio between bugs and lines of code use verbose languages. Look at how many people dismiss continuation-based servers as a design approach. Note that all of them use programming languages bereft of control flow abstractions.

Thus, to truly learn a new tool, you must not just learn the new tool, you must apply it to new kinds of problems. It’s no good learning to replace simple iteration with first-class functions: all you’ve learned is syntax. To really learn first-class functions, you must seek out problems that aren’t easily solved with iteration.

The Why of Y

Which leads me back to fixed point combinators. They appear to have no practical (as in making money) use. And that’s why I’m suggesting to you that you figure out how to make one in your language of choice. The very fact that the problem is far outside of your realm of “practicality” guarantees that you will learn something. You won’t be simply applying your same-old, same-old techniques and patterns to a slightly new problem.

Start your research with Richard P. Gabriel’s The Why of Y. Try porting his examples directly to your favourite programming language. If what you want to use is too brain-damaged to support closures, you may need to do a little greenspunning and build a little functor infrastructure.

Don’t be dissuaded if you have to follow the functor route: you are learning far more about your language and about programming in general than the shmoes that settle for learning five new buzzwords related to the latest WS-* interoperability with XPath 3.x.

If you prefer a fun approach to learning, you can do not better than Raymond Smullyan’s To Mock a Mockingbird: an enjoyable romp through the world of combinatory logic. After reading this book, you will have mastered the S, K, I, Y, and other combinators. Added bonuses include a safe that can only be opened by applying Gödel’s Incompleteness Theorem to its combination. How can you read this book and not learn?

Eating my own dog food

I thought of a few things to say along these lines last week and then I abruptly realizing I was asking you to “Do as I say, not as I do.” What good is recycling problems I first encountered in University textbooks two decades ago? I put this post aside and set to work on a problem of my own.

I set out to write a function for making recursive functions—a function extensionally equal to the Y combinator—in Ruby. The ultimate goal is to take something like:

lambda { |f| lambda { |n| n.zero? && 1 or f.call(n-1) } }

And be able to have it call itself recursively. In this case, to compute the factorial function.

This is trivial, given that Ruby supports named recursion, but if you want to write a fixed-point combinator you want to write a function that makes recursive functions without using the host language’s support for named recursive calls. In other words, you are bootstrapping named recursion out of anonymous first-class functions.¹

There are important theoretical implications of being able to do this, but the killer reason to try it is to learn.

I started my quest for a function for making recursive functions with a rather trivial observation based on OO programming and the Curry function:

require 'test/unit'

class ExempliGratia < Test::Unit::TestCase

  CURRY = lambda { |f, a| lambda { |*b| f.call(a, *b) } }

  def test_recursive_curry
    maker = lambda { |func_with_me|
      CURRY.call(func_with_me, func_with_me)
    }
    assert_equal(120, maker.call(lambda { |me, n| n.zero? && 1 or n * me.call(me, n-1) }).call(5))
  end

end

In OO some language implementations, this (or self) is a hidden parameter passed to each method. Thus, there’s a parameter—me in the example code—that is added for handling recursion. If you write a recursive function—like the venerable factorial—with the extra me parameter, a trivial currying operation evaluates it recursively without any need for names.

This is obviously deficient. As noted above, we want to write factorial like so:

lambda { |n| n.zero? && 1 or f.call(n-1) }

We’ll need an f from somewhere, and just as our Scheme colleagues do, we’ll bind one as a parameter in an enclosing lambda. So we want to write:

lambda { |f| lambda { |n| n.zero? && 1 or f.call(n-1) } }

And somehow this should be transformed into a working factorial function. For the test-driven crowd, we want to write:

def test_clean_up_loose_ends
  maker = ...

  factorial = maker.call(
    lambda { |f| lambda { |n| n.zero? && 1 or n * f.call(n-1) } }
  )
  assert_equal(120, factorial.call(5))

  iterative_factorial = maker.call(
    lambda { |f| lambda { |n, acc| n.zero? && acc or f.call(n - 1, n * acc) } }
  )
  tail_factorial = lambda { |n| iterative_factorial.call(n, 1) }
  assert_equal(120, tail_factorial.call(5))
end

Of course, we need some code for maker. And the iterative_factorial case shows that maker works for functions with more than one parameter. The solution I came up with is:

CURRY = lambda { |f, a| lambda { |*b| f.call(a, *b) } }
maker = lambda { |f|
       lambda { |func_with_me| CURRY.call(func_with_me, func_with_me) }.call(
            CURRY.call(lambda { |inner_func, me, *args|
                        inner_func.call(CURRY.call(me, me)).call(*args) }, f)) }

The source code with each transformation from beginning to end is here (I strongly suspect that this “curry combinator” is actually the Y combinator with a huge amount of cruft hanging off it).

Unique or derivative, crap or craft, the process of getting it to work has enriched my mind by forcing me outside of my usual problem space. I still can’t think of a practical application for what I’ve just written. But I know I’ve stretched myself.

And now back to you: perhaps you’re rushing off to try to implement a fixed-point combinator from first principles. Perhaps your plan is to code the canonical examples in your usual language. Those are both good paths. But whether you follow them today or not, remember the underlying principle exemplified by the fixed-point combinator:

Do not dismiss impractical or weird problems. While you may not have an immediate application for the code you write to solve such problems, you are maximizing your learning when you venture outside of your usual problem space.

---

1. Named recursion is stuff like foo = lambda { |...| foo.call(:bar) }. It takes advantage of the host language’s variable binding to recurse. If you want anonymous recursion, you should be able to assign the same lambda to another name and have it work just as well, as in: fufu = lambda { |...| foo.call(:bar) }. That won’t work if you are relying on Ruby’s name for foo.

p.s. Don’t miss Tom Moertel’s derivation of the Y combinator in Ruby.

Labels: lispy, passion, popular, ruby

  ¶ 12:06 AM

  It's Monday, it's snowing, and I'm actually happy to be at work! I found this link on reddit.com:

“I’m happy to grow up, but I won’t pretend fun things aren’t still fun out of fear of looking silly.”

Ok, I didn’t laugh hysterically at the comic. But I love the sentiment. Raising my son in partnership with my wife is a lot of fun, and that involves a serious amount of growing up.

Why should programming be any different? Business is almost as serious as nurturing a life. It’s generally considered good form to enjoy parenting. Why shouldn’t we enjoy writing software the exact same way?

Sure, we don’t decide to rewrite an entire business application in CPS on a whim because it seems like it might be fun to try. Just as we don’t teach our children to eat Train Cake for every meal.

Let’s not pretend fun things aren’t still fun out of fear of looking silly. Programming is “Grown Up.”

And fun.

Labels: passion

  ¶ 3:43 PM

  The false dichotomy of choosing between your values and expediency Wil Shipley:

Sometimes the most important thing in a fight is which side you are on, and not whether that side can win or not. Sometimes we just have to believe that, eventually, good will win out, and we have ask ourselves, “How will my children judge me when I tell them the story of this time?”

When my kids ask me about the first black man who really had a shot for the presidency, will I be forced to dissemble: ‘Well, see, kids, at the time, I didn’t think he could win, so it seemed expedient for me…’

This blog is not about politics, and it is especially not about politics in another country. If you are one of my American readers, I expect you to decide for yourself what is the right thing to do with your vote, time, and money. That being said, I quoted Wil because I admire something that he is saying there, something that is on topic for this blog.

Let’s start by dispelling one ugly myth about voting “strategically” (as we say in Canada) or for someone “elect-able” (as you say in America): It isn’t about the outcome of the election.

Let’s start by dispelling one ugly myth about voting “strategically” (as we say in Canada) or for someone “elect-able” (as you say in America): It isn’t about the outcome of the election. That’s a little white lie people tell themselves so they can do something they are embarrassed about.

They are really thinking “Here’s this thing that I ought to want to do, vote for the best man who happens to be Black, but—truth be told—I don’t want to do it. But I absolutely don’t want to admit I don’t want to do it, so… I’ll make up this story about expediency and how important it is for a Donkey to beat an Elephant, so I’ll nominate this other guy to run.”

Did you read my “Don’t Feed the Trolls” post? One of the key lessons was that people often are embarrassed about their real motivations for making decisions, so they dress them up in objections that they think sound reasonable to other people. Like winning elections. I’m going to come back to this in a moment. You are a stack-based reasoner, right? So PUSH the idea that what people say about why they are doing things rarely matches why people are actually doing things.

Let’s talk about software development

Think about your software development values. A value is something you are willing to pay for. I can’t tell you what your values ought to be, but I can tell you one thing that absolutely isn’t a value. Making money is not a value.¹ It’s an expediency.

Money is almost always a “reasonable objection to tell other people.” For example, taking a job with a consulting firm because “it pays more money.” When what you really want is the prestige of going into workplaces and telling people what to do. Without bothering to actually, you know, have any real industry experience first.

Synonyms for “making money” include “getting a job,” “sales,” “raising capital,” amongst many others.

Now we have already talked about people that have a value but can’t admit it so they tell a little white lie about “expediency.” It’s not just politics. Do you think our choices in software development are any different? Of course not. We are the same whether we are choosing mates, voting, or picking a programming language. So the way we choose is the same.

But there’s an extra dimension we are not considering. If you read The Wisdom of Crowds, you remember this idea of an information cascade, where information about other people’s choices actually makes the crowd more stupid, not more wise.

Synonyms for expedient include “backwards compatible,” “maintainable,” “easy to understand,” “easy to sell to management,” and many others.

The problem is this: when we see a bunch of people doing something, it seems like a reasonable, sane option. So do you want your fourth Java Certification? It seems reasonable, a lot of people are getting their certification, it’s an expedient way of getting a job.

Or let’s pick something that isn’t such a blatant straw man. Do you want to put your business logic into stored procedures instead of in your model objects? Or perhaps do you want to build yet another rich MVC web application instead of something REST-ful or perhaps one that is continuation-based (like a Seaside application)?

Really? Or is it just expedient to do it that way (synonyms for expedient include “backwards compatible,” “maintainable,” “easy to understand,” “easy to sell to management,” and many others).

The problem with the little white lies

So you’re a reasonable person, and you see a lot of other reasonable people doing something because it’s expedient. You value something, but you are wondering whether you ought to compromise your value in favour of expediency. Life is tough some times, and you have to grow up. What’s wrong with that?

Here’s where we go to the stack: POP. Remember, lots of those other people aren’t actually doing the expedient thing when they say they’re being expedient! They’ve got a whole ‘nother motivation, but they lied about that motivation to you. It isn’t a real reason, they aren’t being honest with you (or themselves) about why they’re doing it, and you go along with it thinking it’s a real, valid option. But it’s smoke and mirrors for other people’s frailties (bless them, they are as human as we are).

For example, the argument that BDUF is the expedient way to manage a project? Look, I’ll level with you: if what you want is to demonstrate to your manager that you have everything under control, that you are in charge, then go for it. But expedient? Sorry, that’s bunk. It’s just a little white lie a lot of people have told you because they fear telling the truth, like:

• They have no idea how to manage software development activities, so forcing developers to specify everything in advance relieves their anxiety that they will have to actually manage things as they go along. Instead, they can just follow “The Plan.”


• They have no idea how to manage software development to a deadline, so forcing developers to estimate every activity in advance—no matter how far the activity is away—relieves their anxiety that they will have to ever explain whether a project is on target for release or not. Instead, they can simply parrot how much of “The Plan” seems to be complete and whether actual times for activities are ahead or behind estimates.


• They have no idea how to manage stake holder expectations and priorities, so forcing stake holders to specify all of their requirements in advance relives their anxiety that they will have to engage stake holders in a continuous and open dialogue over the life of the project.


• The road to senior management in their organization is paved with process documentation, not successful projects.

Yes, we have to grow up and consider expediency. But we also have to grow up and learn that other people are human and don’t always tell us the whole truth about why they do what they do. Which means it is difficult to judge whether something is right for us based on what other people say is right for them.

Be careful when presented with a choice between what we value and what is “expedient.” This choice is almost always a false dichotomy: the “expedient” option is usually a little white lie someone else has told us because they think it will sound reasonable.

We need to be very, very, very careful when presented with a choice between what we value and what is “expedient.” This choice is almost always a false dichotomy: the “expedient” option is usually a little white lie someone else has told us because they think it will sound reasonable, even if it isn’t what they were thinking.

Therefore my exhortation to you is this: If you really, truly want to program in Lisp, or manage your next project with BDUF, or write unit tests before you write code, because these actions reflect your values, then go ahead. I honour you for your choice, even if it’s different than my choice. Rock on!

But… if you really, truly want something else but are doing the expedient thing because that’s where the money is, or the jobs are, or there’s safety from criticism, or any of a hundred or a thousand or ten thousand little white lies, stop it right now. You need to understand that the chattering voices you hear, the ones that sound so reasonable (“Zune sucks, but MSFT always wins in the end, so buying a Zune is expedient”) don’t even reflect what those other people really think or want.

They’re just little white lies other people are telling you because they don’t have the courage to tell you the truth about themselves.

---

1. If you’re reading this blog and telling me that you—to pick just one example—choose to program .NET because “that’s where the money is,” you are misguided. Sorry, but the money is actually on Wall Street or Bay Street where even the poorest analyst makes more money with convexities and gammas than you or I will ever make with objects and tables.

Labels: passion

  ¶ 2:40 PM

  Children should be making things, communicating, exploring, sharing, not running office automation tools

One of the saddest but most common conditions in elementary school computer labs (when they exist in the developing world), is the children are being trained to use Word, Excel and PowerPoint.

I consider that criminal, because children should be making things, communicating, exploring, sharing, not running office automation tools.

Labels: passion

  ¶ 9:59 PM

  A Fellow Traveller's Road to Enlightenment

I began to see that many of the sacred GoF design patterns were not, in actuality, grand universal truths of software engineering, but simply collateral damage caused by incidental limitations in the abstractive power and object model of certain manifestly-typed programming languages.

A wonderful read.

Labels: passion

  ¶ 11:32 AM

  Where were you on Saturday, November 9, 2002? Had you dropped by MIT on November 9th, 2002, you might have seen a rather large collection of hackers, academics, tool vendors, and programming language enthusiasts gathering for the LL2 conference. It was rather like taking the current readership of Lambda the Ultimate and programming.reddit.com—no, I am wrong.

It was rather like taking the top couple of hundred of the people who write papers and blogs cited on Lambda the Ultimate and programming.reddit.com and gathering them in an auditorium together to talk about programming languages.

As the rather Spartan LL2 web page explains, the theme of the conference was:

How to get usable and useful programming tools into the hands of programmers, minimizing dogma, and maximizing flexibility.

Like any other conference, there were speakers. The speakers talked about the languages they had invented, and about how and why people ought to use them.

Had you been there, what would you have thought about the first speaker, Joe Armstrong, and his language, Erlang? Just another weird language for embedded telephone switches? Or an important language that is powering diverse applications such as a distributed, replicating database?

Joe spent a lot of time talking about unreliability. About how computers and programming languages are always designed as if everything always worked and that not working was the exception case, rather than the rule. And about how you can’t build systems out of dozens, hundreds, or thousands of pieces without assuming that everything is unreliable, all of the time.

And how there ought to be a programming language that builds that right into the system. Like Erlang. Would your eyes have glazed over? Or would you have thought about MapReduce and grid computing and would you have taken notes furiously?

Would you have thought that two different talks (one about persistence and another about asynchronous exceptions) were too much for an obscure language called Python? Would you have marked Python down as “One to Watch”?

And what would some random guy know about inventing “the next big programming language”? Oh, he was managing Microsoft’s Programming Languages Systems group. Well, there you go, one guy that Corporate America might care about. I wonder how many of his predictions about the next big programming language have come true? Good thing his talk is on line, we can check…

After lunch, some random guys talk about getting an un-typed language running on top of the Java Virtual Machine. I mean, really, who the hell would think that would ever become important?

Next up, the organizers had flown Yukihiro Matsumoto in to talk about Ruby. Yes, in 2002 the organizers wanted a bunch of Lisp-heads—the Q & A for absolutely every talk included at least one question about when macros would be added to the speaker’s language—to hear from someone who had designed a language known at the time as the most popular scripting language in Japan. Had you heard of Ruby in 2002? Did you care?

And what was with him constantly talking about making programming fun and comfortable? As if a programming language was a user interface for programmers. Everyone knows that programmers love idiosyncratic tools like Emacs, right? Who designs a language for usability? Who indeed.

There were other talks. They were all good, just as good. There was a spirited exchange of ideas over lunch in the lobby, again upstairs in the MIT researchers’ offices, and continuing late into the night at a nearby pub. It was a phenomenal gathering.

Were you there?

That isn’t important right now. If you were there, good for you, and I hope in the four years since you have taken some of those ideas, and a lot of that enthusiasm, and rolled it into what makes this industry great. The fact that anyone—even some guy from the other side of the world—can make something that other people like and use. Whether it’s free or for pay, whether it’s a language, a tool, an application, or a piece of hardware.

But even if you were there, that was then and this is now. 2007. Here’s the important question: Where is LL2 this year?

I asked “where is LL2 this year.” Actually, it may not be a Lightweight Languages conference. It might have nothing to do with programming languages. But somewhere in the world, somewhen in 2007, there is going to be a conference. Or a gathering. Maybe it’s this year’s Startup School. Maybe it’s a DemoCamp in your city.

Wherever, and whenever it is, you are going to see and hear things that are going to matter to you over the next five years.

They may not look like much in 2007, but they’re the seeds of the next Big Thing, the thing that will be big four or five years hence, in 2012.

Finding the conference

In 2002, LL2 didn’t look like much. You probably couldn’t find a job using any of the languages discussed at the conference. Unless you wanted to work on telephony. The lone representative of a mainstream company—Microsoft—wasn’t talking about their products, he was talking about things that would disrupt their products.

So if you’re looking for the right conference in 2007, there are two clues: the agenda isn’t going to include anything you can use to make money, and it isn’t going to feature anybody trying to make money off of you.

That’s right: no consultants, no white papers, no thinly veiled sales pitches. That knocks most Ruby conferences out. Maybe not all, but most. The right conference isn’t going to be about building CRUD applications on top of MySQL with AJAX goodness. That may not be out of date, but it’s so 2007, and we’re looking for 2012.

In 2002, the audience was highly interactive. Most of them were working on their own projects. They were practitioners, not tourists. They weren’t there to get a job, or goof off work. They were there to learn stuff for themselves and share what they knew with others.

In 2007, you should be looking for the same thing. For a conference that will be attended by people who are, as Joel would say, Smart and Get Things Done. Lacking a way to check the door before you show up, how do you find out?

Naturally, you find out by talking to other would-be attendees before going. LL2 had LL1’s active mailing list. In 2005, Startup School had a wiki. I don’t know what the right conference in 2007 will have, but I will bet on this: if there’s no community building in advance around a wiki, a list, or some other peer-to-peer communication, it isn’t the right conference.

The kind of conference where there is a one-way exchange between you and the organization—they tell you where to show up and you send them money—is not the right kind of conference. Look for the conferences that are built on top of active communities.

And when you find the community, pay attention to the action-to-talk ratio. I’m terrible at this, I always have to push myself to work and stop goofing off on my blog. But you know what’s worse? People who don’t have any work to begin with, it’s all just entertainment to them. They are tourists, and they are a liability to a conference.

Here’s why.

When tourists reach critical mass, the organizers start to cater to them. That means safe topics and safe speakers. Consultants. People with a good “patter” but not much to say that would disturb anyone.

Also, the tourists tend to ask the wrong questions, because they aren’t the market for anything new and interesting. Say a tourist raises her hand. She’s going to ask about something she imagines might concern a user, but you know what? She isn’t a user and she really has no idea how a user thinks, because she’s not the type to learn anything new until it’s already popular, and she doesn’t have any idea how such people think.

I think the root of your mistake is saying that macros don't scale to larger groups. The real truth is that macros don't scale to stupider groups.

So she asks something irrelevant. Like about language performance. Or bindings to Oracle. Or whether there is an Eclipse plug-in. Or how it will scale to larger programming teams.

But when a practitioner asks a question, it’s a smart question, it’s important. The speaker learns something important about their tool and the rest of the audience learns too. The next thing, someone asks a smarter question, and the energy builds.

If there are too many tourists, you can’t learn anything significant in the Q & A, and that’s a shame, because Q & A is where the action is.

So, my advice for finding the right conference is, avoid anything to do with either you or the presenters making money, find a conference built on top of an active community, and go the the conference with the highest density of practitioners.

And my question really ought to have been, will you be at this year’s important conference? and better still, will you be presenting?

Labels: passion, ruby

  ¶ 4:57 PM

  Never stop learning

I think every program you write should be the hardest you’ve ever written.

Labels: passion

  ¶ 7:54 AM

  Does the term "Exponential Growth" mean anything to you? Assaf laid the tag-smack on me, so here are five little known facts about me. This blog has a strict theme, so all five are “nerdy”:

1. “My” first computer was the High Speed Job Stream in the Sanford Fleming building at the University of Toronto. It was an IBM something-or-other, but there was no restriction on running punch card jobs in SNOBOL, LISP, or WATFOR. I was twelve at the time, so please don’t be shocked when I say that it took me a few weeks before I became bored of printing calendars with line-printer art of playboy models and progressed to using Fortran to print the multiplication table and Fibonacci series.
   
   My second computer was a Nova 1220. I wrote BASIC programs for it and especially my own highly customized Star Trek game. It needed all 16K of RAM, it was so big. The listing was fourteen feet long. More than a few people tell me with pride that they work on Enterprise software that would probably be fourteen miles long if anyone ever printed all of the source. I change the subject by telling a joke about a truck. Yes, those are front-panel switches for bootstrapping the CPU. Yes, I learned the three 16 bit words needed to obtain an audience with the demon.


2. I grew up in the sixties, when there were still race riots, when companies routinely discriminated against women because “they would just quit to start a family,” and when Toronto hosted a great number of young American men who didn’t want to fight in Vietnam. I helped my parents prepare for protests by colouring placards. I lived on a commune for a short while and I went to “Young Socialists” camp in the Summers.
   
   Although this seems unrelated to nerdy things, I think anyone reading what I have to say about coding as a form of self-expression and about the importance of the programmer in the production of software will see the unbroken thread that has run through my life.


3. My first job was in “Mr. Gamesway’s Ark,” a games and puzzles superstore way ahead of its time. My second was as a haberdasher. my career has oscillated in the same fashion ever since: the pursuit of beauty for its own sake (in math or in dress), the social side of nerdery, and sales. Software development in a startup environment seems to deliver all of the goods in one load. And when someone gives me a friendly tip to “wear a blue suit to the presentation,” it’s always nice to be able to ask in return “the Chalk Stripe or the Plain Navy?”


4. I dream in black and white, although from time to time I am aware of a colour in my head: I know in the dream that something has a colour, but I cannot see the colour and this seems very normal. Like most of humanity, I remember having flying and levitation dreams as well as helplessness dreams.


5. I am an autodidact. This is probably obvious to anyone with a real (as opposed to JavaSchool) education in Computer Science.

Lucas, Chris, Reinier, Rusty, and Dharmesh, you’re it. The ‘rules’ appear to be: five personal trivialities and tag five more victims. Good luck finding five who haven’t already been tagged by this generation of growth!

Labels: passion

  ¶ 3:19 PM

  A Ruby, Io, and Ocaml Programming Pattern

Every 5 minutes you spend writing code in a new language is more useful than 5 hours reading blog posts about how great the language is.

Labels: passion, ruby

  ¶ 1:38 PM

  What colour is your parachute? You know the old saying:

A mind is like a parachute: it works better when it's open.

So: what's with the black-and-white sounding statements I make about subjects like people ("no methodology can save a project staffed with under-performers") or local variables ("bad") or even static typing ("useful, but it doesn't prevent errors that endanger my projects")?

My World

I write about my experience, what has worked for me and what I think will continue to work for me. I'm not presenting a unified theory of everything. I know for a fact that my world is different from your world.

Case in point: everything I write is from the perspective of small teams of skilled practitioners. Yes, I know that there are people who try to build big pieces of software with one hundred and eighty programmers. I'm not one of them. Furthermore, I have absolutely no interest in trying to find techniques or tools that "scale" to hundreds of programmers.

I assume you have a modicum of intelligence and an open mind. I assume you will temper my blacks and whites, you will think them over, and you will figure out whether or how they apply to you.

I did some programming in Scheme a long time ago. Then I learned Java. One day I was responsible for writing approximately four hundred classes in a Struts application. I wrote a code generator that wrote the classes for me. The code generator actually worked off of an Excel spreadsheet that some business analyst used for writing business rules.

Java and Scheme are different worlds. But if you squint a little, and tilt your head just so, you might see some common ground.

Bippety Bop, Bang!

When I write something and say "this is a guideline, use it in these situations but eschew it in these other situations, and temper it with this technique," everyone reads it, nods, and forgets about it. No impact. No passion.

When I say "this is bad, and that is good," I get fan mail. I get hate mail. I get flowers at work. I get dog poop in a burning paper bag on my door step. (Well, not enough fan mail, no flowers, and hardly any hate mail. I'm not that good at writing.) But I get one more thing that I can't ignore: the posts incite debate. People argue for and against. With themselves. With me. With each other.

Because "considered harmful" essays are, by their nature, so incendiary, they are counter-productive both in terms of encouraging open and intelligent debate, and in gathering support for the view they promote. In other words, "considered harmful" essays cause more harm than they do good...

And to me, that's the point. If you are motivated enough to say, "Reg, your idea is no damned good because of A, B, and C," I think we're making some progress. We're talking. We're thinking for ourselves. We don't have some corporation with a billion dollar marketing budget telling us how to program and what's important in our lives.

Do you really think there are no mutable local variables in my code? Come on. But... don't you think I sincerely try to use alternatives whenever I can do so without crippling the code? Yes, I use alternatives in most cases.

University Politics

Sometimes the debates over these things get heated. This is a problem. I think it happens because people have difficulty understanding and articulating their own values. So I say something about static and dynamic typing, and I get people saying I'm stupid or a poor poor programmer (both possibly true).

What's going on? I think that people have some personal investment in these things. We're a little tribal. We think that because we have stars on our bellies, we're better than those without stars. Then someone says "stars are no dammed good."


What happens? We get hot and bothered, because we think that person was really saying "you and your entire value system are no dammed good." But we can't say that, because we're pretending this is about productivity, and methodology, and scaling to two hundred programmers.

That's a shame, because getting in touch with what's really important to us and shaping our lives around it is so incredibly important. I really wish more people could be like this fellow a few desks away from me who programs in .NET because that's where he thinks the money is.

Flames

I had a manager once. He was my boss's boss. When someone presented him with an idea, he would ask for proof it would work. If you wanted to try anything new, he demanded that it be risk-free. You could spend months getting the kind of data he demanded to prove that you couldn't fail.

Now, I don't think he was a malicious man. I doubt he even knew that he was the organization's number one idea shooter-downer. Too bad.

Anyways, a lot of people do this. If they see something new, they seize upon its flaws at once. They point out the missing bits. "It's just superstition, there's no proof that Ruby is more productive or that smaller programs are less buggy, or that static typing eliminates critical bugs, or, ..."

Well maybe so, but do you really think the people complaining about Cargo Cult Programming are rigorous scientists when they look at their own practices? No, they have a blind spot. The finicky rejection of ideas only applies to other ideas, new ideas.

And especially ideas that question some of the ideas that they already have.

Labels: passion

  ¶ 11:27 PM

  Surf's Up!

Every decade or so we have a major revolution in the way we develop software and the environments we develop for. But, unlike the object and web revolutions, we can see the concurrency revolution coming.

Awesome.

I've watched and participated on the periphery of several "waves" of computing progress. Yes, I wrote my very first SNOBOL programs on punch cards. Yes, I wrote a networked database application in MP/M. Yes, I wrote commercial software for Macintosh and marvelled at its Toolbox. And to learn Java, I used it to write a Scheme interpreter. (And yes, it did optimize tail calls and support continuations).

So I've lived on and in this ocean we call the computing industry, or computer science, or programming, for a while. I've seen it roil, I've seen it's placid moments. And I've seen its thundering, express train waves.

And I can tell you, it feels like another big wave is coming. A monster wave. A wave of tow-in surfing epic proportions. It's damn exciting. It feels like if we paddle like hell and try to stand on our boards we could be sucked under and crushed.

Or we could have the ride of our life, teeth bared in the most insanely happy grin we've ever worn.

I know that after the wave hits the shore, it will obliterate half of the vending stalls and the beach blankets and the hordes of sun tanners. And when the water subsides the industry will rebuild the beach houses and the industry will reopen the vending stalls, and the people who are rich today will probably still be rich then.

It's tempting to just wait, wait for the wave to hit and for the water to suck back into the ocean. Wait for others to solve the problems, to harness the energy. Wait until Microsoft and Sun and Apple and Linux and Apache and whoever else decide what we should do. Wait until they give us the frameworks, until they tell us where to swim.

Wait until there's no risk, like many people did with the Web and with Java and with their whole lives.

We can wait, just hunker on the shore and not take any chances out there in the water on our own.

But where's the fun in that?

Labels: passion

  ¶ 1:58 PM

  The Revolution will not be on YouTube

You will not be able to stay home, hacker.
You will not be able to plug in, turn on and cop out.
You will not be able to lose yourself on MySpace and skip,
Skip out for espresso during reloads,
Because the revolution will not be on YouTube.

The revolution will not be brought to you by the
Microsoft Research Group and will not star James
Gosling and Guy Steele or Homer and Marge.
The revolution will not give your startup sex appeal.
The revolution will not get rid of the bugs.
The revolution will not make your code look
better, because the revolution will not be on YouTube, Brother.

There will be no pictures of you and The Woz
dumpster diving for access codes behind a featureless building,
or trying to carry those motherboards on a stolen SegWay.
Fox News will not be able predict the winner at 8:32
or report from 29 districts.
The revolution will not be on YouTube.

There will be no pictures of pigs arresting
hackers in the instant replay.
There will be no pictures of pigs arresting
hackers in the instant replay.
There will be no pictures of ESR being
run off the Firing Range on a rail with a brand new process.
There will be no slow motion or still life of Richard
Stallman strolling through Redmond in a Free
Software tee shirt that he had been saving
For just the proper occasion.

C++, J2EE, and PL
SQL will no longer be so damned relevant, and
women will not care if Gil finally gets down with
Catherine on Crime Scene Investigation because hackers
will be in the Net looking for a brighter day.
The revolution will not be on YouTube.

There will be no highlights on the Yahoo
news and no pictures of Afghanistan
liberationists and Angelina Jolie blowing her nose.
The interfaces will not be designed by Ray Ozzie,
Bill Gates, nor written by Charles Simonyi,
Sergey Brin, Larry Page, Peter Norvig, or Kleiner Perkins
The revolution will not be on YouTube.

The revolution will not appear on ad banners
about enterprise software, enterprise integration, or enterprise sales.
You will not have to worry about a Paperclip in your
window, a Duke in your server, or the Prophet in your database.
The revolution will not go better with FUD.
The revolution will not dumb down the code that may confuse a monkey.
The revolution will put you in the driver’s seat.

The revolution will not be on YouTube, will not be on YouTube,
will not be on YouTube, will not be on YouTube.
The revolution will be no MPEG, hackers;
The revolution will be live.

Labels: passion

  ¶ 10:35 AM

  Just in time for "labour" day

I strongly oppose the false dichotomy of either you program for fun or you program for business. That split is not only artificial, but seems more like a self-serving fairy tale you can tell yourself to why your current environment is making you unhappy. Well, I'm doing Serious Business Stuff, so it has to suck. That's just the way the world spins. Bullocks.

Labels: passion

  ¶ 10:38 PM

  The Difference Between Values and Strategies Put simply, a value is something you are willing to pay for. A strategy is something you hope will pay off for you.

A LISP programmer knows the value of everything, but the cost of nothing.

People mix these things up all the time, especially in business. Have you ever been to a corporate retreat where a select group of the annointed gathers to write a company's "Mission Statement"? Everything is couched in terms of values: "We shall strive for the highest standard of customer service..." "Employees are our greatest asset..." and so forth.



It's a lot of wind. But let's assume that everyone is sincere. Are these things values? You can tell by asking the CEO why you need a mission statement. Here is an answer you might receive:

Companies with Service-Oriented Mission Statements (SOMS) have a P/E ratio 14% higher than their peers as ranked by Booz, Alchie & Weide. Furthermore, customer service has proven to be a competitive advantage: a recent survey showed that 63% of our new accounts selected us because of our reputation for service.

That's a strategy. The CEO believes that she will make more money if they say these things. She may also believe that they will make more money if they actually try to do these things. But either way, it's about the money.

Salient question: if one day they woke up and discovered that they could make even more money by outsourcing their customer service to the third world and driving their customers crazy with frustration, would the 'de-hiring' be done by middle managers or by Bob and Bob?

I'm not saying this is right or wrong, mind you. I'm just saying that if you'd drop service to pursue money, money is your value and service is your strategy for making money. (It might help to think of values as ends and strategies as means).

Something isn't a value unless you... value it. That means you are willing to pay for it. You are willing to give something up to enjoy it. For example, if you absolutely flat-out refuse to work for tobacco companies and you make 20% less money working elsewhere, you have a value.

Or if you insist on top-notch customer service even though it means you have to price your product higher to pay for it, which means you have less market share, and you end up being a niche company in your market instead of vying to be come the next Wal-mart, customer service is actually one of your values.

Now quite frankly, I don't give a hoot about public companies, not even Apple Computer. So let's talk about building great software. What about this quality software thing?

Obviously, quality software can be a value as well as a strategy. We can pretty much take it as given that you can make more money selling oats that have already been through the horse than quality feed. So when you decide that you will make a personal commitment to write the very best software possible, you are acting on your values.

People often have trouble understanding each other when one of them is thinking about their values while the other is thinking about their strategies. Just look at all of the crap people have written about why Apple should license or should have licensed the Mac OS to clone makers.

Could it simply be a values issue? Maybe Apple values the whole user experience and doesn't care if licensing is a decent strategy for gaining market share. (I said if. Allegations that large herds of me-too clone buyers would prefer OS X to stolen copies of Windows are, as they say, unproven in court.)

And so it goes for personal productivity and expression, otherwise known to me as developing software. There are certain aspects of development that incite fractious debate: the people, the processes, and the tools (especially the languages and frameworks).

Honestly,we can have a debate, but let's make sure we're talking about the same thing: are you talking about the best strategies for achieving a certain value? Or are the people, process, and tools part of your values?

It may not work for you, but some people value working on certain kinds of teams. Some people value working in a certain kind of way, be it Agile or BDUF. It's part of their nature. And most especially, some people value expressing their ideas in a certain style, using certain idioms that closely match the way they think of the solution. It's a very deep value.

Constructive Debating

The crazy thing is, a lot of people are resistant to coming clean and saying "Hey, it's important to me personally to do it that way." Perhaps they fear being labelled some sort of difficult prima donna. Yet elsewhere in business, we value people that care about their work enough to set high standards for themselves.

But whether they confess or not, a lot of people have values in this area and choose to couch them in strategic terms. The result is a lot of confusion. The next time you see a flame war going on over REST-ful web service architecture, meta-programming, or whether/how to hire great programmers, step back and try to deduce which participants are defending their values and which are defending their strategies.

You know what? You can go a long, long way towards mutual respect and understanding if you come clean about your own values. Are you constantly defending Java against the Dynamic Typing Visigoths? Try this. Simply say (or write):

You know, this thing I'm using is good enough for my purposes. I enjoy using Eclipse. I like EJB 2.0/Hibernate/Spring. Working with the de-facto standard for Enterprise is one of my values. What you're saying is interesting, and I don't doubt it works for you, but nothing I've heard suggests switching to Rails would be aligned with my values. So it isn't for me.

That would shut me up, right quick. Because I value people actualizing their values.

Practices are values, too

What about practices? For example, are Agile practices values or a strategies?

People get into all sorts of debates as to whether pair programming or project spaces, for example, are more productive than giving programmers private offices. I'm not coming down on either side of the debate here, but I will point something out: there's an implicit assumption that pair programming is a strategy. And that productivity is the value.

That seems obvious. Isn't productivity the goal? Why value anything else?

Well, isn't it just possible that some software developers like the camaraderie of a project space? That some software developers enjoy real-time, high-bandwidth communciation with their customers?

I'll get personal. I get a thrill out of shipping software. Incremental development is one of my values. Even if you could prove it is slower than BDUF, I wouldn't switch. I love the feeling of delivering functionality over and over and over again.

For some people, Agile is a value. Luckily, there's a strong case that it's a good strategy. So a developer that values Agile can work with a customer that chooses an Agile strategy. Click.

Sometimes a value is also a good strategy.

Integrating Values and Strategies

Unless you are a philanthropist, you cannot pay for your values over and over and over again. You have to find strategies that pay for your values. Life is very, very good when you find a way to turn a value into a strategy.

For example. I like working on interesting problems. It's a value of mine. But a career (or a company) can only be built out of solving profitable problems. If I want to integrate my values with my strategies, I have to find interesting problems that are also profitable to solve. Obvious, no?

But before I close on this happy thought, a reminder: even though a value might be a good strategy today, times change and one day the value may no longer be a good strategy in its own right. That's when the real test of commitment will arise: will you have the courage to seek new strategies that still preserve your values, or will both value and strategy be cast aside? It's up to you, and quite honestly nobody knows how they will react until faced with the posibility of making a real sacrifice to preserve their values.

This has been said so many times, in so many different forms, it must be very, very true and simultaneously very, very elusive:

To be successful and happy, seek out opportunities where the strategies for success are tightly aligned with the things you personally value.

Labels: passion

  ¶ 1:20 PM

  Aptitude? We don't need no stinkin' aptitude! Here's an academic paper: The Camel has Two Humps. The authors analyzed programming skills and have come up with an aptitude test that correlates with the success or failure of college students in programming course.



I found the paper's central idea interesting: That the three skills of understanding state change over time (they call it sequence and assignment), recursion, and concurrency are three of the central skills of programming and are ranked in that order of scarcity.

(Revelation: instead of thinking about something as being hard, consider it scarce instead. It's a significant change or perspective).

I expect immediate refutations from people who resist the idea that programming is hard or that programming skill is scarce:

• This only applies to filtering the wheat from the chaff: once students make it through their first year or so of instruction they are good enough to program anything, provided they apply themselves.
• The skills measured do not apply in the real world (wherever that may be), where understanding state change, recursion and concurrency are irrelevant to constructing software: frameworks and IDEs solve all of our problems and all you need to do is wire the pieces together.

What does this mean for you?

Even if you agree with the dissenters, read the paper with an open mind. One benefit you (I am speaking directly to you, not making a grand rhetorical gesture) can take from this is to organize your own self-development along these lines.

Consider athletics. You read a paper saying that future aptitude for sports can be measured at an early age on the basis of running speed, jumping ability, and weight lifting strength. We could drink beer and have a huge argument about their applicability to a specific sport. Maybe fencers would say that heavy bench presses are useless predictors.

But for all round athletes, those three areas are a great foundation for training. Our experience with athletics is that athletes who mix sport-specific skills training with work on speed, leaping, and strength out-perform athletes who just train skills.

So consider this: go out of your way to practice your skills the next time you are writing some code. Use iterators in places where you'd normally use a for loop. Use recursion in places where you'd normally use an iterator. And find a way to decompose operations into parallelizable components.

...

...

...

...

why are you still reading this blog post? hup! hup! get down and give me one recursive routine, then follow the instructions in this paragraph! and I mean all of you, do it at the same time!

Labels: passion

  ¶ 9:08 AM

  It's Friday! Who's having fun? One of the things rock climbers like to say is "the best climber is the one having the most fun."

(Don't worry, this post is not going to go on for pages trying to explain climbing culture and then trying to draw some sort of tenuous parallel to software development.)


Here's one response to the How would you design a program to referee Monopoly? question:

What I love most about the Monopoly question is how it sucks me in. Maybe it's because I'm a gamer at heart, but my mind immediately starts racing through all the different possibilities. It's a little embarrassing to admit, but I'd love nothing more than to sit in a room with another programmer and hash this problem out.

Because it's fun.

Now, I'm not going to tell you that I have some sort of crystal ball for telling you that the guy having the most fun with Monopoly has the most fun with programming as a whole. But you know what? I think it's a significant indicator. (More on why I am probably wrong below.)

One of the things I look for when recruiting is enthusiasm. This is no secret. Pick up any random book about success (or "life hacking" as people call it these days). That word pops up. Read any random biography about somone you admire. That word pops up.

I can't guarantee that a programmer who thinks programming is fun does more and does it better than a dour-faced individual who come in and just works with a Puritan "work == suffering" mentality. But look around. Don't you think employers are looking for enthusiasm?

Now does that mean Monopoly is a great question? No, of course not. Enthusiasm can and does come through when a candidate loves to talk about a design. But enthusiasm can also come through when a candidate talks about a problem they've solved in a previous position or talks about where they want to be in five years, or does almost anything else in an interview... when the candidate is enthusiastic about their chosen profession.

Now what about this quote:

If you're interested enough to ask me in on the basis of my resume, then let's have an interview, not play stupid games or do tests.

Obviously the second person just plain doesn't enjoy this kind of problem. It's not like she said "hey, that sounds like a lot of fun, but what does it have to do with my job?" But that doesn't mean she doesn't have enthusiasm for her work. Maybe the she hates solving design problems and hates writing code in an interview, but ask her about what she accomplished in her last position and her eyes will light up. I don't know.

Interviewers are pattern matching machines. Interviewers have in their head two lists: traits they associate with success and traits they associate with failure. Ideally (IMO), these lists are compiled from personal experience.

If an interviewer has met a number of people who consider solving design problems interesting and who have been very successful designing software, the interviewer will tend to select candidates that enjoy solving design problems in interviews. This interviewer likes Jeff.

Another interviewer may have the exact opposite opinion. They may have known individuals that go into raptures over the architectural options of a design but failed to ship quality software on time. Such an interviewer would weed such candidates out. This interviewer likes the second person.

There is little consistency between interviewers, even within one company. And I'm sorry to say so, but sadly it's true: the subjective things like what you think is important about software development mean so much more than the objective things like how many years of experience doing X a candidate may claim to have.

One great danger is interviewers who (rightly or wrongly) consider themselves successful. They are their own shining example of success. How can such an interviewer avoid selecting people who are just like them? They believe they are a walking example of the kind of person that will be successful.

It's very difficult to shake this, especially when an interviewer has some length and breadth of experience. It is hard to say "I'm not young enough to know everything" and to look at a candidate's potential with a very open mind. Even when their track record and their attitude towards your suggested line of interviewing are not what you prefer.

Given this mess, are we in trouble in our chosen profession? I don't think so. I'll tell you why I'm personally optimistic: my "circle of concern' is well-aligned with my "circle of influence." Or to put it in a triter way, you only need one.

When looking for a job, you only need the one perfect job. It's irrelevant whether you get job offers from half of the places you apply, three quarters, or just one. What you need are the following:

1. The perfect place for you is hiring;
2. The perfect place for you interviews you;
3. The perfect place for you makes an offer.

You only need that one interview and that one offer. All other interviews and offers are actually waste by-products of your search for the perfect job.

So when going to interviews, you can afford to be choosy. By all means, if you hate stupid games, walk on out. That company isn't your kind of place. It will be full of people who are very different than you. They all got their jobs because they think that solving that kind of problem is fun. And if you like programming but you have to ask a hiring manager if they want to see you juggle, don't worry about getting an offer from that place.

Do you really think you'll have fun in a company where you don't have fun in the interview?

Here's the thing I've seen in my career: some companies do a better job of interviewing than others. But whether they interview well or poorly, the company's character and style are always front and centre on display.

If you walk out of an interview feeling enthusiastic about working in that place, you have a much better chance of enjoying yourself at work there than a place where you walked out feeling ambivalent.

After all, if you want to be the best programmer...

Wait for it...

You have to be the one having the most fun.

Labels: passion

  ¶ 9:02 AM

  Let's be "Real Mac Users"

Hence the difficult situation faced by small-minded Windows users who do not get the appeal of the Mac; to admit that Mac users'’ strong preferences are reasonable would be to admit that they (the Windows users) are unable to perceive something that Mac users can.

That to concede that Mac users are reasonable wouldn'’t just imply that Mac OS X is in at least certain ways much better than Windows, but that Mac users in certain ways have a more refined sense of taste than Windows users, which in turn cuts way too close to implying that in certain ways Mac users are smarter, which is where things turn ugly because those certain ways are, to Mac users, the ways that really matter, and any chance at reasonable discourse evaporates because both sides feel deeply insulted by the other.

—--John Gruber, "And Oranges"

As goes the Mac vs. Windows debate, so goes the One True Programming Language debate.

I'm deeply suspicious of Mac users who claim to be perfectly happy with Mac OS X. Real Mac users, to me, are people with much higher standards, impossibly high standards, and who use Macs not because they'’re great, but because they suck less than everything else.

--—John Gruber, "And Oranges"

And when we hear of new languages, new tools, or even improvements to approaches we have discarded in the past, let's approach them with an open mind. We may find that they contain little of interest, but to an open mind, anything at all of interest is an opportunity to learn and improve.

We are far better off finding one good idea in a new tool than blogging about 999 ways it is inferior to what we already have. It takes us closer to achieving our goals.

That's what matters.

Labels: passion

  ¶ 4:32 PM

  How to make programming hard for yourself I posed a simple question: why do we have trouble admitting that programming is hard?

This generated a tempest in a teapot of comments. Most focused on the red herring of whether programming is an innate talent. That's actually irrelevant to the fundamental question.

Is programming actually hard?

The evidence is in that software development as an engineering practice is very hard. One person on reddit commented that he had worked for three successive game companies that shipped working software on time. Other than that comment, my own experience, the experience of everyone I've ever talked to, and the industry folklore is that software projects are almost always late, short of promised functionality, buggy, or all three.



Now I admit that lots of people think that this is a management problem. Their view is that the actual programming is quite tractable, it's just that getting people to work on the right things at the right time is the hard part of software development. This argument takes the same world view that the Egyptian engineers had about building pyramids: the labour is so easy that ignorant slaves can do it.

My view is that this is true for a certain class of programs. Most of these can be found in current business contexts. Quite honestly, most business applications are what I would call wide and shallow. There's a lot of work to be done, but little of it requires deep thinking. The challenges are around managing requirements and user expectations. These are hard problems, but they are not in the same class as, say, figuring out how to implement a fast join on two distributed hash tables.

I think that easy programming problems drive a lot of decisions about people and tools. If you perceive that a problem isn't very hard, why hire the 'cream' of the programming crop to solve it? And if you eschew the cream for a plain glass of 1% milk, won't you insist that the program be written in Blub instead of Python, Ruby, or Lisp?

This becomes a feedback loop. The 'B' players become hiring managers one day and they hire 'C' players. The 'C' players are asked to design applications, and they design things that they know how to build. This second effect is a huge lever pushing programming problems downwards in business.

Programs are written to solve problems posed by people, be they programmers, program managers, business analysts, or what-have-you. The difficulty of writing the programs is driven by the difficulty of the problems. The difficulty of the problems is limited by the imagination of the people posing the problems.

If your feedback loop is pushing difficulty downwards, the only problems you will have are the ones that can be solved by anyone who can pick up a Learn Blub in Twenty-One Days book.

Is this a problem?

Quite honestly, no. If what you want is a steady paycheque, and you can sleep at night knowing that your job might be outsourced one day, there's nothing wrong with chasing the easy work. It's no different than playing saxophone in a dance band at a wedding. Charlie Parker did it, you can do it.

But just because almost everybody does easy work doesn't mean there isn't hard work to be done. Hard problems are their own reward for some people, myself included (in my case I mean "hard for me," I know more than a few people that consider my hard problems to be recreational diversions). And some hard problems have lucrative consequences if you have a taste for starting companies.

Why it's hard to find hard problems

It's hard to find problems that are so challenging that you don't know how to solve them. It's especially hard if you also want a problem that can actually be solved. And it's very, very hard to find a hard problem that makes business sense, even in a startup. Most good startups solved the right problem rather than the hardest problem.

Very few people have the luxury of writing requirements for a program that they have no idea how to write, much less the imagination to see the results. Jump in your time machine and zoom back to 1980. Start asking people do sketch a word processor design on a napkin. How many people specify bit-mapped graphics and mouse-driven selection?

I compared programming to music and athletics. I suggested that programming is hard and the reason we don't see that is because we aren't competitive enough. That may be true, but now I sense a deeper challenge: in programming, we tend to pose questions for which we already know the answer.

In music, most people's reach far exceeds their grasp. It's quite easy to find pieces of music that are difficult or impossible for someone to play but nevertheless pleasing to that same person's ear.

If you are a wanna-be bassist and you listen to Jaco Pastorius' harmonics on "Portrait of Tracy," it is immediately obvious to you that you could be better and that being better is worthwhile. I leave it to your imagination to draw the same parallel to sports and athletics.

But in programming... we have a nasty habit of being very satisfied with the tools we already know. After all, they are good enough for all of the programs we have ever written. And, surprise surprise, they just might be good enough for all of the programs we will ever want to write.

How to make life hard for yourself

Most of us need outside forces to shake us up. Many bassists were shocked in 1976 when they heard Portrait of Tracy. Harmonics were used as accents here and there, but nobody dared to play an entire piece with harmonics as the main technique. This shook things up, and now playing harmonics is considered a basic skill on electric bass (although it's still fiendishly hard to play Portrait of Tracy!)

For all of my professed admiration of Ruby on rails, I personally don't think that easier and more productive CRUD application writing will shake things up. I personally care very much about writing applications in a tenth of the time, but using Rails is like listening to Jaco Pastorius. The real learning experience comes when you try to duplicate the feat.

If you want Rails to help you improve as a programmer, you need to look at the source code and figure out how to use Ruby to write Domain Specific Languages. Next, you'll have to write a few DSLs to become immersed in how programming changes when you can write DSLs. Now you're ready to look for a problem that is barely possible if you can solve it with a DSL.

Before you knew how to write your own languages, you probably couldn't design a program that couldn't be written without a DSL. Now you can.

This general template works surprisingly well: Learning things that force you to solve existing problems in new ways will help you find hard problems that can only be solved with the aid of your new techniques.

So when you're done learning how to write DSLs just like Rails, go ahead and write a web application using continuations as the basic metaphor. And don't forget to look under the hood and figure out how the framework makes continuations scale in a web context.

Come to think of it, figure out how to write continuation-based Rails applications that live on a Google-scaled grid and you might be headed towards a place where the inhabitants have no fear of hard.

I'll see you there.

Update: I found an early take of Jaco playing Portrait of Tracy (MP3). Nice.

Labels: passion

  ¶ 2:36 PM

  Why do we resist the idea that programming might be hard?

Don’t blame me for the fact that competent programming, as I view it as an intellectual possibility, will be too difficult for ‘the average programmer’, you must not fall into the trap of rejecting a surgical technique because it is beyond the capabilities of the barber in his shop around the corner.

Programming something even half decent requires considerable intellectual ability. That’s not all there is to it, but if you’re not smart, it will be much harder for you to be a decent programmer. We’re mostly in denial about this and would like to believe that programming can be done by anyone or can be made somehow unnecessary. It’s the opposite view we take to managerial or executive ability, where we pay through the node for talent. Yet it’s been known since we started building software that some programmers are much better than others. Maybe all disciplines are like this, but it is very obvious in programming. Naturally this intelligence makes people who don't have it uncomfortable, particularly if that person is supposed to lead or organize very smart people to do something that they don't fully understand, even though that person is probably better paid, more socially equipped and has nothing to worry about.

As with meat-markets this is not something we like to talk about a lot. It’s not egalitarian and intelligence is something we’re sensitive about culturally. But consider that it’s not controversial to say being a good musician or athlete requires a level of ability, some of which is innate. Or that our educational systems are based around being ranked smarter than the next person anyway. So we should get over ourselves a bit and accept the fact the good software requires more than stringent process; it requires well-above average intelligence to create it.

The statements that make people mad are the ones they worry might be believed. I suspect the statements that make people maddest are those they worry might be true.

Addendum:

I have recieved a few emails and comments suggesting that anyone can become a good programmer with sweat and dedication. Here're my personal feelings:

I don't believe that anyone can become a good programmer with dedication, practice, and experience. In case you haven't guessed from the third quote above, my interest is not in whether there is an innate talent involved with programming, but in people's state of denial.

Let's look at what it means to improve and to be good.

I suppose there's some room for argument about whether almost any infant or toddler, if exposed to the right stimuli, can become a good programmer. Our knowledge of genetics and early brain development is primitive.

However, by the time people reach their teen years, I believe that much of their potential has been established.

With dedication and practice, people can improve almost anything, however in most fields this improvement shows diminishing returns. It may be aymptotic, so in theory they may eventually become a good programmer or a chess master, but in practice this may take longer than the number of productive years available.

With respect to everyone who has emailed to say that anyone can be a great programmer, how many of you play Bridge or Chess? These are fields where no serious practitioner believes that round the clock study and fanatic devotion will take you to the top: winning requires both potential and the will to fulfill that potential.

Likewise, how many of you who write or email play a musical instrument and especially compose music? Many people have fanatic devotion to practice, yet every generation yields remarkably few geniuses that will stand the test of time.

Why do people consider fields like Bridge, Chess, Athletics, and Music different from programming?

I think the salient distinction is that we habitually consider games like Bridge, Chess and Athletics very competetive. And if you're paying close attention, we consider Classical Music and Jazz equally competitive, if not more so (where do you think we got the word "Diva"?).

When something is competitive, it's irrelevant how good you can become through practice. If someone else can become even better with the same amount of practice, you will be unable to surpass them.

Of course, you can still surpass those that don't study, practice, or learn from their mistakes. That's important. But if the field is desirable enough that there are a significant number of innately talented people willing to study, practice, and learn, then only those with innate talent will occupy the top spots.

I believe that if programming jobs were more demanding and that if the industry were more competitive, we would rapidly lose all pretensions to egalitarianism.

But what do I know?

If you found this interesting, you may want to read How to make programming hard for yourself.

Labels: passion, popular

  ¶ 9:45 AM

  Torcamp I went to TorCamp yesterday. Wow. First off, a message to David Crow: thanks!

It was good to hang out with people who are actually doing things. Everyone I met was running their business, advocating change in their organization, or contributing code to open source. Or whatever. The point is, everyone was in gear and moving.

(Tip: if you ever go to a networking/user group where everyone talks about such and such but they don't actually use it in their own projects or at their own workplace, you're among tourists, not users.)

I'd post a run-down of the event, but Kate did a fantastic job. Click on over there. She's awesome.


I gave a presentation on "Web 2.0 beyond ajax." I posted the presentation here. I tried a very different style of presentation, and I give myself mixed reviews. At best.

Sutha Kamal had the best suggestion for improving the presentation: use Peter Norvig's example of statistical translation rather than search to stress the reason that a broad corpus is better than deep ontology for most "intelligent" applications.

(The most interesting flaw is the screen capture of del.icio.us I took on Friday night. For some reason I didn't get the "recommended tags" feature and I pointed this out. People in the audience were all mystified and quite sure that you should get popular tags at the bottom of the page. That's true, but you should get "recommended tags" just below the save button, and I didn't get either. I'm quite mystified as to where they were when I took the screen shot.)

Labels: passion

  ¶ 2:08 PM

  A quick word about prima donna hackers Is every prima donna really one hundred times as productive as the middle of the Gaussian distribution who have a degree? Maybe, maybe not. But this post is about those prima donnas who really are from another planet, and the people who hire and manage them.

This is not a team. It's not a boat, not a machine that has a lot of parts of it that have to work together. The metaphors are all crap. This is a business - that's all it is.

Vogler in "House," Season One Episode Fifteen, "Mob Rules"

Read the sports section of the newspaper. Or do a Google News search for Terrell Owens. He's a superstar, one of the truly gifted who can change a football team dramatically. He's also incredibly high maintenance. And he's just been fired.

The bottom line? His team is losing. Here's the way it really works. Let's make things simple and say that Terrell is worth four wins to the team. If the team can make the playoffs without him, four extra wins makes the difference between making the playoffs and making it to the Superbowl.

When a team is losing, the same four wins are only the difference between just missing the playoffs and being out of it a month before the season ends. Although the difference in number of games won might be the same, the "utility" is far lower.

Now consider the cost. When the team is doing well, most of the players, fans, and management are basically happy. The distraction of a loose cannon is just that. A distraction. But when the team is losing and everyone is unhappy, a loose cannon is more like a bouncing hand grenade. Everyone is sore.

And even if a coach or manager is detached enough not to take a difference of opinion conveyed through the media personally, it's hard to turn a team's culture around and make changes when your authority is so visibly undermined. Basically, a prima donna provides less value to a losing team and costs more.

And yes, I'm going to say that the same thing applies to software development. If you're rocking and rolling in a growth environment where you're shooting the lights out, prima donnas provide value. They can make the difference between "just shipping" and hitting a liquidity event. They provide more value and cost less.

But if things aren't going well... a prima donna is going to be a distraction at a time when others have very little tolerance for a contrarian viewpoint.

And that's why you often see prima donnas flourishing in start ups but exiting when the company goes enterprise and development is no longer the value add.

Labels: jobs, passion

  ¶ 4:26 PM

  I'm not young enough to know everything Paul Graham has alluded to the difficulty of doing a startup when you're, well, not young. He puts the upper bound at thirty-eight:

So who should start a startup? Someone who is a good hacker, between about 23 and 38... I don't think many people have the physical stamina much past that age. I used to work till 2:00 or 3:00 AM every night, seven days a week. I don't know if I could do that now. Also, startups are a big risk financially. If you try something that blows up and leaves you broke at 26, big deal; a lot of 26 year olds are broke. By 38 you can't take so many risks-- especially if you have kids.

Paul Graham, How to Start a Startup

While what he says is true, lately I've become aware of an even bigger threat to starting a startup. Experience can be a handicap.

Let's start with a digression. What makes weblog posts popular? Most people say things like "insightful" posts become popular. Or "well-written" posts become popular. Adjectives like that are tautological: if someone likes a piece of writing, they generally will find nice adjectives to apply to it.

One model for popular writing is that it panders to the reader's prejudices. Plain and simple. People like writing that validates them and especially their ideas. I'm no different, and as a result I tend to focus my research on things that I already think I know.

When you're twenty-two this isn't much of a problem because you know you don't know. You're "consciously incompetent." So you're far more likely to find something unfamiliar and try to understand it, to change your way of thinking to match what you learn rather than applying a "bozo filter" to it in advance.

But at forty-two (or three!), it's easy to think you know things. You're at incredible risk of thinking you know things when you've achieved some measure of success, no matter how modest. You become "unconsciously incompetent." You don't know, but you don't know you don't know.

I was personally reminded of this at startup school. As Chris Sacca pointed out:

The glow of screens (from a refreshingly Powerbook-dominated audience) revealed an array of real-time collaborative note-taking for a virtually assembling the room's minds in a concurrent recording and discussion of the event.

Chris Sacca, Startup School - An Inspiring Room Full of Hackers

Sitting in that room, I was hyper-aware that I was no longer in Kansas. It struck me that if I didn't want the next generation of hackers to wipe me out like a Tsunami, I needed to stop paddling, climb on my surfboard, and accept the risk of being dashed on the rocks.

I immediately made a commitment to myself to let go of things I used to think I knew.

Matz, Jonathan Ives, and A Narrow Road to a Far Province

One commitment was to try Ruby on Rails instead of Lisp. Every time I've looked at Ruby, I've thought "nice, but it doesn't do anything I couldn't do in SmallTalk back in 1980."

I've been the most pompous hypocrite. I mean, I often poke my Windows apologist friends and tell them that efficiency and user interface is not measured with check boxes ("mouse? check. icons? check. well, they must do the same thing.") I've extolled the virtues of design, of the interaction between the parts, of the frictionless user experience of a Macintosh.

Do not seek to follow in the footsteps of the wise. Seek what they sought.

Matsuo Basho

Ok, Ruby's blocks and classes and closures are the same things that Lisp has given us since 1959 and SmallTalk has given us since 1980. But maybe... Maybe... Maybe Ruby on Rails is easier to use than Lisp in the very same sense that OS X is easier to use than Windows.

As for the Rails thing, it's awfully popular and I have been uneasy about anything so hype-ridden. But how is that different from any of a thousand funny quotes deriding the new new thing? I'm tempted to say that there's a world market for maybe five Rails applications. But maybe there's more to it.

Maybe I should find out.

Thomas Bayes, Joel Spolsky, and Richard Feynman

Joel Spolsky dropped an incredibly provocative anecdote into one of his well-written and insightful posts:

A very senior Microsoft developer who moved to Google told me that Google works and thinks at a higher level of abstraction than Microsoft. "Google uses Bayesian filtering the way Microsoft uses the if statement," he said. That's true. Google also uses full-text-search-of-the-entire-Internet the way Microsoft uses little tables that list what error IDs correspond to which help text. Look at how Google does spell checking: it's not based on dictionaries; it's based on word usage statistics of the entire Internet, which is why Google knows how to correct my name, misspelled, and Microsoft Word doesn't.

If Microsoft doesn't shed this habit of "thinking in if statements" they're only going to fall further behind.

Joel Spolsky

I have an entire career built on top of experience building applications that manage ontologies, that are built out of objects and classes and tables and relations and all sorts of things that boil down to if statements. Or at least, they boil down to classifying things in advance rather than building systems that learn over time.

I've known about Bayesian classification for years. And I've always thought of it as a specialized tool. It's incredibly disruptive to think of it as an every-day tool, as a general-purpose tool, as something that can replace the if statement.

"Your old-fashioned ideas are no damn good!"

Richard Feynman

Yet when I step out of my comfort zone, I realize that I've seen this before (experience can be handy at times). Many of you young-uns have never known a programming world where there was no polymorphism (although judging by some of the code that has caused me to say "WTF?", not as many as I would like). Before messages, virtual functions, and method calls there were the switches, cases, and if statements.

There was an "aha!" moment for me when I suddenly grokked polymorphism. When I understood that switch statements were junk. Maybe Bayesian inferences can change programming the same way that polymorphism changed programming.

I need to find out.

And now I'd like to quote the other Steve, the one who isn't a hacker and wasn't presenting at startup school (Psst! Steve! Do you want to sell coloured MP3 players for the rest of your life or do you want to change the world again?)

One more thing

For a very long time I've been carrying a conjecture around. Paul Graham supplied me with the framework for thinking about the conjecture:
The "Graham Question" is: Can we predict the future of a software development project with objective observation?

Let's take a simple example, one that ran through my head this morning. I've worked for several companies that used issue tracking systems. These systems have had a little widget/enumeration for declaring the priority of an issue. I've also worked with Scrum-like teams that maintained priority with a master list or backlog.

You want to know which issues will be fixed/implemented/done by a certain date. What is the significance of the priority?

Well, this is management 101. Start with the number of hours available for development, then take the highest priority issues and estimate how much time is required to do them. Your prediction is that the team will do as many as possible of the highest priority items in the time available.

All well and good, but in reality "Spolsky" isn't in the dictionary and neither is "Braithwaite." For that matter, neither is "p.s.", and why should I have to click "add to dictionary" after the program has watched me type this thing and not correct it hundreds of times?

And lo, if we watch an actual software project we see that over time priorities change and new issues are added to the mix and sometimes low priority items are done first for some reason, and humans just can't seem to follow the damn plan, but software emerges from the other end anyways.

It's easy to say, "your old-fashioned priority is no damn good." But maybe we are not young enough to know everything. Maybe we should ask a question: "what good is the priority if you can't construct a nice if statement around it?"

Maybe this is like Spolsky and Braithwaite and Error IDs and Help Text. Maybe there is no formula up front but we can watch what people do hundreds of times.

Maybe Thomas Bayes knows the significance of the priority.

Labels: passion, popular

  ¶ 1:08 PM

  Fully Engaged by Example First, a personal note. I'm feeling fully engaged. You know that feeling, when it seems like what you're working on occupies 100% of your attention, when you have to fight to think about anything else.

This weekend I went climbing in Kentucky. On Saturday night I was at a rocking party with some of my most attractive friends dancing up a storm. I was in a corner, writing down thoughts about how my project would help people communicate about software functionality using examples. And I love to dance!

Sometimes a dance is just a dance, but an idea is a dream coming to life. I didn't want to miss the moment. Do you ever feel like the entire Universe is conspiring to move you forward? It isn't really, but it feels that way because you're so positively attuned to your idea that you can find help from even random outside events.

And after a weekend working on examples, what do you suppose I spied on a church billboard driving home?

Okay, now about examples. Here's the pain point in development. How do we communicate about what we're going to build and why over the life of a project, not just in version 1.0?

Jonathan Edwards does a terrific job of explaining the problem:

Programmers in the real world will tell you that specs aren’t worth the paper they are written on. They are half-baked informal descriptions that are too abstract to be understood by the users, and too imprecise to be useful to the programmers. They are full of internal inconsistencies and factual errors, because there is no way to test them. They are obsolete the day they are published, because it is too difficult to rewrite them as the system becomes better understood. Things are no better with the other standard commodities of development: project plans, schedules, cost estimates, and documentation.

Jonathan Edwards, The Currency of Development

Jonathan is trying hard to build mechanical ways of getting teams to work together. Have a look at his ideas: he's an original thinker. I'm interested in a slightly different take on the same problem. As he puts it, "They are obsolete the day they are published, because it is too difficult to rewrite them as the system becomes better understood."

How do we make it easy to rewrite functional specifications? Some cultures try to solve the problem with command-and-control ("thou shalt not make a code change without an accompanying tech spec").

What I've observed is that all the information we want is out there in emails, PowerPoint decks, screen shots, and documents. And that's just the stuff produced by customers/analysts/product managers. What about the comments in the source code and the source code control system that describe how the changes close a specific bug or implementa specific feature?

Everything's fragmented and gets more and more unwieldy as projects move forward.

Using examples as the central communication idea is just part of the solution. We need to make it easy to write examples and read them. We need to track the history of changes. And we need to tie the examples closely to tasks and releases.

Labels: passion

  ¶ 1:43 PM

  Just do it Eric Sink has some terrific advice:

You could spend a lot of time trying to figure out if this idea is any good, and at the end, you wouldn't really know. Alternatively, you could spend the same amount of time implementing this idea, after which you will have the opportunity to really find out if it's any good.

Small Ideas

I could link this over to my own struggle to get my idea out of the garage and (at the very least) sitting in the driveway where my neighbors can think up polite ways to ask "WTF?"

But really, this principle is so simple we should look for it in far more places. Isn't this the foundation of iterative development? Can you make a wireframe or mockup or prototype faster than you can draw a pretty UML diagram?

What about selling agile practices? Which ones can you just do? Wouldn't it be easy to say "we've been trying X for four weeks and we've achieved Y and Z"? Isn't it far better to risk asking forgiveness than campaign for permision?

Why not just try pair programming? What's stopping you from just doing test-driven development on your own personal tasks? Is there any reason you can't use Cruise Control on your workstation even if the team doesn't buy into continuous integration? What's wrong with writing all of your tests in FITnesse and sharing the results?

I'll make you a deal. I'll stop wasting time writing this blog entry now if you will go ahead and think of one thing you could just go ahead and do.

You've got your idea? Ok, here's my 'end' of the bargain.

Labels: passion

  ¶ 10:34 AM

  Yester dreams This post is one of the hardest things I've ever sat down to write. If it seems pained, or stilted, or just plain embarrassing... That's because I'm "sitting down at a typewriter and opening a vein" (Red Smith said it so well).

Two years ago I began to work on starting a new business. Two years ago! At a time when businesses go from napkin drawing to users in months. What has held me back? Why aren't I shipping?

Well, the biggest mountain I've been climbing is the fact that business is very personal for me. I admit, I could use some detachment at times, some uninvested third-party perspective. But not too much! As I just read today:

All good businesses are personal. The best businesses are very personal.

Mark Cuban, "It's not just business, it's personal"

My first idea for this business was, I think, a reasonable business concept. I think there was a good chance of making money. It might even have been fundable, had we gone that route. But I held back. For quite a while I wondered if I was 'afraid of success.'

In the end, I decided that there was something bothering me about the business plan. There was something I didn't quite like, something... I'll be blunt. I thought there was something that sucked about that plan. Today, I can put my finger on what it was.

If you want to do something that's going to change the world, build software that people want to use instead of software that managers want to buy.

Jamie Zawinski, "Groupware Bad"

That plan was top-down. It was all about creating a tool managers would buy and dictate to development teams. This kind of businesses has made rather a lot of money for rather a lot of people. All you do is grab any old product and flog it to death.

But even though I still feel it's achievable, that plan just doesn't seem like fun to me. I had trouble admitting that to myself for the first year or so.

It's not that I don't like selling. I've spent a good portion of my career in sales, and I'm still proud of repeatedly being top salesperson and also being successful sales trainer. But looking back, one of the reasons I was successful was that I sold something I absolutely loved. I had a passion for my customers. It was the 1980s, and microcomputers were changing the world. What's not to like?

For example, I had some clients who were Investment Bankers. We set them up with Macintoshes, accelerated CPUs, as much memory as we could stuff in a system (eight megabytes!), and big screens. They rocked and rolled and got stuff done faster than their competitors. They loved it. I loved it.

And they were happy to pay a premium. I helped them make money, they were happy to pay for tools and services that helped them make more. Win win.

Ah, happy memories. I'm smiling while I write these words. And you know what? That story pretty much describes what I want out of a business.

1. We have to sell something people love. Specifically, the people who use it must love it.
2. We have to make money and lots of it. Money isn't the game, but it's how we keep score. More on this in a moment.
3. We have to provide a premium solution. We should provide a high-quality niche product. We are Ferarri, we are Macintosh, we are Single Malt, we are Aeron, we are the Monte Cristo #2, we are RIM.
   

About the money: I'm not venal. But perhaps I can describe it another way: we can't not make money. Not making money is a kind of cop-out, saying "this isn't for real." I think in software it's a way of avoiding the crucial moment when you have to look someone in the eye and say "How about it? Is that worth $10 a month?"

Trying to make money with software means facing rejection. Without money, you can brag about how many downloads you get per month. With money, you are forced to talk about conversion: how many people who tried it ponied up the bucks?

My first plan was a little heavy on the money and light on developing software people will actually use. So guess what? I ditched the plan and I started somewhat afresh. Why settle for less than 100% passion for my business?

The new business plan has some basic principles to live by:

1. We're selling bottom up. We sell to people who use our software. If it's good enough, if it solves their problem, they'll pay for it and they'll sell it upstairs to their manager. If it's just another ho hum, me too, people won't pay and they won't put their credibility on the line to evangelize it. That puts the pressure on us.
2. We charge from day one. There's no plan to get 1,000,000 copies into the world and then try to charge for version 2.0. That doesn't mean there won't be ways to try it for free, or use it for free under certain circumstances (such as the fashionable 'free if your product is free, pay if your use is commercial' approach). But it does mean that on the very first day I demonstrate it, I will ask people for money.
3. We make the best product possible. That means we can't make the best possible product. It means having fewer features than more, having features that fit together to work for our customers instead of having features that were cool or fun to program. It means a product with fewer checkboxes for the manager but more enthusiasm from the user.

So where am I with this? (I use the "Royal We" for principles but the "Singular Personal Pronoun" when there's work to be done).

• I'm churning out user stories. The second best way to find flaws in a product design is to write a story from end to end where a persona solves a problem with the product (the best way is to ship the product and ask for feedback).
• I've written several prototypes and proofs of concept. The best way to find flaws in a technical design is to implement it.

What's next? That's easy. I need to finish the user stories and circulate them to actual users. I may even publish them here. And then I need to start coding again, this time on version 1.0.

When I write these words now, it's starkly apparent how much work I have to do. That hurts! But at least I'm now working on something I love. I hope you feel the same way when you have a setback: it's a mighty burden, but it's your burden, you're building your dream.

I'm back on track again. Pax.

update: homoiconic.com.

Labels: passion

  ¶ 4:30 PM

  The other side of the passion argument "Anyone trying to select a product would be dimwitted in the extreme to choose one run by a hobbyist who is ruled by whimsy and mood and fears social contact and views professionalism as an insult to his artistic sense."

Labels: passion

  ¶ 2:35 PM

  Heresies, Episode I Dear reader, you are an intelligent person. So I'm going to save us both some time and skip several paragraphs where I regurgitate what consultants and books on creativity have said for decades, namely that there's a benefit to thinking about things that might be very, very wrong.

Though the wind is not made by leaves flapping, as some children guess, the theory is sufficiently profound that it should not be dismissed out of hand. In fact, disassembling erroneous concepts is one of the best ways to find new ideas.

Nicholas Negroponte, Creating a Culture of Ideas

I'll also skip the part where I list the number of things that were known to be very, very wrong and turned out to be right.

The statements that make people mad are the ones they worry might be believed. I suspect the statements that make people maddest are those they worry might be true.

Paul Graham, What You Can't Say

And so after taking several paragraphs bragging about how much writing I was going to skip, here are the ground rules for this little creativity game:

1. Find the extremes. We're interested in the outliers, the data points and ideas that live in the fringe. If somebody says that Extreme Programming works in 98% of the organizations that adopted all of its practices for a complete project, forget about what those 98% can teach us. Don't even argue with the number 98%. But those 2% that still failed... What's up with that?
2. Play the ball, not the (wo)man. Disregard the author of any source I quote. This works both ways: if you respect the author, don't forget they may be wrong. Or right. Or it doesn't matter, because the benefit is in thinking about the idea, not patting yourself on the back for agreeing with or disagreeing with someone else. And this is really the same point: Popularity is irrelevant. Disregard whether an idea is popular. You may be suspicious of anything popular. Let go of that. Again, the benefit is in thinking about the idea, not patting yourself on the back for agreeing with or disagreeing with someone else.
3. And finally... create extremes. Become a reductionist. Embrace the Tar Pit. Reject postmodernism. Look at an idea. Take one element of it, right or wrong, good or bad. Preferably wrong (see above). Ask yourself: "if this were not just true, but universally applicable, what would that mean?"

Ok, so here is the first heresy: software architecture sucks wind. Or blows chunks. It's resume-driven design.

• You aren't going to need it;
• Architecture Astronauts deliver Leaky Abstractions;
• Lamp Users Need to Grow Up.

What do you think?

Labels: passion

  ¶ 4:56 PM

  Great Reasons Not to Work on Your Idea

"It's been done before"
"The market is crowded"
"There are free alternatives"
"Nobody's making any money off it"
"It isn't Enterprise-y"
"Nobody's funding companies in this space" and/or "It isn't a space"

These are all excellent reasons to procrastinate another year. Absolutely no-one will blame you if you don't get started on your idea, especially if you don't even mention that you have an idea.

Meanwhile, in another part of the forest...

"Yes, similar services already exist. That never stopped me before. I'd like to point out that Fog Creek has been doubling in revenues every year mostly thanks to bug tracking software, and it's not like we invented bug tracking software."

—Joel Spolsky

So, if a fellow with a blog can be successful with bug tracking software, what's holding you back?

Labels: passion

  ¶ 10:42 AM

  Working code attracts people who want to code Charles Miller made a profound observation:

Working code attracts people who want to code. Design documents attract people who want to talk about coding.

He happened to be talking about a new open source initiative, but I couldn’t help but relate this simple principle to my experience with agile methodologies.

I have found that Big Design Up Front environments attract people who want to talk about software development, while iterative environments attract people who want to develop software. Is it any surpise that in companies where BDUF is predominant, nobody wants to stoop to coding? Everyone wants to be an “architect” or a “Business Analyst” or a “Product Manager.”

Recently I met a very bright fellow who is implementing a big new framework initiative. Developing within the framework seems to be quite frustrating and time consuming. Of course, there are many buzzword-compliant benefits promised from the framework. But there’s a certain “smell” to the project.

After some reflection, I realized that the problem with the framework is that the “architects” behind it have little or no interest in software development. Unlike today’s flavour-of-the-moment open source framework,* the developers of the framework do not actually use the framework. They’re architects, they don’t actually code applications. They don’t eat their own dogfood.

Their framework was developed for people who like design documents, buzzwords, and PowerPoint slides. And for that reason, there is lots of documentation and white papers. But there is precious little in the way of working example applications.

I suspect—although I don’t actually know—that the framework began with a white paper, and its destiny was forged then and there by that simple act.

Turning back toward the light, I think there’s a powerful lesson. Begin each project as Scrum begins, with the simplest chunk of code that delivers a benefit. If you’re building a framework, start with an example application in mind. Don’t make it “hello, framework.” Begin with something you can use and make sure your framework actually makes your life easier.

I’ll close with a joke:

A programmer is someone who is so irritated by small inconveniences that they will spend weeks writing a script, so they can spend days debugging same script, so they can save five minutes of their time, which they will waste reading slashdot on reddit.

Just remember that if your work actually saves you some time you’ve created something of value. And the best way to make that happen is to start with working code.

* Ruby on Rails (link above), is the foundation of Basecamp, a software service for distributed project management.

Labels: passion, popular

  ¶ 10:17 AM

  Tune in, turn on, drop in Over on slashdot there's a fairly predictible thread discussing Steve Jobs' alleged advice to students: drop out of college.

There're the usual arguments about why a college education is more than the coursework (it had better be: many CS undergraduate programs would be right at home in a Java Vocational College). There're the usual arguments about how intelligent, drivien indivduals would succeed without a degree. And of course more than a few people shrilly deride Steve for giving dangerous advice.

What I found interesting about the quote is that Steve doesn't seem to be saying college is useless and that you should quit college and get a job. Have a read over at Wired:

He said his real education started when he "dropped in" on whatever classes interested him -- including calligraphy.

Jobs said he lived off 5-cent soda recycling deposits and free food offered by Hare Krishnas while taking classes.

He told the graduates that few friends could see the value of learning calligraphy at the time but that painstaking attention to detail -- including mastering different "fonts" -- was what set Macintosh apart from its competitors.

I take this as meaning that the key is to drop in on stuff that ignites your imagination, regardless of the cost. I take this as espousing internships, networking, volunteering, hacking, fiddling, puttering, fiddling, hobbying, auditing, experimenting, and yes, writing code in your garage.

p.s. Here's an unrelated email I've been circulating:

Hello:

Some friends of mine are looking for a talented individual with deep Enterprise Java knowledge. Their environment is excellent: laid back atmosphere, plenty of agility, and long-standing relationships with blue chip clients like Apple and ING Direct.

I can share references from employees and contractors who just love working with this company. Of course, they're looking for the same: an individual with a knapsack bulging with enthusiastic references from people who have just loved working with her/him.

Their office is a stylish loft just steps from Roy Thompson Hall and their kitchen is stocked with caffeine and sugar. Get in touch with me directly if you'd like to explore an excellent opportunity.

Warm regards as always...

p.s. Naturally, almost everyone I know who has the chops to impress me has a great situation going really well. But almost everyone I know who is "smart" and "gets things done" seems to know two or three other people who are smart and get things done... and maybe this email might be perfect for them. Could you please pass this on? Introducing two friends is a great way to get two people to owe you a favour in one easy step :-)

Labels: passion

  ¶ 12:12 PM

  You and Your Research Richard Hamming on solving important problems:

In order to get at you individually, I must talk in the first person. I have to get you to drop modesty and say to yourself, ``Yes, I would like to do first-class work.'' Our society frowns on people who set out to do really good work. You're not supposed to; luck is supposed to descend on you and you do great things by chance. Well, that's a kind of dumb thing to say. I say, why shouldn't you set out to do something significant. You don't have to tell other people, but shouldn't you say to yourself, ``Yes, I would like to do something significant.''

This, in my experience, is the whole story. Everything else falls out of this personal commitment. It's a commitment to try. To strive. To sweat. To make the effort. To care.

Without this commitment, you have made a de facto decision to be on the receiving end of life. To float down the river like a leaf, now hither, now thither, carried by currents and wind.

Drop the modesty. There's nothing wrong with wanting to do something great, even insanely great.

But I'm forty-two going on forty-three...

Age is another factor which the physicists particularly worry about. They always are saying that you have got to do it when you are young or you will never do it. Einstein did things very early, and all the quantum mechanic fellows were disgustingly young when they did their best work. Most mathematicians, theoretical physicists, and astrophysicists do what we consider their best work when they are young. It is not that they don't do good work in their old age but what we value most is often what they did early. On the other hand, in music, politics and literature, often what we consider their best work was done late. I don't know how whatever field you are in fits this scale, but age has some effect.

But let me say why age seems to have the effect it does. In the first place if you do some good work you will find yourself on all kinds of committees and unable to do any more work... When you are famous it is hard to work on small problems. This is what did Shannon in. After information theory, what do you do for an encore? The great scientists often make this error. They fail to continue to plant the little acorns from which the mighty oak trees grow. They try to get the big thing right off. And that isn't the way things go. So that is another reason why you find that when you get early recognition it seems to sterilize you.

In my own experience, having some modest success early opens doors. But do they lead in the right direction? Sometimes not. I am constantly getting offers to manage teams of programmers working on uninteresting applications. I have taken such jobs, and I can say with certainty that while I have been successful most of the time and unsuccessful some of the time, uninteresting projects have never magically transformed themselves into important work.

It's hard to plant acorns when you've grown an oak. But it's necessary, very necessary. You can't get buried in management meetings and UML diagrams and presentations to resellers (you especially can't even get buried in writing about software development).

Todd Proebsting

A few years ago I went to a conference at MIT called "Lightweight Languages 2." All sorts of very bright people were there. Joe Armstrong gave a talk on Erlang. Matz gave a talk on Ruby. And a charismatic speaker from Microsoft named Todd Proebsting gave a talk about Disruptive Language Technologies.

In his talk, he gave a story about Richard Hamming. And here's the same story, in Richard's own words:

Over on the other side of the dining hall was a chemistry table. I had worked with one of the fellows, Dave McCall; furthermore he was courting our secretary at the time. I went over and said, ``Do you mind if I join you?'' They can't say no, so I started eating with them for a while. And I started asking, ``What are the important problems of your field?'' And after a week or so, ``What important problems are you working on?'' And after some more time I came in one day and said, ``If what you are doing is not important, and if you don't think it is going to lead to something important, why are you at Bell Labs working on it?'' I wasn't welcomed after that; I had to find somebody else to eat with! That was in the spring.

So...

• What is the most important problem in your area of interest?
• What are you working on?
• Why aren't they the same?
  

Labels: jobs, passion

  ¶ 12:33 PM

  Are you waiting for someone else to validate your worth?

"Can't sing. Can't act. Slightly balding. Also dances."

Paramount Pictures' screen test report on Fred Astaire.

Labels: jobs, passion

  ¶ 4:36 PM

  Passionate Communication What do Steve Jobs and John Abele have in common?

Both founded enormously successful companies: Apple Computer (AAPL) and Boston Scientific (BSX). Actually, both co-founded successful companies. And both are enormously talented communicators. Steve is famous for his "reality distortion field."

I recently attended a breakfast meeting of the Toronto Venture Group where John was the featured speaker. His presentation was riveting. He spoke with passion about his years turning the surgical establishment on its head. To hear him speak, Boston Scientific was David to the establishment's Goliath. He spent decades criss-crossing the globe, relentlessly selling the world on his dream.

He showed us his own heart x-ray. He showed us the basement where Boston Scientific was founded (he called it their "garage"). He showed us a picture of the conference room where he and his partner slept when there were no rooms at the inn. He spoke about values, about management, about integrity, and mostly he spoke about how to share your dream with the world even when the establishment is trying to defend itself against your disruption.

Being passionate about creating new software is about creating new software. But it's also about sharing your dream with people. Even if you're already comfortable with writing, presenting, and selling, invest in yourself and learn even more about selling your dream.

Here are two places to start:

1. Guy Kawasaki. Like many successful authors, he tends to re-write the same book with variations every time. But each new book has at least one revelation that has made me thankful I bought and read it. There's no better place to start than "The Macintosh Way."
2. Cliff Atkinson's "Beyond Bullets." Cliff is a PowerPoint specialist. There are very good reasons to hate PowerPoint, but Cliff's unorthodox approach and tips gets to the heart of communication so you can inspire and lead. "Zen and the Art of PowerPoint" is especially provocative.
   

Labels: passion

  ¶ 8:52 AM

  What I've learned from failure

I have been fired from more jobs than most people have had.

Why does failure matter?

It's a funny thing. After almost twenty years of drawing a paycheque for creating software, people generally want to hire me because they want me to duplicate the successes I've had. The model seems to be "do the things you've done successfully before, and you'll be successful now."

My experience is that this has never worked on its own. Success in software development is at least as much about avoiding failure modes as it is about "best practices." I conjecture it's because software development on a commercial scale is so hard that almost any mistake will sink a project if left uncorrected or even worse, actively encouraged.

We tend to seek easy, single-factor explanations of success. For most important things, though, success actually requires avoiding many separate causes of failure.

With that in mind, I've taken a little time to jot down some thoughts about situations where I've personally failed. I'm not going to tell you about some theoretical anti-pattern, or relate some broken thing I've fixed, I'm going to share things that caused me to leap from the deck of a burning boat to avoid drowning.

If you decide to run with the ball, just count on fumbling and getting the shit knocked out of you, but never forget how much fun it is just to be able to run with the ball

Some of them, in retrospect, would be comical if it wasn't for the human misery, damaged careers, and money wasted on failed projects. Or worse, in my opinion, the opportunity cost of putting good people to work on things that never end up delighting the world. I weep for what might have been.

The four most important causes of failure

Things which matter most must never be at the mercy of things which matter least.

The first thing I've learned from failure is that the four things which matter most are:

1. The quality of the people doing the development
2. The expected value of the product to its stakeholders
3. The fitness of the proposed solution
4. The quality of project management and expectations communication

In my experience, you need all four working to have a successful project. I've personally failed when even one of those four things was bad and not corrected immediately. If two, three, or all four were wrong, my discovery is that I've been unable to avert disaster. (This list obviously doesn't cover all of the factors needed for business success: I'm just talking about getting the software to ship).

Now that I've learned this, I have four new things to evaluate when placed in charge of a new project. And regardless of what I'm told, I'm going to investigate these four things every time, right away, without fail.

I've never seen a project where strength in one area made up for weaknesses in others. I've never personally seen a great technology platform, for example, that magically enabled low-quality developers to produce commercial-quality results.

And don't talk to me about XP being a magic bullet: all of the good XP teams I've seen happened to have quality developers, a valuable objective, decent technology, and yes, good project management.

People

I think the root of your mistake is saying that macros don't scale to larger groups. The real truth is that macros don't scale to stupider groups.

I've been involved with strong teams and weak teams, and the weak teams always failed. Weak teams have individuals whose performance is weak. The strongest indication of a weak team is the realization that if you were to quit and start your own business, you wouldn't try to poach any of your colleagues.

Painful experience has taught me some of the signs that a team doesn't have the chops to perform up to par. The first sign of a weak team is poor hiring practices.

Developing software is a difficult job. It requires a panoply of strengths. Hiring good people is never as simple as interviewing three people with "five years of J2EE" on their résumés and making an offer to the best of the three. Strong teams have almost impossibly high hiring standards. Strong teams will always leave a desk empty rather than settling for less than the best.

Another sign of a weak team is poor development hygiene. There are dozens of development practices that seem trivial to the inexperienced outsider or to the manager focusing on "big wins." Examples of development hygiene include source code versioning, maintenance of an accurate bug or issue database, significant use of automated testing, continuous integration, and specifications that are kept current (whether incredibly detailed or high-level overviews).

One team I audited were not just unwilling, but were actually unable to build a product that was in sustaining development. In other words, the product was in the field, in use by customers, and the team were not able to rebuild it from source. They were issuing all of their bug fixes as patches on their existing binaries. This was not a good sign.

Does this mean that nothing can be done if the team is weak? Not exactly. Some of the time I've had the authority to replace members of the team. I've always had the ability to set an example and suggest practices. But sometimes I've thought that an organization would be unreceptive to calls for change. And for want of courage, projects have been lost.

The bottom line is that when I've failed to recognize weakness in the team and/or failed to take immediate and decisive action to bring the team up to world-class strength, I've failed.

Argue with idiots, and you become an idiot.
If you compete with slaves you become a slave.

I've mentioned that I've failed with weak teams. Would you believe I've compounded this failure by failing with weak stakeholders? Whenever I've had stakeholders who didn't have the horsepower or the will to recognize that a project was in trouble, I've wound up in the E.R. having the brick dust removed from my forehead.

A chicken and a pig decided to open a diner together. The pig asked the chicken what they should call their new restaurant. The chicken suggested "Ham and Eggs." The pig thought about it for a while, then decided she didn't want any part of the venture. "You," she told the chicken, "would only be interested in serving breakfast. I'd be committed."

Getting away from weak teams, another source of failure is the omnipresent threat of "chickens." A chicken is not necessarily a weak individual, but a sign of a weak management structure. A chicken is an individual who has significant authority over your project, but does not make a personal commitment to the success of the project. Significant authority includes the authority to impose constraints on the team.

Even a single chicken can take a project out. Chickens are a special case of "external dependencies." Special, because they are often politically entrenched. I've worked with teams where the pay scale was determined by an edict from H.R. They were literally prevented from hiring top talent, and it wasn't a question of budget: they did not have the freedom to replace three mediocre programmers with two good programmers for the same price.

Another situation involved a team that were continually pestered to include functionality and architecture for "strategic" reasons by a Business Development person. Although senior management made the importance of the strategic functionality clear, they were unwilling to relax tactical requirements like the ship date or the target revenues. They had to constantly manage the "chicken" in order to succeed.

I've managed around chickens here and there, but I've failed to deliver a successful project whenever I've failed to limit the effect of chickens on the management of projects.

Action

Always dive down into a problem and get your hands on the deepest issue behind the problem. All other considerations are to dismissed as "engineering details"; they can be sorted out after the basic problem has been solved.

The next thing I've learned from my failures is something familiar to the test-driven development crowd. It's mandatory to fail early. You need to know you're in trouble right away. That's essential when taking over an existing project or starting something new. You have to find out how you're doing within weeks. Not quarters, not months. The longer you wait, the more inertia the failure will have.

You have to come in, take over, and establish some incredibly short-term goals and be prepared to take action based on the project's performance. I've learned that there's no such thing as too little time between milestones. Looking back at projects where I've failed, many contained some uncertainty or risk that I didn't address immediately.

In one case there was a critical piece of functionality that was so important the entire architecture was designed around it. The CTO and every developer swore it was the greatest thing since sliced bread. I made a back-of-the-envelope risk calculation and scheduled testing of that functionality to begin three months before the project was to be delivered.

A week before the function was to go into test, the technical lead informed me that it didn't work, had never worked, and that no attempt to fix it would work, because there was a major, glaring flaw that had been overlooked. A rewrite would be required.

The stakeholders agreed and appointed a new team to handle the rewrite. Needless to say, the old team's job security suffered a major hit.

Today, older and wiser, I would demand immediate proof of feasibility of all critical pieces of the product, no matter how obvious things may be to everyone else. I should have said "Great! It's a slam dunk! Wonderful, let's schedule a demo next week." At least we would have felt the pain early.

Details

Whenever I've allowed the details of a project to escape me, I've failed. On one project, the technical lead was a Ph.D. and refused to describe his work, saying that although I was the managing product development, he wasn't going to try to explain his rarified code or architecture to a layperson.

No, I'm not presponsible for what happened. I'm accountable for how we dealt with it, but I'm not responsible for it.

Needless to say, I was unable to ship a successful project. On another project the CEO would ask me the same question every few days: "draw on my whiteboard who's working on what." I had no trouble with this on that particular project, but looking back there have been projects where I was not tracking people's work on a day to day basis.

And I can tell you, whenever the details of a project have slipped from my grasp, the project has started to drift into trouble. I make no apologies for now insisting on knowing exactly who, what, where, when, and why. There's a big difference between being asked to explain your work in detail and being told how to do your job.

My personal experience is that attention to detail has always accompanied successful projects.; losing track of the details has always accompanied failing projects.

The Schedule

In most companies if a good quality project ships late then the managers will still get it in the neck whereas if poor quality project ships on time then the managers say "we did our best - obviously the dev team seem to be of a poor standard".

Dates are sacred. I've learned this lesson in good times and bad. Stakeholders treasure good dates. Stakeholders despise bad dates and the people who make flawed promises. That would have been me, more than once.

Every time, the lesson has been clear. Don't get the dates wrong. I'll confess: I don't really think Scrum is an order of magnitude more effective than anything else at producing beautiful, world-changing software. It may be worse. But it does produce software every month, month after month.

And every time I've delivered software on schedule milestone after milestone, my influence and standing with stakeholders has grown. And every time I've missed a date, I've suffered, regardless of whether the late software was demonstrably better than what was originally planned for the missed date.

If documents don't serve to avoid stupid things, mitigate risks or calculate budgets then what are they for? They're to show you have a "process" and a "paper trail" so that you can get ISO certified. That's all they look for, they don't care if you read them or not.

Back to the measurable processes. I've learned from failure that stakeholders like to know what's going on. I hate producing useless documentation. The net result is that I've tried to find the happy medium where I generate weekly management reports on projects.

A management report is something that is used to actually make a decision. Everything else is garbage. I've learned that when I haven't had management reports for a project, failure has resulted. Worse, sometimes I've had documents and metrics that were used to justify bad decisions that sank the ship.

So my lesson from these failures is that every project needs a set of regular reports that contain information you'll actually use to make decisions.

Software

Few projects are cancelled because their designs and implementation weren't complicated enough. Many are cancelled because they become so complicated that no one can understand them anymore; any attempt to change or extend the system results in so many unintended side effects that continuing to extend the system becomes practically impossible.

One of the reasons people associate me with "Agile" development approaches is that I'm always trying to simplify, simplify, simplify. This is because almost every time I added something to a milestone, I've gotten burned. It seems like it's always better to say "just finish what we planned, get that 100% functional, and then we'll add foobars."

I recently got burned twice in the same project adding functionality in between milestones. Both times I was sure that the changes were low risk. Both times I burned myself. Now I'm licking my wounds and swearing I'll never, ever break my agile principles of restricting scope changes to between increments.

It's important to remember that when you start from scratch there is absolutely no reason to believe that you are going to do a better job than you did the first time. First of all, you probably don't even have the same programming team that worked on version one, so you don't actually have "more experience". You're just going to make most of the old mistakes again, and introduce some new problems that weren't in the original version.

Here's something that I've screwed up repeatedly. Sometimes I've bounced back, sometimes the project has paid the ultimate price. The grand "this time we'll get it right" mantra is absolute garbage.

It had taken 3 years of tuning to get code that could read the 60 different types of FTP servers, those 5000 lines of code may have looked ugly, but at least they worked.

Don't talk to me about porting to Java, or new design patterns. If you must refactor, refactor here, and there, and there to solve this, and that, and the other specific problem that has a specific feature or bug attached to it. And show me that you had 100% unit testing coverage on the affected code and completed each refactoring in a day or so and then ran all the unit tests and got a green light.

If you can't do that, you're going to fail. I know it, because I've failed when I didn't do that. And when I cried on a friend's shoulder, he told me "I also made that mistake once, and I suffered the same horrible fate."

Thinking that a major rewrite is going to solve all of your problems is just revisiting my four things that matter most and planning on having one, the fitness of the proposed solution, overpower defects in the people, expected value, and process. It won't happen.

A major rewrite should produce a major new product that offers an order of magnitude more expected value. And you'll need to be 100% sure your team has the horsepower to get the job done and is going to use a process that can handle the load. I say this because I've tried and failed to rewrite entire applications, and I've taken over other people's rewrite projects and failed there too.

Power

Some days you are the bug, some days you are the windshield.

I've learned a little about politics from failing. What I've learned is that if you stick your neck out and evangelize change, you will be blamed if you do not achieve results. You may or may not care about that. But be aware of the fact that making changes involves spending your personal credibility. If you don't want to lose it, don't ante up: get out of the project.

Don't have good ideas if you aren't willing to be responsible for them.

And if you decide to make changes, have the courage to go 100% with your gut. I've failed more than once when I watered down my convictions in order to appease dissenters. The only thing worse than evangelizing change and failing is looking back and realized you might have succeeded if you'd held firm on your convictions. What a waste!

Making an employee work and profiting from that work are two different things.

I've seen a number of "sweat shops," and I've worked in several places where long hours and rhinoplastic intimacy with the grind stone were demanded of the team. I can honestly say that hard work makes no long-term difference to failing software development teams.

I disagree with those who say that long hours are 100% detrimental to software development: I''ve seen lots of situations where people worked around the clock, motivated by passion. But those were successful projects.

I've learned that redoubling effort when a project is in trouble has not fixed the project. The conclusion I draw is that although teams have worked long hours on many successful projects, there is no causal relationship between long hours and success (It's another example of the fallacy of "best practices": copying a single element of a successful project does not guarantee that another project will be improved).

My experience with failing projects is that the exhortation "Ahh, I'm going to have to go ahead and ask you to come in on Sunday, too..." has always been interpreted as punishment, not a meaningful way to fix the project. It has had no effect on under-performing members of the team and tends to strongly demotivate the people who are pulling more than their fair share of the weight.

It is impossible to sharpen a pencil with a blunt axe. It is equally vain to try to do it with ten blunt axes instead.

Good luck convincing stakeholders of this. One of the reasons people love to hand out overtime like candy is that hours in the office are measurable. The team's behind? Make them stay until midnight every night. Even if it doesn't work, the executive handing out this order can be sure she can measure compliance.

The bottom line is, it's easy to measure how many axes you're using to sharpen a pencil. When you discover that a blunt axe isn't sharpening the pencil, how do you propose to measure "sharpness"? How do you measure "working smarter"?

If we wish to count lines of code, we should not regard them as lines produced but as lines spent.

So another thing I've learned about failure is that when things start to go wrong, stakeholders want two things:

1. New processes
   
2. A way to measure compliance with the new processes

Overtime meets both criteria nicely, as do other simple panaceas like generating reports with every build or compliance with coding standards.

Fixing failing projects demands lots of things that are easy to measure and some that aren't. I've learned that if you don't control your stakeholder's expectations around change, you'll find yourself fending off demands for things like overtime and reports.

History

Even when my proposals are seen as significant improvements, they are often rejected on the grounds that they are not intuitive. It is a classic Catch-22: The client wants something that is significantly superior to the competition. But if it is to be superior, it must be different. (Typically, the greater the improvement, the greater the difference.) Therefore, it cannot be intuitive, that is, familiar. What the client wants is an interface with at most marginal differences from current practice... that, somehow, makes a major improvement.

When I've been brought in specifically to "work out" a failing project I've failed when I didn't have the authority and support to make major changes. This is saying the same thing I've said several times already, but it needs to be repeated.

Often, the stakeholders have just finished casting someone into the darkness and think that they've cast failure into the darkness with him. Take a moment and look up the definition of the word "scapegoat." They may have symbolically cleansed the project of sin, but the sins remain, and I have inherited them whenever I've allowed the project to continue to do business as usual.

The most damaging phrase in the language is, It's always been done that way.

I've heard dozens of variations on the same line (yes, I've failed dozens of times!) The line is "Well, so-and-so failed because he didn't x. But now you're here, you'll get x cleaned up, and we'll start succeeding right away. We were hardly failing, really, a little behind, nothing serious."

Every time I've been told that, things have ended up being seriously dire. No, it wasn't as simple as implementing monthly sprints or formalizing acceptance tests or nightly builds. The rot went right to the core and the stakeholders were usually (unwittingly) enabling it by not understanding or being in denial of the real problems. When people don't see the depth of the problem, they don't accept the importance of making changes.

It has boiled down to something so simple that you've probably heard it described in jest as the definition of insanity. If a project has been doing things a certain way, and the stakeholders are not 100% happy with the results, doing things substantially the same way will not produce substantial changes in the results.

Finishing

There are two kinds of people in the world: those who finish what they started.

Getting back to failing early, I've learned it's important to completely fail. Get fired. Shoot the project, then burn its corpse. Melt the CVS repository and microwave the backup CDs. When things go wrong, I've often tried to play the hero from start to finish. Guess what? Some projects are doomed no matter what. Some need skills I don't possess. And some need a fresh face.

The best way to fix a bad project is to not be part of it.

I've ridden more than one project down in flames, and as painful as it is to 'fess up and admit defeat, it's important to know when to fold your cards and quit. Yes, that sounds defeatist. But most success stories are comebacks from personal failures, not wondrous turn-arounds.

Sometimes you shouldn't finish what you started. Sometimes you shouldn't finish what somebody else started.

Every essay ought to end with a summary. Since this isn't an essay, I'll end with an adaptation of a Taoist story:

A musician performed a new piece he had written for his best friend. The friend sat in wonder and listened to the entire piece. When it was over, he nodded and told the musician that the music was wonderful. But what, he wondered, did the piece mean?

The musician nodded at this question and bent over his instrument, then played the entire piece again from the beginning.

Labels: agile, passion

  ¶ 7:33 PM

  Are you a perfectionist? Before you shout a resounding "YES!", stop for a moment and reflect. Be careful with the words "perfectionist" and "perfectionism." These words have very specific meanings in industrial psychology that may not reflect your intentions.

Are you:

• Thorough?
• Detail-oriented?
• Committed to writing solid, bug-free code?

These qualities do not make you a perfectionist.

Perfectionists make unreasonable demands of themselves and their environment. The 'super high standards' of the perfectionist give them an excuse not to fail: since they cannot reach succeess by their own standards, they never really try and can never fail.

Perfectionists obsess over trivial details as a way of avoiding new challenges. A classic perfectionist behaviour is to spend time meticulously optimizing every last line of working code instead of moving on to implement something new.

The key here is not whether optimization is worthwhile, but rather why the perfectionist is spending so much time on it: the perfectionist is afraid of moving forward and uses perfection as justification for holding back. "I'd like to get started on that, but I need to finish this work first."

Another perfectionist behaviour is called "blame and shame": they can't reach their own high standards because of the shoddy work and bad attitudes of their colleagues.

Again, the root motivation is avoiding failure: the perfectionist could write great code, if only her teammates would get their act together. But since her teammates are such shmucks, she spends all her time fixing their code or agitating for new methodologies instead of firing up Emacs and writing code.

Is it wrong to agitate for improvement in your organization? Of course not. But to the perfectionist, it isn't really about improvement, it isn't even about excuses for failure. The perfectionist didn't even try to succeed because she knew that the project could never reach her standards, so she didn't "waste her time" even trying.

This link discusses perfectionism amongst students. You might find it illuminating:

http://www.kidsource.com/kidsource/content2/perfectionist.k12.4.html

(From the old Joel on Software Forum, with some edits for clarity)

Labels: passion

  ¶ 7:55 AM

  Maxine on "Being Great"

"I used to think that all eyes were on me, just waiting for me to be great!... I thought everyone really cared! I thought it all mattered!

"But... ...what if I wasn't great?"

...Read the rest in the comic strip Maxine [link].

Labels: passion

  ¶ 10:09 AM

  Returning to Macintosh When I was a teen-ager I was in love with computers. I wrote games for our school's Data General minicomputer in 16K BASIC, I hung around the University of Toronto's High Speed Job Stream writing batch jobs in Snobol, Watfor, and Lisp, and daydreamed about writing two programs that could converse with each other (an idea from a Michael Crichton book).

Things gradually became more and more familiar, more and more mundane as I drifted out of school and made the massive mistake of thinking that computers were for business applications. Then, twenty years ago, I saw Byte Magazine's Macintosh issue. The scales fell from my eyes and my passion was rekindled.

It took me a while to actually get into programming for Macintosh, but I was now hooked on the idea that computers were more than just a neat environment for puzzle-solving. Computers could actually mean something.

I wound up "making big bucks in my spare time" writing Macintosh software: one creation, Tableau, managed classified ads for PageMaker and Quark users. I still own a working Mac SE and an SE/30. Although I've used "desktop" Macs and the enigmatic Duo, I've never lost my love for the original all-in-one design with the integrated carrying handle. For years, my web site was hosted on Macintosh systems.

It's twenty years later. I've owned a Windows desk side system for more than five years, and it's fair to say I've been contemptuous of Windows the entire time. Saying that "it's popular" and "it's good enough" is like saying that everyone should eat at McDonalds. Two years ago, I bought a Toshiba Portegé Tablet Computer, and things got better: my Tablet PC is the "first Windows computer good enough to criticize."

But I still admire Macintosh. And I finally found an excuse to buy another one.

My fiancé and I are expecting [Update: March 2007]. And that means a complete reorganization of our apartment, including moving our desk into our living room. I've been gradually relaxing my minimalist aesthetic over the years, but the idea of putting our TV, stereo, speakers, DVD player, VCR, computer, and monitor in one room was too much. As I wrestled with how to integrate as many of these components as possible, Apple announced the new iMac G5 (Now superseded by the Core Duo iMac).

I felt a visceral desire for this system. And luckily, I had an excuse for buying one: cleaning up the living room. I contacted Anthony Lewin at Computer Systems Centre and ordered a 20" iMac. I plan to add Bluetooth, AirPort, and a USB TV tuner. I already have about half of my CDs ripped to MP3. We can play the rest and/or rip them too. Our VCR tapes can be recorded to disk, so the VCR will go too.

So I have a rationalization. But who's kidding whom? I'm buying an iMac strictly because it's beautiful and my stereo is ugly, ugly, ugly. I'm making a decision based on passion.

And it feels good.

Labels: passion

  ¶ 9:13 AM

  Would you hire yourself as Chief Geek?

"Technical skills are relatively easily taught and learned. How people communicate with one another, their drive, their sense of responsibility, their problem-solving abilities—those skills are valuable parts of each person."

Labels: passion

  ¶ 12:13 PM

  The Dark Side of the Force In job interviews, there's a cliché question: "tell me about your weaknesses." It's so over-used, there's a crafty stock answer: "Well, I've been told sometimes I work too hard and I need to slow down. And I'm a bit of a perfectionist, and I have to make sure my colleagues don't resent my pushing for their best work."

In a high-pressure situation, you may prefer to be, as they say, "economical with the truth" when asked a question like this. But privately, everyone has real, actual weaknesses. And every weakness is an opportunity for self-enlightenment and growth.

I don't want to waste your time talking about how I enjoy a leisurely start to the day and have trouble being in the office before 7:00am. Instead, I'm going to generalize and share an observation: every personality trait is simultaneously a strength and a weakness.

For example, I enjoy writing about my feelings on this blog. However, I have to watch the positive to negative ratio: when I spend my time slamming the hoi-polloi, I'm giving in to the negative side of my passion. I'm not alone: there's an entire industry of academics slamming each other's work, so much so that I found this meta-essay: "Considered Harmful" Essays Considered Harmful.

Likewise, I'm passionate about solving hard problems in elegant ways. Bingo! There's a weakness, a hacker mentality, a "he'd rather be slinging Lisp on Linux than grinding Java on Windows NT" attitude. It's true, that's how I usually feel about my work (although I don't use Lisp: a real hacker can write Scheme programs in any language).

By the way, this attitude is not restricted to language weenies. The next time you see a J2EE application with factories, persistence mechanisms, XML-ification of everything, and Inversion of Control, you'll see the handiwork of someone passionate about J2EE architecture.

The trick, I guess, is taking stock of your traits and making sure you minimize the deleterious effects of them while maximizing the benefits. Last year I joined a team working on a J2EE/Oracle application. There were some hard problems to solve, and I was more excited by the 2% of the work that were the hard problems than dismayed by the 98% of the work that was routine.

And there I'll stop. The important thing is to constantly look for the 2% of your world that fuels your passion in a positive way!

Labels: passion

  ¶ 4:26 PM

  You're Smarter Than You Think Further to my exhortation to go ahead and write software, I found this on Dan Bricklin's blog:

When it comes to the traditional press writing about blogging, I'm reminded of programmers reacting to developments like the spreadsheet when VisiCalc came out. Sure the "programs" people wrote with it, and the "databases" they kept as lists, were not up to the standards of "real" programmers. But "every-person programming and databasing" has proved a boon to society and has not really threatened the profession of "programmer". It has, though, changed the role of programmer by allowing many of the detailed, area-of-expertise-centric applications to be done quickly, effectively, and inexpensively. Likewise, personal online publishing, such as blogging, is providing a means for communicating feelings, facts, experience, and opinions that we're even seeing the benefit of in this first try on a national stage. Bravo!

In short, lots of stuff that comes out "sucks" compared to the generally accepted standards. Yet, it sometimes changes the world. If you have an MBA, you can explain the whole thing in terms if "disruptive" forces. Or you could simply say that when someone writes software with a specific person in mind instead of a "market segment" or "analyst" or "their boss" in mind, the software is accessible, valuable, and generally delightful.

And that is why you're smarter than you think: you know an awful lot more about what you want to write and why it needs to be written than IBM, Microsoft, Dan Bricklin, Sun, Slashdot, and everyone else put together.

I'll close with some words of advice from George Carlin:

It's not enough to know what notes to play: you have to know why they need to be played.

p.s. Ethan made an amazing suggestion yesterday. When discussing "The Innovator's Solution" with me, he pointed out that the author's focus is defending the entrenched turf, instead of how to be disruptive. His suggestion is that the author's consulting practice is with the lumbering giants that get eaten by the innovative disruptors. Anyways, one very strong possibility is that it's way easier to be disruptive than to defend against disruption. Another reason to be passionate instead of analytical, I say...

p.p.s. Yes, I blatantly snarfed the headline from Nike. It's about time they got back on track and started using their influence to help people build their self-esteem instead of using one group of disadvantaged brown people to manufacture gaudy baubles for another group of disadvantaged brown people to wear on their feet.

Labels: passion

  ¶ 12:32 PM

  Taking Work Personally I take software development personally. Developing great software is part of my identity. Doing a fecal, semi-hemispherical (make that "shitty, half-assed") job is, in effect, living my life in a shitty, half-assed way.

Lots of people in our industry jump all over this and talk about how you have to compromise on principles in order to make money, or succeed in company politics. Well, no you don't. Let's get this straight:

Shipping great software is the art of making compromises, of satisfying constraints. However, what makes one project great and another mediocre is the quality of the compromises. Deciding to ship the first Macintosh with 128k of RAM is an example of a high quality compromise. Deciding that it's too time-consuming to fix regressions before the end of the project is an example of a low quality compromise.

The same idea holds for making money, being effective in a team environment, or balancing work and home life. Compromise is the art of the possible, and being passionate is about putting yourself in a position to make high quality compromises.

So when people sigh and tell you that their projects are late because of compromises, they're copping out. The truth is, the people whining about being "reasonable" and "pragmatic"* are actually trying to make themselves feel good about being unprincipled, about having no particular interest in whether their work is great, fair, or poor.

They never had the passion, or they had it but sold out, or it was beaten out of them early. So they tell you how they'd like some principles, but they can't afford any. They're lying to themselves! The worst part of these whimperers is that a few of them secretly want you to fail! It would crumple their world view if you decided to work nights and weekends on some unpaid passionate work and you ended up creating the next friendster, oddpost, or groovy.

On the other hand, I don't mind if they all make tons of money. I'd rather they be happy where they are than sit at their desk all day drinking coffee from a TGIF mug and wishing they were rocking and rolling in a start-up. I've found that the passionate people I know feel the same way: they have an abundance mentality. The only thing I resent about passion-less programmers is their desire to pull us down to their level. It's a herd mentality, and I don't feel particularly bovine.

I'm thinking about this a lot right now. My partner and I are expecting a son this October. A few people who don't know me very well have suggested that now that I'm going to be a father (going to be? as my partner points out, the baby is already here!), I'm going to have to settle down and stop all this passion stuff.

Update:

As if! My feeling is that when you're twenty-one, with no kids and no long-term relationship, you can afford to take a job for the cash. Your life is work, very little sleep, and partying**. Who cares if the nine to five gig is rewarding? But when you're a parent, you're a role model. Now is the very worst time in the world to be working a job that doesn't reflect your values in a deep and meaningful way.

Kids learn from what you are, not what you say or even what you do. No matter what you tell them, they'll grow up to be like you in a fundamental way. Do you think for one moment I plan to have my children learn that work is something you do to pay the bills, but you really don't take any pride in it?

Next I'd have to tell them that you go to school because you need a degree to get a job, instead of an incredible opportunity to learn and meet brilliant, motivated people. Bah!

Okay, time to wrap this up, I've got to get to O----, where my friend Ethan Henry starts today. We're raising the bar another notch! This week I'm interviewing more candidates for a development position, and I'm looking really, really hard for people who feel the same way.

So, thank you for 'listening'. I feel better now, and I hope you do too: I really, truly hope you're going to spend today as passionately as possible. Not throwing a temper tantrum because you have to write your app in Java instead of Lisp, but striving to make those good compromises, the ones that push you closer to shipping a great piece of software.

* I do not mean the pragmatic programmers: those guys have so much passion they practically sweat passion onto every page of their book!

** I know this is old fashioned, but you can get a lot of software written if you avoid anyone who uses the word "party" as a verb :-)

Labels: jobs, passion

  ¶ 7:28 AM

  How To Write Software With Art, Independence and Spirit

Everybody is talented because everybody who is human has something to express

Are you working on a personal project right now? Are you working on something that will have an unmistakable stamp of you in it? If not, why not?

Are you holding back because you don't have that million-dollar idea that will change the world?

This is a subtle way of beating yourself down, of not giving yourself credit for the talents you already possess. You are talented, you have something that should be shared. It may not be a big hit. Of course, it may be a big hit no matter how zany it sounded when it started.

Dave Winer started a company to sell a scripting environment. He ended up with a company that helped launch the weblog revolution. This shouldn't have surprised him: he once wrote a Pascal-specific programming editor and ended up with MORE than he expected, inventing the outliner. And don't get me started talking about the guy who created a web site for selling Pez dispensers.

Wait a second, what about the millions of metric tons of books published with the advice to research, research, research before you start your business?

I have a longer answer about market research below. But the short version is that your estimate of the value of the market for your idea is in no way related to the talent you have to express that idea in software. So express yourself: write your software.

Teachers, critics, parents and know-it-alls, when they see you have written something, become at once long-nosed and finicking and go through it gingerly sniffing out the flaws. AHA! a misspelled word! as though Shakespeare could spell!

Absolute crap sells when it brings a new idea to the world. The quality of the idea is more important than the quality of the software. Don't horde your software, polishing it for years lest someone file a bug report or tell you that you can't code anything.

Worse is Better: release early, release often. Get it into the world and refine it with feedback. It's easier for critics to find flaws than to appreciate the enormity of the change you are proposing. When Apple released Macintosh, what did people say? How about "It sucks, it only has 128k." Only 128k? Well, of course they had to add more memory. Didn't the first PC come out with 64k? That isn't the point.

The point is that the most important thing is to ship your software. Start it. And then release it. You'll have many chances to polish it later. Look at Open Source: the number one thing people look for in freely available software is that it is maintained, that it received regular attention from its authors.

If you wait until your software is perfect, there will be nothing left to add, and it will sit on the shelf gathering dust like a precious gem that is too valuable to wear daily. Get it out early, don't worry about the criticisms, and let it surprise you.

A great musician once told me that one should never play a single note without hearing it, feeling that it is true, thinking it beautiful

Consider everything in your software. Make a list of requirements and features. So much to do! And it always feels like making your software better will take more time. What to do?

This one's easy. Go over your list of requirements. How many are there because they capture some essential truth in the software? And how many are there because there's some 'should' or 'ought to' that drove the decision?

For example, if you're writing a web application, does it really need to be .NET or J2EE? What's with the XML/XSLT layer? Do you really need XML/XSLT to implement skins? Or would XHTML+CSS get the job done faster and simpler? There's a whole debate on faster, lighter development. Let's skip over that. The point is, unless the core of your idea is about XML and XSLT, you don't need it.

You may choose it if it helps you express your fundamental idea more succinctly. But if it's a lot of extra work, what are you writing? Your software? Your business plan? Or your resume? Stick to the core ideas that are true, that are beautiful, that will make your software sing like a Stradivarius.

Men spend their lives adding and subtracting and dictating letters when secretly they long to write sonnets and play the violin and burst into tears at the sunset

Remember the question about market research? Why shouldn't you do market research before you write your software?

By all means research the market when you're ready to sell. And maybe at that moment you'll make some changes. But don't start with the research. Market research will kill any idea and furthermore, you'll get depressed thinking about how spectacularly useless your ideas might be.

Another reason to skip the market research is that anything worth doing creates its own market. Pretend it's 1979. Microcomputers have been around since 1977. What's the market for another micro to go head to head with Apple Computer? Good thing you didn't do your homework, because you're IBM and if you start now, in 1981 your new micro coupled with 40% of your entire worldwide marketing budget will revolutionalize the business.

Okay, it's 1982. What's the market for a mouse and bit-mapped graphics? Good thing you skipped the market research, because you're Apple Computer and in 1984 you will change the world with Macintosh. It's the third kick at the can: Xerox's customers have voted thumbs down on their Star workstation and your own customers have told you that "Lisa Technology" sucks.

Anyways, here's the real issue: market research is adding and subtracting and dictating letters. It's trying to fit in with the business world so that you can feel like you're part of the herd, one of the crowd, so that you don't stick too far out.

I'll give you a hard, hard truth to face: when you try to write a piece of software around market research, you're subtly trying to justify your own failure. You're making excuses in advance.

What happens when you research the market? Well, one possibility is that you don't write your software ("nobody wants to pay for it"). But you take solace in the fact that it wasn't your fault: it's the market. And if you tweak and pull and twist your ideas to fit the research, then you've got another excuse: it wasn't your fault the program failed, that wasn't your original idea, but the market research said you had to do things that way.

Forget the market research. Forget the analysts. Write what's in your heart, and let's find out.

This post was inspired by a chapter in Guy Kawasaki's book The Computer Curmudgeon. That chapter reprinted a magazine column he wrote that was inspired in turn by Brenda Ueland's book If You Want To Write. The quotations in bold are taken from Brenda's Book. If you'd like more inspiration, buy Guy's book The Macintosh Way, where he shares his feelings about beautiful software and beautiful companies.

If you like this, please try If You Want To Write Software, where I tried to express how I feel about writing software, using Brenda Ueland's quotes to introduce the ideas.

Labels: passion

  ¶ 7:41 PM

  If You Want To Write Software This post was inspired by a chapter in Guy Kawasaki's book The Computer Curmudgeon. That chapter reprinted a magazine column he wrote that was inspired in turn by Brenda Ueland's book If You Want To Write.

Everybody is talented because everybody who is human has something to express: don't worry about market share, customer needs, or the admiration of your peers. Ask yourself, "what is inside me straining to get out and manifest itself as a piece of software?"

Teachers, critics, parents and know-it-alls, when they see you have written something, become at once long-nosed and finicking and go through it gingerly sniffing out the flaws. AHA! a misspelled word! as though Shakespeare could spell!: It is not important that your code be well-formatted, refactored mercilessly, or even be entirely bug-free. It is only important that when you look back at what you have written, you can say with certainty: "that's what I was thinking of when I wrote it."

A great musician once told me that one should never play a single note without hearing it, feeling that it is true, thinking it beautiful: Everything you write should embody some great truth you hold dear. Ignore things you know in your heart to be false, such as integrations driven by Business Development, or features added to make your business plan more marketable.

Men spend their lives adding and subtracting and dictating letters when secretly they long to write sonnets and play the violin and burst into tears at the sunset: does the world need another XML-based, J2EE powered, SQL-Server? Or does it need Konfabulator, software so beautiful it inspires legions of admirers? What makes your heart beat? What would you write if you had hundreds of millions of dollars? Go ahead: write it anyways.

No writing is a waste of time,—no creative work where the feelings, the imagination, the intelligence must work. With every sentence you write, you have learned something. It has done you good. It has stretched your understanding: during every development there comes the awful realization that nobody is going to want what you are building. Sometimes you are wrong and people will download your creation like hot cakes. Sometimes you are right and your software will pass unnoticed as the world careens down its crazy, unpredictable, unfair path. The thing is, it doesn't matter if your software changes the world. Writing it will change you for the better.

Two roads diverged in a wood, and I-
I took the one less traveled by,
And that has made all the difference.

The Road Not Taken, by Robert Frost

The moment I read Van Gogh's letter I knew what art was, and the creative impulse. It is a feeling of love and enthusiasm for something, and in a direct, simple, passionate and true way, you try to show this beauty in things to others, by drawing it: you and I think software has beauty, an aesthetic value, in both form and function. We have a sense of taste for software. Writing software is a way of sharing your feeling with the world, of telling the world what beauty means. Tell us! We want to know! We want, for one shining moment, to feel your love for your work.

At last I understood that writing was this: an impulse to share with other people a feeling or truth that I myself had. Not to preach to them, but to give it to them if they cared to hear it: thank you for reading this article. The words of others have helped me to explain how I feel about—how I've always felt about—writing software.

The quotations in bold are taken from Brenda's Book. If you'd like more inspiration, buy Guy's book The Macintosh Way, where he shares his feelings about beautiful software and beautiful companies.

Labels: passion

  ¶ 1:41 PM

  Chris Crawford on Game Development Chris Crawford is a game developer. He created many of my favourite Macintosh games, including Balance of Power, a sophisticated game of geopolitics. He later wrote a book about Balance of Power, explaining the game's creation and mechanics.

I re-read his book over the weekend, and was struck by Chris' insights into software design. First person shooter games are all about rendering and trajectories. But Chris' games perform the difficult feat of modelling "soft" subjects like geopolitics, trust, or economics.

His perspectives struck me even more relevant today than they were in 1986 when he wrote the book (not to mention 1984 when he began work on the game). Today we are obsessed with "styling," animated menus and translucent interface elements. But where is the "Knowledge Navigator"? How much progress have we made getting computers to help us with our lives or jobs?

Of course, computers are pervasive. That's a measure of economic success. But to be honest, I'd say that computers in the office spend most of the time solving problems that other computers have created. I recently made a presentation with animations. But is there any content in that presentation that wouldn't have been just as effective if I'd written things out with Word 2.0 for DOS, my editor in my University days?

So back to Chris. His perspectives on turning soft concepts into games apply directly to developing software for helping people solve problems. The problems we want to solve are incredibly soft.

Take software development itself. Do you really believe it's an empirical science? That we just assemble the estimates, draw some pretty diagrams, follow the ISO something-or-other standard, and software emerges from the process? Software development is a soft science. And I hope that some of Chris' principles for developing games around soft concepts will apply directly to developing software to assist software development.

I captured some of Chris' words from the book. I hope he doesn't mind my sharing them with you here:

Simplification to achieve clarity is the essence of my work; clarity can be extracted from a muddy reality only by denying some of reality's richness.

A game is to a simulation as a painting is to a blueprint. A painting of a house gives you an emotional impression of the house; a blueprint of a house tells a carpenter where to put the windowsill.

What is important is the principle, not the instance, and principles are processes. The actual amount of the GNP of Ghana is less important, for the purposes of a game on geopolitics, than the manner in which that GNP changes over time... This concept—which I call *process intensity*—is the organizing principle of [my book]... I give the facts themselves short shrift. Facts are transitory, while processes are the enduring truths.

You can't interact with a fact. It's like a dead fish—it just lays there. But you can interact with a process. You can shape it, change the parameters that affect its behaviour. Ultimately, you can learn about it. facts are best relegated to books and other static media, and computers are best applied to problems involving processes, for computers are not *data* processors, but data *processors*.

I did not start out with a fixed set of notions and then express those notions directly through the computer. Instead, the attempt to express my thoughts on the problems of geopolitics helped refine and correct them.

Verbs are all-important in game design. They are the allowed actions, the permissible commands that are available to the player. A good set of verbs allowes players to do everything they would need or want to do. A poor set of verbs will either confuse them with its arbitrainess, or lock them in a frustrating straightjacket. The game designer spends a great deal of design effort worrying about the verbs to provide in the game.

A programming tool is like a freeway: It takes you somewhere in the universe of results. All programming tools are to some extent generalized to handle the needs of a large number of programmers. They are like freeways that take you to the most popular beaches or the most crowded resorts. He who would climb the remote peaks must forsake the freeway and make his way by foot. The exertion of a week's simple sweat can place the programmer on a mountain peak from which are visible new territories of creative opportunity invisible to those who veer away from steep grades.

And my favourite quote:

Using paratroops is like putting yourself into a deep hole to see if you can dig yourself out of it. I did much the same thing with Balance of Power, setting a goal for myself that I had no reason to believe I could attain. Then I publicly and financially committed myself to attaining that goal... It was a tough, frightening experience. When it was over I was physically and spiritually spent. But what is the point of undertaking anything less than the most demanding of efforts?

Labels: passion

  ¶ 10:10 AM

  How to get your first job developing software HumbleCoder asked:

I've seen a number of people recommend doing programming work for free in order to gain experience. This can involve either volunteer work for a local charity or school, or working as an intern for a for-profit corporation. I'm not sure if this actually works or not, so I am wondering if anyone out there can comment on this strategy for getting experience. Email me if you have used this technique to help you land a first job.

I have been hiring developers for almost a decade, and one of the things I look for is a habit of writing free or nearly free software.

In my experience, good developers love to write code. In fact, the very best developers don't know how to "not write code". When they were in high school, they wrote MUDs in Basic (well, that's what they did in the 80s). When they were in college, they developed Perl and Tcl scripts that scanned Usenet for updates to threads of interest. Even if they hold a "lowly" summer job working in a computer store, they have a Linux system at home running Apache and mod_perl or PHP.

Even after they get paying jobs working 70 hours a week, good developers can't stop coding. For example, Phil Greenspun started Ars Digita, yet he wrote lots of free tools on his home server (I use two of them on my own weblog). Paul Graham ran Franz and Viaweb (now Yahoo! Stores), yet he is busy developing a new language, Arc. In my own case, I was busy as a consultant in the early nineties but couldn't help writing a PDA application and a dynamic web service "on the side".

When I am asked how an inexperienced person can get their foot in the door, I give two pieces of advice:

First, I tell them to just start writing code. If they don't know what to write, I don't suggest anything specific, I just tell them that they should pick stuff they enjoy, using tools they like. It isn't important to use the latest "industry standard" tools or work on something "commercial".

Second, I tell them to think in terms of a "portfolio" instead of a "résumé". Even if they don't take a physical portfolio to job interviews, their résumé or cover letter should include an appendix with samples of their work. If an inexperienced person wants to stand out from the crowd, screen shots, links, and code snippets will appeal to a technical hiring manager.

Is any of this guaranteed? No. Some companies hire HR people that are not qualified to tell the difference between a talented candidate who compiles and builds their own development environment at home and a mediocre college graduate that didn't expand their own horizons.

I personally wouldn't want to work for a company that gives significant authority over its intellectual property to such an HR person, but not everyone is as picky as I am :-)

Labels: jobs, passion

  ¶ 7:57 AM