Haggling about the price A few minutes ago I saw this interesting post: Ajax Web Developer: $240k per year… with only one catch. If you want the executive summary, the job location is Iraq, a battle zone.

What I find interesting about jobs like this are the people who debate whether $240,000 is enough to compensate for the very real possibility of being killed:

Unfortunately, the whole 12/hr/day, 7/day/week deal works out to only about $52/hour…

I think that for $240k a year, I could shift my sleep schedule to iRaq time…

I think it would be a cool opportunity, but the 12 hour days, 7 days a week drops the money down to just over $50 an hour. If there is the possibility of being shot at, I need at least $80 per hour :-P

I would take 300k, though probably (?) not 200 to work in Iraq…

And lots of people were debating the tax-free status of this position, as if that’s the only deciding factor between staying or going. I’m sure many of the people commenting were being facetious, however Winston Churchill really put things into perspective in this apocryphal exchange with a socialite:

Churchill: “Madam, would you sleep with me for five million pounds?”

Socialite: “My goodness, Mr. Churchill… Well, I suppose… we would have to discuss terms, of course…”

Churchill: “Would you sleep with me for five pounds?”

Socialite: “Mr. Churchill, what kind of woman do you think I am?!”

Churchill: “Madam, we’ve already established that. Now we are haggling about the price.”

If you’re going to sacrifice your life on principle, I commend you for your bravery.

But when it comes to money, either you incorruptible, or you have decided that your life is a commodity, something that can be bought or sold, traded or bartered, played as a pawn and sacrificed so that some contracting company somewhere makes another billion dollars for its shareholders.

but I’m not working in a battle zone…

This all seems very surreal, I’m quite sure few or none of my blog readers are wearing combat fatigues and body armour right now.

But the principle applies elsewhere. If you ever find yourself being subjected to personal abuse, or manipulative behaviour by your manager, or asked to lie to clients, or outright cheat such as double billing, you are in the exact same position as someone asked to sacrifice their life. You have to decide whether you are sacrificing your self-esteem for a noble cause, or whether it’s about the money.

And trust me on this one, if it’s about the money, you are on the road of perpetual unhappiness, where every well is dry and every inn is full, and your journey never ends.

If you have a dream, and making that dream a reality involves tough business decisions, good for you. I commend you for having the courage to duke it out with VCs, lawyers, and whomever else will try to get you to mortgage your honesty to fill their coffers.

But if you are dragging yourself to work every day, if you hate what your boss makes you do, if it’s about the money and the vacations and the toys, but not about fundamental satisfaction with your job, you need to stop right now, sit down with the people you love and trust, and reëvaluate your choices.

Trust me on this one. If $100,000 isn’t enough, neither is $200,000, and $300,000 won’t do it either. We’ve established what they want you to become. All you’re doing is haggling about the price.

the sun also rises

The good news is that unless you have signed up for a two year tour of duty with the army, you can stop any time. You can get off the treadmill. You can just stop doing things that aren’t consistent with your values.

I’m not saying you shouldn’t work long hours, or compromise your technical integrity by using a for loop instead of a map function. Making tough choices is part of growing up.

But you do not need to compromise your integrity, ever. And the good news is, you can just say no. Tell your boss to handle all communications with the client if he doesn’t want you to mention that you only put in twenty hours on their project last week because he has you working for two clients at once.

Tell your client that no, you cannot submit an invoice for $2,500 but accept only $2,000 in payment, with her skimming the extra $500 as a secret commission.

It’s incredible, but this little word, no, it really works. The world does not stop. You don’t get dragged from the room and imprisoned without trial. If you are let go, you will find another job, a better one, a job with people who like you and respect you and want to make money in an honourable fashion.

We have a lot of freedoms. The point of those freedoms is to have an almost completely unrestrained ability to pursue happiness. All you have to do is make the right choices. And the first choice is to simply say no to anything you cannot abide.

That is the road that leads us into the sunshine.

Labels: passion

  ¶ 3:37 PM 20 comments

  Still failing, still learning The good news: I’m still learning.
The bad news: …from failure.

This post officially announces that my side project (originally named cause & effect and later named certitude) is over.

What I wanted to achieve

For those of you who weren’t subjected to one of my enthusiastic rants, here was my Graham Question: Can we predict the outcome of a software development project with objective observation?

I have always believed that the answer is Yes. And I manage projects that way. But I also strongly believe that the only way to prove that we have an objective understanding of an algorithm is to mechanize it, to write a piece of software that executes your algorithm.

So I set out to write a piece of software that, pure and simple, would look at a software development project and show you a traffic light: a green light would mean that the project looks like it’s on track, a yellow light would mean that the project needed help, and a red light would mean that there is no hope.

I won’t go into a lot more detail, mostly because this isn’t a story about teeming hordes of customers and me not being able to finish by deadline. It’s a story of inventing a great solution to a problem nobody cares about. But if you absolutely must visualize the application, think of something that gobbles up your MS project files, your bug tracking data, even your burn down spreadsheets, and belches out that traffic light when it’s done. That’s it.

So how and why did I fail?

Project management software is social software

Reason zero, just as Paul Graham warned, my age. Really. I remember how much code I could crank at age 22, and now that I am double the age, I write an order or magnitude less. I’d like to say that my code is that much better that it makes up for the lack of volume. That might be true if I start a project with a specific end in mind, but the act of experimenting, tinkering, and exploring benefits from being able to turn large amounts of code around in short amounts of time.

This is reason zero because I wanted to get it out of the way before explaining why I think I still would have failed if I were 22. But it’s still important to understand, because I might have known that I was going to fail two years ago, instead of today.

Reason one that I failed—and this is the most important reason I failed—is that I was trying to tackle a social problem with technology. This can work—ask any dating site zillionaire—but it is a very high-risk strategy for software. Not just for making money, but for what really matters, adoption.

Project management is a social problem.

Project management is a social problem. It is 99.5% about getting everyone who knows something about the state of the project to share what they know with everyone else. Getting all the relevant information is 99.5% of the problem, analyzing the information is 0.5% of the problem.

Joel likes to brag about how much trouble FogBugz goes to to make it easy to enter bugs, about how certain kinds of reports are not available to discourage punitive social behaviours like punishing developers who generate too many bugs. This is a strong hint that getting good information is all about managing people’s perception of the likelihood of punishment for telling the truth.

Sitting here typing this, I think the company who can do the best job of predicting the outcome of software development projects is Inkling Markets. That’s because their entire business is about finding a way for people to communicate what they really think of something, not just what they think other people want them to say about something. I think Todd Proebsting would agree.

This problem isn’t limited to dysfunctional environments where people cower in fear of saying anything except “Sir, Yes Sir” when told to change the laws of physics.

Lemons, damned lemons, it’s always lemons

Project management suffers from a real lemon problem. I quoted Bruce Schneier at length about this already, so this time I’ll explain things directly:

Most managers, especially those with limited experience shipping software on a predictable schedule, do not know how to correlate what they’re told about the project with the likelihood that the project will succeed.

Some information is valuable, some is junk. The problem is, managers “buy” information. They trade favours like letting you keep your job for information about how well you are doing your job. So there is a market for information just like a market for cars.

They also “sell” information, literally: they have to make a report or a presentation to their superiors, or to stake holders, or to their fellow founders at the YCombinator dinner.

When a manager cannot tell the difference between information that is useful for predicting the outcome of a project and information that is not useful for predicting the outcome of a project, she thinks about the next best thing: the “resale value” of the information with people one step removed from the project, like her own manager. So she values things like pretty PowerPoints about the architecture higher than finished pieces of functionality.

(This is why I have always sweated my heart out to give good presentations. My teams have depended on me being able to take good information and sell it upstream just as if it were CMM Level Five Buzzword-Compliant Junk).

Do managers further removed from a project always value pretty junk more than good, solid information? Not always, but often. And that’s enough for people to be pressured to give the bad information that sells to their manager, while hiding the good information that doesn’t sell. Exactly like the owners of good cars taking their treasures off the market.

Lemon and bay leaf crème brûlée

Why does junk information outsell good information? Nice PowerPoint isn’t a good explanation by itself: there are nice PowerPoints explaining Agile, but managers still prefer waterfall.

Consider a not so big design. Let’s call completing that design good information: we’ve done a good job finding out what’s really important for the project and making a design that emphasizes the way this project is unique, not the technology stack.

Now consider a typical technology design, emphasizing frameworks and technologies. Fully buzzword-compliant.

Which one sells better? The technology stack does. Why? Well, for starters, managers have been exposed to seventeen billion dollars worth of advertising talking about the benefits of technology stacks. Nobody is advertising the specific ways the not so big design helps the project. How could they? Those are specific to the project, that’s the whole point.

And managers are like anyone else, they compare what you are doing to successful projects they have seen in the past. Once again, the not so big design doesn’t have anything in common with other projects, but the technology stack does. (There are lots of failed projects with technology stacks, of course. But who cites those when bugging the team about whether they will use Hibernate as their ORM?)

How did this happen? How did things that have no correlation to the success of a project become more attractive than things that do?

Quite simply, people have an incentive to look successful. So they imitate the outward appearances of successful projects. We have a really simple way of completing successful software projects: we put successful people on them. But we have a broken way of thinking about it: we don’t like to think of the people as being special, we think that what the people do is successful.

And by that logic, we can take anyone, have them do the same things as successful people, and our projects will succeed.

In a manager’s mind, the measure of whether information is good or not is, Does it measure whether people are doing the same things that successful people have done on projects I’ve been told were successful?

This is not the same thing as measuring whether the project is on its way to success at all. This measures the outward appearance of a project. Things that can be measured easily are rarely the most significant things. Behaviours that can be “gamed,” like how many hours a team is working, will be gamed.

And as above, even if a manger knows better, does her manager know better? If not, good information will be difficult to sell and she will be under a lot of pressure to serve Lemon Pie.

Off balance sheet transactions

There’s another important reason that projects have bad information: the best information is off balance sheet. That’s an expression meaning something a businessperson sweeps under the rug. Try Googling Off Balance Sheet Transaction. It’s never pretty.

In essence, project plans and reports never include the most important information about the likelihood of project success. Never. (I mean never in the same sense that Joel Spolsky means “nobody,” as in “fewer than 10,000,000 project plans”).

Let me give you a recent example. I was discussing a project with a certain manager in a client organization, and there was a rather linear series of releases. Her question was, Can’t we shorten the plan by looking at the dependencies and starting some releases in parallel with others?

Of course we could, but in doing so, we increased the project risk. Just as a single developer has a problem keeping multiple tasks in one brain, a team has the same problem: when working on several unrelated pieces of software at the same time, the individual developers may only work on one thing at a time, but the managers and the testers and especially the stake holders who are thinking about functionality and exercising change control are overloaded, so they will make poorer decisions.

And worse, although you might think that there are no dependencies, you only think that at the outset of a project because of assumptions you are making now, before you fully understand what you are getting into. All in all, there’s a reason Agile projects have a practice of working on iterations with single themes, and the reason is to reduce risk.

Does this mean that manager wasn’t making a reasonable trade off between risk and time? Maybe she was making a very reasonable choice. But let me ask you this: where on the plan would you see the risk component of the decision? If you were comparing Plan “A,” with the linear releases that finish in August, and Plan “B” with some parallel releases finishing in July, how do you decide which plan is better?

Right, you can’t see anything except the difference in dates, so you choose Plan B. The risk component of the plan is off balance sheet. Enjoy your lemon.

(And it is very hard for a manager’s people to explain, for the twentieth time, that it is a mistake to try to optimize a project by having everybody working at once, because that crystallizes certain architecture decisions too early, and haven’t you read any Eliyau Goldratt, whose novel The Goal explains what’s wrong with optimizing resource engagement when you should be managing project risk? The incentive is for them keep their own counsel and put their résumés up on the web. Such is life.)

A more striking example comes from another project where, for various reasons, there was nearly 100% turnover of the senior staff, and finally an outside firm was brought in to clean it up. Do you think there was a notation on the plan for staff turnover? That has to have a huge risk implication, but the plan where you use the same team from start to finish looks exactly like the plan where new people are brought in for each phase or iteration.

Why is risk off balance sheet? I think it’s because people have a blind spot for risk in projects. After the fact of a project, you can always do a postmortem and say, “if we had done this, and this, and if we didn’t do that, we would have succeeded.” So you blame yourself for poor execution.

There’s a culture of Boolean thinking about projects and plans. They worked or they didn’t. We’ll be done in July or August. Not “With Plan A, we’re 90% August, 8% September” and “With Plan B, we’re 12% July, 56% August, but 23% September and 7% October!”

Decisions that add risk to a project, such as excessive parallelization, or the use of unproved people, or the use of team who have never worked together in the past, or of forcing decisions prematurely, all of these things are not reflected in the plan.

(This is another reason there is pressure downwards on developer competence in many organizations. Consider two managers: the first staffs up quickly by selecting available developers, who may not be particularly good. In fact, they are less good then the existing team average. The second moves more cautiously, only adding to the team when the addition improves the average competency of the team. Do you think the second manager will keep their job long enough to finish a project? No, because the ticking time bomb of hiring certain types of developers is off balance sheet, but being understaffed is out there for everyone to criticize.)

I had this hope that through a kind of collaborative filtering I could have a piece of software notice that linear plans or plans involving hiring too quickly or whatever had less variance than parallel plans, and adjust the traffic light accordingly. Regardless of what people would say, it would shrug its shoulders and remorselessly remark on the actual historical averages.

Nice idea? No, it’s a dud. (Or at least, my execution was a dud!)

Somebody set up us the bomb

The second major reason I bombed is that I couldn’t find any buyers. Quite simply, the people who understand these principles are running successful projects. I know lots of people who agree that there is a problem and agree this would help, but they don’t think they need help.

I couldn’t find anyone holding their head in their hands, saying, “If only I could get the truth about our projects, I could open my budget and shower you with gold.” The people who were aware of the problems with project information were busy using decidedly low tech tools—like hiring good people and having daily meetings and estimating tasks in spreadsheets—to solve their problems.

And the people who could use some help quantifying the consequences of their choices—like managers who insist on calcifying decisions very early in a telephone book design document so that they can demonstrate progress in the form of an architecture—they don’t think there’s anything wrong with their approach.

Why is that? My conjecture is that people want help with things that fit their model of what’s important. Someone who uses skilled practitioners and XP thinks that limiting risk is important: that’s why they use two week iterations and merciless refactoring rather than up-front design.

Someone using a classical BDUF management approach thinks maximizing resource allocation and managing task dependencies is important, that’s why they spend all of their time looking at GANNT and PERT charts. When you try to sell either of them something, they want to know how it will fit the model they have in their head about how to succeed with software development, not why they should consider a new model.

And I’m not sure they’re wrong about what’s important to them, personally.

If one of those BDUF projects fails due to the architecture being a poor fit for what the team discovers is really important about the projects, or from poor decisions being made in haste at the beginning of the project, well… managers will say that the problem was with the people making poor decisions. Such managers are not shopping for tools to help them understand the trade-offs, because they do not believe they are making trade-offs.

Something I learned from selling Macintoshes back in the day is this: only make one sale. Convincing someone they have a problem is one sale. Convincing them you have the solution is another. And convincing them that today is the day to act is a third. If you have to do all three at the same time, you are doomed.

This is why experienced companies distinguish sales from marketing. The first two steps are marketing, the third is selling. When you are a new company, you don’t have the resources to market and sell. You have to work with an established pain point (eliminating the first hurdle), then use PR and limited marketing funds to get the word out that you have solved the problem (the second hurdle). You only have time and energy for the third sale, separating customers from their money.

But trying to convince managers that they’re doing it wrong (tactfully), then convince them that they want your product rather than some big rigid waterfall thing or three by five cards and XP, and then convince them that they should write a cheque today…

Bad idea. I should know, I discovered that I was trying to do exactly that.

Now you understand why I have used so much of this post to rant about the state of project management in software development. When you understand what most companies are doing and why, you understand what will sell, and why. And when I understood that the things my project were measuring and reporting were of little interest to the people who were my market, well...

The not so big business plan

I think there are lots of things that will sell, that do sell into this market. When you understand that this is a social problem, when you understand that most information is bad and that the incentives are to value bad information, and for managers to value activities that produce bad information over activities that produce good information, you can make something people will buy.

Like books, lectures, methodologies, and video training. If people have a social problem, a social solution is a natural fit.

There were interesting possibilities like selling this to BigCo for “process improvement.” But whenever I thought about such ideas, I noticed that the software wasn’t in there. I could have written a book and lectured on these ideas. I could have done a “search and replace” of agile for certitude.

A good business plan is one that is really specifically about you and your software. If your plan could be executed by someone else, or with a different project, it simply isn’t the right plan. And all variations of turning this into ConsultingWare were excellent ways of monetizing Reg Braithwaite the brand, but not ways of spreading the adoption of certitude, the product.

And so… and so I looked the sunken cost fallacy straight in they eye and said, no more. As much as I hate to lose, I have folded my project and I am sharing with you the important lessons I learned.

Lessons learned

Well, the good one is this: Paul Graham is right. If you phrase your venture as a question, you will walk away with something very valuable. I now know a lot more about project management and the culture of software development than I did when I started the project, and I wasn’t exactly a Spring Chicken. And set out to learn about Bayesian Networks and Supervised Classification, but I ended up learning about Unsupervised Clustering and Collaborative Filtering.

What a journey.

And there is the one I ignored for far too long: Always Be Selling. I can thank friends like Leila Boujnane, Sutha Kamal, and Erich Finkelstein for reminding me about this, incessantly. Asking people to “buy” your idea forces you to make something people want. That being said, I think there’s a right amount of “push.” You can’t invent by polling the market. Quickly now, jump in your time machine and go back to 1981 or so. How many people wanted a mouse for their computer? But yet… Always be selling!

(There’s a bunch of other stuff that is far more personal, and maybe some of it will appear one day in public, but I wanted to keep this post to essay length, so it’s almost entirely about products and markets.)

So, here I am. Older, wiser, and ready for life’s next adventure. It might be more consulting for BigCo. It might be joining a start up and helping someone else’s dream flower. I don’t know, but if you have an idea...

My brain is open.

Labels: agile, passion, popular

  ¶ 6:49 PM 30 comments

  Hard-Core Concurrency Considerations Kevin Greer responded to What Haskell teaches us about writing Enterprise-scale software with some insightful emails about the pros and cons of using purely functional data structures for managing concurrency in a multi-processor environment. I’ve reproduced them (with his permission) below.

Now, your first reaction might be, “Ah, Reg is not nearly as smart as he thinks he is.”

If you feel like giving me some credit, you can keep in mind that I was not writing the definitive essay on designing concurrent software, just pointing out some interesting overlaps between what I consider to be the most academic programming language around and the most practical requirements in business applications.

But there’s something more important to take from this.

The original post was a response to someone asking whether there was any value to the stuff you read on programming.reddit.com. His query was whether reading about Haskell was practical. My response was, yes it is, and here are some examples of where functional data structures have an analogue in concurrent data structures. My thesis (that’s a two dollar word for “point”) was that many “impractical” things have a lot to teach us about things we will encounter in the pragmatic “real world.”

Of course, most people read the post and thought, “Cool, some neat stuff about concurrency.” Nothing wrong with that. If you value tips on concurrent programming (and I certainly do), you’ll find Kevin’s emails very insightful.

And if you are still wondering whether you should look at “impractical” and “academic” material like Haskell, Erlang, or other things not being promoted by MegloSoft, consider that one of the papers Kevin cites describes a data structure running on a 768 CPU system. Is this in a University lab somewhere? No, it is in a company that promotes itself as an Enterprise-Scale Java company.

You simply can’t assume that everything the industry tells you to learn is everything you need. Or that any one source (Cool! Raganwald has a new post on Lazy Evaluation) has the definitive answer.

You need to commit to life-long learning to be a software developer. Some of that learning is straightforward MSDN Magazine stuff, simple updates to things you already know. And some of that learning is a little further out on the curve.

Without further ado, here is one of the most comprehensive technical replies to a post on my blog I’ve received to date.

Concurrency Considerations

Hi Reg,

I was just reading your article on concurrent collections and have a few comments:

1. Just because readers do not update the structure doesn’t mean that they don’t need to synchronize. This belief is a common source of concurrency bugs on multi-processor systems.
   
   Unless you synchronize on the root of your data-structure (or declare it as volatile), you can’t be sure that your cache doesn’t have a stale version of the data (which may have been updated by another processor). You don’t need to synchronize for the entire life of the method, as you would by declaring the method synchronized, but you still need to synchronize on the memory access. You hold the lock for a shorter period of time, thus allowing for better concurrency, but you still have to sync.
   
   If you fail to synchronize your memory (or declare it as volatile), then your code will work correctly on a single CPU machine but will fail when you add more CPU’s (it will work on multi-core machines provided that the cores share the same cache). If you have a recursive datastructure (like a tree) then you will need to actually synchronize on each level/node, unless of course you use a purely functional data-structure, in which case, you’ll only need to synchronize on the root.
   
   Given that you need to make a quick sync anyway, this approach is not much better than just using a ReadWrite-lock (it is slightly better because you can start updating before the readers finish reading (not a big deal for get()’s but potentially a big deal for iterators()), but then again updates are more expensive because of the copying).


2. I don’t think that you should be using Haskell as a model of “Enterprise-scale” anything. “Enterprise-scale software” usually entails “Enterprise-scale hardware” which means, among other things, multiple-CPU’s. The problem is that Haskell purely-functional model doesn’t support multiple-CPU’s (it’s true, check the literature (except for specialized pipelined architectures, but not in the general case)).
   
   The whole processing “state” is essentially one large purely-functional data-structure. The problem of synchronizing your data-structure appears to be eliminated, but it has really just been moved up to the top-level “state” structure (monad), where the problem is actually compounded. This is worse, because not only would you need to lock your structure in order to make an update, but you would in fact need to lock “the whole world”.
   
   Some people will advocate launching one process per CPU and then using message passing to communicate between them. This is just a very inefficient (many orders of magnitude slower) way of pushing the synchronization issue off onto the OS/Network-stack. (Q: What do multi-core systems mean for the future viability of Haskell?)


3. You forgot to mention the common technique of using “striping” to improve concurrency. Basically, what you do is create multiple sub-datastructures which each contain only a sub-set of the data. You can partition the data based on the hash of the key being stored. You then wrap the whole thing up so that it still has the same interface as the single data-structure.
   
   The advantage of this approach is that when you use a ReadWrite lock you only need to lock a small portion of the data-structure, rather than the whole thing. This allows multiple updates to be performed in parallel. This is how Java’s concurrent collections work. See: ConcurrentHashMap: Java creates 16 sub-structures by default but you can increase the number when even more concurrency is required.


4. Have a look at Azul’s non-blocking HashMap implementation. They can do 1.2 billion hashMap operations per second (with 1% of 12 million/sec of those being updates) on a 768 cpu machine (the standard Java ConcurrentHashMap still does half a billion ops/sec which isn’t bad either) . This shows how scalable non-functional methods can be.
   
   I’ve never read of any Haskell or other purely-functional implementation being able to scale to anywhere near these numbers.

Summary: Using a purely-functional data-structure does make it easier to support multiple readers, but you still need to sync briefly at the “top” of the structure. Striping has the added advantage of also supporting multiple-writers as well, and in practice, this is a much bigger win. Haskell is inherently limited to a single CPU, which doesn’t match modern hardware; especially of the “enterprise” class. Shared-memory models of concurrency have demonstrated much better performance than purely-functional models.

Best Regards,

Kevin Greer

p.s. I've actually implemented a b-tree structure for very large data-sets and chose to use a purely-functional data-structure myself. My reason for doing so was that I have some very long-running read operations (~8 hours) and I obviously can't afford a ReadWrite lock that's going to starve writer threads for that long.

Another nice advantage of using purely-functional data-structures is that they make it easy to implement temporal-databases that let you perform time-dependent queries.

I just wanted to point out that if all you have is quick-reads then purely-functional is no different than a simple ReadWrite lock and that a super-pure implementation, as with Haskell, doesn't scale to multiple-CPU's or to highly concurrent updates. However, it can be used to good effect in combination with striping or other techniques.

(A tangential note: Java's GC is getting so good in recent releases that P-F data-structures are becoming much more efficient (given that P-F generates more garbage).)

p.p.s. One more advantage of functional data-structures (the largest advantage for me actually):

They map well to log(or journal)-based storage. Functional data-structures never update old data, but instead, just create new versions. If your data-structure uses disk-based storage then this means that you never need to go back and overwrite your file; you only need to append data to the end of the file. This has two advantages:

1. This works well with the physical characteristics of hard-disks. They have incredibly high transfer rates (10’s of Megs/sec) but very slow seek times (~ 200 seeks/sec if 5ms access time). If you are adding say 1k objects to a data-structure which requires 2 updates on average to a file then you’re looking at about 100 updates per second. If on the other-hand you write all of these updates to the end of the file then you’re looking at say 20,000 updates per second!


2. You can’t corrupt old data with interrupted writes. The old data is always still there. The only place that a corruption occur is at the end of the file, in which case you just scan backwards until you find the previous good head of your data-structure.

---

This is fantastic stuff to share, thanks, Kevin!

What other tips can readers contribute for people building highly concurrent software (besides the frequent use of JProbe Threadalyzer, of course)? What online resources (papers, articles, books) do readers recommend for the professional developer?

Labels: java, passion, popular

  ¶ 2:17 PM 16 comments

  Having fun

I’m just going to keep trying to find interesting problems to work on, and if one of them happens to change the world, then great. And if not, at least I’ll have had fun trying to solve some hard problems.

Labels: passion

  ¶ 9:32 AM 0 comments

  Quote. Unquote.

I admit I live in a bubble. The thing is, my bubble is the one where they develop the technologies that you’ll be using in your bubble in ten years.

Basically, I hang around with people who are good programmers. I realize this is not a random cross section of computer users. But it is a disproportionately important subset, because what they use, other people will be using later. These were the people who were using microcomputers in 1980; now everyone is; the people who were using email in 1988; now everyone is; the people who were using the Web in 1995; now everyone is; etc etc.

Ok, take a deep breath. Forget about whether you find these statements arrogant or elitist. Forget about whether you think they are true or not today. Forget about whether they were once true (I know that on Saturday, November 9th, 2002, they were absolutely true).

The question I ask myself, and that I think you might want to try asking yourself, is this:

Am I developing the ideas, technologies, and products today that the rest of the world will be using in ten years?

If not, what’s stopping me?

p.s. Don’t miss the comments!

p.p.s. And of course, this is not an orginal thought on my part.

Labels: passion

  ¶ 2:59 PM 7 comments

  I wanted to be on the side that was right, even if I didn’t win

This is where I am great. I am great at being very, very stubborn and ignoring all sorts of reasons why you should change your goal, reasons that many other people will be susceptible to. Many people want to be on the winning side. I didn’t give a damn about that. I wanted to be on the side that was right, and even if I didn’t win, at least I was going to give it a good try.

Labels: passion

  ¶ 9:46 PM 1 comments

  Programmers are Sages Raymond Smullyan does much more than write about Infinity, Combinatory Logic, Recursion Theory, and Gödel’s Incompleteness Theorems. And he does much more than teach these concepts delightfully by turning them into stories and puzzles to solve. He also teaches philosophy by way of insightful essays and anecdotes.

I’m going to paraphrase one of his down-home insights. This particular little story has helped me through many dark periods in my life. Raymond was talking about Taoism, and he explained that his two pet dogs embrace the Tao. I think he described them as “Sages.”



Raymond explained that like many dogs, they preferred rich chunks of meat and gravy to dried kibble. And like many people, Raymond had tried making the kibble more palatable by mixing in some meat with the dry dog food and pouring gravy over the whole thing.

The dogs, of course, would carefully pick the meat out and lick up all the gravy, leaving nothing but the dried kibble behind. But did they resent this? Did they howl and whine about the kibble ruining the meat? Of course not! They would wag their tails and frisk about, joyous in finding all that good meat and tasty gravy in their bowls. It was like the kibble existed in some alternate dimension that the dogs cold not perceive.

Aaah… Dogs truly are Sages.

Kibble

Every time someone writes about interviewing and hiring, they are writing about judging people. That’s really deep and meaningful to us as people: being picked or not picked, picking or not picking is pretty much all of our social structure. So when you discuss it, you are reaching deep into people’s souls.

I know this, because it hurts me when I interview for a job and the response is “thanks, but no thanks.” It hurts when people say, “you don’t have the right kind of degree” or when they tell me that I forgot to use GROUP BY when writing a SQL query to sum the sales of automobiles by brand, or that they don’t think a Black engineer would go over well with Southern clients (all true stories).

There’s almost no way¹ to write about picking people without getting some heartfelt “thank you for recognizing my worth” responses. And some invective from people who feel you’re saying that you would pass them over. The fact is, there’s some unpleasant dried dog food in every post about interviews and hiring. Even if you pick something you think is obvious and unassailable.

Delicious chunks of meat!

Of course, some programmers just eat the delicious meat and ignore the kibble. Look! A tasty programming problem!

I’m one of these. What was my first thought when I saw a certain notorious screening question? I thought, “Hmmm… Instead of a loop with three conditions, this could be expressed as the composition of three functions, let’s try writing it that way.”² A tasty chunk of thinking about programming meat!

You know, it never even occurred to me to wonder whether I fail the test and be rejected and be cast out of the herd.³ I mean, honestly, was the original author interviewing me for real? No! So why eat the dry dog food? Honestly??

But in even the most trivial problem there’s some meaty thinking to enjoy. Composing functions. Yummy.

Covered in Gravy

Based on responses, the overwhelming majority of readers of this blog are Sages.

Most people respond to a mixture of dried kibble and meaty chunks friskily, wagging their tails and leaping about with programming solutions and discussion about how best to solve the problem. Yummy stuff indeed.

Some people might wonder if we’re playing Stairway to Heaven. Of course if you walk into a music shop and yell “nobody can play Portrait of Tracy,” everyone with a bass is going to start hitting harmonics.

That isn’t because we’re insecure and need to prove ourselves. It’s because we love playing and you just named a tune.

What can I say, I’m delighted with how many people love to play music. And I’m delighted with how many programmers love to program. For its own sake.

So thank you all for reminding me that life is all about the tasty chunks and delicious gravy.

I really appreciate it. Thank you. Especial thanks to Tom and Eric for constantly telling us all about the tender and sublime taste of Haskell. It’ll be on the menu one of these days.

Wonderful, the universe is covered in gravy!

---

1. Well, there is one way to write about picking people without offending anyone: if you really say nothing at all. For example: At NiceCo, we conduct our interviews as conversations. We discuss each candidate’s experience and accomplishments, we canvas their views and share our objectives with them, working collaboratively to identify the most appropriate contribution they can make to the team’s success.
   
   Now see, you can’t criticize that because it didn’t actually say anything. Maybe it’s a wonderful process involving nice people, or maybe what really happens is that they decide whether to hire you in the first thirty seconds based on some triviality like whether you have body odour. If you don’t say anything, there’s no kibble and no meat.
   
   It’s an empty dish.
   
   This does not mean it’s a bad process or a bad statement. But the process isn’t reproducible from the statement alone. That’s the exact sense in which I say that the statement is empty.


2. This separates the concern of how to generate the list from the concern of how to transform the list. Someone posted a variation with different conditions: you can simply generate a new transformation function by composing two of the exiting functions with two new functions, more or less. And if someone asked for the same thing in Ancient Rome, you could write a different generator that generated the numbers from I to C.


3. Don’t bother writing to say that my confidence is the reason why: I recall failing a technical interview when I brain cramped and was unable to write a program to print the first 100 prime numbers. I’m not that smart.

Labels: passion

  ¶ 1:23 PM 13 comments

  But Y would I want to do a thing like this?

Choose life. Choose a job. Choose a starter home. Choose dental insurance, leisure wear and matching luggage. Choose your future. But Y would I want to do a thing like that?

Writing about first-class functions and their compatibility with object-oriented programming naturally leads to the Y combinator. And that is the point where eyes glaze over and soft, snoring sounds rise from RSS readers everywhere.

But please bear with me, this essay is not really about the Y combinator, it’s about learning new things and expanding our capacity to think.

Sharpening the saw

Years ago I picked up Steven Covey’s book The 7 Habits of Highly Effective People. If the book is a test out of seven, I really wasn’t doing very well.

If you’ve read the book, you probably remember that he talked about “Sharpening the Saw,” investing in your own abilities. That’s incredibly important, but I don’t need to tell you that. If you exercise with programming katas, or learn a new programming language once a year, or pick up a book like The Reasoned Schemer and actually go through the exercises, then you are already in the top 1% of software developers for personal skills improvement. (Sorry, certifications don’t count. They are the classic case of doing the wrong thing for the wrong reason!)

New ideas—by which I mean, new to you—are an important way to sharpen your saw. If for no other reason than this: the brain needs regular exercise to perform at or near its potential. Learning new things keeps you sharp, even if you don’t directly use the things you learned.

Others have suggested that learning Lisp is beneficial to your programming skills in its own right. That’s one good way to sharpen your saw. But I add to that an important caveat: to obtain the deepest benefit from learning a new language, you must learn to think in the new language, not just learn to translate your favourite programming language syntax and idioms into it.

Think different

The interesting thing about that is that almost by definition, if you see something in, say, Lisp that solves a problem you already have, you won’t learn much from the Lisp code. It is tempting to think that Lisp (or any other language) will somehow do what you’re already doing in some wonderfully magic way that is obviously better. But no, that isn’t how it really works.

For your problems are tuned to your existing tools. You simply can’t imagine problems that your tools can’t solve well, much less can’t solve at all. That’s why there are so few continuation-based web servers. Who’s going to invent one unless they have a programming paradigm with continuations?

And worse, when a new tool is applied to a problem you think you know well, you will probably dismiss the things the new tool does well. Look at how many people dismiss brevity of code. Note that all of the people ignore the statistics about the constant ratio between bugs and lines of code use verbose languages. Look at how many people dismiss continuation-based servers as a design approach. Note that all of them use programming languages bereft of control flow abstractions.

Thus, to truly learn a new tool, you must not just learn the new tool, you must apply it to new kinds of problems. It’s no good learning to replace simple iteration with first-class functions: all you’ve learned is syntax. To really learn first-class functions, you must seek out problems that aren’t easily solved with iteration.

The Why of Y

Which leads me back to fixed point combinators. They appear to have no practical (as in making money) use. And that’s why I’m suggesting to you that you figure out how to make one in your language of choice. The very fact that the problem is far outside of your realm of “practicality” guarantees that you will learn something. You won’t be simply applying your same-old, same-old techniques and patterns to a slightly new problem.

Start your research with Richard P. Gabriel’s The Why of Y. Try porting his examples directly to your favourite programming language. If what you want to use is too brain-damaged to support closures, you may need to do a little greenspunning and build a little functor infrastructure.

Don’t be dissuaded if you have to follow the functor route: you are learning far more about your language and about programming in general than the shmoes that settle for learning five new buzzwords related to the latest WS-* interoperability with XPath 3.x.

If you prefer a fun approach to learning, you can do not better than Raymond Smullyan’s To Mock a Mockingbird: an enjoyable romp through the world of combinatory logic. After reading this book, you will have mastered the S, K, I, Y, and other combinators. Added bonuses include a safe that can only be opened by applying Gödel’s Incompleteness Theorem to its combination. How can you read this book and not learn?

Eating my own dog food

I thought of a few things to say along these lines last week and then I abruptly realizing I was asking you to “Do as I say, not as I do.” What good is recycling problems I first encountered in University textbooks two decades ago? I put this post aside and set to work on a problem of my own.

I set out to write a function for making recursive functions—a function extensionally equal to the Y combinator—in Ruby. The ultimate goal is to take something like:

lambda { |f| lambda { |n| n.zero? && 1 or f.call(n-1) } }

And be able to have it call itself recursively. In this case, to compute the factorial function.

This is trivial, given that Ruby supports named recursion, but if you want to write a fixed-point combinator you want to write a function that makes recursive functions without using the host language’s support for named recursive calls. In other words, you are bootstrapping named recursion out of anonymous first-class functions.¹

There are important theoretical implications of being able to do this, but the killer reason to try it is to learn.

I started my quest for a function for making recursive functions with a rather trivial observation based on OO programming and the Curry function:

require 'test/unit'

class ExempliGratia < Test::Unit::TestCase

  CURRY = lambda { |f, a| lambda { |*b| f.call(a, *b) } }

  def test_recursive_curry
    maker = lambda { |func_with_me|
      CURRY.call(func_with_me, func_with_me)
    }
    assert_equal(120, maker.call(lambda { |me, n| n.zero? && 1 or n * me.call(me, n-1) }).call(5))
  end

end

In OO some language implementations, this (or self) is a hidden parameter passed to each method. Thus, there’s a parameter—me in the example code—that is added for handling recursion. If you write a recursive function—like the venerable factorial—with the extra me parameter, a trivial currying operation evaluates it recursively without any need for names.

This is obviously deficient. As noted above, we want to write factorial like so:

lambda { |n| n.zero? && 1 or f.call(n-1) }

We’ll need an f from somewhere, and just as our Scheme colleagues do, we’ll bind one as a parameter in an enclosing lambda. So we want to write:

lambda { |f| lambda { |n| n.zero? && 1 or f.call(n-1) } }

And somehow this should be transformed into a working factorial function. For the test-driven crowd, we want to write:

def test_clean_up_loose_ends
  maker = ...

  factorial = maker.call(
    lambda { |f| lambda { |n| n.zero? && 1 or n * f.call(n-1) } }
  )
  assert_equal(120, factorial.call(5))

  iterative_factorial = maker.call(
    lambda { |f| lambda { |n, acc| n.zero? && acc or f.call(n - 1, n * acc) } }
  )
  tail_factorial = lambda { |n| iterative_factorial.call(n, 1) }
  assert_equal(120, tail_factorial.call(5))
end

Of course, we need some code for maker. And the iterative_factorial case shows that maker works for functions with more than one parameter. The solution I came up with is:

CURRY = lambda { |f, a| lambda { |*b| f.call(a, *b) } }
maker = lambda { |f|
       lambda { |func_with_me| CURRY.call(func_with_me, func_with_me) }.call(
            CURRY.call(lambda { |inner_func, me, *args|
                        inner_func.call(CURRY.call(me, me)).call(*args) }, f)) }

The source code with each transformation from beginning to end is here (I strongly suspect that this “curry combinator” is actually the Y combinator with a huge amount of cruft hanging off it).

Unique or derivative, crap or craft, the process of getting it to work has enriched my mind by forcing me outside of my usual problem space. I still can’t think of a practical application for what I’ve just written. But I know I’ve stretched myself.

And now back to you: perhaps you’re rushing off to try to implement a fixed-point combinator from first principles. Perhaps your plan is to code the canonical examples in your usual language. Those are both good paths. But whether you follow them today or not, remember the underlying principle exemplified by the fixed-point combinator:

Do not dismiss impractical or weird problems. While you may not have an immediate application for the code you write to solve such problems, you are maximizing your learning when you venture outside of your usual problem space.

---

1. Named recursion is stuff like foo = lambda { |...| foo.call(:bar) }. It takes advantage of the host language’s variable binding to recurse. If you want anonymous recursion, you should be able to assign the same lambda to another name and have it work just as well, as in: fufu = lambda { |...| foo.call(:bar) }. That won’t work if you are relying on Ruby’s name for foo.

p.s. Don’t miss Tom Moertel’s derivation of the Y combinator in Ruby.

Labels: lispy, passion, popular, ruby

  ¶ 12:06 AM 17 comments

  It's Monday, it's snowing, and I'm actually happy to be at work! I found this link on reddit.com:

“I’m happy to grow up, but I won’t pretend fun things aren’t still fun out of fear of looking silly.”

Ok, I didn’t laugh hysterically at the comic. But I love the sentiment. Raising my son in partnership with my wife is a lot of fun, and that involves a serious amount of growing up.

Why should programming be any different? Business is almost as serious as nurturing a life. It’s generally considered good form to enjoy parenting. Why shouldn’t we enjoy writing software the exact same way?

Sure, we don’t decide to rewrite an entire business application in CPS on a whim because it seems like it might be fun to try. Just as we don’t teach our children to eat Train Cake for every meal.

Let’s not pretend fun things aren’t still fun out of fear of looking silly. Programming is “Grown Up.”

And fun.

Labels: passion

  ¶ 3:43 PM 3 comments

  The false dichotomy of choosing between your values and expediency Wil Shipley:

Sometimes the most important thing in a fight is which side you are on, and not whether that side can win or not. Sometimes we just have to believe that, eventually, good will win out, and we have ask ourselves, “How will my children judge me when I tell them the story of this time?”

When my kids ask me about the first black man who really had a shot for the presidency, will I be forced to dissemble: ‘Well, see, kids, at the time, I didn’t think he could win, so it seemed expedient for me…’

This blog is not about politics, and it is especially not about politics in another country. If you are one of my American readers, I expect you to decide for yourself what is the right thing to do with your vote, time, and money. That being said, I quoted Wil because I admire something that he is saying there, something that is on topic for this blog.

Let’s start by dispelling one ugly myth about voting “strategically” (as we say in Canada) or for someone “elect-able” (as you say in America): It isn’t about the outcome of the election.

Let’s start by dispelling one ugly myth about voting “strategically” (as we say in Canada) or for someone “elect-able” (as you say in America): It isn’t about the outcome of the election. That’s a little white lie people tell themselves so they can do something they are embarrassed about.

They are really thinking “Here’s this thing that I ought to want to do, vote for the best man who happens to be Black, but—truth be told—I don’t want to do it. But I absolutely don’t want to admit I don’t want to do it, so… I’ll make up this story about expediency and how important it is for a Donkey to beat an Elephant, so I’ll nominate this other guy to run.”

Did you read my “Don’t Feed the Trolls” post? One of the key lessons was that people often are embarrassed about their real motivations for making decisions, so they dress them up in objections that they think sound reasonable to other people. Like winning elections. I’m going to come back to this in a moment. You are a stack-based reasoner, right? So PUSH the idea that what people say about why they are doing things rarely matches why people are actually doing things.

Let’s talk about software development

Think about your software development values. A value is something you are willing to pay for. I can’t tell you what your values ought to be, but I can tell you one thing that absolutely isn’t a value. Making money is not a value.¹ It’s an expediency.

Money is almost always a “reasonable objection to tell other people.” For example, taking a job with a consulting firm because “it pays more money.” When what you really want is the prestige of going into workplaces and telling people what to do. Without bothering to actually, you know, have any real industry experience first.

Synonyms for “making money” include “getting a job,” “sales,” “raising capital,” amongst many others.

Now we have already talked about people that have a value but can’t admit it so they tell a little white lie about “expediency.” It’s not just politics. Do you think our choices in software development are any different? Of course not. We are the same whether we are choosing mates, voting, or picking a programming language. So the way we choose is the same.

But there’s an extra dimension we are not considering. If you read The Wisdom of Crowds, you remember this idea of an information cascade, where information about other people’s choices actually makes the crowd more stupid, not more wise.

Synonyms for expedient include “backwards compatible,” “maintainable,” “easy to understand,” “easy to sell to management,” and many others.

The problem is this: when we see a bunch of people doing something, it seems like a reasonable, sane option. So do you want your fourth Java Certification? It seems reasonable, a lot of people are getting their certification, it’s an expedient way of getting a job.

Or let’s pick something that isn’t such a blatant straw man. Do you want to put your business logic into stored procedures instead of in your model objects? Or perhaps do you want to build yet another rich MVC web application instead of something REST-ful or perhaps one that is continuation-based (like a Seaside application)?

Really? Or is it just expedient to do it that way (synonyms for expedient include “backwards compatible,” “maintainable,” “easy to understand,” “easy to sell to management,” and many others).

The problem with the little white lies

So you’re a reasonable person, and you see a lot of other reasonable people doing something because it’s expedient. You value something, but you are wondering whether you ought to compromise your value in favour of expediency. Life is tough some times, and you have to grow up. What’s wrong with that?

Here’s where we go to the stack: POP. Remember, lots of those other people aren’t actually doing the expedient thing when they say they’re being expedient! They’ve got a whole ‘nother motivation, but they lied about that motivation to you. It isn’t a real reason, they aren’t being honest with you (or themselves) about why they’re doing it, and you go along with it thinking it’s a real, valid option. But it’s smoke and mirrors for other people’s frailties (bless them, they are as human as we are).

For example, the argument that BDUF is the expedient way to manage a project? Look, I’ll level with you: if what you want is to demonstrate to your manager that you have everything under control, that you are in charge, then go for it. But expedient? Sorry, that’s bunk. It’s just a little white lie a lot of people have told you because they fear telling the truth, like:

• They have no idea how to manage software development activities, so forcing developers to specify everything in advance relieves their anxiety that they will have to actually manage things as they go along. Instead, they can just follow “The Plan.”


• They have no idea how to manage software development to a deadline, so forcing developers to estimate every activity in advance—no matter how far the activity is away—relieves their anxiety that they will have to ever explain whether a project is on target for release or not. Instead, they can simply parrot how much of “The Plan” seems to be complete and whether actual times for activities are ahead or behind estimates.


• They have no idea how to manage stake holder expectations and priorities, so forcing stake holders to specify all of their requirements in advance relives their anxiety that they will have to engage stake holders in a continuous and open dialogue over the life of the project.


• The road to senior management in their organization is paved with process documentation, not successful projects.

Yes, we have to grow up and consider expediency. But we also have to grow up and learn that other people are human and don’t always tell us the whole truth about why they do what they do. Which means it is difficult to judge whether something is right for us based on what other people say is right for them.

Be careful when presented with a choice between what we value and what is “expedient.” This choice is almost always a false dichotomy: the “expedient” option is usually a little white lie someone else has told us because they think it will sound reasonable.

We need to be very, very, very careful when presented with a choice between what we value and what is “expedient.” This choice is almost always a false dichotomy: the “expedient” option is usually a little white lie someone else has told us because they think it will sound reasonable, even if it isn’t what they were thinking.

Therefore my exhortation to you is this: If you really, truly want to program in Lisp, or manage your next project with BDUF, or write unit tests before you write code, because these actions reflect your values, then go ahead. I honour you for your choice, even if it’s different than my choice. Rock on!

But… if you really, truly want something else but are doing the expedient thing because that’s where the money is, or the jobs are, or there’s safety from criticism, or any of a hundred or a thousand or ten thousand little white lies, stop it right now. You need to understand that the chattering voices you hear, the ones that sound so reasonable (“Zune sucks, but MSFT always wins in the end, so buying a Zune is expedient”) don’t even reflect what those other people really think or want.

They’re just little white lies other people are telling you because they don’t have the courage to tell you the truth about themselves.

---

1. If you’re reading this blog and telling me that you—to pick just one example—choose to program .NET because “that’s where the money is,” you are misguided. Sorry, but the money is actually on Wall Street or Bay Street where even the poorest analyst makes more money with convexities and gammas than you or I will ever make with objects and tables.

Labels: passion

  ¶ 2:40 PM 7 comments

  Children should be making things, communicating, exploring, sharing, not running office automation tools

One of the saddest but most common conditions in elementary school computer labs (when they exist in the developing world), is the children are being trained to use Word, Excel and PowerPoint.

I consider that criminal, because children should be making things, communicating, exploring, sharing, not running office automation tools.

Labels: passion

  ¶ 9:59 PM 0 comments

  A Fellow Traveller's Road to Enlightenment

I began to see that many of the sacred GoF design patterns were not, in actuality, grand universal truths of software engineering, but simply collateral damage caused by incidental limitations in the abstractive power and object model of certain manifestly-typed programming languages.

A wonderful read.

Labels: passion

  ¶ 11:32 AM 2 comments

  Where were you on Saturday, November 9, 2002? Had you dropped by MIT on November 9th, 2002, you might have seen a rather large collection of hackers, academics, tool vendors, and programming language enthusiasts gathering for the LL2 conference. It was rather like taking the current readership of Lambda the Ultimate and programming.reddit.com—no, I am wrong.

It was rather like taking the top couple of hundred of the people who write papers and blogs cited on Lambda the Ultimate and programming.reddit.com and gathering them in an auditorium together to talk about programming languages.

As the rather Spartan LL2 web page explains, the theme of the conference was:

How to get usable and useful programming tools into the hands of programmers, minimizing dogma, and maximizing flexibility.

Like any other conference, there were speakers. The speakers talked about the languages they had invented, and about how and why people ought to use them.

Had you been there, what would you have thought about the first speaker, Joe Armstrong, and his language, Erlang? Just another weird language for embedded telephone switches? Or an important language that is powering diverse applications such as a distributed, replicating database?

Joe spent a lot of time talking about unreliability. About how computers and programming languages are always designed as if everything always worked and that not working was the exception case, rather than the rule. And about how you can’t build systems out of dozens, hundreds, or thousands of pieces without assuming that everything is unreliable, all of the time.

And how there ought to be a programming language that builds that right into the system. Like Erlang. Would your eyes have glazed over? Or would you have thought about MapReduce and grid computing and would you have taken notes furiously?

Would you have thought that two different talks (one about persistence and another about asynchronous exceptions) were too much for an obscure language called Python? Would you have marked Python down as “One to Watch”?

And what would some random guy know about inventing “the next big programming language”? Oh, he was managing Microsoft’s Programming Languages Systems group. Well, there you go, one guy that Corporate America might care about. I wonder how many of his predictions about the next big programming language have come true? Good thing his talk is on line, we can check…

After lunch, some random guys talk about getting an un-typed language running on top of the Java Virtual Machine. I mean, really, who the hell would think that would ever become important?

Next up, the organizers had flown Yukihiro Matsumoto in to talk about Ruby. Yes, in 2002 the organizers wanted a bunch of Lisp-heads—the Q & A for absolutely every talk included at least one question about when macros would be added to the speaker’s language—to hear from someone who had designed a language known at the time as the most popular scripting language in Japan. Had you heard of Ruby in 2002? Did you care?

And what was with him constantly talking about making programming fun and comfortable? As if a programming language was a user interface for programmers. Everyone knows that programmers love idiosyncratic tools like Emacs, right? Who designs a language for usability? Who indeed.

There were other talks. They were all good, just as good. There was a spirited exchange of ideas over lunch in the lobby, again upstairs in the MIT researchers’ offices, and continuing late into the night at a nearby pub. It was a phenomenal gathering.

Were you there?

That isn’t important right now. If you were there, good for you, and I hope in the four years since you have taken some of those ideas, and a lot of that enthusiasm, and rolled it into what makes this industry great. The fact that anyone—even some guy from the other side of the world—can make something that other people like and use. Whether it’s free or for pay, whether it’s a language, a tool, an application, or a piece of hardware.

But even if you were there, that was then and this is now. 2007. Here’s the important question: Where is LL2 this year?

I asked “where is LL2 this year.” Actually, it may not be a Lightweight Languages conference. It might have nothing to do with programming languages. But somewhere in the world, somewhen in 2007, there is going to be a conference. Or a gathering. Maybe it’s this year’s Startup School. Maybe it’s a DemoCamp in your city.

Wherever, and whenever it is, you are going to see and hear things that are going to matter to you over the next five years.

They may not look like much in 2007, but they’re the seeds of the next Big Thing, the thing that will be big four or five years hence, in 2012.

Finding the conference

In 2002, LL2 didn’t look like much. You probably couldn’t find a job using any of the languages discussed at the conference. Unless you wanted to work on telephony. The lone representative of a mainstream company—Microsoft—wasn’t talking about their products, he was talking about things that would disrupt their products.

So if you’re looking for the right conference in 2007, there are two clues: the agenda isn’t going to include anything you can use to make money, and it isn’t going to feature anybody trying to make money off of you.

That’s right: no consultants, no white papers, no thinly veiled sales pitches. That knocks most Ruby conferences out. Maybe not all, but most. The right conference isn’t going to be about building CRUD applications on top of MySQL with AJAX goodness. That may not be out of date, but it’s so 2007, and we’re looking for 2012.

In 2002, the audience was highly interactive. Most of them were working on their own projects. They were practitioners, not tourists. They weren’t there to get a job, or goof off work. They were there to learn stuff for themselves and share what they knew with others.

In 2007, you should be looking for the same thing. For a conference that will be attended by people who are, as Joel would say, Smart and Get Things Done. Lacking a way to check the door before you show up, how do you find out?

Naturally, you find out by talking to other would-be attendees before going. LL2 had LL1’s active mailing list. In 2005, Startup School had a wiki. I don’t know what the right conference in 2007 will have, but I will bet on this: if there’s no community building in advance around a wiki, a list, or some other peer-to-peer communication, it isn’t the right conference.

The kind of conference where there is a one-way exchange between you and the organization—they tell you where to show up and you send them money—is not the right kind of conference. Look for the conferences that are built on top of active communities.

And when you find the community, pay attention to the action-to-talk ratio. I’m terrible at this, I always have to push myself to work and stop goofing off on my blog. But you know what’s worse? People who don’t have any work to begin with, it’s all just entertainment to them. They are tourists, and they are a liability to a conference.

Here’s why.

When tourists reach critical mass, the organizers start to cater to them. That means safe topics and safe speakers. Consultants. People with a good “patter” but not much to say that would disturb anyone.

Also, the tourists tend to ask the wrong questions, because they aren’t the market for anything new and interesting. Say a tourist raises her hand. She’s going to ask about something she imagines might concern a user, but you know what? She isn’t a user and she really has no idea how a user thinks, because she’s not the type to learn anything new until it’s already popular, and she doesn’t have any idea how such people think.

I think the root of your mistake is saying that macros don't scale to larger groups. The real truth is that macros don't scale to stupider groups.

So she asks something irrelevant. Like about language performance. Or bindings to Oracle. Or whether there is an Eclipse plug-in. Or how it will scale to larger programming teams.

But when a practitioner asks a question, it’s a smart question, it’s important. The speaker learns something important about their tool and the rest of the audience learns too. The next thing, someone asks a smarter question, and the energy builds.

If there are too many tourists, you can’t learn anything significant in the Q & A, and that’s a shame, because Q & A is where the action is.

So, my advice for finding the right conference is, avoid anything to do with either you or the presenters making money, find a conference built on top of an active community, and go the the conference with the highest density of practitioners.

And my question really ought to have been, will you be at this year’s important conference? and better still, will you be presenting?

Labels: passion, ruby

  ¶ 4:57 PM 5 comments

  Never stop learning

I think every program you write should be the hardest you’ve ever written.

Labels: passion

  ¶ 7:54 AM 3 comments

  Does the term "Exponential Growth" mean anything to you? Assaf laid the tag-smack on me, so here are five little known facts about me. This blog has a strict theme, so all five are “nerdy”:

1. “My” first computer was the High Speed Job Stream in the Sanford Fleming building at the University of Toronto. It was an IBM something-or-other, but there was no restriction on running punch card jobs in SNOBOL, LISP, or WATFOR. I was twelve at the time, so please don’t be shocked when I say that it took me a few weeks before I became bored of printing calendars with line-printer art of playboy models and progressed to using Fortran to print the multiplication table and Fibonacci series.
   
   My second computer was a Nova 1220. I wrote BASIC programs for it and especially my own highly customized Star Trek game. It needed all 16K of RAM, it was so big. The listing was fourteen feet long. More than a few people tell me with pride that they work on Enterprise software that would probably be fourteen miles long if anyone ever printed all of the source. I change the subject by telling a joke about a truck. Yes, those are front-panel switches for bootstrapping the CPU. Yes, I learned the three 16 bit words needed to obtain an audience with the demon.


2. I grew up in the sixties, when there were still race riots, when companies routinely discriminated against women because “they would just quit to start a family,” and when Toronto hosted a great number of young American men who didn’t want to fight in Vietnam. I helped my parents prepare for protests by colouring placards. I lived on a commune for a short while and I went to “Young Socialists” camp in the Summers.
   
   Although this seems unrelated to nerdy things, I think anyone reading what I have to say about coding as a form of self-expression and about the importance of the programmer in the production of software will see the unbroken thread that has run through my life.


3. My first job was in “Mr. Gamesway’s Ark,” a games and puzzles superstore way ahead of its time. My second was as a haberdasher. my career has oscillated in the same fashion ever since: the pursuit of beauty for its own sake (in math or in dress), the social side of nerdery, and sales. Software development in a startup environment seems to deliver all of the goods in one load. And when someone gives me a friendly tip to “wear a blue suit to the presentation,” it’s always nice to be able to ask in return “the Chalk Stripe or the Plain Navy?”


4. I dream in black and white, although from time to time I am aware of a colour in my head: I know in the dream that something has a colour, but I cannot see the colour and this seems very normal. Like most of humanity, I remember having flying and levitation dreams as well as helplessness dreams.


5. I am an autodidact. This is probably obvious to anyone with a real (as opposed to JavaSchool) education in Computer Science.

Lucas, Chris, Reinier, Rusty, and Dharmesh, you’re it. The ‘rules’ appear to be: five personal trivialities and tag five more victims. Good luck finding five who haven’t already been tagged by this generation of growth!

Labels: passion

  ¶ 3:19 PM 1 comments

  A Ruby, Io, and Ocaml Programming Pattern

Every 5 minutes you spend writing code in a new language is more useful than 5 hours reading blog posts about how great the language is.

Labels: passion, ruby

  ¶ 1:38 PM 5 comments