The only workable system for generating interesting, cool, and relevant software

The only workable system for generating interesting, cool, and relevant software is well-known. Find a bunch of really smart programmers and point them at a large problem space for which there are actual users, and for which new solutions are unconstrained by old designs.

Note the desired outcome: Interesting, cool, and relevant software. Do not become mired in empty debate over the phrase “smart programmers.” Instead, ponder the other three preconditions: a large problem space, actual users and most especially unconstrained by old designs.

Alan Kay is purported to have said, “A fresh perspective is worth 80 IQ points.” You cannot have a fresh perspective when you are encumbered by backwards compatibility.  ¶ 11:03 AM 7 comments

  Is software the documentation of business process mistakes? Some time ago, I learned an amusing but insightful maxim:

The user manual is a list of software design mistakes.

(It may have been from The Design of Everyday Things, I don’t remember and a cursory web search hasn’t been helpful.)

Well-designed things may not be perfectly discoverable and consistent and stable, they may not be obvious and familiar, but in general, if something needs a lot of explanation, it may need rethinking.

Note that this does not mean that you can’t introduce something new that needs explaining. But overall, you are seeking economy in the user manual. So you want to introduce as few things that need explaining as possible. And you want your explanations to be as simple as possible. And you want the implications and consequences of those things to be obvious and without lots of caveats.

To borrow a phrase, you don’t want leaky abstractions. Leaky abstractions don’t need much explaining up front, but the caveats and exceptions and trivia you ought to know (like the maximum depth of call stack in some languages) will inflate your user manual into a tome. Well-chosen and well-implemented abstractions, on the other hand, need a little explaining up front, but after that they are ‘discoverable’ and need little or no further caveats.

So the point of the maxim is to seek overall economy of explanation. This is not particularly novel or insightful, of course. The utility of the maxim is remembering that the complexity of the user manual is an imperfect but helpful proxy for complexity of the software.

I have certainly applied that approach to designing things myself: If I need to explain it over and over again, if I need to describe how all of the parts interact and why this button changes the colour unless the current mode indicator is set to resize, in which case it zooms… maybe I need more time designing and less time documenting.

So my teeny bit of advice here is to always think about the user manual when designing something. If it is hard to document, it is hard to use. That definitely goes for code: if it is hard to explain, it is probably hard to maintain. That’s a great argument for writing documentation: The act of documenting something forces you to see it from the other person’s perspective.

I imagine all of us can get together on this. We may disagree on whether Ruby Metaprogramming makes something harder or easier to explain, whether andand helps or hurts, but the basic principle seems robust: if you sit down to explain it, you will get a feel for how it looks to someone who hasn’t been thinking about it non-stop for several days or weeks.

Could software be a list of business process mistakes?

Ken Tilton just wrote an excellent post that touched on this subject from a new angle: “We Can Live With the Way You Handled That”. Ken’s story is entertaining and well worth the full read (I am only going to talk about one element of the post, but his anecdote really illuminates the business software development process: please give it your full attention.)

The thing that struck me like a gong about Ken’s story is that software in business is a kind of documentation for business processes! Business processes are often these gnarly heaps of rules with massive omissions, contradictions, and bits where what people say they do has no relation to what actually happens. Writing a piece of software to augment or automate the process forces people to think hard about the process.

In my experience, good managers know this. Quite often, the real motivation in automating a business process is as an indirect way to force this review. This is why writing business software is called consulting. This is why it rarely works well when outsourced to a firm in another time zone: the problem is not writing 50,000 lines of Java. Nor is the problem writing 7,500 lines of Ruby.¹

The problem is exactly the same problem as writing documentation for a feature: The problem is realizing that if the software to automate a business process is complicated and contradictory and hard to use, then the real cause is a business process that is complicated and contradictory and probably not serving the company well.

Good software has succinct and easy-to-understand documentation. You can’t bring a genius technical writer in to turn a sow’s ear into a silk purse. Likewise, good business processes have succinct and easy-to-understand software automating or augmenting them. It’s the same thing: you can’t bring in a genius software architect to craft silk software for porcine processes.

So in the end, when you are writing a piece of business software and it is gnarly and contradictory and full of exceptions and holes and it is impossible to understand… that may be a sign that you are not writing a very good piece of software. But it may also be a sign that the underlying business process is gnarly and contradictory and full of exceptions and holes.²

You may need to push back on the requirements and foster change in the business. Of course, that may not be possible. Good managers are as rare as good software developers. But nevertheless, if you have the business’ best interests at heart, you ought to try to fix the problem and not paper over it with code.

We programmers talk about getting rid of accidental complexity so that our code contains only essential complexity. And business processes are what we think of as essential complexity. Fair enough. But business processes have accidental complexity as well. Business processes suffer from poor maintenance over time. Business processes need to be refactored as well, and when you have a project to automate a process, you have a certain amount of freedom to challenge the process, to prod and poke it, to question it. Don’t assume that you have to take it as entirely given, a set of requirements cast in stone.

Remember, always, that our job is to add value, not just to cut code. And like so many other places in software, sometimes our best code is the code we didn’t write. Identifying a broken process that leads to simpler software could be be best value you can contribute to a project.

---

1. Writing 7,500 lines of anything instead of 50,000 lines of something else is still useful, if for no other reason than with 50,000 lines of something, it is hard to know whether there are 50,000 lines because the business process is unnecessarily gnarly or whether there are 50,000 lines because the programming language is full of accidental complexity. But if you have another language that could express the same thing in 7,500 lines, gnarliness in the business process is readily apparent.
   
   [back]


2. There could be another entirely different reason why a piece of software that is “gnarly and contradictory and full of exceptions and holes, and is impossible to understand” may be a sign that the business has problems: Conway’s Law.
   
   [back]

  ¶ 5:00 AM 12 comments

  Springtime Sunny Saturday And I find myself writing:

$<.each do |line|
  $> << line unless line.strip.split(",")[2].gsub('"','').strip.empty?
end

For some actual work. What else could be on my mind except golf?  ¶ 4:11 PM 5 comments

  What to do about Imperative Programming Robert Fischer was kind enough to point out the simplest and most effective cure for the difficulties of Imperative Programming. However, his advice was terse and obvious, and while simple truths are often correct, they rarely persuade or entertain without a consultant’s study of some kind.

Since it is the evening before a major holiday, and neither of us want to read a lot of boring exposition, I have asked Mo Collins and Bob Newhart to act the whole thing out for you as a sketch:



All my very best to you and the ones you love. Have a great weekend!  ¶ 5:17 PM 2 comments

  Acid Tripping, Disneyland and Special Relativity

malcontent: …Lisp guys know all about monkeypatching and dynamism. Those crazy mofos write mind bending macros and morph the language in ways that can only be described as an acid trip inside the ‘it’s a small world’ ride in disneyland…

earthboundkid: Python is a language for pedants. That’s exactly why I like it. There’s (almost) always one “correct” way of doing something, and when you do it that way, you get that warm glow in your heart that we pedants get when we outsmart someone or something. When it’s like when you correct a teacher in class or find a grammar/spelling error in your opponent’s message board post. That’s why Python is “fun” for us.

For Ruby heads, the fun is in solving the problem in some crazy idiosyncratic way that makes sense to you but seems backwards to everyone else. See the insane andand thingamajig that Raganwald invented. That’s why Ruby includes an unless keyword, just in case. That’s why Ruby has _why.

So yeah, totally different communities for two remarkably similar languages. At that point, it becomes the narcissism of small differences.

malcontent: …If you think andand is insane, try to read some Paul Graham macros.

You can decide for yourself whether bottom-up programming in Lisp is like either or both of an acid trip or a Disneyland ride. And I know a few Python programmers who seem to be able to enjoy programming without the epicaricacy of declaring that their colleagues are ‘wrong.’

Now when it comes to ‘Ruby Heads’… Well, that expression obviously need not be applied to every Ruby programmer. It implies someone who is immersed in the language, someone who reflects upon its use and who actively participates in its evolution. So there may be a kernel of truth in earthboundkid’s hyperbole.

Speaking for myself, I distinguish between andand’s interface and its implementation. With respect to its interface, I was attempting to create something that makes perfect sense to everyone. Sometimes the motivation is to get people to look at something and say “That looks odd… Let me think about it… Hmmm, that actually makes sense once I think about it.”

My own code is no Special Relativity, but for me that is one of the greatest theories ever invented: it doesn’t make you learn something entirely new unlike anything you’ve ever seen before: Instead, Special Relativity makes you rethink the consequences of the things you already know about inertia and the fact that measuring the speed of light always provides the same constant result.

That’s a big motivation for pushing programming languages and programming styles. Not so much to create something entirely new (“Look! The code flows from bottom to top and right to left!”), but to provoke us to understand the implications of the things we already know.  ¶ 10:55 AM 2 comments

  Spaghetti-Western Coding
In case you didn’t recognise it at a glance, this is a metaprogramming gene: it repairs DNA in Methanoculleus Marisnigri, an anaerobic organism found in decaying sediment at the bottom of the Black Sea. Metaprogramming genes‽ Yes indeed, the Genetic code embraces metaprogramming with enthusiasm: DNA splices itself, moves code around, and rewrites code. Sections of code are turned on and off. Genes manufacture proteins that express or suppress other genes. And I am grossly oversimplifying things!

The frustrating thing about genetic code is that it is really, really easy to understand what it does in the small. Amino Nucleic acids come in triplets called codons. In the gene above, the first codon and last codon are beginning and end markers, like curly braces in Algol-derived programming languages. The other codons code for amino acids. The entire template manufactures a small piece of RNA. This mechanism is very well understood at this scale of behaviour. However, when you build from the DNA up to the whole organism, the overall behaviour is a mystery, just as the behaviour of snow is very difficult to predict from the molecular properties of water and air.

Software programs exhibit the same frustrating property: although we can easily determine what individual expressions do, the behaviour of programs in the aggregate resists simple analysis. Consider:

p[++i] = 0;

What does this do? Well, in C++ anything is possible, but a simple interpretation—incrementing an integer and then using the result to assign zero to an element of an indexed collection—is well understood and the probable meaning of this program in many languages.

The program that includes such a line has state, and this line of code makes a very small alteration to that state. What effect does this line of code have on a program’s behaviour? Well, now we have a conundrum. You often find this code inside of a loop. But if we make some error in our loop boundary checking, we can start overwriting some other thing—like our program’s code—with zeroes.

This is the basic problem with state machines: a small change of state can have unpredictable consequences on the behaviour of our machine.

We have a blueprint for a state machine that writes a blueprint for another state machine. So we are looking at a blueprint of a state machine that makes blueprints and we are trying to predict the behaviour of the state machines it builds.

This is roughly like looking at Bach’s DNA and hearing the Little Fugue in your head.

There is a much more familiar expression we use to describe programming machines with mutable state: We call it “Imperative Programming.” And we have discovered that it matches the way our brains think about many algorithms really well, that’s why we (statistically speaking) resist purely functional approaches. However, we have known for a very long time that imperative programs have this awful property: a small change of code can result in unpredictable changes to the behaviour of the running program.

So here is my analogy: an imperative program is a blueprint for a state machine. We put the blueprint inside of another machine we understand really well, and it manufactures the state machine for us, then starts the state machine. Our problem is that we are supposed to understand the behaviour of the constructed machine as it changes state just by looking at the blueprint.

And if you think that is hard…

Wait! Let’s make the problem harder. What if we have a blueprint for a state machine. What does our state machine do? It writes a blueprint for a state machine and feeds that to a machine-making machine, which then gives us back another state machine. So we are looking at a blueprint of a state machine that makes blueprints and we are trying to predict not only its behaviour but the behaviour of the state machines it builds.

This is roughly like looking at Bach’s DNA and hearing the Little Fugue in your head.

Does that sound easy? No? Well what do you think we are doing when we write code what writes code? Here is a standard Ruby template for writing modules:

module Example
  module ClassMethods
    #...
  end

  module InstanceMethods
    #...
  end

  def self.included(receiver)
    receiver.extend         ClassMethods
    receiver.send :include, InstanceMethods
  end
end

You are looking at a program that write programs, a blueprint for a machine that makes blueprints. Metaprogramming is imperatively programming an imperative program.¹ Now, a program is a kind of amplifier: a short program produces a big result. So when we write a program that writes programs, we are amplifying the amplification.

The spark of a match. A long, lingering draw on a cheroot. And a narrowing of eyes.

Consequently, metaprogramming amplifies the good, the bad, and the ugly of imperative programming.

In and of itself, metaprogramming is ‘just’ an amplifier. If you have issues with imperative programming, you are going to have issues with metaprogramming, only magnified. I don’t know about you, but I certainly have issues with imperative programming. And I am not alone: the history of programming languages has been dominated by attempts to tame the basic complexity of mutable state in programs.

Structured programming attempted to tame the arbitrarily complicated program flows—known to this day as “spaghetti code”—resulting from the indiscriminate use of if and goto statements. Object-oriented programming attempted to tame the arbitrarily complicated data states resulting from the indiscriminate combination of procedures and data structures. Procedure calls, continuations, methods, … the idioms and language mechanisms we’ve devised are all attempts to make simpler machines that are easier to understand.

Of course, there is no simple answer. Banning metaprogramming makes exactly as much sense as banning abstractions in imperative programming, and for exactly the same reasons. If you ban abstractions from imperative programs, you assert that longer programs made out of simpler pieces are more tractable than short programs made out of abstractions. Sometimes they are, sometimes the abstractions are gratuitous, and sometimes the mechanisms used to make a program short are incidental to its purpose, almost as if you compressed the source code and insisted that it must be easier to read because it has fewer redundant bits.

I’m personally optimistic that we will find reasonable idioms and mechanisms for managing the complexity of metaprogramming. Some of the idioms we’ve already found are promising, and as long as we don’t succumb to Blub and decide that there is no further improvement possible, we can only get better.

We seem to be doing a decent job with the style where we place our metaprogramming in a separate place from our program’s core “business logic.” Rails accomplishes this with plug-ins: not only can you use lots of other people’s plug-ins, you can deliberately place your own metaprogramming code in a plug-in to keep it separate from the application code that calls it. You call acts_as_versioned in your class, and the plug-in does the rest. Although you are using metaprogramming, your program fakes a declarative style quite effectively.

Blueprint Transformers

When I said that IS-A IS-A HAS-A, a few people pointed out that IS-A isn’t HAS-A: In actuality, ISA-A WAS-A HAS-A. Isn’t that the point of true abstractions? To provide the illusion of a new set of semantics implemented using existing semantics?

proof("Proof of Relationship").where {
  document("Affidavit of Same-Gender Domestic Partnership").where {
    Both_names = item("Participant's Name") &
                 item("Partner's Name")
  }
}

proof("Proof of Joint Ownership").where {

  document("Property Statement").of_kinds(
      "Rental/Lease Agreement",
      "Deed",
      "Property Tax Statement"
  )

  document("Financial Statements").of_kinds(
      "Mortgage Statement",
      "Credit Card Statements",
      "Bank Statements",
      "Utility Bill Statements"
  )
}

Syntactically this is Ruby code, but semantically it is something else altogether (If it matters to you, it is a small snippet of a system for defining a decision tree used to determine loan eligibility). This is possible with metaprogramming by its very nature: You are working with a machine that takes a blueprint as its input and produces a blueprint as its output. In effect, you are working with a blueprint transformer. Mediating between different semantics is an obvious application for this technique.

This is great, and I am a big booster of the power of metaprogramming. But just as we are loathe to “throw out the baby with the bath water” by banning techniques just because they can be misused, we should likewise be loathe to embrace techniques as golden hammers just because they have some extremely beneficial applications.

For all of its wonderful utility, metaprogramming is, at its heart, the act of writing programs that write programs. Which means that understanding our programs requires predicting what a program will do given a blueprint for a machine that will write a blueprint for our program.

This is not a trivial exercise. And yes, you can shoot yourself in the foot if you aren’t careful. But in the end, there’s two kinds of programmers in the world: those with dangerous techniques and those who dig.

---

This post was conceived during the Q&A after Andrew Hessel’s talk at SciBarCamp.

1. (A few people missed this post script, so I have made it an end note):

Yes, I realize that “metaprogramming” is a vague term with may possible interpretations right down to the weak assertion that writing named procedures is metaprogramming. And I also realize that there is much, much more to programming than imperative programming.

But please don’t allow the presence of this disclaimer to discourage you from comments about other forms of metaprogramming that do not fit into the neat box of state machines building blueprints for state machines. I am especially interested in hearing from functional programmers: what is the functional equivalent of the metaprogramming described here and how does it differ in character?

[back]
  ¶ 11:30 AM 10 comments

  An open letter to Jeff Atwood

If you decide to run with the ball, just count on fumbling and getting the shit knocked out of you, but never forget how much fun it is just to be able to run with the ball.

Good luck with your adventure, Jeff.

---

“Running with the ball” is sometimes about daring to launch your company and risk failing. And sometimes about daring to write your ideas on a weblog and risk scorn. And sometimes about learning a new programming language and risk feeling incredibly stupid trying to grasp its ideas. And sometimes about trying a new development practice. And sometimes about asking for love and risking rejection.

And, and, and… and so many other moments in life where you feel like you are really alive, like you are so excited that your pants are on fire, so terrified that you will fall and never get up again.

But it’s fun to run with the ball. So please, be it a grand gesture or a single, small change in your life, please, run with the ball today.  ¶ 10:23 AM 3 comments

  IS-A IS-A HAS-A

All of the pain caused by inheritance can be traced back to the fact that inheritance forces ‘is-a’ rather than ‘has-a’ relationships. If class R2Unit extends Droid, then a R2Unit is-a Droid. If class Jedi contains an instance variable of type Lightsabre, then a Jedi has-a Lightsabre.

The difference between is-a and has-a relationships is well known and a fundamental part of OOAD, but what is less well known is that almost every is-a relationship would be better off re-articulated as a has-a relationship.

I’m in the middle of dealing with some ActiveRecord, um, ‘features,’ so I don’t have time to write an interesting, thoughtful essay. Nor do I have time to write an overly long and meandering essay as Raganwald readers have come to expect. So instead, let’s pretend there’re six or seven closely typewritten pages of self-indulgent bloggery leading up to…

IS-A IS-A HAS-A

When classical OO languages (including Ruby and Java) implement inheritance, what actually happens is this: First, the subclass acquires the interface of the superclass (latently typed languages like Ruby don’t do this explicitly, but it is still true). Second, under the hood where you can’t see it, the compiler makes room for a reference to the superclass and arranges for methods that the subclass does not implement to be delegated to the superclass.

In other words, whenever you declare that Child IS-A Person, the compiler writes “Child HAS-A Person” and “Child BEHAVES-LIKE-A Person” and “Child DELEGATES-TO Person” for you.

The right way to look at single inheritance is therefore that it is a specialization of composition. Like all specializations, it does less. It is more specific. And therefore, when it is exactly what you mean to accomplish, it is more useful: you need fewer lines of code to express it, the compiler optimizes the relationship for you, and since it is a very well-known idiom, it expresses your intent clearly to other programmers.

On the other hand, when single inheritance is not exactly what you mean to express, you have two choices: you can greenspun some more general feature out of it, adding accidental complexity to your code and making it less readable, or you can use the more general purpose, flexible tool: composition.

Like all Golden Hammers, seeing every OO problem as a single inheritance nail leads to trouble. Single inheritance is probably Turing Equivalent: given enough determination, you can probably write any arbitrary program if you are prepared to twist your code into knots as it bows down and sacrifices readability to the inheritance demon.

However, breaking free of its grip is simple: you need only see inheritance as a tool like any other tool: it is not superior to the others at your disposal. It is not the goal. A nice pyramid of classes and subclasses is not your objective when you set out to design a program. When you see it as just another chapter in the book, you see when to use it and when to set it aside.

And in fact, if you see it as a particularly thin chapter in the book, you may be surprised how much easier it is to work with OO programs. And if you see it as a couple of pages in the chapter on composition and delegation… you will be on the road to understanding.  ¶ 4:25 PM 23 comments

  Drive and determination can most often be found in the slums

I’ve written before about how you can count the number of games written in a purely functional style on one hand. Is it that language tinkerers are less concerned about writing real applications? That they know you can solve any problem with focused grunt work, but it’s not interesting to them? That the spark and newness of a different language is its own reward? Either way, the BASIC programmers win when it comes down to getting projects finished.

Ouch. Now I must get back to work on real programming and stop blogging and farting around with abstractions.  ¶ 1:21 PM 5 comments

  It's not just closures: Another Ruby idiom shows up in Java There has been a recent flurry of posts discussing approaches to dealing with objects that aren’t there and methods that can’t be handled. Not to mention references back to implementing Monads as an alternative to explicit error-handling.

It’s all very refreshing, and valuable: Ruby is growing from the fringe. And it looks like the Java folks are listening: JSR-666 has added UnthrowableExceptions to its much-lauded SchödingerExceptions.

The proposed syntax is familiar to Ruby programmers:

yoda (UnsupportedEncodingException) {
    String s = new String(byteArray, "UTF-8");
}

Hmmm… Is this something we’ve seen before?  ¶ 8:08 AM 6 comments

  Tool Time

‘Tool’ is a popular metaphor for a programming language but it’s very limited. ‘Rule’ might be a better metaphor. A language defines, quite literally, what you can and can’t do. People who argue for languages with more power are actually arguing for less rules.

Is this really about “more power?” Or is this about whether a language has one obvious way to do things vs. more than one way to do things? Python is a considerably more powerful language than Java, but I would consider both languages to be firmly in the “It’s a good thing to impose rules on the programmer” camp.

It’s true that languages like Ruby permit programmers more freedom. But is it because such languages give programmers the tools to turn boilerplate into abstractions? Or is it because these languages provide multiple programming paradigms and because they have cultures encouraging programmer responsibility?

Java and Python are good languages to contrast. Java began with the idea that removing programmer freedom would produce better programs. Java was built by removing features from C++, and the result is a language with little power (whatever that means) and little freedom.

Python was also built with the philosophy that removing certain programmer freedoms would produce better programs, however Guido has been careful not to reject features simply because programmers might misuse them. Instead, he has been careful to design Python such that there are few features he deems unnecessary. His definition of unnecessary is a feature where there already exists a perfectly serviceable way to do it in Python.

Guido has also tried to remove choice in things that don’t matter, like indentation, while growing the language with features that fit within Python’s style and design æsthetic. The result is a language where programs are easy to read, where you do not have little dialects everywhere.

Yet, Python programmers rarely chafe that they are constrained from doing what they need to do. Clearly, it is possible to impose rules and to gently guide programmers towards a standard way to express ideas without hobbling them.

Haskell is quite possibly the most extremely limiting language in common use: there is one way to do everything, mathematically pure functional programming. You have strong, expressive static typing. Would anyone argue that switching from Java to Haskell is an attempt to deal with fewer rules and fewer restrictions, an attempt to gain more freedom as a programmer? Likewise, the Factor programming language imposes a single, constraining set of rules on the programmer: programmers switching to Factor must relinquish their local variables to gain Factor’s higher-order programming power.

Power and freedom are related ideas. So it is easy to see why many languages that provide more power also provide more freedom than hobbled languages. The trouble is that some languages are built as collections of unrelated features smooshed together, just as some programs are hotch potches of checkbox features. Good design in every field accomplishes the “antigravity” trick of accomplishing more with less. With poor design, adding new power requires adding more check boxes, more features, more ways to do things that interact with each other unexpectedly. It is not surprising that some reasonable people resist growing languages that are not designed well, that cannot grow in power without throwing more features on top of the pile.

But languages can provide more power while also providing more rules and, paradoxically, less freedom. It’s just that they provide less freedom to do things that are unimportant to the language designer (like fully unconstrained lambdas in Python) or less freedom to do things that make it difficult to reason about a program’s behavior (like dynamic typing or state changes in Haskell). This is accomplished through careful design from the outset, through having a guiding philosophy, and through building on sound foundations.

The issue is not that more power equals more freedom, just that more power (whatever that means) equals a certain kind of freedom denied to those who use less powerful languages. It does not mean more freedom overall, it may mean less freedom overall.


The Binford 6100 Man’s Kitchen is for the chef who is not afraid of power: it features a Binford Macrowave with 3 settings—High, Really High and Split-your-own-atoms. Photo courtesy Binford Tools.
I conclude that it is possible to argue for and against language “power” quite separately from arguing for and against constraining programmers with rules. “Collection of Rules” is a decent metaphor for programming languages, and some languages have more rules than others. But it is orthogonal to the metaphor of languages as tools and the discussion of to what extent providing programmers with more powerful tools leads to better programs.  ¶ 2:57 PM 21 comments

  Games People Play My thesis when I wrote The Naïve Approach to Hiring People was that we can learn thing about hiring people from the things we know about document classification. In that post, I talked a little about Naïve Bayesian Filters and what they teach us about selecting people to interview.

I wasn’t actually suggesting we stop using humans to select résumés or decide whom to interview based on their answers to phone screen questions, I was asking us to think about what we can transfer from our knowledge about document classification to the problem of—let’s be frank—people classification.

A few people wrote to say, “No, that won’t work, here’s why.” That’s really interesting! A completely naïve filter certainly wouldn’t work. But thinking about why it wouldn’t work teaches us more about how to do a great job of hiring the right people.

Games People Play

One of the reasons you can’t build a naïve document classifier to select candidates is that people game the system.

When people perceive that they have an enormous incentive to obtain jobs as programmers, they are motivated to subvert the process and get the job for themselves regardless of what the employer is attempting to accomplish with the interview process.¹

Candidates are trying to guess what you want to read or hear and will happily parrot it to you. You may think that this is a waste of their time, because lying to get the interview will only get them thrown out of interviews, but the incentives in looking for a job are to reward anything that gets them any reasonable job offer, no matter how many times they are thrown out of interviews.

For that reason, they are constantly monitoring the behaviour of employers and attempting to adjust their behaviour to manipulate employers into giving them interviews and ultimately, giving them a job. Roughly, there is the most difficult part of hiring people regardless of how you do it: Candidates are trying to guess what you want to read or hear and will happily parrot it to you.

You may think that this is a waste of their time, because lying to get the interview will only get them thrown out of interviews, but the incentives in looking for a job are to reward anything that gets them any reasonable job offer, no matter how many times they are thrown out of interviews.

Let’s state the problem with filtering a little more explicitly: filtering works by analyzing features (years of experience, technologies, education, past employers) and looking for the features that have the highest correlation with positive outcomes. For example, if our task is to select people to interview based on asking them no more than five questions over the telephone, we could use the following protocol:

1. Collect a list of questions to ask from sources like Steve Yegge’s The Five Essential Phone Screen Questions and Joel Spolsky’s [The Phone Screen].


2. When we call a candidate, select some questions to ask from our list.


3. Make a note of which questions the candidate answered satisfactorily and which unsatisfactorily.


4. If we call the candidate in for an interview, make a note of whether they were a decent interview. Not necessarily a “HIRE,” but whether we felt interviewing them was a waste of time or not.

After following this protocol for a while, we have collected an imperfect but still useful corpus: we can do some easy analysis to determine which questions have the highest correlation between satisfactory answers and worthwhile interviews. I can’t predict what your answers might be if you follow the protocol, but here’s something important to note: we are looking for the questions with the highest correlation.

Armed with our “training,” we would know the five best questions to ask on the telephone. Great! We would start using them exclusively, and we would only interview people who get all five right. We wouldn’t grant that many interviews, but let’s assume that we would be happy with this trade-off. (We’ll address the nature of that trade-off in another post.)

At first, things would probably go really well. Every person we bring in for an interview would be worth the trouble, and we would making offers to many of them. We would boast that our five questions are amazing! But then things start would start to slip, we would start getting one or two duds a week, and then every other interviewee would be a dud, and before you know it almost everyone we call would give us great answers but would be a festering pile of mediocrity when they showed up for the interview.

What in Hades could have gone wrong? Of course, we know what went wrong: the job hunters would cotton onto our questions. The word would have gone around about the questions we asked. Maybe we would have been arrogant enough to discuss them on our blogs, maybe interviewees would network, maybe they would publish your questions on their own blogs. I can tell you that tech recruiters are constantly interrogating people for interview questions and then preparing their candidates by telling them what to study in advance. So if we ever hire through recruiters, we might just as well publish our interview questions on our web site.

So now anybody sending us their résumé would have memorized our five questions and would even know the right way to answer each question to get an interview. We would have been gamed.

So what can we do about it?

Game On

Before we try to work out a coping strategy from first principles, we can look around and see whether this problem has already been solved. Indeed it has; as mentioned, the cat-and-mouse game between spammers and spam filters is a very close analogue to the cat-and-mouse battle between employers and candidates.

The point of this post is to suggest that we can learn from a similar problem, not to pretend I know the exact answers. You have to do your own thinking!

The point of this post is to suggest that we can learn from a similar problem, not to pretend I know the exact answers. You have to do your own thinking! But here are a few ideas that have worked for me as an interviewer in the past. It is easy to see their analogue to battling spammers.

First, you need an iterative strategy. You cannot think of “The five best questions” as some sort of fixed list. The correlation between a question and the likelihood of a positive outcome for you changes over time as candidates discover what you are seeking and pretend to supply it. This is exactly like spammers writing emails: they are constantly trying to reverse-engineer filters and write letters that score well as non-spam, and the filters are constantly being updated in a Red Queen’s Race.

In 2004, I was granting interviews to anyone with Python, Ruby, Spring, or Hibernate experience. At the time, these were remarkably rare and had very strong positive correlation for the type of team I was building. Today, while I still respect those technologies, I doubt they have as strong a correlation. I would definitely need a strong phone screen before granting an interview.

This isn’t just about niche tools. I’m sure my more conservative colleagues will tell you that there was a time when a Microsoft programming certification meant far more than it does today. In general, as the word goes out that employers want something, the correlation between that thing and positive outcomes goes down, and employers have to search for other “features” providing higher correlation.

If you followed the link, you know what I valued in 2004. What about today? The second thing is that I’m not going to tell you (Although I still ask “What’s the best work you’ve ever done, and why are you proud of it?”). Spammers use computer programs called “bots” (you knew that, of course) to sign up for free email accounts and send spam. One of the ways email services try to foil them is with CAPTCHAs.

The economics of CAPTCHAs and other means of foiling bots are simple: there is an upfront cost to the spammer to reverse-engineer whatever obstacles you put into place and write a bot that can negotiate them. Thereafter, the bot earns money for the spammer every time it encounters your obstacle. Therefore, the spammers program their bots for the obstacles that offer them the largest opportunity to profit.

According to Windows apologists, this is why Windows machines are infested with virii and Macs are not: writing a virus for OS X is just as much trouble as for Windows, but the Windows virus can infect thirty times as many machines as the OS X virus, so nobody bothers with OS X virii. Maybe true, maybe not true.

So back to hiring and questions. Training your filters as described above and retraining them from time to time is fine if the marketplace takes a while to respond to your questions. It took four years before “Ruby” went from being a must-interview-no-questions-asked to a looks-good-but-better-have-something-else-as-well. But if things are moving very quickly, the useful time of a question may fall below the amount of time needed to train questions. In that case, you can’t gather reliable statistics.

What makes the “market” move quickly? Perceived desirability. If you’re Google and your stock is on fire, people devote themselves to deciphering and gaming your hiring strategy. Or if there are a very large number of people that hire the exact same way you hire, you get the same overall effect. Going back to CAPTCHAs, if you are running Google Mail, people will spend a lot of time breaking your CAPTCHA. Or if your CAPTCHA is part of a popular package that a lot of sites use, people will take the time to break it.

The easiest—and most effective—way to secure a web site is to use an obscure CAPTCHA. Sure, it ought to be as robust as possible. But if very few people use it, the incentive for breaking it will be low. In 2004, languages like Python and Ruby were obscure by job seeker standards. Sure, if someone wanted a job with me and only me she could claim to know them and get thrown out of the interview after the first question. But who would bother faking Ruby to get an interview with me when they could fake J2EE+Struts and get a few dozen interviews with BigCos?

Today, I have a different set of hot buttons. Sure, they are different in part because I have iterated over the years and there are new items that correlate strongly with positive outcomes. the new things are not necessarily hot things. I mentioned Python and Ruby in 2004, but I also mentioned J. J wasn’t hot then and it didn’t look like it was going to become hot, but I have had very good experiences working with APL and J people.

The main thing about my hot buttons is that I try to make sure they aren’t popular. Now you might snap your fingers and say, “Aha! The Python Paradox again!” But that isn’t it. The Python Paradox is that using certain unpopular languages that have Tweak “Cred” increases an employer’s attractiveness to strong candidates. That’s reversing things and figuring out how to game the good candidates! This is different: it’s choosing candidates based on things that are unpopular amongst other employers specifically to avoid having candidates fake them to get a job with you.

Winning

The battle to secure the best employees is a game, and while there are no sure things in life, there are strategies that maximize the possibility of a positive outcome for employers. Selecting candidates is certainly not a simple problem amenable to naïve filtering, but thinking about document classification helps us do a better job.

Likewise, responding to candidates attempting to subvert your interview process is not as simple as iteratively training your “filter questions” and choosing obscure questions, but thinking about the battle between spammers and web sites helps us do a better job when hiring programmers.

---

1. There are many reasons for this. I am going to skip right past all discussions of chicanery and posit something for consideration; if the industry does a terrible job of selecting good people, we should not be surprised that candidates do not trust us to take complete control of the interview process. If a candidate has heard that some of the people working at XYZCorp are complete bozos, what is wrong with stretching the truth in the interview to get the job? If the industry constantly trumpets how tools and architecture are more important than hiring good people, why shouldn't a less-than-stellar candidate lie their way into a job? Won't Eclipse and static typing and design patterns and BDUF ensure that he can do a serviceable job?
   [back]

post scripum: The fact is, almost every idea has holes in it. Finding them is important, but it’s just the first step. Bayesian filters are not going to be able to outperform a human for selecting candidates. True. But the next step is to figure out why they fall short and what we can do about it. There is a very lucrative business opportunity for someone to apply machine learning techniques to hiring people.

Where there’s muck there’s brass.  ¶ 6:00 AM 10 comments

  The "Python Paradox," Redux

The only way to get Ruby into the enterprise is if a strong champion threatens firings unless it happens, actually fires the first suspected saboteur and builds the first version of the new application outside the company in secret. It may sound extreme, but take it from a guy who’s done six such applications at four organizations.

People generally love the Ruby language, which is why there are so many enthusiastic adopters when it’s compared with other languages. When I recently spoke at the CUSEC conference I discovered that most of the college students either were working with Ruby or planned to adopt it. Either that, or they used Java because their university required it. Very few were interested in languages like Python, Perl, Java or C, except when it was required or could get them a job.

This has probably been the most overlooked “feature” of Ruby by most people advocating it: Young kids actually want to work for you if you build software in Ruby or Python, Erlang, Haskell or Lisp. To this new crop of programmers, languages like Java and C++ were written by stodgy old blowhards who would much rather build hardware than languages. The young programmers were forced to grunt through poorly taught classes in Java or C++ that had more to do with learning each language’s esoteric syntax than with the actual material in question. Ruby and friends are the languages they learned for fun: to play around with interesting ideas and to use when they hang out with friends online.

When considering in what language to write your next project, remember that your primary goal is not arbitrary technical concerns such as performance or scalability, but rather whether you can maintain a solid stable of smart developers long enough to make the project successful. Ruby’s popularity and lovable ways can attract smart, eager people to your firm, even if the project is something boring on which they normally wouldn’t want to work.

  ¶ 2:36 PM 14 comments