Thank you for writing such a heartfelt critique of the Fizzbuzz screening question

Dear Shelley Powers:

Thank you for writing such a heartfelt critique of the Fizzbuzz screening question. I’m not writing to argue with your observations: I think we both agree with your points.

First, I think we agree that is not true that 199 out of 200 programmers cannot actually program. Anyone who draws this conclusion from observing that 199 out of 200 (or any lesser ratio down to 1 out of 2) applicants for a programming job cannot program has omitted a very simple analysis.

Second, I think we agree that it is not necessary to ask screening questions, and that interviewers who fail to take advantage of alternatives when they are easily available is unprepared.

Third, I think we agree that some people place far too much emphasis on a screening question like Fizzbuzz, trying to extract all sorts of inferences about someone’s professional software development capabilities form what is—in every respect—a toy problem with almost no relationship to the work we do on a daily basis.

Fourth, I think we agree that some excellent candidates will feel anxiety in an interview situation, and that their “performance” on a test in front of a panel of interviewers will fall short of their performance under workplace conditions. And I certainly agree with your premise that workplace pressure is often not the same thing as performance anxiety.

All of my comments below are based on the premise that we are talking about “blind” applicants, people who have responded to a posting or people with CVs up on a job board.

Can 199 out of 200 programmers actually write programs?

Let’s perform this simple thought experiment (if you know anyone who is single and actively seeking a partner, they can carry out the experiment for real and report their results to us).

We register with a dating or personals site. We use the search facility to perform a fairly simple screen for “qualifications” like age, gender and gender orientation (presuming you care). Now, we contact every single person matching our search. How many are worth a second date? How many are, let’s be honest, completely unready to date?

After a few months of “dating,” people can become very cynical. “Where are all the good men/women/womyn?” They plaintively ask. The answer, we all know, is that good people are rarely looking. They are already in relationships. If they become single, after a reasonable period to recover from their experience, they date very briefly and then get into another relationship. They have a good network of friends and family that introduce them to eligible mates, so they have very little need for dating web sites.

People with “issues,” on the other hand, spend a lot of time dating and searching for mates but relatively less time in relationships. Their friends and family are reluctant to introduce them to potential partners because, deep down, their friends and family know they are not ready.

The net result is that on a dating site, the majority of the people looking for mates are not the people you want to meet. And if you try to judge people as a whole from dating sites, you will get a poorly skewed distribution. It is not a representative sample.

The fact is, the vast majority of people are wonderful partners. You cannot judge people as a whole from the people you find looking for blind dates.

And the vast majority of programmers can program.

It’s understandable that someone interviewing programmers from the available pool (like résumé boards or blind submissions to the company’s jobs@bigco.com email address) will become cynical and wonder “where are all the good programmers?” But it’s wrong to conclude that the vast majority of programmers are incompetent based on observations of people looking for programming jobs.

Is a screening question mandatory?

If there is no reason to worry that someone may not be an outstanding, qualified programmer, it is appropriate to screen them before investing time and energy in a full interview. Just as an applicant ought to be carefully screening the companies where she submits her CV. Are there alternatives to a screening question like Fizzbuzz? Of course there are.

Before I touch on a few, let me describe one that is not. Years of experience indicated on the CV, even if not falsified, is not evidence that an applicant can actually perform the work in a competent fashion. They must still be screened and interviewed.

It’s a paradox related to the dating analogy: if someone has five or more years of experience, why are they looking for a job? Why haven’t they compiled a network of friends and contacts? For five straight years recruiters have been calling them every week offering them a job. Why haven’t the recruiters found them a new position?

This person may be an incredible find who is looking due to unusual circumstances. Or they may be a wonderful developer with poor networking skills. But they may also be incompetent, and that’s the reason their friends, contacts, and recruiters haven’t placed them in a job.

So what alternatives do we have to asking a screening question? The best alternatives are entirely within the candidate’s control.

First, a candidate can belong to the “Insider’s Club.” Very simply, candidates who do not wish to be asked screening questions should only go on interviews where there has been a warm, personal referral from a trusted friend. I don’t mean Joe told you that Bob’s company is hiring and Bob sends your CV to HR. I mean you know Greta well and she takes you to lunch with Gwen, who is the hiring manager.

Not everyone is a great people person. That’s why it helps to have nature or nurture on your side. But everyone can become a better networker through practice. And nearly everyone should. Not just to avoid screening questions, but because a regular lunch with Greta is fun, and playing Ultimate with Joe’s team on Tuesday nights is a good workout.

If an interviewer still insists on screening such warm referrals… I don’t know what to say about that.

Second, I like your proposal that candidates bring samples of their work to interviews:

Having candidates bring in samples of code and having the interviewer and interview team review such, and question why decisions were or weren’t made is an excellent way of getting insight into the person’s problem solving skills.

For the last decade I have carried a portfolio to interviews with code samples, product brochures and awards, references, and other opportunities for the type of interaction you suggest. I would love it if every candidate brings such a portfolio to me when I’m interviewing them.

Even though I suggest it in nearly every invitation to an interview, most candidates show up with nothing except an eager expression on their face. I wish they would listen to you on this subject. No matter, by the time they have arrived I have already Googled them to see what they have on their home page, what they’ve written on their blog, and what they’ve contributed to OSS. Just as you suggest.

Alas, I often come up empty. (Not always: one candidate arrived with a book… that he had written! Another brought an actual shrink-wrapped Adobe product he had a part in developing). But note that you have a blog… I would never need to ask you about Fizzbuzz, I would much rather ask you about what you have written.¹

So candidates can avoid a lot of screening questions by blogging, contributing to open source, participating intelligently in on line professional forums, and bringing samples to interviews.

If an interviewer wants to ask me to write Fizzbuzz after reviewing my code that implements tail call optimization in Scheme… let’s just say that I will be amused.

Root of minus one, what I actually do in many cases, especially with new graduates, is mail people a problem. They can work on it in their own time without any pressure and get back to me when they have a solution. I have posted one such problem on my blog: designing a deck of cards.

I’m still waiting for the candidate who has Googled me before an interview and comes prepared with code for a deck of cards. That person I would hire: even if they copied the code off the internet, they have initiative and will make a contribution somewhere!

The work at home problem is still used to screen people out, so it’s kind of unfair to lump it under “ways to avoid a screening question.” But it isn’t a screening question in the sense that I plan on drill down and ask candidates questions about their code in a face to face meeting.

In some ways it’s worse than a screening question in the sense that it requires a larger investment on the part of the candidate and there’s more pressure to do well. It’s not just enough that it work, it must somehow please the interviewer.

I personally find that tough, more on that a few headings below.

Is the screening question a significant predictor of talent?

If someone doesn’t have a warm referral, and they don’t have the type of material you suggest, I need to know if they’re the person on their CV. Something dead simple like Fizzbuzz is impossible to screw up in an interview. I actually don’t believe you would fail this in an actual interview, even with your disclaimer.

But let’s pretend that I was too busy to Google you and you forgot to bring some samples and I ask you to write some code or diagram something. Let me be very clear: I’m not judging you on how well you write it, or whether you are nervous. Or even a trivial “nit” like whether you start at zero or forget that numbers divisible by fifteen should print “fizzbuzz.”

And I’m not going to put you on a pedestal if you write it in Haskell, or use recursion, or somehow torture Perl’s regular expression engine into doing the work. Although some interviewers insist that you write such tests in the lingua franca of their office, I always ask a candidate to write programming tests in whatever language they find easiest. PHP is acceptable.

All I’m trying to do is separate you from the people who don’t know that you want a loop (or recursion, or a range initializer, or whatever) and can’t even get started. This is a very Boolean gate.

When it’s done we’re not going to have along chat about your approach. I may even stop you before you’re finished if it looks like you know what you’re doing. I don’t want to waste time on it: once I know you aren’t impersonating the person on your CV, I want to forget about Fizzbuzz as fast as humanly possible and get to the real interview.

So really, the important thing about a screening question is to remember that it’s only a screening question. It’s not a be-all and end-all opportunity to judge a candidate’s coding style, or ability to elicit requirements, or cultural compatibility.

I’m not going to narrow my eyes into slits and demand that you “find the bug.” I’ll glance over your one page of code and then turn it face down. Or I’ll erase the whiteboard. And move on.

Do good people really get anxious enough to fail screening questions?

I agree that good people can fail a screening test from time to time. Do you want to hear a story about the time I was given a lukewarm referral into a company and somehow I was unable to write a program to print out the first one hundred prime numbers? My contact cursed me for choosing that day to have a brain cramp and devaluing his credibility as a judge of talent. We all have our days… this kind of thing keeps my feet firmly on the ground.

Every single technique has some kind of false negative. Interviewing is a statistical model: you employ a number of techniques in the aim of making sure you don’t accidentally toss out a great person or hire a duffer by mistake.

From my experience, the most important thing is not to fall prey to the mistake of making up your mind in the opening sixty seconds of an interview and using the remaining time to justify your prejudices. There are people with my experience—perhaps even me—who are duds. There are people with a funny accent or perhaps with experience in a programming language you despise who are brilliant thinkers and excellent communicators.

A good interview requires an open mind.

This is why the title of my own post on the subject of Fizzbuzz was Don’t overthink Fizzbuzz. It’s just one extremely brief part of a process that takes multiple meetings and many hours when I am looking for a great team mate. But that being said, I still look for people who can finish the problem, no matter how elegant or haphazardly they do it.

I can tell you what frightens me more than another “print the first one hundred primes” problem:

What frightens me is an interviewer cross-examining me with no apparent system or objective. Perhaps someone who has already pre-judged me as superficial because I brought code samples on a Macintosh instead of a Dell. Or who ignores my Java experience to spend the entire interview grilling me about how I could use an allegedly “unsafe” language like Ruby.

I personally find testing of any kind—whether a formal test like a programming problem or an informal test like an interview—produces anxiety. What increases the anxiety for me is not knowing how the test is graded. It’s like being asked to write a program without requirements from the client. That’s very stressful.

Quite honestly, this is a major flaw with the way some people administer screening questions. They ask you to write a program that does such-and-such. But they don’t say anything else. Are they looking for unit tests? Build scripts? Do they want golf-like brevity of code? Or do they want you to baby. talk. the. code. so. that. any. body. can. mouth. the. symbols. one. by. irritating. one.? Do they want comments from top to bottom? Or just comments to explain why not what?

Faced with this anxiety, a lot of candidates feel like they are playing “Guess the program I’m thinking of.” They worry that the interviewer is thinking of the one perfect answer (I wonder if it’s by strange coïncidence exactly what the interviewer would write themselves?). The candidate can think of a few things to write, but the main problem is choosing the right thing to write.

I personally think that the anxiety can be reduced by being very specific about the problem and what is being tested. I think it’s not just anxiety-provoking but outright harmful (to the interviewee and to the interviewer) to say little or nothing beyond the desired output (fizz… buzz… primes…) but attempt to draw conclusions about a candidate based on the code they wrote on the spur of the moment with no further guidance.

I agree it’s ridiculous to expect that the code for Fizzbuzz would resemble production code for a mission-critical web application, and one of the sources of anxiety for some candidates might be trying to “complexify” such a toy program to make it look like it belongs on BigCo’s heavy iron.

Thank you

Once again, thank you for taking the time to remind everyone not to glorify screener questions like Fizzbuzz. Although I have written this partly in support for your points and partly in defence of my own interviewing practices, your post has reminded me to be extra observant the next time I am interviewing. I do not want to let a good person become nervous: even if they “pass,” the extra pressure will put the meeting off stride.

I also want to thank you for your implied criticism of the tone of many blog posts. Yes, I have been frustrated with the quality of applicants the last few times I have been interviewing. But nevertheless, I need to be more clear about the difference between saying that 199 out of 200 applicants cannot code and 199 out of 200 working programmers cannot code.

Cordially,

Reginald Braithwaite

p.s. I am going on my very first diving holiday later this month. Your links to the ocean’s wonders are a timely discovery. Thanks! I am going with my partner, whom I happened to meet on a dating site! So I know that good people are looking and that they deserve the benefit of the doubt until proven otherwise, no matter what the previous 199 dates were like.

---

1. For example (and I say this gently, I am very pleased with your post), I might probe into whether you are the type who “does not suffer fools gladly,” based on the use of phrases like “arrogant twats” and “idiocy travels up the food chain” in your own post. Just as, I am sure, you would have plenty of questions to ask me if you took a few minutes to peruse my own blog.

Labels: jobs

  ¶ 12:02 PM