raganwald
(This is a snapshot of my old weblog. New posts and selected republished essays can be found at raganwald.com.)

Wednesday, January 10, 2007
  An interesting case of bias


One of the subjects that seems to raise people’s hackles is how to conduct a technical interview. I recall being subjected to personal attacks because I happen to like asking senior candidates a design problem as part of the process.
Oh, I’m a terrific software architect if you want a three tier, MVC business rules-driven application where I have days or weeks to suss out requirements and fully investigate the options. But my performance designing a distributed email server, or a game, or anything else in just a few hours does not reflect my performance under actual job conditions.
I’m not going to argue the point here. But I have observed a very, very common theme to the arguments against solving puzzles, writing code, and designing ‘toy’ systems in an interview (let’s call all of these activities “juggling”): in almost every comment I’ve read objecting to asking a candidate to juggle, the poster admits that they dislike juggling, or are poor at it under the time and pressure constraints of an interview.

The argument against testing is cloaked in terms of “Oh, I’m a terrific software architect if you want a three tier, MVC business rules-driven application where I have days or weeks to suss out requirements and fully investigate the options. But my performance designing a distributed email server, or a game, or anything else in just a few hours does not reflect my performance under actual job conditions.”

The premise is since the candidate thinks they are good, and also thinks they would do poorly on the test, the test must be invalid.

I’m not saying they’re wrong. I just find it interesting how few people make the claim that they are incredibly skilled at ‘juggling’ in technical interviews but do not think the results are particularly significant.

update: Do you dislike juggling but you’d like to work for the company anyways? Take control of your interview.
 

Comments on “An interesting case of bias:
Recruit people who blog and write open source. That way you can easily gauge what they're passionate about, how they think, how they make long term decisions.

The "juggling" tests are not even a close approximation.

I make fun of them by playing with the interviewer. Like writing a test-case before writing the string reversal routine.

Or asking if I the ball weighing algorithm should be recursive or iterative.

Unless they catch on and admit the tests are stupid, I won't show up for the second interview.
 
I make fun of them by playing with the interviewer. Like writing a test-case before writing the string reversal routine.

Or asking if I the ball weighing algorithm should be recursive or iterative.


Maybe it's just me, but that kind of behaviour tells me an awful lot about your technical acumen and about your approach to working with people in the workplace.

Isn't that interesting?

When I ask people about Monopoly, I have absolutely no difficulty finding out how they think, how they make long-term decisions, and whether they have any passion.

Of course, I also Google candidates.

People who blog and write open source might be getting a leg up on people who obviously don't take an interest in software outside of their 9-5 gig.
 
Frankly, I'm scared to death of that type of interview. I really have to sit down and think about a problem before I start it. To be fair, I'm not much of a coder either, and I certainly wouldn't qualify myself as such.
 
assaf: The small programming problems may be stupid, but you wouldn't believe how many candidates bomb them.

The hardest question we ask is "reverse a singly-linked list." And that's for senior programmers, in a shop where we do some functional programming.

For Unix sysadmins we ask something like "find all the lines starting with (whatever) in this log file."

And the scary thing is, even after aggressive phone screening, about 1/3rd of candidates bomb these simple questions--despite otherwise excellent interviews. So we keep asking.

The questions themselves may be stupid (deliberately), but the rationale behind them isn't.
 
I'll cop to writing a comment like that recently on reddit.

But I'll stand by my point on logical grounds: Why test with a silly logic problem when it's so easy to test with a real design problem? Logic problems aren't much like programming.

Of course, I only dislike logic problems on the ground that they aren't much like programming, they're only like a particular caricature of programming. Doing a design test seems perfectly reasonable to me. In fact, doing anything that actually looks like the job is perfectly reasonable to me. Unless you are really going to pay me to solve made-up logic problems, though, let's leave those out of the interview and substitute for real code discussions.
 
I saw the discussion about logic puzzles on reddit. I admit I'm undecided about puzzles.

I love recreational puzzles and I think that the ability to solve a logical puzzle is a good indication that you have basic programming talent.

However, I am not convinced that the converse is true: someone who is a poor puzzle solver may not have poor programming skills.

As you may have guessed from my writings, I prefer asking people to juggle. That means something scaled down from the actual job.

What kinds of scaled down tests do you think reflect the skills of a successful software developer?
 
(same anonymous)

Not to bootlick or anything (since I am posting as anonymous after all), but I really agree with the Monopoly question discussion.

One additional benefit: You can learn how to find the light/heavy coin, or the solution to any other given logic puzzle. In fact, learning the solution will teach you little. But even if you tell your interviewee in advance that you're going to ask them the Monopoly question, it doesn't do them any good. Heck, even if the question becomes so popular that people try to write guides on how to BS your way past it, it still wouldn't help the interviewee if the interviewer is at least moderately competent. (Even if the interviewee can regurgitate an answer to a "why" question, it'll still be clear that they don't really know why. I can't put what that will look like into English, but I'd know it when I saw it.)

That is one attribute I'd look for in an interview question: The inability to game the question even if I tell you two months in advance that I'm going to ask it to you. Given the way good questions seem to have a way of getting around, that's probably a good thing to shoot for anyhow.

(I'm actually not a poor puzzle solver; I dislike them despite being pretty good at them. I do dislike the "gotcha!" variety of puzzles, as it's not that I have a hard time guessing the gotcha so much as I can come up with a long list of potential gotchas, all equally valid, and all equally unlikely to be the particular gotcha you had in mind.)
 
To anonymous: I believe. The type of candidates you get through recruiters ... some have "years of experience", but can't even swap two variables. Sigh. We used to circulate by e-mail a questioneer with five tests and expect a response the next business day. It took people on my team all of ten minutes to solve them, yet most candidates could only solve the first question.

The first problem, you need to decide whether the recruiter is sending you a developer or a warm body. It's called screening. Think of it as CAPTCHA for resumes.

The point of the interview is to find out if the candidate is a great fit with the company, culture and development needs. When you're sitting for fifteen minutes in front of a person (or even on the phone) waiting for them to answer a question ... you'll never see those fifteen minutes again, so they better be good and telling questions.

You realize people who get these screening questions by e-mail can Google for an answer, or get someone else to help solve it. If they did, at least they'll show some ingenuity and problem solving skills. Ironically, the ones who fail the screening questions don't, and the one who pass don't need to.



Reginald: "Isn't that interesting?"

Not really. You indirectly got two interesting datapoints. If you asked questions about the last project I worked with, you'll get more datapoints in the same fifteen minutes. You'll see if I understood the overall architecture and business problem, as well as any details you care to drill with. What I liked to do, and what I hated. Even whether I have rage issues and how I communicate with others.

But ... those are all unstructured questions, you can't checkbox them. They differ for each interview, there's no repeatable list. You never know where it goes, so you need to be quick on your feet. And, for the duration of the interview, you need to care about the candidate's past job in a different company working on a different product.

I think you can learn much more, but you also need to invest much more.

I don't think people who blog and write open source are getting "a leg up". If someone has passion, explores the world around them, can communicate with the world, they're a better employee. The blog/code is just a paper trail they leave behind. It doesn't give them an advantage, but records the one they already have.
 
Assaf:

If I am not really, really familiar with the actual environment and actual business issues, I have to take their word for it that the issues they mention were important and that the approaches they advocated were appropriate.

Whereas when I pose the problem, I know it cold and I have an objective view of their proposed solution.

My bottom line is that it takes more than fifteen minutes to judge a potential colleague. It's never either/or. So...

I want to see you write some actual code, not just talk about code. I will bend on this if you walk in with actual code I can look at or I get a super-strong reference from someone who worked with you on your code.

I want you to walk your way through a design problem that I pose. Always.

And I want to talk to you about your past successes. Not either/or. I am not hiring anyone on the basis of a coding sample and a design problem.

So really, we're in violent agreement. It's absolutely important to collect the data points you suggest are important. I just want those and more.

Is there some reason not to ask for code and design and past experience over a phone screen, a first interview, and a follow-up second interview?
 
If someone has passion, explores the world around them, can communicate with the world, they're a better employee. The blog/code is just a paper trail they leave behind. It doesn't give them an advantage, but records the one they already have.

I was thinking of an advantage getting a job. Like a degree, a paper trail of accomplishments is usually a positive indicator, but lack of same is not a reliable negative indicator.

(I realize that not everyone agrees that a lack of a degree does not strongly correlate with lack of ability. Strangely enough, those with degrees tend to feel it is more important than those without. Hmmm, I should blog about that...)
 
Like they say, you don't trust in others what you don't trust in yourself. I think the code I produce in an interview is nothing like the code I write in a project. Much the same as I don't usually dress that well, or show up on time :-) You may wonder whether it's crappy because of the interview pressure, or superb because the problems are simple and I do well under pressure. Either way, it's not a good indicator.

I once applied for a position related to stock trading. They gave me one company's prospectus, and 24 hours to come back with a report. It took most of the work day to get it done, so no big pressure, but more than a little juggling act. If you're a designer, you're expected to present a portfolio at the first interview, and they might ask you to do a sample project for the next interview.

Why should the software industry be any different? Well, we do have this "intellectual property" problem that helps companies from bleeding employees. And during bubbles we go on warm body hiring sprees. And, some people detest looking at other people's code. So we have this idea that you don't bring your work to a job interview, the most you can show is reversing a string.
 
Reading the comments to your Monopoly question on the other blogs was fascinating. Almost to a one I can say that the attitude projected by those who think the question was a "jumping through hoops" makes me think of them as automatic "no hires".

I've been in interviews where I've been asked questions by tech seniors who were obviously interested in proving that they were smarter than me and I've been in interviews where it was clear that the question really was an attempt to discern how I thought, how I approached, and how I worked.

It's not the question, it's the attitude of the interviewer. I'll happily spend two hours figuring out Monopoly with an interviewer who's willing to participate, play devil's advocate, give prompts, etc. In other words, works with me.

And I too won't give ten minutes to an interviewer who's asking tricky, pitfall-laden question and then simply sits back and waits to count the errors.

Maybe I'm clouded by my personal friendship with Reg here. Though we've never worked together, I'm quite certain Reg is the kind of interviewer that would riff off that question with good candids so they were both having fun with it.

I like it as a question, but the question doesn't make it a good one per se. I think the interviewer does.
 
It's not the question, it's the attitude of the interviewer

Yes! That works both ways, as you point out. Isn't it possible that an interviewer in a negative mood (maybe they didn't like the university on your resumé) can ask a question about past experience and use it as an opportunity to look for “gotchas” as well?

I admit, if an interview gets off to a bad start I can also be a poor interviewer. Human nature.

I’ll bang this drum one more time: Take control of your interview. If the interviewer seems to be in a bad mood, you may be able to rescue the interview using a coat tail.
 
I like juggling. I like anything that rewards cleverness and quickness.

Web development is a very different beast than application development. Often you will need to work on live code.. if you break it, hundreds of people will notice immediately.

I guess it depends on the position.. but for web developers, you better know how to juggle.. and you better not drop the ball.
 




<< Home
Reg Braithwaite


Recent Writing
Homoiconic Technical Writing / raganwald.posterous.com

Books
What I‘ve Learned From Failure / Kestrels, Quirky Birds, and Hopeless Egocentricity

Share
rewrite_rails / andand / unfold.rb / string_to_proc.rb / dsl_and_let.rb / comprehension.rb / lazy_lists.rb

Beauty
IS-STRICTLY-EQUIVALENT-TO-A / Spaghetti-Western Coding / Golf is a good program spoiled / Programming conventions as signals / Not all functions should be object methods

The Not So Big Software Design / Writing programs for people to read / Why Why Functional Programming Matters Matters / But Y would I want to do a thing like this?

Work
The single most important thing you must do to improve your programming career / The Naïve Approach to Hiring People / No Disrespect / Take control of your interview / Three tips for getting a job through a recruiter / My favourite interview question

Management
Exception Handling in Software Development / What if powerful languages and idioms only work for small teams? / Bricks / Which theory fits the evidence? / Still failing, still learning / What I’ve learned from failure

Notation
The unary ampersand in Ruby / (1..100).inject(&:+) / The challenge of teaching yourself a programming language / The significance of the meta-circular interpreter / Block-Structured Javascript / Haskell, Ruby and Infinity / Closures and Higher-Order Functions

Opinion
Why Apple is more expensive than Amazon / Why we are the biggest obstacles to our own growth / Is software the documentation of business process mistakes? / We have lost control of the apparatus / What I’ve Learned From Sales I, II, III

Whimsey
The Narcissism of Small Code Differences / Billy Martin’s Technique for Managing his Manager / Three stories about The Tao / Programming Language Stories / Why You Need a Degree to Work For BigCo

History
06/04 / 07/04 / 08/04 / 09/04 / 10/04 / 11/04 / 12/04 / 01/05 / 02/05 / 03/05 / 04/05 / 06/05 / 07/05 / 08/05 / 09/05 / 10/05 / 11/05 / 01/06 / 02/06 / 03/06 / 04/06 / 05/06 / 06/06 / 07/06 / 08/06 / 09/06 / 10/06 / 11/06 / 12/06 / 01/07 / 02/07 / 03/07 / 04/07 / 05/07 / 06/07 / 07/07 / 08/07 / 09/07 / 10/07 / 11/07 / 12/07 / 01/08 / 02/08 / 03/08 / 04/08 / 05/08 / 06/08 / 07/08 /