23

After seeing this question today on variation of the FizzBuzz interview question, as well as seeing other questions related to it around, I have to ask: is there any formal research on how effective this is as an interview technique? There are plenty of folks who say that it filters out some crazy-sounding percentage of candidates (95%+ in some cases) but how good is it for finding good programmers/developers?

There has been some research that shows that soft skills might be better indicators than hard skills for choosing a good candidate for a job role. As well, there might be some intermediate reasons for so many candidates failing this test (nervousness, thinking "it's a trick", etc).

Based on the comments so far, it looks like the idea behind FizzBuzz questions is to determine whether or not the person has good (enough) coding skills to be further considered. I'm not totally convinced FizzBuzz actually does this. Also, remember that FizzBuzz filters a very large percentage of candidates very early in the hiring process, which could mean that significant numbers of good candidates are being removed early on because they can't solve silly problems fast enough.

Also, I'll point out that looking at FizzBuzz effectiveness is something that can completely be done in a quantitative manner. This is unlike looking at "fit" or "passion" which are highly variable and highly unscientific (but still quite important for new hiring decisions).

Community
  • 1
joshin4colours
  • 3,678
  • 1
  • 24
  • 37
  • 1
    Not a formal research: http://www.codinghorror.com/blog/2007/02/fizzbuzz-the-programmers-stairway-to-heaven.html AND http://www.codinghorror.com/blog/2007/02/why-cant-programmers-program.html – wleao Sep 16 '11 at 14:20
  • 17
    I wouldn't want to hire a candidate that can't code FizzBuzz while nervous. – Eric Wilson Sep 16 '11 at 14:26
  • That's not what the article you link too indicates. It indicates that the ones that are hired there is a high drop out rate. **But** it has absolutely **no data** about the ones that were rejected. **To me this indicates** the companies are probably doing a good job at weeding out the non technical people (as that was the focus of their interviews) that could not do the job, but they need to do additional filtering for the people that refuse to learn or refuse to behave well as team memebers – Martin York Sep 16 '11 at 14:56
  • @Tux-D But is there any evidence that the converse is true? That rejected candidates would've been better hires if their hard skills (ie solving FizzBuzz) was weighted more evenly with their soft skills? – joshin4colours Sep 16 '11 at 15:19
  • From an HRM standpoint, you aren't going to find any scientific studies (or even case study analyses) of particular interview questions. However, you might be able to find studies that focus on a particular interview technique and it's effectiveness for assessing certain qualities of a candidate. – Thomas Owens Sep 16 '11 at 15:29
  • 2
    @JMG: You need to be more precise (the converse of what exactly). If they can't do the technical part of the job (Hard Skills) why would you think they had any chance. If you hire people that by definition can't do the job they are hired for don't you think the drop out rate would be 100%. In our interviews processes we have a specific sections to see if the candidate will be a good fit for the team and company. But no matter how much we like them personally there is no point in hiring a person without the coding skills. – Martin York Sep 16 '11 at 15:38
  • 3
    So as an answer to `if their hard skills was weighted more evenly with their soft skills`. **No**. The hard skill will be one filter. The soft skills is an additional filter. Its not a you case where you can have `A or B` you must have `A and B`. Which is the way I read the article you linked. They have already filtered for A they have just done a bad job in filtering for B. – Martin York Sep 16 '11 at 15:46
  • Which FizzBuzz? Even sticking to the same code requirements, you can have a relaxed low pressure fizzbuzz - or a fizzbuzz by a panel with a time limit and distractions. And should a candidate ace the FizzBuzz, it's a good opening toward testing or change/refactoring questions - is that part of FizzBuzz? (I particularly love folks that do a great job convincing me they're all about TDD and never wrote a single line of untested code, then produce an untestable, (or faulty) fizzbuzz and get stuck writing even a single unit test for it - in their language of choice) – ptyx Jun 22 '15 at 22:30
  • It is not that they can't solve it fast enough. It is that they can't solve it at all. – BЈовић Jun 24 '15 at 06:32

3 Answers3

34

The purpose of fizzbuzz isn't to find good programmers, it's to find a certain class of bad programmers, which is people who can't implement a simple algorithm. Your question is sort of like asking how many Nascar champions are identified during their driver license exams. There's a lot of middle ground between someone you definitely don't want to hire and someone you definitely do.

There are other classes of "bad" programmers that require different methods of identification, like those who aren't proficient in a given language or have behavioral issues. However, "fizzbuzz-bad" programmers are almost universally undesired, which is why the test is so frequently recommended. Whether a programmer who is overly territorial or one who needs to transfer skills to a new language is "good" or "bad" depends much more on the individual employer's needs and preferences.

Karl Bielefeldt
  • 146,727
  • 38
  • 279
  • 479
  • 2
    But what makes a "bad" programmer? Would you want someone who is an amazing coder but removes any checked-in edits to "his" code on a project? Or someone who might need a little help going from C to Java but will dedicate themselves fully in the project? – joshin4colours Sep 16 '11 at 15:10
  • 4
    @JMG: There's a school of thought that informs us "You don't want either". If they have some uncooperative habits like the latter, then "No Hire"; if they don't have common but needed skills "No Hire". If there's something about the candidate that makes you hesitate at all about hiring them *for this project* "No Hire" – SingleNegationElimination Sep 16 '11 at 15:15
  • @TokenMacGuy I guess that school of though should be called the "Near Infinite Time and Resources" school :) – joshin4colours Sep 16 '11 at 15:38
  • @Karl can you incorporate that back into your answer? –  Sep 16 '11 at 17:06
14

It's been a long time since I read up on job interview research, (so I can't site specifics, for which I apologize) but the general rule is that people hire on the basis of soft skills but there is no correlation between that and performance. Testing for specific skills can work, for personality does not (according to measurable data).

Almost everyone believes their special interview techniques work, the data says you should probably not bother interviewing (the "tell me about yourself" part of the interview, as opposed to objective testing) since candidates selected by interviewers are not any better but interviewers do show biases towards extroverts, people demographically similar to the interviewer, tall people, attractive people, etc. (Arguably you should hire people who do well on tests and bad on the rest of the interview, since they are apparently just as good on the job but may find it harder to leave your company).

I kind of expect to get flamed for this, usually anyone I tell this to says "but we use method X and it really works for us", but unless you

  1. hire some people who don't pass your interview process,

  2. don't tell their coworkers who passed and who didn't, and

  3. compare performance of those who passed with those who didn't, using previously agreed on objective criteria

you really have very little idea whether your process works. Unfortunately the effectiveness of interviewing is inherently difficult to evaluate.

If you are wondering, experiments do things like ask people to evaluate candidates based on interview transcripts, from candidates who have actually already been hired (among many other techniques). Ideally experimenters would have people do interviews, pick candidates, erase the memory of the interviewer, randomly select who to hire from among the candidates, and compare the candidates performance using accurate, objective criteria versus the ratings given by the interviewers, but that experiment is hard to pull off.

So, though I haven't seen FizzBuzz specific research, it is the kind of thing which can show effectiveness in scientific testing. If it screens out 95% of candidates then I'd be curious as to why - it was invented to be so absurdly easy that only an incompetent programmer could fail to pass it, and it's hard to see how to dumb down FizzBuzz. I wonder what percentage pass the "Hello world" test? The "make the computer do something, anything" test?

psr
  • 12,846
  • 5
  • 39
  • 67
  • +1 That's why I ask the question: Hiring is inherently very unscientific, but the effectiveness of FizzBuzz could at least be tested out quite quantitatively. – joshin4colours Sep 16 '11 at 17:46
  • @psr: In my experience interviewing new graduates in the US, BSCS or MSCS, about 90% of them can't program. Why? Because the institutions have a financial incentive to keep them in the program paying tuition. – kevin cline Sep 16 '11 at 18:05
  • 1
    @kevin cline: Kind of getting off topic, but in my MSCS I don't think they were that focused on the bottom line. They were, however, extremely concerned with distinguishing themselves as an *academic* program vs. a trade-school, and so many professors looked down a bit on mere programming as opposed to, say, proving really minor things about Turing machines. – psr Sep 16 '11 at 18:15
  • I think you're overlooking the fact that the goal of the folks doing the hiring is not so much to find the best possible match, as just trying to avoid disaster. Even in good economic times people offering jobs are deluged with applicants, many of them hopelessly unqualified. You simply must have some sort of quick filter to winnow down the number of applicants to a manageable number. The original point of the fizbuzz test was to do it as part of a phone screen. If the pool of people who pass Fizzbuzz is even slightly enriched for folks who can actually code, it's a help. – Charles E. Grant Sep 16 '11 at 18:27
  • @Charles E. Grant - I understand FizzBuzz - I actually use questions like that when interviewing (most everyone passes though). I was mostly responding to the "soft skills might be indicators than hard skills" (sic), which I *have* read relevant research for, and in the research soft skills helped in an interview, but could not be shown to help on the job. Or, more accurately, interviewers could not identify candidates that would be effective on the job, if they tried to identify them based on soft skills. – psr Sep 16 '11 at 18:34
5

FizzBuzz isn't about identifying good programmers, it's about eliminating bad ones/fakers. If a candidate can't pass the FizzBuzz test then they don't know how to program and shouldn't be applying for a programming position.

nicholaides
  • 793
  • 1
  • 5
  • 12