14

I think our company can create challenges designed to find software engineer candidates who are:

  • Good at solving problems, not at wowing recruiters.
  • More likely to be afraid to come up to us at a career fair.
  • More likely to be underutilized in their current programming job but are too introverted to do anything about it.

For an example see this article that discusses Facebook hiding an email address in an image using Piet.

I just can't find any studies or hard data on whether this actually works or not.

gnat
  • 21,442
  • 29
  • 112
  • 288
JoeB
  • 249
  • 1
  • 4
  • I disagree. It's not unusual to have two-line titles on SE websites, and this specific title is very clear as is. Shortening it may make it more confusing. – Arseni Mourzenko Feb 08 '12 at 00:53
  • 1
    I can't imagine a serious study. How do you decide whether the ones, which are hired after such challenge are better than those who would have been hired otherwise, or vice versa? Programmers differ, their education changes much over the decades, recruiter change, the challenges change, a usable scoring system is hardly imaginable. – user unknown Feb 08 '12 at 03:28
  • @user unknown : How do we know that smoking kills and drink driving is unsafe, because by your argument, these are also impossible to prove. – mattnz Feb 08 '12 at 04:04
  • 1
    Hi Joe: requests for studies tend to do poorly here: our expertise is not in information retrieval. If this was phrased as, "How effective are programming challenges in the recruiting process?", it'd likely do much better. –  Feb 08 '12 at 04:13
  • 1
    @mattnz: I don't see where your conclusion comes from. You can perform animal test with tobacco smoke with mice. You can measure reaction speed in a simulator after people drank some alcohol. How can we transfer these methods to hiring programmers? – user unknown Feb 08 '12 at 04:29
  • 3
    @mattnz, number of deaths per 10,000 persons over a specific period of time, whether due to lung cancer or to road accidents, is a (more or less) objectively measurable quantity. Goodness of a developer, or success of a SW project, are not even well defined terms. – Péter Török Feb 08 '12 at 09:27
  • @userunknown: You can't easily compare those you hired with those you didn't hire (assuming someone else hired them, you could theoretically compare their job performance, but it'll be hard to get that data), but you *can* compare job performance of people you selected using interview technique A with those selected using interview technique B. Or you could use different interviewing techniques to predict job performance of the people you hire and check the correlation with actual job performance a year later. I'm not aware of such a study, but it's certainly not impossible. – nikie Feb 08 '12 at 11:09
  • Your assumption that there is an appreciable number of highly competent but underemployed programmers seems doubtful. If you want to hire capable programmers you are going to have to pay the market rate. – kevin cline Feb 08 '12 at 14:44
  • @nikie: It is certainly impossible. Nearly nobody hires a mass of developers, big enough and similar enough to be comparable, and in a double-blind randomized process, which could lead to viable data. – user unknown Feb 09 '12 at 00:18
  • @userunknown: Have you read Peopleware? It's full of quotations to studies that would be impossible, by your reasoning. And mattnz is right, if a double-blind randomized process were the only valid statistical method, we couldn't decide about the ill effects of smoking and drink driving either. Or how would you design a double-blinded test where neither the driver nor the tester knows that the driver is completely drunk? – nikie Feb 09 '12 at 06:50
  • I know how to measure reaction time, or the ability, to keep on track, but we're still waiting to see how to measure how good a programmer is, and how he fits for a certain job. If you can do it in a few hours, I would suggest replacing the recruitment process with that test entirely. – user unknown Feb 09 '12 at 07:11
  • @MarkTrapp, edited based on your comment. All - I've gotten some initial data by reaching out to companies that specialize in this type of thing and asking for their marketing materials. I'd still love to see some real studies, but will eventually share my findings here regardless. – JoeB Feb 09 '12 at 15:38
  • @MarkTrapp Some of us actually do have expertise in information retrieval ;) - or the design and running of studies. I think the bigger problem is the assumption that there are simply "studies" on X or Y random problem. Even if its a measurable thing to study, you'd have to find funders, and some poor organization willing to go against what they clearly consider "best practice" enough of the time to have a meaningful control group. – Fomite Feb 09 '12 at 22:58
  • See also [Do coding puzzles make good interview questions?](http://programmers.stackexchange.com/q/131232/20011) and [Tricky logic puzzles - Are they really useful in assessing programming skills?](http://programmers.stackexchange.com/q/68120/20011) – Caleb Feb 10 '12 at 23:19

3 Answers3

7

Like any tool, they can be extremely helpful, or extremely dangerous. A power drill will make your life so much easier - until you drill through the top of your hand and land yourself in the ER. The same is true with programming challenges in recruiting.

The Good: This may be an effective way to detect someone who, on paper, might not be all that compelling as a programmer. The one with a degree in something that has very little to do with what people normally consider "programming" related fields - Biology, Political Science, Art History...

If they blow through your challenges, then great. They learned programming, somehow, and it's apparently stuck. If they get bogged down, their application may really just be something that slipped through HR.

The Bad: A poorly written programming challenge doesn't actually assess programming skill. It tests puzzle solving via programming skill. The problem is the later is a two variable question - are you good at puzzle solving, and can you do said puzzle solving via code. It's possible to have a perfectly talented programmer who utterly fails at the puzzle solving part.

Most programming challenges I've seen also fail at detecting people who are close to what you want, depending on how it's written.

There's ways to mitigate both of those. For the latter, I'd consider accepting "partial credit" in the form of solutions that don't seem to quite be getting there, "Here's how I would solve this..." etc. if you're genuinely looking for problem solvers. After all, very few people code all alone, and if their answer would have been right if they could ask a senior colleague "Hey Jim, do you know a good way to implement X?", that may very well be someone you want on your team.

The former is somewhat harder, because the burden for that is on you. Pick puzzles/problems/challenges that matter. If no one in your group has ever come up against anything even remotely resembling the Traveling Salesman problem in their work, don't make some clever spin on Traveling Salesman the challenge you come up with. That way, if they're failing at the problem solving aspect of "solve the problem and code it", they're at least failing at something that will actually come up, rather than some arbitrary bit of cleverness your team spitballed over lunch.

Fomite
  • 2,616
  • 6
  • 18
  • 20
6

Very effective.

... as long as your recruitment process doesn't just contain programming challenges. While I fret and hate doing the technical assessment of any interview, it does act as a simple gauge to filter out idiots. And filtering out idiots is the crux of the recruitment process, so you can spend more of your time on those who are fit for the role.

When interviewing, I deem it very important to see what people say under pressure. If they're incline to spit out a bunch of blatant crap, it's easily identifiable and I'll know this person isn't worth my time.

More likely to be afraid to come up to us at a career fair.

This isn't a bad thing. If your potential candidate isn't willing to bet that he's worth being employed there, do you really want to recruit him anyway?

Simon Bergot
  • 7,930
  • 3
  • 35
  • 54
J.K.
  • 13,063
  • 1
  • 40
  • 56
  • 1
    There may be other reasons why someone would be afraid to come up to them... For instance, some people find it difficult/scary to try to sell, or even just present, themselves. Feelings get in the way. That doesn't mean they wouldn't be brilliant and/or valuable once they get past that initial making contact fase. – Supr Feb 10 '12 at 13:06
0

I am assuming you want someone to work as part of a team - as such the better programmer is the person who works better with the existing team members. You want to get a group of people together that can effectively communicate to each other, who actually get along well with each other (they don't have to be friends, but they need good rapport and respect), who are willing to agree and use a common development standards (code and process), who are willing to help their colleges when they deal with a new issue or have a mental block ( four eye theory ). You also need to find a mix of personality types, so if you have a team of introverts that rarely talk, then bringing in a more talkative team member may well improve team dynamics which will make the team more productive. On the other hand, you want to avoid a person with a overly strong personality because they will dominate the team and that will reduce quality and productivity - particularly with introverts in the team.

After you get a person to fit into that mix, then consider technical skill/ability. These too need to complement. Every one has different areas that they are strong at, others they are OK at and some they have no clue. So you need to get a mix of strengths together relevant to the project at hand. Remember that an intermediate coder who works well with a good coder will have the level of their work raised by the stronger person. The weak link in the chain is relationships, not skills (provided the skill is in the team)

Good luck in putting that together.

adam f
  • 380
  • 1
  • 3