How relevant are Brainbench scores when evaluating candidates?

Question

I've seen many companies using certification services such as Brainbench when evaluating candidates. Most times they use it as a secondary screen prior to interview or as a validation to choose between candidates.

What is your experience with Brainbench scores? Did you try the tests yourself, and if so do you feel the score is meaningful enough to be used as part of a hiring process?

Difficult choice.

Consensus seems to be that BB cert are not very good as a certification. The biggest argument was around the fact that some of the questions are too precise to form a good evaluation. This view can probably be tempered somewhat but still, to hold someone's future solely on the results of this evaluation would be irresponsible.

That said, I still think it is possible to use them properly to gain additional objective knowledge on a candidate's level of expertise provided the test is done in a controlled environment ensuring that all taking it stand on equal footing.

Thus I went with the answer that best reflected this view keeping in mind that it is still just an hour long 50ish multiple choice question to evaluate skills and knowledge that take years to acquire. To be taken with a grain of salt !

In short, the tests have value but whether or not they are worth the money is another debate.

Thanks all for your time.

Related: [Are certifications worth it?](http://programmers.stackexchange.com/questions/44/are-certifications-worth-it). I've restricted this question to be specifically about Brainbench scores to prevent it from being a duplicate of that question and because it's what everyone answered to. — , Dec 05 '11 at 03:38
Regarding changing this to Community Wiki: we don't use community wiki for that purpose. For more information about what it's used for, please check out [The Future of Community Wiki](http://blog.stackoverflow.com/2011/08/the-future-of-community-wiki/). — , Dec 05 '11 at 03:39
If you're not in a position to evaluate somebody as a peer, I'd recommend hiring not a tech recruiter but an honest to god specialized head hunter with some actual technical expertise. If certs like this were any good anybody with a relevant college degree would be an excellent programmer. — Erik Reppen, Feb 27 '13 at 03:06
For what it's worth, I once took a BB test for Cold Fusion, having never even *heard* of it before, and scored high enough to be certified. My friend who was considering using it to evaluate candidates decided that it wasn't very effective. At a later job, I got stuck on a CF-basedproject because having seen it before while taking that test, I was the most qualified person in the company... — Bobson, Mar 15 '13 at 18:43

score 23 · Answer 1 · answered Dec 02 '11 at 14:53

23

I took BrainBench tests for C# and Java some years ago and thought they were completely useless. I scored well, but the questions were largely about details of syntax and the class library. Incompetent programmers can memorize this information and pass these tests. Competent programmers learn the features of the language (e.g. C# has delegates, Java has anonymous classes), and the general contents of the standard library, and look up the details as needed. In my opinion, these tests measure exactly the wrong thing. If you want to hire a programmer, you need to see them program.

answered Dec 02 '11 at 14:53

kevin cline

33,608
3
71
142

good point, to look at ones code is probably the best way to assess ones ability. However few will have code to show as most of work done prior was proprietary to the previous employer and few interviewers will actually take the time to read it should it be available. – Newtopian Dec 05 '11 at 03:00
3

@Newtopian: I don't need to see hundreds of lines of code. I can learn everything I need to know by watching a candidate program on a whiteboard for five minutes. Usually, all I have to do is ask "what do you like and dislike about **?" Good programmers are full of well-reasoned opinions. Mediocre programmers have nothing to say. – kevin cline Dec 05 '11 at 04:29
2

also, the questions many times don't have the correct answer as an option! – Jun 26 '12 at 17:21

score 11 · Accepted Answer · answered Dec 02 '11 at 10:16

I think BB certs make good sense as part of a hiring process - provided you can guarantee that the guy passing these is indeed the one you're interviewing (at BB it is easy to screw up identity and make someone pass the test for you).

My first encounter with BB was at interview when they gave me OO concepts test. Since it was at their office, and since the BB session was established at their account, this procedure left no chance to trick exam results. Based on my further experience with BB I believe this is the only reliable way to certify a candidate.

I did not receive my results back then because these were delivered to guys who interviewed me, but anyway exam questions and overall BB approach got me curious enough to give it more try...

I tried all the free exams that made me interested, 5 or 6 or maybe 7 (it was few years ago, I don't remember how many) - Java SE and EE, RDBMS, business writing, typing.

Scores I received felt quite relevant: I got Master's in areas where I felt my knowledge was solid, average scores where I was not so good, and could not certify at exam for typing which has proven that I am not a professional. Also, Java SE score at BB felt pretty consistent with one I've got at SCJP - which in turn felt pretty accurately reflecting my strengths and weaknesses.

My overall impression of BB is that as interviewer I would feel comfortable using these certifications as part of candidate screening - provided that these were taken in a way that guarantees that candidate identity was not screwed.

Thanks nice answer ! agreed with the supervision though, probably the best way to use them would be to have candidates take the test as part of the interview with the company's account and give all candidates the same resources (books, internet access etc). Personally I think I would be even more interested in looking at if and how the candidates look for references than the actual test score. — Newtopian, Dec 05 '11 at 02:51
@Newtopian I think your point about givng candidates access to books / online resources is **very** well taken. I've been using these at BB and I really liked that. I firmly believe that an option to quickly check the references makes test more realistic - after all that's exactly how I do things when at work — gnat, Dec 05 '11 at 05:34

pdr · Answer 3 · 2011-12-02T12:45:43.500

9

Honestly, from the few people I've interviewed with Brainbench "qualifications", I would say that the 20-minute multiple-choice test that I built myself, on eSkills, was a more effective filter. And that wasn't much of a filter at all, it just saved me phone-interviewing people who were going to embarrass themselves.

The only way that I use Brainbench as a filter is to filter out CVs where people try to pass it off as a professional qualification like an MCPD. That seems dishonest to me.

edited Dec 02 '11 at 12:45

answered Dec 02 '11 at 10:21

pdr

53,387
14
137
224

Did you try them yourself ? At first I had a similar feeling towards it. However after trying some of the tests myself I found that their assessment were relatively accurate on how I would rate myself. Not that self assessment is [necessarily more accurate](http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect) but to go as far as filtering candidates just for putting such reference... I'd love to hear more if you could argument the case a bit. – Newtopian Dec 05 '11 at 02:23

score 7 · Answer 4 · answered Dec 02 '11 at 15:10

I'm not a big fan of these tests because they more of a trivia contest than an actual test of programming ability. Some people simply don't perform well on these tests for various reasons not related to their programming ability. I tend to have trouble with them myself due to my vision problems and my typical programming thinking process.

I also think they're biased toward recent college grads who'll have trivia they learned in class not that long ago more clearly in mind and are used to taking tests on a regular basis than someone who graduated 10+ years ago or who is largely self-taught.

I would consider using such a test for a junior level developer position but I wouldn't necessarily use it as the sole determination of who gets an interview or not. I probably wouldn't use it for a mid-level or senior developer position. For these candidates, a phone or face-to-face interview where they are given an open-ended opportunity to discuss their previous projects and programming knowledge are more productive. I think asking such a programmer why they chose X over Y in developing a project is more revealing than their ability to regurgitate 40 multiple choice questions on programming that may not be relevant to the work you need them to do.

true, some of these questions, especially for tests about frameworks will have trivia questions attached to them. The more specialized the test the more chances of getting strange corner cases few have heard about. — Newtopian, Dec 05 '11 at 02:55

score 5 · Answer 5 · answered Dec 02 '11 at 15:22

5

Brainbench is a good way for a non-technical person (e.g. recruiter) to provide a first screen before dropping a resume on a technical manager's desk. They shouldn't substitute for a true tech screen however.

Also, I've taken a few brainbench certs and found many of the questions are too mired in the minutiae of the framework. I passed nonetheless but don't think the questions asked really qualify a candidate as being proficient with development.

answered Dec 02 '11 at 15:22

Michael Brown

21,684
3
46
83

1

good point, HR is often a wall of buzzwords that one needs to go through. Integrating a more objective way to gather info could help bridge the gap. However I also see this becoming yet another wall put up by HR should they take over this part of the interview as well. Double edge sword. – Newtopian Dec 05 '11 at 02:53

score 3 · Answer 6 · answered Dec 05 '11 at 04:23

I took the Brainbench Java and XML tests in 2003. I thought they were kind of hokey. But in a lot of job interviews, the homebrewed 20 multiple choice questions tests are even worse. They're invariably composed and edited in Word, and the auto-capitalization "feature" makes "string" into "String" and "boolean" into "Boolean". The latter can really make a difference in Java code, and turning "int" into "Int" can render C code where you're supposed to pick out the error into a confusing maze.

As an interviewee, I take those tests with a grain of salt. The goofier the test is the less I want to work there. So: testers beware. You're giving away information, and unless you've formatted extraordinarily carefully, your candidate may find out something undesirable about your company.

score 1 · Answer 7 · answered Dec 02 '11 at 15:12

I had to take one not long ago and I think this type of test is more for weeding out candidates than to see whether you want to hire them or not.

I had 30 questions, 3 minutes for each and the test encouraged the use of references, like books or the Internet. Most of the questions were too specific and I would have not been able to answer them by myself, but with the use of references, I did very well. The test was great for deciding whether a candidate is great at quickly researching what they need to answer a multiple choice question, but nothing more.

I would definitely call someone in for an interview if they did well on the test. Would I reject someone who did really badly (below 60%?) Probably! Would I reject someone if they did mediocre? (70-85%) Probably not. So I guess it does have some weight in the decision making process, but not much.

score 0 · Answer 8 · answered Mar 28 '12 at 21:21

I have also took one Brainbench exam regarding J2EE. As a first impression it appeared to me not much related to actual programming. This is like knowing everything about J2EE/Java which in reality is impossible and one cannot claim ever in life.
As per my experience a developer merely touches one aspect/side of any programming language like as my example, I have mainly worked on applications that have a front end in the face of HTML forms and using some framework example Struts/Hibernate I manipulate that information from/to some database.
I guess so far I have touched just one dimension/aspect/corner of java and I don't think that I ever would encompass all features of the ever growing Java.
Point is, the core concepts regarding language semantics and in particular OOP constructs should be very clear to any experienced developer.
If he/she knows what to implement where or which is needed where, then he can be a successful guy. No one has the memory to cover everything related to any language. In fact you just need to strengthen your foundation concepts. Once they are clear, rest is research and development (RnD) work. It all boils down to the fact, how much effort you contribute towards problem resolution using an initial set of tools and then working your way gradually towards the more sophisticated ones.

score 0 · Answer 9 · answered Feb 27 '13 at 02:21

I've worked in to the Information Technology fields for 20+ years. I started when there were not "any" IT Certifications. I actually have a few BrainBench certifications as well as other "Vendor Neutral" certifications, and I use them on my resume or on web sites. My Information Technology experience spans not only security, networking and hardware and software but also programming. My real world experience is what companies look for first, followed by the certifications that I have. The tests are comparable to other IT vendors, and are worth taking, no matter what your current skill set is.

score 0 · Answer 10 · answered Mar 15 '13 at 18:14

Working as a technical lead at NASA on the Space Station, it became necessary for me to become certified in IT Security as well as Solaris Sys Admin. NASA required a Brainbench certification at that time, and paid for the three certs I was required to take. This in and of itself speaks volumes as to the opinion NASA has for Brainbench. Since then, I have lost my certs after 3 years, and now NASA does not pay for certifications, and even my employer will not pay for them, so since I have to pay for it out of pocket, I find their prices reasonable. Note that most certification sites do not meet the criteria for reimbursement from employers, since they do not issue CEUs, and do not have traditional accreditation. My employer will pay for prepping for the exam, but not for the exam itself. The yearly subscription at Brainbench is a great deal, if you need multiple certifications ($199 normally, but I got it on-sale for $99.50). Their single certification costs is $49. By comparison, if you want an MSCE for instance, that will cost $72 if you use the outfit recommended by Microsoft. Brainbench requires an extra $15 for a nice printed certificate, so the total price is normally about $65 with certificate. The subscription allows you to take as many certs within one year as you want, but you still have to pay the $15 for the certificate. The tests generally are difficult, and certainly demonstrate the proficiency of the test taker.

How relevant are Brainbench scores when evaluating candidates?

10 Answers10