It seems that everyone who has ever been involved in hiring programmers has been amazed at how many people there are out there who are unable to solve even basic problems with code. The most rigorous certifications and famous names in the employment history are no sure sign. Even an interview with a manager can often let through people who should not be allowed anywhere near an IDE. Given this, it’s amazing that some companies are still basing their programmer hiring around CV + interview with a manager.
We’ve been working on our hiring process for a few years now, and while I’m sure it’s far from perfect, we’ve had some good outcomes.
The initial problem is that typically the first thing you see of a candidate is their CV. Now, I don’t want to give the wrong idea; CVs do sometimes have useful information on them, but I have found them to be very poor predictors of whether or not we want to hire someone. If you imagine having a default belief about whether you want to hire someone, an amazing CV doesn’t shift that belief much.
For this stage, I generally have a spreadsheet with the following headings:
- Academic Qualification / From where (if any)
- Evidence that they are smart.
- Evidence that they are enthusiastic.
- Relevant experience in the technologies (beyond what is expected, or note here if their experience is less than expected).
- How they did on the technical test we sent them.
- General comments
I summarize the relevant information into the columns. Of course, this means that while a CV may strike me with its beauty, unless I think it’s relevant, it won’t make it into the spreadsheet and it’s unlikely to affect my decision.
Disappointingly, many CVs don’t give me much to go on with headings 2 and 3. I absolutely am interested in your blog (if you ever write about technical topics), your github account, your stackoverflow account, if you’ve done any side projects, if you’ve submitted changes to significant projects that have been accepted (no matter how small), but CVs tend to be straightforward listings of previous work.
One part of the CV that I put absolutely no weight on at all is the ‘Skills’ section. A good candidate can be productive in an unfamiliar technology very quickly (especially if given good guidance), while a poor candidate will not contribute much even if they have years of experience. But usually I don’t even get as far as worrying about that, because although the ‘skills’ section is not usually entirely dishonest, it is normally a a grab bag of acronyms that the candidate has ever so much as smelled and speaks more about their desire to avoid being kicked out by someone who doesn’t understand the job and is making their decision based on keyword matches than on how much experience they actually have in an area.
Given that the CV is usually fairly useless in making a decision about whether to proceed, we need something else, that is relatively low cost to administer but gives us a bit more information. When we first receive a CV, we set up a technical test, which involves us emailing the test to the candidate at a particular time, and we expect to receive a response within just over an hour. We have a few different tests depending on the seniority of the position being applied for. It’s difficult to get it all done in an hour, but that tests a candidates ability to prioritize (some questions are worth a lot more than others, a fact that is indicated on the test). We usually ask about agile, testing, data structures, design patterns, that kind of thing. The only single question that you can immediately fail on is marked as such and is a simple programming problem. The rubric makes a point of allowing the candidate to challenge the questions. We’ve done our best to make the questions unambiguous and fair, but I’m immediately put off a place to work by stupid questions (e.g. ‘continue the sequence’), so making it clear that we don’t mind being challenged is an investment in keeping bright people interested.
When I’m marking the programming problem, I’m looking to weed out two kinds of candidates – the ones who can’t actually code at all and the ones who do not have the right instincts for a developer. Generally speaking, there is no way to tell if someone can code or not without actually seeing them code. There are people with top flight academic qualifications and people with 10 years of experience in industry (and stellar CVs) who can’t actually solve problems with code. If we can spot these people quickly and get them out of the process, that’s a massive win.
There are also people who have a moderate ability to solve problems with code, but do not have the right instincts. For example, if someone starts writing out a 100 line switch statement with no awareness that mechanically repeating steps is actually what computers are for, that indicates someone who doesn’t get it, and while we don’t mind teaching technologies (especially to juniors or graduates), we don’t usually want to have to actually teach developers what programming is for.
Given that the test is not done in a controlled environment we also need to, as far as possible make it so that copying an answer from google without understanding is not a valuable way to approach it. There are various things you can do with the questions, but assuming their test was interesting enough we also check people have actually understood what they have written by asking them about it.
From a technical point of view, we use the Phone Screen to validate the test and the CV. We talk to them about some of the decisions they made on the test, perhaps challenge them on weak areas to see their response and see if they recognize a better approach if it’s suggested. If they’ve got something publicly visible on their CV, I normally pull up a browser and ask them to take me through it.
This is also an opportunity to talk to them a bit about the role too. People often apply for a job because they need a job, but we ideally want to hire someone who is also interested in the technical area, and is aware upfront about the different responsibilities they will have to take on. We also want to find out a little about their career plans. Are we likely to be able to accommodate them within the company? Are they likely to clear off after a year or less? We’ve hired extremely valuable people who then didn’t stay very long because of travel, so if they do not plan to relocate and have an unusually long commute, that might be relevant too.
The next stage is to actually get them to come in and spend some time with us. We usually kick off with a 1 hour ‘pairing test’. Obviously it’s not true pairing, because the pair is evaluating you as well as helping you. Lots of people haven’t coded in front of other people before and it can be very difficult to think under pressure, so we try to be very non-confrontational and put people at their ease as much as possible. We choose pairing tests that provide opportunity to talk about decisions to see why people are doing what they are doing, and we encourage the pair to talk about their thoughts as they work. I usually don’t mind if the candidate has strong opinions that I disagree with – the fact that they care enough to have formed opinions counts in their favor, especially if the opinions are fact based and they show that they have considered a number of possibilities. People often find it difficult to get started, and freak out that they might not be approaching things in the optimal manner, but that isn’t so important; I’m always happy to start with something suboptimal and iterate, or talk through possibilities.
How a candidate uses the tools can also be quite informative. How good are they at searching for answers to their questions online? Do they use the IDE well (I try to give people the environment they’re used to)? If you suggest a better way of doing a task, do they use it later?
A recent innovation for us is to get the candidate to give a 10 minute presentation on a subject of their choice. We instituted this partly because many of our developers will, at some point, have to work at a customer site and present to customers, but it’s a valuable skill even if they never leave the office. I’ve found this to be very useful so far and there have been a number of very interesting presentations. We generally get most of the people involved in the interview process to come along for it, or pull anyone else in we think might be interested.
I’m looking for people who can communicate and talk coherently to a group of people, express their enthusiasm for a topic, and answer any questions that come up.
The technical interview is with two developers, with at least one of them an Architect or Tech Lead. We give them a free rein, but usually the candidate will be asked to talk about some of their recent projects, discuss the more theoretical side of things and solve a simple coding problems on a whiteboard. If they’ve indicated a particular interest or expertise in say MVC, they might be asked to discuss it and related patterns in depth.
For more senior roles we’re also looking for the extent to which they’ve been able to organize and lead teams, mentor others, take responsibility for getting things done and make process improvements.
Development Manager Interview
At this point, we usually have a good idea about what’s going to happen to them. If a candidate has failed miserably at the pairing test, we’ll send them home immediately, but assuming they got through to the Technical Interview, there’s a checkpoint with the result being either send them home, we think we want them, or they’re completely amazing and we must have them.
If we’ve decided to send them home, I try to give as much feedback as possible so that they know what to work on. If we’ve decided that the we absolutely must have them, the formal part of this interview is often cut short and we end up with something close to a sales pitch for the company and position.
Otherwise we go ahead, with possibly another code problem on the white board and more general discussion of their background. The Development Manager will usually round this interview off by trying to get a feel for how likely they are to say yes and when they can start if we do want to hire them since that helps our planning. This is usually around working out what other interviews / offers they have, and whether they would accept an offer from us immediately or wait for results from other companies.
For an exceptional candidate we may be prepared to offer more money than planned, but often a budget will have been preagreed and if we blow that budget, we need to get approval.
Finally, if everything is still looking good, I’ll walk the candidate around the office, pointing out some of the perks and try to give them a sense of how the teams work.
After I show them to the lift, I then walk around everyone who has been involved in the interview process and get more detailed feedback. A single strong negative from anyone is usually enough to reject a candidate. People are often nervous of giving strong opinions if they know that it might be acted on, so some form of synthesis of opinions is normally left to me and the Development Manager. We try to get back to the candidate or their agent the same or the next day.
I’ve come to the conclusion over the last few years that our industry has changed significantly. While I used to think of it as an engineering discipline where people would get qualifications that would prove they could do the job, it’s become increasingly obvious that there are no qualifications that can do that for software development. Indeed, having a certification on your CV isn’t always regarded as a good thing by interviewers and while University degrees have value, they’re by no means a guarantee that the person in question can actually solve problems with code.
I think we’ve moved to a different place, more like art and design, where the portfolio is king. I often moan about how useless CVs are for making decisions – an interesting portfolio (a well organised github account is often enough) is a much stronger advert for a candidate than the most beautiful CV.
4 thoughts on “Our Quest to Recruit People Who Can Actually Code”
People who mainly worked on closed source projects might not have a representable public portfolio …
Then I think they should spend more time creating a representable public portfolio than polishing the way their CV looks. And a portfolio can include things like screenshots, etc of non public things.
Well written walkthrough of a very thorough way of conducting tech interviews.
I believe that the technical exercise via email, and more importantly seeing the candidate code are important, because they give you certainty that he can do it and also instill in the candidate’s mind the fact that he is expected to write code. I’ve seen candidates be put off by an emailed simple exercise, although they claimed to be proficient at coding during, for instance, an initial phone interview.
A lot to think on here, both in the context of previous interviewing experiences, and in consideration of future ones. Good post. Thanks!