Q&A highlight
Posted3 months ago
Key Takeaways
Isn’t this where it would likely unravel?
The interviewer will know what the interesting parts of the exercise are, and ask the deep questions about them. Observe some more: do they know how to use an IDE, run their own program, cut through code to the parts that matter. Basically, can they do the things someone who wrote the code should trivially be able to do?
Since it was mentioned in a sibling comment: Even if the candidate used an LLM to write the code at home, I don’t care, so long as they ace the explanation part of the interview.
(Though you have to watch out for folks that are using the AI to answer your questions.)
In fact, I'm okay with people using AI to solve coding problems, as long as that is acceptable behavior at work as well. That should all be spelled out in the interview expectations.
Heh I do think that happened once (that I was aware of), but it was on a topic I knew a lot about, and it fell apart after layer one. Still, pretty lame, I’d much prefer a “I don’t know,” which I usually say if they start guessing.
They also need to be able to reason well about why they made the choices they did. Something useful when talking to them can be asking questions like "If X changed, how would that impact your design?". If they were reliant on AI for vibing (rather than just using it as a tool), then those can be more difficult questions to answer well.
I think at least half of all interviews are a popularity contest, and the other half are your qualifications.
The key is to make them stop the process, not you; so you won't have to deal with HR or management telling you that they are brilliant and you should not reject them just because "you" don't like their behavior.
Scaring jerks out is quite easy: just be excessively nice during the interview. Tell them how much time and energy everybody take caring for each other's well-being and feelings. Tell them that in the team you have lunch together every single day. And you do stuff together after works. And during the weekends.
They'll just run.
Sure, you're lying because you're working in a regular team where people are simply respectful professionnals, but they don't need to know that. You'll do everybody's a favour. The jerk included.
I have no clue whether he'll care and help or pretend to work and drag everybody else down.
There's a huge number of incredibly capable developers who could pass any interview but then spend days playing video games and sabotaging projects and teams.
I really don't believe in technical interviews, I'd rather base the relationship on trust, if you tell me you're good/experienced at X I trust you to be. If it was bs you'll be shown the door with ease.
Instead many companies make it insanely hard to get you hired, but also incredibly hard to cut you out even if you're impact is a very net negative.
To be clear, I'm not saying worker protections are bad, just that if firing is much more expensive than hiring, you can't really afford to hire any warm body that walks through the door. These days everyone and their mother is in CS, there are many more talented people than ever, but also more duds than ever.
It's the reason why most jobs, even in Europe, have a probation period. During this oeriod firing is inexpensive. In my current employment the probation period was 6 months.
If in 6 months you still can't figure out if a hire was good or not, interviewing won't save you.
It's also the reason why e.g. Italians don't change jobs for 10/20% higher salary, you don't want to go through probation.
So... The solution is putting your most senior people doing week-long interview rounds for each candidate?
With the caveat that I have not been on the hiring side of things myself, my feeling about this is that it can—if a) the person you're interviewing is acting fully in good faith, b) they're not the type of person who gets so nervous about interviews that it skews their whole personality temporarily, and c) your interview process is actually decently-designed (I don't think leetcode or any type of "gotcha"-questions will give you a good picture of how professional a candidate is, nor will an interview for any kind of technical position designed entirely by nontechnical people with no domain input).
The big, big problem is the genuinely small proportion of people who come into a job interview in bad faith, seeking not to demonstrate but to hide their true skill and personality, because (rightly or wrongly) they believe it will hinder their chances of a good position.
We don't have that for software development.
I'd say that the problem is that the diplomas (degree certificate, ...) are useless when hiring software developers.
A doctor who has a diploma is much more likely to be a useful/good doctor than a person with a "computer science" diploma will be a good developer.
A few things to consider:
What we practice is "software engineering." Computer science is closely related to software engineering; but not the same thing. (It's like the difference between a degree in physics vs mechanical engineering.)
Doctors still have to be board certified, which requires self-study of topics that aren't taught in class. Some people do get their medical diploma and fail their board certifications, in which case they can't practice.
My computer science degree overlooked a lot of topics that are very important when developing an industrial strength application; specifically, an application that is not an academic exercise, which gets back to:
> A doctor who has a diploma is much more likely to be a useful/good doctor than a person with a "computer science" diploma will be a good developer.
If we, as an industry, had something similar to doctors getting board certified, we could put pressure on ourselves (and academic institutions) to make sure that ample opportunity is provided in schools to learn how to be a good developer.
Back then, CS degree programs didn't even teach higher level languages than C/++ as they were thought to change too quickly. Whatever you learned during your four years wouldn't apply after you graduated. Instead, the programs focused on the low level implementation details with the theory that was where the engineering and science were.
Now, the same school I went to has courses and tracks for web technologies, so who knows.
It takes 2 quarters (ie. 6 months) to go from recognizing a problem employee to firing said employee.
This makes the risk of hiring the wrong candidate significant as a hiring manager, because a bad hire reflects badly on you and eats up your budget thus preventing a backfill.
On top of that, firing individual employees can lead to litigation risk (even if frivolous), thus requiring hiring managers to go through the extreme song and dance of the PIP process and documented reprimands in order to provide counsel if litigated.
Yet, most of the interviews put way too little focus on the soft skills and way too much focus on the hard skills.
Because the vast majority of job interviews are with terrible candidates, even if the majority of candidates are excellent. This apparent paradox has a simple explanation: excellent candidates selectively apply to a few companies and get interviews/offers at almost all of them. On the other hand, terrible candidates are rejected at every step of the hiring process, and have to constantly reenter the interview pool.
Suppose 90% of candidates are excellent and 10% are terrible. If the excellent 90% only need to interview at one company, whereas the bad 10% need to interview at 20 companies, then only 0.9/(0.1*20+0.9)=31% of interviews will be with qualified candidates. To retierate: almost 70% of interviews will be with terrible candidates, even though 90% of people applying for jobs are excellent.
Because the cost of a bad hire is so consequential, the interview process is not designed to efficiently handle a minority of qualified candidates, but rather efficiently weed out a majority of horrible candidates. It is therefore a terrible process for the people actually qualified to pass it.
That great candidates are not out there doing blind interviews, only those of us who are average are interviewing and going through hoops. Engineers below average are almost constantly interviewing.
> Because the cost of a bad hire is so consequential,
This is stated all the time and I feel it is true. But is there any way to make it less consequential? That was my main argument for contract-to-hire (though I know there are downsides to that approach).
Are there any other ways to make hiring less risky?
You invest all that time into interviewing Bob, but then if they don't get the offer, you never reach out to them again. I don't get it.
I don't think I've ever seen this done well.
Professional licensing.
Many other fields require professional licenses. I don't understand there's so much opposition in our field.
(Ok, I do understand.) In general, licensing has some risks:
The lemons will get excluded from the field. (Which is kind of the point.)
Or, the lemons will decide what the criteria for a professional license is; which turns it into a BS hurdle.
---
That being said, the article gets closer to the point of what a professional license is for: "An interview is like running 100m and a job is like a 10k.". If the license is more like running a 10k, then interviewers can rely on it to do a better screening than they could ever hope to do.
In things like civil engineering, there's usually mandated context. You have to work within certain parameters, so it's not too difficult to test with real-world criteria.
With software engineering, it's all over the place. In fact, one of the most exciting things that I used to look for, in potential candidates, was people who were not bogged down with dogma, and would bring alternative viewpoints to the team.
Since anyone coming into my team would require a ton of training; regardless of their seniority, I always had a nice, long on-ramp, in which I could evaluate people.
Maybe have online application pages, that can’t be submitted, unless you fizzbuzz. :(
I did this because I was part of a project that failed because the leads did not know how to work with a database. They thought the ORM would just magically load objects. Without understanding the basic limitations of data readers, they painted themselves into a corner.
Licensing basically needs to filter this kind of thing. IE, you don't get a license to work with a SQL database unless you can work with a data reader.
---
The irony was that we very rarely touched SQL, but the question was one of the greatest filters out there! It showed who understood basic concepts, who was just BSing their way though, and who would get stuck in high-level abstractions.
* Licensed (allowed to practice in that state)
* Credentialed (degrees and experience verified)
* Enrolled (able to be imbursed via insurance)
* Priviledged (authorized to perform certain tasks/roles)
This could take weeks or months per physician and there is usually an entire team (Medical Staff Office/MSO) dedicated to the work.
That seems like a better alternative to what we have now! Some companies search forever for the "perfect" candidate, or otherwise have to sift through so much SPAM resumes that they are already taking "weeks or months."
(1) The expectations for the position are clearly defined (2) The hiring team members coordinate on questions and expected responses, and they are consistent in interviews. (3) The hiring team members know how to spot potential issues (e.g., excessive bad-mouthing of previous employers, etc.) (4) The hiring teams effectively leverage their networks for references. Ideally, there are not-too-distant trusted relationships between the candidate and the hiring team. In the absence of this, references are followed up on carefully (this has become an art form in modern times.)
These reduce the risk of someone slipping through the cracks. Hiring teams also get better with experience, so any mistakes should be carefully analyzed and corrective actions incorporated into the hiring process.
Paradoxically, a higher bar for hiring increases these consequences for everyone. A bad hire is only consequential in the first place because hiring managers are slow to cut them loose. Managers are slow to cut loose because they are morally culpable for the consequences to the individual they hired. When a manager extends an offer, they are accepting some responsibility for a significant change in a person's life. It's very difficult to walk that back when it's a bad fit, knowing that hiring is a slow process and every other company out there is scared of making a bad choice. But at the end of the day, interviews are an approximation of the candidate/company fit in what is ultimately a matching problem. More attempts make for better matches. Companies and candidates both would be better served by being faster to hire and faster cut loose.
Maybe not great mitigations, but that’s what I could come up with
Of the interviews I've had I would say about 3/4 have tried to catch you out with some inane gotcha that you would never see in the wild or have a very specific solution in mind without exploring or discussing. Sometimes both.
In another job where I was hired, I was tortured for over 10 minutes as one interviewer tried to get me to say the magic word. Fortunately, his manager was present, and shut down the questioning at that point. I never did learn what word the subordinate was trying to get me to say, but I was fully qualified for the job, and hired.
> I’m exaggerating a lot, but the point is, when you select 1 out of 200 applicants, the other 199 don’t give up and go into plumbing (although I wish they would… plumbers are impossible to find). They apply again somewhere else, and contribute to some other employer’s self-delusions about how selective they are.
----
“The majority of interviews being statistically rejections “is not obviously related to a need to make the interview process more demanding than the actual job - Testing for the role would also weed out bad candidates.
Here’s some alternatives:
- engineers testing for the idealized version of their job, rather than the realities of it - “a true engineer must know how to balance a binary tree! Nevermind that I spend my time dealing with support tickets regarding a null pointer that slipped by in a code review”.
- companies using long processes as negotiation tactics - “you went through two months of trials so you’re less likely to reject our offer now and start over”
- design by committee - “every interested party and team must give an approval in their own step”
- interview as marketing - “see? We deal with tough challenges, this is an interesting place to work at”.
Do you have actual data showing this? Or is this just your intuition?
Because if it's the latter, my intuition is pretty different.
No one is an "excellent candidate" for every position, and many people who are "excellent candidates" for a given position might not even recognize that excellence in themselves. Therefore, they're not going to be only applying for positions at the very best companies; they're going to be applying for any position they think they have a chance at, assuming they think they can actually be OK with the job (eg, they might not want to apply for a job in adtech if they are personally repulsed by the ethics of surveillance capitalism).
Additionally, your scenario sounds like it paints the candidate pool for jobs in general as a bimodal distribution, with one peak of "terrible candidates" and one peak of "excellent candidates", with very little in the middle. My intuition says that it's much more likely to resemble a normal distribution.
No; what's much, much more likely is that most people are decent candidates for many jobs in their field, and excellent candidates for a few, but their chances of actually getting the opportunity to apply to those few (between the position being open and them searching at the same time, and them finding out that it's open) are slim, so they have to apply for a great many of the positions that they're only decent candidates for. That means that they'll try many times before finding something. This can lead to a lot of frustration and even desperation, creating a willingness to engage in some shady techniques to actually get a human to talk to you and recognize your value.
Then there are a few people who are, indeed, nothing but shady techniques, and they are likely to flood all channels with as many AI-generated or otherwise low-quality applications as they can manage.
So no; even if most applications any position sees are terrible applications, most interviews are likely to be with decent-but-not-excellent candidates, and most people are still going to have to interview with a few or even tens of companies before they are actually offered a position.
Our best hires are nearly always coming from the network of a team member or people we contracted with and decided to hire full time.
Most of my time in interview nowadays is spent understanding what the candidate has done before, explaining to them what we do and asking open questions to see how they would approach our issues and how they link them to their experience. If it seems to fit, we hire. My country standard contract offers a fairly long probation period for new hire and we don't hesite about parting with people when it's not working after a quarter. We are very explicit about this policy.
I love this approach.
However, at least in the USA, there are substantial costs to candidates for this approach:
- no health care while contracting (unless the candidate pays for it)
- being a contractor is a different level of risk than moving from FTE to FTE
- if the employer decides it isn't a fit, candidate has to find another contractor
Do you have any great candidates who approach you and then, finding out your approach, pass?
We never generalised the contractors to employee thing. It's just that we hire contractors fairly often to fill in temporary needs and we generally extend offer for full time employment to the ones we would like to keep with us when we can. They generally say no because they like being contractors.
For candidate applying, I have yet to see one explicitely refuse an offer because they know we don't always keep going after the probation period. Then again, the standard probation period for engineers in my country is 3 months which can be renewed once so they would get the same offer anywhere.
But I have never seen someone being surprised when we parted after the 3 months. If someone told me they were during our offboarding meeting, I would take it as us having done something seriously wrong during our onboarding.
To be honest, we don't have that many hirings which end up not working and of them I can only remember one being due to an actual lack of skill from someone who clearly padded their resume yet managed to go through the interview. Some people needed a bit more time than others to reach the level of delivery we expect but that's ok. We don't always need rock stars, just professional who can deliver consistently and are open to learning new things. When we need specialists and don't have someone sufficently skilled internally, we contract. Working together with a specialist tends to raise the level of the team as a whole.
Most what I consider our true hiring failures have come from a mismatch between what the person expected and what the job actually is. That's why I now take time to ensure the person I'm interviewing actually has a good idea of what we are doing.
The other nice thing about this is your interviewee generally doesn't leave feeling like a failure; it's not like I have three questions and you can get them all wrong. Unfortunately there are some people who end up spinning on the very easiest question for the entire session, and, uh, well... I can only do so much, if you really don't know anything about programming at all. This is at least the exception, though.
I have not had to do an interview in the age of practical AI yet, though. In person I don't think I'd have to change much, I've always interviewed with a policy of "I'm not worried about whether the string split command takes its parameters in this or that order, I just want to know you know it exists" and I can basically serve as an AI in the same way I was already being the API reference. Remotely, I'm not sure what I'd do yet.
I've hired a lot of the past ~5 years and my take away: it sucks for everyone - on both sides - except the early HR-led functions, and they don't add any value or much help. They may actually make most of the challenges much worse.
It may not be how you approach coding problems but it tends to accurately reflect how people program in real life, so I don't think it's all that much of a stretch to ask people to show it in an interview. Nobody gets an assignment to write a scalable, testable, blah blah blah 100 pages of requirements project and just sits down and writes it linearly from start to finish. Everything is incrementally developed.
"it takes an awful lot of the interviewer's time to prepare,"
Perhaps in general, but not for me past the first couple of times. However, that's because I do strongly agree with...
"You need a very strong technical resource to deliver this style of interview"
Yes, definitely.
However I don't believe there's a way around this. I don't believe you can get high-quality results for technical hiring from any possible "interview" you can hand to a non-technical programmer.
Moreover I don't even know why people think this is or should be possible. Other than the popular "I want it therefore it should be possible" argument, which the universe doesn't particularly respect. I wouldn't expect that even armed with a prepared formal interview that I could interview an industrial chemist, oil rig engineer, or a plumber. There still seems to be a certain amount of residual contempt for programming in some circles and that includes the idea that a non-programmer should be expected to be able to interview a programmer at anything more than a super-basic level. No, programming is at this point as technical as any other career out there. It really wasn't 30 years ago, but it is now, where the basic standard for the simplest possible project now encompasses a suite of technologies and competencies just to get started and the complexity goes up from there.
Also, while a senior engineer perhaps shouldn't be doing full time interviewing, interviewing is extremely high value work. It determines the course of the entire team for years at a time. The interviews I did years ago still determine the course of my day-to-day activity. Huge impact.
> I'm not worried about whether the string split command takes its parameters in this or that order, I just want to know you know it exists
I’ve run quite a few “can you write code” interviews in the age of practical AI, and I don’t know if I’ve been lucky, am good at breaking through nonsense, or if internet claims are exaggerated, but I can hardly tell the difference between now and the before times. You get someone on a call, you explain a problem, you see how they approach it, you probe along the way. I don’t work for a giant FAANG-like, maybe that’s part of it.
When there are too many candidates and not so many jobs, and absolutely no regulation, employers are free to exploit applicants without consequences. Employers will do this even when there is no actual job being offered, so they can use the availability of applicants to apply downward pressure on the salaries of the workers they already have.
Basic, boring, line-of-business web application shops are asking candidates to implement Boyer-Moore and k-d trees. When the applicant is going to be hired to mostly update JSX templates and figure out why the customer is frustrated. It's like every business needs Olympic athletes when they're running a hot-dog stand.
It's one thing if your business is writing a RTOS for a custom platform you develop. Asking a candidate to implement a slab allocator might be a fine exercise. It fits the work being done.
Businesses need to get off their high horse and look in the mirror. Most of them are doing mediocre things that need average, good-natured people who are willing to work.
Actual jobs offer much more time for people to learn to sync and communicate.
Also, on both sides of the interview table, people have varying strengths when it comes to short and long term communication skills. Plenty of interviewers are not good at interviewing, just as plenty of candidates are not good at their side of the process.
In short, interviews are a very poor approach to choosing who belongs long term.
Ultimately, regardless of the interview process used, every job is ultimately a long term “try and see” interview. You only know how someone fits by trying them for a while.
(Not saying this happened to me but it's a common story I've heard in the last few years)
I agree with the author that it is hard to assess someone's skill if you have a list of 100 people to interview and you know nothing about them. The bigger the stack of applications the easier it is to treat them like data and not people.
I know plenty of software developers who write, maintain, or contribute to OSS libraries; they write blog posts, give talks at conferences/meetups, and make videos in their spare time (or sometimes as part of their work). I've rarely walked into an interview where the hiring manager or the technical interviewer hadn't just read my name off my resume as I joined the meeting. Get to know the candidate's work before assuming they know nothing!
Maybe people involved in the hiring process should be given more time to properly research a candidate. Relying on ATS' and putting the burden on applicant's to do the work of proving themselves is causing a lot of folks to burnout just trying to get a job.
I refuse a lot of these. I've put a lot of time into many of them, just to be ghosted.
Typically, when the take-home comes with a laundry list of frameworks and 3rd party libraries, I walk away. Coming up to speed with all of them is too time consuming for the probability of getting ghosted.
It takes a lot of work on the interviewers side to come up with a good take-home. It requires discipline to reject the temptation to just throw in a laundry list of requirements.
I tended to do the "Historical Interview" one. In my case, it worked well.
I think take-home tests are probably damn near worthless, these days. People will just feed the assignment into an LLM, and return the results.
I am not a fan of LeetCode, because I don't want to waste time, studying stuff that won't have any relevance to what needs to get done. I like to keep my dance card full, and wasting precious time to make someone else happy, isn't my idea of a good time.
That said, I understand why they are there. I just feel that they aren't really something that I'd use to judge senior-level talent; which was what I used to hire.
Not affiliated with Hacker News or Y Combinator. We simply enrich the public API with analytics.