Software Engineer Assessment

Why Software Engineers are hard to hire well

The industry standardised on a screening format roughly fifteen years ago and has been quietly unhappy with it ever since. Algorithmic puzzles are cheap to administer, easy to grade and genuinely predictive of one thing: how much time the candidate has spent practising algorithmic puzzles. Some excellent engineers practise. Some excellent engineers refuse. The signal that survives is mostly about preparation budget, which correlates with career stage and free time far more strongly than with capability.

The deeper problem is that the job has almost nothing to do with writing code from a blank file. Most engineering work is archaeology: reading something somebody else wrote under different constraints, forming a theory about why it behaves as it does, and changing it without breaking the four things that depend on it. That skill is invisible in every format built around producing new code in an empty editor.

There is also the seniority illusion. “Software Engineer” is a title that spans a candidate three years in who has shipped continuously in a well-run team, and a candidate eight years in who has shipped the same year eight times. Time served tells you almost nothing, and the interview signals that feel like seniority, vocabulary, confidence, opinions about architecture, are the easiest ones to acquire without the underlying experience.

What separates the best from the rest

The best engineers reduce the number of things that can go wrong, rather than handling more of them. Given a function with six edge cases, a competent engineer writes six branches and tests them. A strong one asks why the input can be in six states, and frequently finds that three of them should be impossible by construction. This instinct compounds: a codebase built by people with it stays tractable, and a codebase built by people without it grows conditionals forever.

They are also unusually specific about failure. Ask a weak candidate what went wrong with a project and you get a narrative about deadlines and requirements. Ask a strong one and you get a mechanism: this cache key included the user ID but the invalidation did not, so we served the wrong data for eleven minutes. The ability to hold a precise causal model is the same ability that makes someone fast at debugging, and debugging is where the hours actually go.

The third difference is in review. Strong engineers give feedback that is actionable and proportionate, and they distinguish between a preference and a defect. Weaker ones either wave everything through or turn every pull request into a style argument. Both are expensive, and neither shows up in a technical screen, because a technical screen involves no other people.

Why interviews alone fall short

An interview measures a candidate’s performance in a synchronous, observed, adversarial conversation lasting under an hour. The job is asynchronous, mostly unobserved, collaborative and measured in quarters. The correlation is weaker than anyone comfortable would like, and it is systematically biased against people who think slowly and carefully, which is a description of many of the best engineers you will ever hire.

Take-home exercises fix some of that and introduce their own distortion. They measure available evening hours, which excludes candidates with caring responsibilities and rewards candidates who over-invest, and they are graded inconsistently because the grader’s own preferences fill the gaps in the rubric. A structured situational assessment sits between the two: every candidate faces identical decisions about testing, debugging, review and trade-off, in a fixed time, scored the same way. It will not tell you whether someone is brilliant. It will reliably tell you how they think, which is what the first stage should be for.

Common hiring mistakes in software engineering recruitment

• Screening on the current stack - hiring for the language rather than the reasoning produces a team that cannot change its mind about the language
• Treating years of experience as a proxy for level - the distribution of capability at any given tenure is wide enough that the average is meaningless
• Never testing how someone reads code - the loop asks candidates to write all day and never once asks them to explain something they did not write
• Interviewing exclusively for the greenfield case - almost nobody joins a team with no legacy, and candidates who only shine on blank pages struggle from week one
• Letting each interviewer ask whatever they like - unstructured loops have well-documented reliability problems and reward candidates who resemble the panel

The work this role is assessed against

• Implement features and fix defects across the stack with high-quality code
• Write and maintain unit and integration tests, and automate quality checks in CI/CD
• Review pull requests, refactor legacy code, and uphold coding standards
• Design data models and REST or GraphQL APIs, and manage schemas and migrations
• Instrument applications, create dashboards and alerts, and participate in incident response

Tools and outputs this role works with

Visual Studio Code, IntelliJ IDEA, Git, GitHub, GitLab, Bitbucket, Jira, Confluence, Jenkins, GitHub Actions, GitLab CI/CD, Docker, Kubernetes, Postman, Swagger/OpenAPI, PostgreSQL, MySQL, MongoDB, Redis, Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP), Terraform, Prometheus, Grafana, and Sentry.