Here is the 15 sec coding test I used to instantly filter out most applicants

That should avoid messing things up for people with screen readers while still trapping the copy+pasters.

I'm pretty sure the intent is to weed people out well before they get to a point where you could share a screen with them. He mentioned a few people "resubmitted the application", so sure this is probably an initial step.

One of the best business analysts I worked with (also a profession, mind you) was almost fired when working under an old, grumpy and clearly underskilled one.

I was hired once without interview into a unicorn, was loved by colleagues but hated the work, the business and the industry, then left rather quickly.

See? There are mismatches and unknown unknowns, not just bad or good developers.

Can you re-frame your process or the prompt to only elicit those specific responses?

So instead of a whole exercise of building a React app or a whole backend API, for example, what would really "wow" you if you saw the candidate do it in the submission for a project? Could you re-frame your whole process so you only target those specific responses and elicit specific outputs?

Now that you've taken what was previously a 2 hour coding exercise (for example) and distilled down to 3-4 key questions, you can seek the same outputs in 15-30 minutes instead.

There are several advantages to this:

1) Many times, candidates know the answer but they actually can't figure out what you're looking for when there's a lot of cruft around the question/problem being solved. You can end up losing candidates that know how to solve the problem the way you want, but because of the way the question was posed, the objective is opaque.

2) It saves a lot of time for both sides. Interviewer doesn't have to review a big submission, candidate doesn't have to waste time doing a long project.

3) By condensing the cycle, you can evaluate more candidates and you can hopefully select a top candidate before they get another opportunity. You shorten the cycle time because the candidate doesn't have to find free time to do a project or sit down for long interviews, you don't need to have people review the code submissions, etc.

There was story like this on NPR recently where a professor used this method to weed out students who were using AI to answer an essay question about a book the class was assigned to read. The book mentioned nothing about Marxism, but the prof inserted unseeable text into the question such that when it was copy&pasted into an AI chat it added an extra instruction to make sure to talk about Marxism in relation to this book (which wasn't at all related to Marxism). When he got answers that talked extensively about the book in Marxist terms he knew that they had used AI.

There's a bizarro version of this guy who rejects people who do it in their head because they weren't told to not use an interpreter and he values them using the tools available to solve a problem. In his mind, the = is definitely part of the code, you should have double checked.

It also excludes users of Lynx, cURL, possibly people using accessibility tools, those with custom/overriding style sheets.

Thus, if you get the wrong answer, you "cheated" (or used reader mode)

Not just an "AI solver", a Python interpreter will also do it "wrong". The idea is that it's so simple that anyone qualified should be able to do it in their head, so they get the answer without the equals sign (but IMO a qualified applicant might also know it takes 5 seconds to run it in the repl and it'd be better to be correct than to use the fewest tools, or might be using a screen reader).

Just to be clear: the main problem is not that they did not know what `:-1` was - there are many weird syntax additions with every version - understandable.

IMHO the problem is that there's usually a single interviewer that decides go/no go.

We all have biases, so leaving such an important decision (like hiring an EM) to one person is, (again IMHO) ...stupid .

So for them, yes, it would clearly be faster to run the code than to work through it manually.

What you're doing here is selecting for candidates who are less comfortable with using the tools that they'd be expected to use every day in the role you're hiring for. It's likely to provide a negative signal.

Considering there's no (explicit) instruction forbidding or discouraging it, I'd consider the REPL solution to be perfectly valid. In fact some interview tests specifically look for this kind of problem solving.

I get it still, I'd expect some valuable signal from this test. Candidates who execute this code are likely to do so because they really want to avoid running the code in their head, not just because it's more straightforward, and that's probably a bad sign. And pasting that into an LLM instead of a REPL would be a massive red flag.

I just don't think answering "-11" here is a signal strong enough to disqualify candidates on its own.

OR:

I could run the code in the interpreter and be 100% certain.

I know what attitude I would prefer out of my developers.