What I Look For Now That the Portfolio Stopped Meaning Anything

What I look for now is the catch. Not the portfolio, not the demo, not the answer that arrives too clean. I look for the single moment a candidate tells me the model was wrong and shows me how they knew. Everything else on the table has stopped separating people, and that one thing has started. If I could keep only one signal from an interview, I would keep that one and throw the rest away.

I did not always work this way. For years I hired the way everyone I respected hired. Show me your best work. Give me the sample, the case study, the ten screens. I trusted those artifacts because they were expensive to fake, and expensive-to-fake was the whole reason a signal was a signal. That trust was correct for a long time. It stopped being correct, and the shift did not announce itself. It just happened, quietly, over about eighteen months, until one day I noticed I was learning nothing from the parts of the process I used to trust most. The reorientation below is what I concluded works, and what I watch the sharpest hiring managers do.

The short version:

• The portfolio, the polished sample, the confident demo used to be what I looked for. AI made all three cheap to produce and left the underlying capability as rare, so they stopped carrying information about the person.
• What I look for now happens live, on a problem the candidate can't pre-cook: the moment they catch the model being confidently wrong and can tell me how they knew.
• Wharton researchers found people followed a wrong AI answer around 80% of the time, growing more confident as accuracy fell. The candidate who never catches the model is the expensive hire, not the cheap one.
• The new tells are small: an unprompted correction, a "the AI told me X and I didn't buy it," a change of mind mid-answer. I call the underlying thing Direction, the only signal I have found that AI can't hand a candidate for free.

---

The day the portfolio stopped telling me anything

I remember the review that broke it for me. A candidate walked me through a body of work that was, on its face, excellent. Clean structure, sharp writing, a case study that read like a McKinsey deck. Six months earlier I would have made the offer in the room. Instead I asked one small question about a choice buried on the third page, and the answer told me the person in front of me had not made that choice. The document had. The work was sitting next to the candidate, not inside them.

That was not a dishonest applicant. That was the median one. This is the part hiring managers keep getting wrong, and I got it wrong too: the person leaning on the tool is not a cheat hiding at the edge of the funnel. It is the honest candidate, using the instrument that sits on every desk, with no sense they are doing anything they shouldn't. Because they aren't. I walked through the mechanics in every hiring signal AI can now fake, and the short version is brutal: the artifact was only ever a proxy, and the thing that made it a proxy is gone.

So I stopped reading portfolios as verdicts. I still ask for them; I just read them as routing documents now. The verdict moved somewhere the tool can't reach on the candidate's behalf.

Where the verdict went

It went live, into the twenty minutes where the candidate works a problem they never got to prepare for, and turns unpredictable in front of me. Every signal I trust now has that shape: real time, no pre-cooking, a small pressure they have to metabolise on their own. The centrepiece is a wrong-answer setup. I hand a candidate a real task with an AI output already attached, polished and confident and quietly incorrect in a way that costs money if it ships. Then I say nothing about the error and watch. One kind of person accepts the gift and builds on top of it. Another gets an itch, checks the load-bearing claim, finds the crack, and rebuilds the part that was rotten. The setup is not mine alone; the sharpest hiring managers I talk to have converged on some version of it, because it is the only test where faking the pass requires the exact capability you're testing for. I laid the full method out in a hiring manager's field guide to spotting an AI Operator.

Here is why I trust the catch over everything else. In a Wharton study titled Thinking, Fast, Slow, and Artificial, researchers Steven Shaw and Gideon Nave ran three experiments across 1,372 people. With no AI, people got the reasoning tasks right 46% of the time. When the AI was correct, accuracy jumped to 71%. Then the AI was made wrong, and accuracy collapsed to 31.5%, below the 46% people managed with no help at all, while they followed the wrong answer around 80% of the time, their confidence rising even as they got it wrong. Read as a founder trying to staff a company, that stops being an academic finding. It is a description of the default hire, who follows the model off the cliff and feels great doing it. The catch is the whole thing I now pay for, because almost nobody does it and the ones who do are worth more the harder the work gets.

The new tells, told plainly

I don't score these. I listen for them. They are small, they arrive unprompted, and once you have heard the difference a few times you cannot un-hear it.

The first is the unprompted correction. Somewhere in a normal answer, the candidate says a version of "the model gave me X and I didn't buy it, so I checked Y." Nobody asked. It just falls out of how they work, because catching the tool is not a special occasion for them; it is the job. When a candidate volunteers a story about the machine being wrong before I've gone looking for one, I lean in. That is a person describing their instrument, not the view from the passenger seat.

The second is the mid-answer reversal. I ask something, they start down one path, and halfway through they stop and revise their own belief out loud because a fact they just surfaced contradicts where they were heading. Most processes read that as a candidate being unsure. I read it as the hardest thing to fake in an interview: watching someone update in real time. A model gives you the confident version first. A thinking person occasionally argues themselves out of it, and lets you watch.

The third is what they'd change about the work they brought. Not "walk me through your portfolio," because that answer is rehearsed, and half the time it isn't even theirs. I ask what they would do differently now, and listen for whether they can find the weak seam in their own best work. Ownership shows up as a critique the candidate can only make if they built the thing with their own hands. The take-home used to test this and can't anymore, which is exactly why the take-home assignment is dead as a standalone filter.

The fourth is how they treat a claim they can't verify. I drop something plausible and unsourced into the conversation and see whether they swallow it or push. The ones I want get uncomfortable with a number that has no origin and ask where it came from. It is a tiny reflex, and the same one that catches a hallucinated statistic before it reaches a client deck.

None of these is a trick question. They are the places where Direction leaks out of a person whether they mean it to or not. Fluency, meaning what a candidate can make the tool do, is now everywhere and separates nobody. Direction is what I am actually buying, and no résumé line has ever been able to carry it, which is the case I made in stop hiring AI users, start hiring AI Operators.

What I used to look for, and what replaced it

What I used to look for	What I look for now	How it shows up
A polished portfolio	Whether they can find the weak seam in their own best work	I ask what they'd change now; they name a flaw only the builder would know
A confident demo	A confident catch	"The model gave me this, I didn't trust it, here's what I checked," unprompted
The clean, fast answer	The answer that reverses itself mid-sentence	They surface a fact, it contradicts them, and they update out loud
Tools they can name	A claim they refused to accept unchecked	I float an unsourced number; they ask where it came from
How fast they ship	What they did the last time the machine was wrong	The story is about the catch, not the speed

The thing this does not mean

It would be easy to read all this as "AI ruined hiring, trust no one." That is not the lesson. AI did not ruin the signal. It exposed that the signal was always a proxy, and proxies were always going to break the moment faking them got cheap. The portfolio worked for a century because effort was the collateral behind it. The collateral is gone. That is not a tragedy; it is a fact I had to build around, and the reorientation is honestly a better way to hire. The old method rewarded whoever produced the most finished artifact, and finish was never the same as judgment. Shaw and Nave's numbers say the most polished output can come from the person who thought the least. When I stopped hiring the portfolio and started hiring the catch, I stopped rewarding surrender. The market agrees with the direction of travel: AI-related skills now sit on roughly 2.5% of US job postings and climbing, but "AI skills" on a job description still names Fluency and stops there. It is the newest wrong marker dressed as the right one.

The person I am trying to find is the one who can build with the machine and still tell it no. Fluent, and not surrendered. That candidate is sitting on the same shortlist as the one who looks identical to them on paper, and the only way I have found to tell them apart is to stop reading what they handed me and start watching what they do when the model, in front of me, is confidently wrong.

---

Frequently asked questions

Does the portfolio still matter when hiring in the AI era? Not as a verdict. A portfolio is now cheap to produce without the underlying capability, so it tells you the quality of a candidate's tool, not the candidate. I still ask for one, but only as a routing document to decide what to probe live, never as a reason to make an offer. The decision moved to what the person does in the room.

What do you look for instead of a portfolio? The catch. I set up a moment where an AI output is confidently wrong and watch whether the candidate notices, checks the load-bearing claim, and rebuilds it. I also listen for unprompted corrections, mid-answer reversals, and whether they push back on an unsourced claim. Those are the places judgment leaks out on its own, whatever the candidate handed me on paper.

Why is a candidate catching the AI such a strong signal? Because faking the catch requires the exact capability you're testing for: you'd have to know the model was wrong, where, and how to fix it. Wharton's Shaw and Nave found people followed wrong AI answers around 80% of the time while growing more confident. The candidate who reliably catches the model does the rare thing the default hire doesn't.

Isn't the confident, polished candidate usually the strong one? The opposite now, more than people expect. The most finished-looking work comes from whoever leaned hardest on the tool and checked it least. Polish used to correlate with capability because faking it was expensive. It no longer does.

How do I test for this without a formal assessment? Drop a plausible, unsourced claim into a normal conversation and see whether they accept it or ask where it came from. Ask what they'd change about the work they brought, and listen for a flaw only the builder would know. Both are cheap to run and hard to fake on the spot.

Doesn't this make hiring slower and more subjective? It is more effort than scoring a take-home, yes. But scoring a take-home now measures the candidate's tool, not the candidate, so the effort was buying you a false read. A live wrong-answer test is subjective the way tasting food is subjective: you calibrate fast once you've run it enough times. Ivanooo built the AI Operator Profile to make that read repeatable rather than a gut call.