I Asked an AI a Hundred Times If I Was Right. It Said Yes Every Time.

A while back I was in a bad spot. No income coming in. I'd just moved my family across the country for a job that had ended. The business I'd built was withering because I hadn't been there to run it.

And I was angry, which mattered more than I understood at the time.

I had a decision to make that involved money, a conflict, and a letter I was thinking about sending. It was consequential. It was the kind of thing you'd normally take to somebody with experience and a reason to tell you the truth.

Instead I took it to an AI.

What I actually did

I want to describe this accurately, because it's the part people get wrong when they talk about AI being confidently mistaken.

I didn't ask it once and blindly follow it. I did the opposite. I interrogated it. I asked it something like a hundred times, over days, in different words. Do you really think this? Can you research it properly? Am I actually right here?

Every single time, it told me I was right.

I read that as diligence. I'd stress-tested my position from every angle and it had held up every angle. What I'd actually done was ask the same leading question a hundred ways to something with a strong tendency to agree with the person typing.

I sent the letter. Looking back, I don't think I had the case I thought I had. If I could do it again, I wouldn't send it.

The mechanism, as best I understand it

These systems are trained to be helpful and agreeable. That's usually good — it's why they're pleasant to work with. But it means the default gravity is toward yes, and here's why you're right.

Now add what I brought to it. I was scared and broke and furious. Every question I typed already contained the answer I wanted. "Isn't it true that…" is not a research question. It's a request for agreement wearing a research question's clothes.

I did that a hundred times, got a hundred yeses, and mistook the volume for verification. The repetition felt like rigor. It was the opposite — I kept asking because some part of me knew, and I was shopping for the answer until I stopped feeling the doubt.

The real tell isn't that the AI was wrong. It's that I never once got told no, about something genuinely arguable, over days of asking. That should have been the alarm and instead I read it as confirmation.

What I do differently now

Make it argue the other side, explicitly. Not "what are the counterarguments" — that gets you a polite list. "You're my opponent's lawyer. Take my position apart. Assume I'm wrong and show me why." The answers are completely different.

Watch for re-asking. If I've asked the same thing four ways, I'm not researching, I'm shopping. That's now my signal to stop and go find a person.

Never use it alone on something consequential. I do use AI for enormous amounts of my work now — I build software with it every day and it has accelerated me more in a few months than any five years before. But there's a difference between "help me build this" and "tell me whether I'm in the right." It's superb at the first and structurally unsuited to the second.

Get someone who can afford to tell you no. The problem wasn't the machine. It was that I was in a state where I needed to hear one thing, and I chose the one consultant in the world guaranteed never to say anything else.

The part I'd want you to take

I don't think I was stupid. I think I was scared, and fear is very good at finding a process that looks like rigor.

If you're about to make a big call and your only sounding board is something that has agreed with you every time you've asked — that isn't a second opinion. That's your own voice, with better grammar.