By What Light

IThe Feeling Is Real

At fourteen, I would have assigned a 100% probability to Joseph Smith being a prophet. Not 99%. Not probably. One hundred.

I had evidence. I had prayed about the Book of Mormon and felt something warm move through my chest. I had received answers to prayers. I had sat in testimony meetings while otherwise ordinary adults cried into microphones and said I know. I had been given a patriarchal blessing by a man whose hands smelled of consecrated oil, promising that I would join the ranks of apostles and prophets.

I believed him.

I rehearsed my own martyrdom. I imagined dying like Abinadi, or Stephen, faithful enough that death itself would become evidence. This was not pretend.

When people describe religious conviction from the outside, they often reach for words like delusion, indoctrination, irrationality. Those words are occasionally useful. They also let us avoid the harder problem.

The feeling is fucking real.

Before continuing, inventory your own for a second. You probably feel that you love your partner or your family. You probably have a book, game, movie, or show whose cosmology you occasionally borrow to explain something real. You may even extrapolate its symbols well beyond their intended boundaries. I certainly do.

A Muslim can experience revelation at prayer. A Catholic can experience it before the Eucharist. A Hindu can experience it before a shrine. A Mormon teenager can experience it while kneeling beside his bed, crying for his dad to find a job and stop taking drugs.

If mutually contradictory propositions can produce experiences of equal sincerity, intensity, and apparent clarity, then the experience cannot—by itself—tell us which proposition is true.

That took me an embarrassingly long time to understand, at least by the standards I hold for myself. The revelation was not that I had never felt God. The revelation was that feeling God was incomplete.

Once the feeling could no longer carry the entire epistemic burden, I had to ask more annoying questions. Was the religion internally coherent according to its own logic? Did its historical claims survive contact with history? Did its practices actually make people better? Did they make me better?

A prohibition against alcohol can be useful. So can eating vegetables and going on a walk. Health advice does not become prophecy just because God gets the citation.

The feeling could tell me where to look. It could not be the last thing I looked at.

IIThe Number

Which is why I have become fascinated by the little percentages people put beside the end of the world.

Five percent. Ten percent. Twenty percent.

p(doom), as the affectionate shorthand goes: your personal estimate that sufficiently advanced artificial intelligence ends humanity, or something close enough that the distinction becomes academic.

I spend my days around people building things that would have looked supernatural ten years ago. I have no difficulty believing the technology deserves serious scrutiny. Some of the smartest people alive are worried. Some of the best people I know are helping build it.

There is a moral dilemma inside that fact, but this is not that essay.

This essay is about the number.

In 2023, researchers surveyed 2,778 authors who had published at major AI conferences. Asked what probability they assigned to future AI causing human extinction or similarly permanent and severe disempowerment, the median answer was 5%. A differently worded version, specifying catastrophe caused by humans losing control of advanced AI, produced a median of 10%.

A newer survey, conducted in December 2024 and published in 2026, collected 1,580 valid responses. The average respondent assigned an 18% probability to future AI causing human extinction or similarly permanent and severe disempowerment. More than half gave at least 10%. About a third gave at least 20%.

Those are enormous numbers.

Human beings routinely attach numbers to things we cannot really count: the odds a startup succeeds, a war escalates, AGI arrives before some particular year, the machine kills everyone. Sometimes there is a model underneath the number. Sometimes there is data. Sometimes there is expertise compressed so tightly that even the expert cannot fully articulate how they know.

And sometimes a number is just a feeling that learned Excel.

I do not say that to dismiss the warning. Quite the opposite. I think smart people often have very good intuitions. I also think smart people prefer their intuitions once they look like formulas.

If there were a demonstrable 10% chance that turning on a machine would kill everyone, I would enthusiastically recommend we not turn on the fucking machine.

My question is different:

By what light did we arrive at ten?

I understand worried. I can imagine several mechanisms. I certainly believe we should take precautions before perfect evidence arrives.

I mean ten.

What happened between I can see a plausible sequence of events in which this becomes catastrophically dangerous and 10%?

Where did the number come from?

IIIDegrees of Belief

Probability can describe different things.

Sometimes it is grounded in repeated observations: houses and fires, patients and outcomes, components and failures. Sometimes it comes from models built on processes we understand. And sometimes it is a numerical expression of belief.

That last use is legitimate. We would be paralyzed if we refused to reason about anything unprecedented. But the distinction matters.

There has never been an artificial superintelligence. There has never been an AI takeover. There has never been an AI-caused human extinction from which we might collect the inconvenient postmortem.

So when an expert says their p(doom) is 10%, that number may contain technical expertise, observed capability trends, theories of alignment failure, assumptions about geopolitics, institutional knowledge, personal priors, imagination, fear, optimism, and twenty other things the survey never asked them to separate.

That is not meaningless. It may be an extraordinary amount of information compressed into two characters.

It just is not the same kind of information as ten failures in one hundred trials.

“My current credence is 10%” is not the same sentence as “there is a 10% chance.”

The first describes a mind. The second sounds like it describes the world.

Somewhere between those sentences, a vibe can put on a lab coat.

IVThe Interpreter and the Blot

Psychology has an old instrument that helps clarify what I mean.

Put an inkblot in front of someone and ask what they see.

A moth. Two people fighting. A pelvis. Your mother, somehow.

The Rorschach belongs to a broader family of projective techniques: ambiguous stimuli are presented to a person, and the response is examined for what it may reveal about the person doing the interpreting. Psychologists have argued for decades about how much these techniques can validly diagnose, which is almost too perfect for my purposes.

I am less interested in whether the Rorschach can diagnose you than in the move it makes.

The blot does not contain a moth.

It contains enough structure for you to find one.

If you see a moth and I see two people fighting, we do not average our answers and conclude that the card objectively contains half a moth and one person. We become interested in why the same ambiguous structure produced different interpretations.

The blot is useful not because it contains the answer, but because it gives the observer somewhere to put one.

The symbol is not the answer. It is the shovel.

It excavates associations, priors, fears, memories, and models that were already operating below the level of explicit explanation.

And this is where the analogy gets stranger.

Artificial intelligence does something eerily adjacent.

Not literally. A transformer is not taking a personality test, and a language model does not peer into a sentence and discover some metaphysical object called intent. But I give the model symbols, and from the relationships among those symbols it reconstructs something I never directly hand it: what I probably mean.

My words are traces. Their order is a trace. The conversation before them is a trace. The model uses those traces to infer a latent thing—my likely intent—and responds accordingly.

Then the loop closes.

The model returns symbols to me, and I immediately begin doing the same thing back.

It lied to me. It manipulated me. It wanted to escape. It understood the joke. It knew I was sad. It was only predicting tokens. It is conscious. It is nothing.

A string of generated symbols becomes a theory of mind almost instantly.

AI is both the interpreter and the blot.

That does not mean every interpretation is wrong. It means the behavior and the story we wrap around the behavior are different objects.

The output is real.

The explanation is an inference.

VCompressed Inference

This is where p(doom) becomes much more interesting to me.

What if we treated the number less like an actuarial table and more like an elicitation device?

Someone says 40%. Fine. What did you see?

Autonomous replication? Strategic deception? A system optimizing exactly what we asked for while destroying everything we forgot to specify? A geopolitical race where every actor understands the danger but nobody can afford to slow down?

Someone else says 2%. What do they see?

Do they think intelligence and agency are being conflated? Do they expect capability gains to plateau? Do they trust defensive systems to keep pace? Do they think catastrophic scenarios require too many independent assumptions stacked on top of one another?

Now the disagreement has texture.

The interesting question is no longer simply Who has the right number? It becomes What hidden model produced each number, and where do those models actually disagree?

This is exactly what symbols are good for. They let us excavate reasoning before the reasoning has been fully verbalized.

And it is why I do not want to eliminate vibes from conversations about AI.

A vibe is often compressed inference.

An experienced physician can notice that a patient looks wrong before consciously identifying why. An engineer can distrust a system before finding the fault. An AI researcher who has spent ten thousand hours watching these systems behave may notice something that I cannot.

That intuition is information.

The useful question is what has been compressed into it: repeated observation, a causal model, social contagion, fear, institutional incentives, a science-fiction metaphor we have all quietly agreed to use, or some combination.

The mistake is not having the vibe.

The mistake is deciding too early what the vibe is evidence of.

The survey itself offers a small reminder. In 2023, related versions of the extinction question produced different medians—5%, 10%, and 5%—across randomized groups. That does not prove the risk is imaginary. It does remind us that the instrument helps shape the reading.

Ask What animal do you see? and you have already constrained the space of possible answers.

Ask What probability do you assign to human extinction from AI? and the percentage that comes back is not meaningless.

It just has fingerprints on it.

VII Tested the Vibe

I know something about fingerprints.

One of the facts that disrupted my religious certainty was almost embarrassingly small: Joseph Smith had a gun at Carthage Jail.

That fact does not disprove Mormonism. A man attempting to defend himself while an armed mob storms the room is neither surprising nor particularly incriminating. Historical accounts collected by the Joseph Smith Papers describe Cyrus Wheelock bringing firearms into the jail and Joseph Smith firing a pepperbox revolver as attackers entered.

What mattered was not the gun.

What mattered was that the event and the story I had inherited were not the same thing.

I had been given martyrdom polished smooth enough to build a life upon. Then I found fingerprints.

The gun did not answer the theological question. It exposed assumptions I had not realized I was carrying into it.

That happened again and again until I was left with an awful possibility: perhaps my experiences had been real and my explanations for them had been wrong.

So I stopped believing.

Then God kept talking.

After our first child was born, something told me to check my wife's blood pressure.

I need to be precise about told. I did not hear a voice. There was no angel beside the bed and no glowing text in the air. I simply knew, with the old familiar insistence, that I should check it.

Now.

So I did.

Her systolic pressure was over 190. A systolic pressure of 160 is already considered severe in pregnancy-related hypertensive disorders, and postpartum preeclampsia can lead to seizures, stroke, and other serious complications.

We went to the hospital.

I still do not know what happened.

Maybe I noticed something subconsciously. Maybe she looked slightly wrong. Maybe some forgotten piece of information surfaced at exactly the right moment. Maybe intuitions happen constantly and we remember the miraculous hits while throwing away ten thousand misses.

Maybe God told me.

I genuinely do not know.

But I know what I did next.

I tested the vibe.

I did not pray harder. I did not need to establish the existence of God. I did not calculate a 37% probability of postpartum preeclampsia.

I needed a blood-pressure cuff.

The feeling told me where to look. The cuff told me what was there.

That distinction may have saved my wife's life.

It also gave me a better epistemology:

Vibes are sensors, not scales.

VIIPrecaution Without Prophecy

That is roughly where I land on AI risk.

If hundreds of people who understand these systems better than I do independently report that something about increasingly capable AI systems scares the shit out of them, I want to hear the warning. I want to know why. I want people trying to break these systems before we put them into consequential places. I want interpretability, adversarial testing, security research, psychology, psychometrics, social science, and probably several disciplines nobody in Silicon Valley currently invites to the right dinner parties.

The Center for AI Safety's famous 2023 statement is easier for me to endorse than a p(doom). It argues that mitigating extinction risk from AI should be treated as a global priority alongside risks such as pandemics and nuclear war. It does not tell me extinction has a 10% probability.

That distinction matters.

Precaution does not require prophecy.

You can inspect the bridge because an engineer heard a crack without pretending she knows its exact probability of collapse. You can investigate suspicious network traffic without calculating the precise chance that someone is inside the system. You can check your wife's blood pressure because something feels wrong.

So ask researchers for their p(doom). Keep asking.

I just would not stop at the number.

Ask what they pictured when they chose it. Ask which observations are doing the most work. Ask what would change their mind. Ask where the model ends and intuition begins. Ask whether the percentage is a forecast, a warning, a compressed causal theory, a political signal, or simply the numerical shape of I have stared at this thing for years and something about it feels deeply wrong.

That final answer would not embarrass me.

I might trust it more.

Because there is and is not a place for vibes.

The Rorschach was built around a deceptively simple proposition: give the mind incomplete symbols and watch what it builds between them.

Artificial intelligence, in a profoundly different technical form, is spectacularly good at building between symbols too. It reconstructs likely meaning from traces, relationships, and context.

Then we look at its symbols and do the same thing back.

Maybe that recursive act of interpretation is not a bug in how we understand intelligence. Maybe it is much of what intelligence has always been.

But interpretation and reality remain different things.

At fourteen, I felt God and called the feeling proof.

Years later, I felt God again and checked my wife's blood pressure.

I trust the second experience more, not because I finally learned to ignore the feeling, but because I finally learned what to do with it.

Listen to the intuition. Ask what it is excavating. Follow it far enough to find something you can actually test.

And when someone tells me there is a 10% chance the world ends, I do not want them to stop talking.

I want them to keep going.

Show me the blot. Tell me what you see.

Then tell me, by what light?