We live in an interesting technological era, with so many questions about the morality and legality of using AI for creative content. As such, many researchers have taken it upon themselves to find patterns in what AI generates. They tried to put into numbers what we feel when we read or see something AI-generated.
Basically, they provided an answer to the question, “can something be too neat? Too clean? Too to the point?”
To answer this, let’s take a step back and imagine reading a short story about a woman returning to her childhood home after twenty years.
“She finds a letter hidden behind a loose floorboard. The letter reveals a family secret. She confronts her mother, finally understands why her childhood was so complicated, and learns an important lesson about forgiveness”
At first sight, that sounds like a perfectly reasonable storyline! The plot makes sense. The characters seem to have logical motivations, and the ending ties everything together.
Great, but what if we propose another version of that same story? What if the story opens at the mother’s funeral? We jump backward twenty years (because who doesn’t love a time jump?!), then forward three days, then sideways into a memory that initially seems irrelevant. The letter contradicts something we learned earlier. The daughter makes a decision that might be brave or selfish, depending on whom you ask. And the ending refuses to explain what any of it means.
Now, can we say for sure which one sounds more human?
That question is subjective, of course. Many human writers love tidy endings, and many AI-generated stories are delightfully strange. But what if these intuitions reflect measurable patterns?
In this article, I want to walk you through a fascinating paper I read recently that answers that question. The paper is titled StoryScope: Investigating Idiosyncrasies in AI Fiction. Instead of looking for suspicious vocabulary, repetitive sentence structures, or everyone’s favorite AI punctuation mark (and mine!), the em dash, the authors investigated the narrative decisions that shape a story.
And what they found was quite interesting! Without further ado, I’ll walk you through the paper.
Contents
The problem with current AI detectors
A lot of AI detection work focuses on how text is written. They look for certain words, sentence predictability, and whether the prose follows recognizable statistical patterns.
These signals can be useful. The problem is that they can also change. A language model can learn to avoid common AI clichés, a human editor can rewrite a suspicious paragraph, and a good prompt can make the machine-generated prose sound much more natural.
We can represent a piece of writing using a feature vector:
Where each component might represent something measurable about the text, such as word frequency, sentence length, punctuation, or syntactic structure, but if a new model changes its vocabulary and sentence structure, the feature distributions shift, and our previously useful decision boundary may no longer work as expected.
StoryScope asks whether we can look beyond these surface-level features by asking questions like:
-
Who drives the plot forward?
-
How predictable is the sequence of events?
-
Does the story contain subplots?
-
How much does the narrator explain the meaning of events?
-
Does the story jump around in time?
-
How morally straightforward are the characters’ decisions?
These features describe a story’s architecture, and changing it is a much bigger job than replacing a few words.
Turning stories into data
Here is where the research becomes interesting from a data science perspective, because how do we quantize “creativity”? The StoryScope researchers built a pipeline to do exactly that.
They used a dataset that contains 10,272 writing prompts. For each prompt, they paired a human-written story with five AI-generated versions from Claude, DeepSeek, Gemini, GPT, and Kimi. So, 61,632 stories!
The stories averaged roughly 4,753 words, long enough to study narrative structure at a level shorter texts might not support. They then developed a 3-step process to extract 304 narrative features from each story.
Step 1: Turn prose into structured representations.
An AI model extracts information such as characters, motivations, events, settings, plot structure, and temporal organization. Instead of comparing two stories solely through their original wording, the system creates structured representations of their content.
Stage 2: Compare stories written from the same prompt.
Because each human story has five AI-generated counterparts, the researchers can compare stories that start from the same premise. This helps control for contextual differences.
Stage 3: Convert narrative observations into measurable features.
The researchers identify recurring differences and formulate them as questions with defined answer options. For example, a feature might ask whether the protagonist’s choices or external events drive a story’s resolution, or how often the narrative jumps across time.
The result is a feature vector describing a story’s narrative decisions, which the researchers then used XGBoost classifiers and feature-importance analysis to identify the decisions that best distinguish human and AI writing.
The results
The researchers trained a classifier using narrative features while deliberately excluding stylistic features, which achieved a 93.2% macro-F1 score on the human-versus-AI classification task.
The model also achieved 68.4% macro-F1 when distinguishing among all six sources (human authors and the five AI models), which is harder to do! Because detecting whether something is AI-generated is not the same as identifying which AI model generated it.
You might be wondering now: what did they find? They distinguished three patterns AI uses when writing fiction.
1. AI likes to explain the point
They found that AI-generated stories explicitly stated their themes 77% of the time, compared with 52% for human-written stories. Think about the difference between these two endings:
She finally understood that forgiveness was not about forgetting the past, but about finding the courage to move forward.
And:
She folded the letter, placed it back beneath the floorboard, and left the house before sunrise.
The first explicitly spells out the moral of the story, and the other leaves it open to the reader’s interpretation. Sometimes a story benefits from spelling out its message, but when a model repeatedly chooses explicit explanations, that tendency becomes a measurable pattern.
2. AI prefers a tidy plot
They also found that AI stories were more likely to have straightforward causal chains and fewer subplots. About 79% of AI stories had no subplots, compared with 57% of human stories. AI-generated stories also more frequently resolved conflicts through the protagonist’s choices: 69% versus 46% for human-written stories.
By comparison, human stories were more likely to embrace morally ambiguous characters, nonlinear timelines, and unresolved tensions. For example, consider two mystery novels: In the first, every clue points to the same conclusion. Each revelation follows logically from the previous one, and the detective explains the solution at the end.
In the second, the first chapter describes a funeral. The next takes us fifteen years back. A seemingly unimportant conversation becomes crucial halfway through the novel, and the final revelation makes an earlier scene mean something entirely different.
The second structure is harder to summarize as a chain of cause and effect, which human writing tends to follow. This makes AI writing have an interesting narrative shape, a very uniform shape.
3. AI can be dramatic!
One of the more amusing findings was about sensory descriptions! AI-generated stories more frequently conveyed emotion through bodily sensations and physical metaphors: 81% versus 38% for human stories.
Think of the familiar literary combination of a tightening chest, cold sweat, trembling hands, and the room suddenly seeming darker. In their dataset, human writers often used explicit emotion labels rather than physical descriptions.
The pattern persists
Many people generate AI-written text and then edit it to sound more human. They can remove clichés, vary sentence lengths, replace generic transitions, and delete paragraphs that repeat what the previous paragraph already said.
But what happens if you improve the text without changing the story underneath?
The researchers answered this using a rewriting method called LAMP (an editing approach proposed by different researchers in May 2025). They applied surface-level edits that remove common AI tells to 278 Gemini-generated stories. This achieved 95.5% macro-F1 on the original stories and 93.9% on the edited versions! Not a very significant difference because it only changed the writing style, not the underlying story structure.
One important qualification, though: the editing experiment covered 278 stories from one model, Gemini. It is evidence that narrative features can survive a particular kind of stylistic editing, not proof that every AI-generated story will remain detectable after every kind of revision.
Can we actually use this to detect AI writing?
As a researcher, I state a study’s limitations first. No study is perfect or capable of encompassing the whole picture. So the most important thing here is that StoryScope studied long-form fiction. Its results don’t automatically apply to emails, essays, documentation, social media posts, or short articles.
Second, the dataset is constructed in a specific way: it pairs stories from human writers with AI-generated stories based on reconstructed prompts. This is a controlled comparison, but it doesn’t reflect every way people use AI in the real world.
Third, the researchers assign narrative features using language models! Finally, as always, a high classification score doesn’t prove that a particular person used AI.
The study’s 93.2% macro-F1 result applies to its evaluation setup. It does not mean a detector can examine any novel text and reliably confirm its authorship and originality.
The paper also reports stronger results from supervised text-based baselines, including near-perfect classification scores. That matters because the contribution of StoryScope is not that narrative analysis beats every other detector. Instead, interpretable narrative features retain substantial predictive power without relying on surface style.
Final thoughts
It is tempting to turn research like this into an oversimplified conclusion: “AI-generated text is always bad.” I don’t think the data supports that. AI-generated fiction can be enjoyable; a linear plot can be exactly what a reader wants. Explicit themes aren’t a literary flaw, and ambiguity isn’t automatically a sign of genius.
Language models can produce convincing sentences while still showing systematic preferences in how they organize stories. We see that all the time when we deal with ML models! A model can perform well on a task while exhibiting biases in the patterns it uses to get there. Looking only at the final output score can hide those patterns.
The authors argue that evaluating generative AI requires more than asking whether its output sounds plausible; they also encourage us to think about how to make a story more interesting by going beyond replacing predictable words with unusual ones, and instead question the structure itself.
Perhaps the next step in generative AI development isn’t just producing more convincing words; it is learning when not to take the obvious narrative path. Which, I would argue, is a much more interesting problem than counting em dashes. So writers (I mainly mean me) can go back to using em dashes without being accused of using AI.
Reference
Russell, J., Rajendhran, R., Pham, C. M., Iyyer, M., and Wieting, J. (2026). StoryScope: Investigating Idiosyncrasies in AI Fiction. Conference on Language Modeling (COLM 2026).