> In 2029, AI will not be able to read a novel and reliably answer questions about plot, character, conflicts, motivations, etc. Key will be going beyond the literal text, as Davis and I explain in Rebooting AI.

Can AI actually do this? This looks like a nice benchmark for complex language processing, since a complete novel takes up a whole lot of context (consider War and Peace or The Count of Monte Cristo). Of course the movie variety is even more challenging since it involves especially complex multi-modal input. You could easily extend it to making sense of a whole TV series.

▲

idreyn 3 hours ago | parent | next [-]

Yes. I am a novelist and I noticed a step change in what was possible here around Claude Sonnet 3.7 in terms of being able to analyze my own unpublished work for theme, implicit motivations, subtext, etc -- without having any pre-digested analysis of the work in its training data.

▲

postalrat an hour ago | parent | prev | next [-]

No human reads a novel and evaluates it as a whole. It's a story and the readers perception changes over the course of reading the book. Current AI can certainly do that.

▲

the-grump 3 hours ago | parent | prev | next [-]

Yes they can. The size of many codebases is much larger and LLMs can handle those.

Consider also that they can generate summaries and tackle the novel piecemeal, just like a human would.

Re: movies. Get YouTube premium and ask YouTube to summarize a 2hr video for you.

▲

falloutx 3 hours ago | parent [-]

Novel is different from a codebase. In code you can have a relationship between files and most files can be ignored depending on what you're doing. But for a novel, its a sequential thing, in most cases A leads to B and B leads to C and so on.

> Re: movies. Get YouTube premium and ask YouTube to summarize a 2hr video for you.

This is different from watching a movie. Can it tell what suit actor was wearing? Can it tell what the actor's face looked like? Summarising and watching are too different things.

▲

pigpop 4 minutes ago | parent | next [-]

Yes, it is possible to do those things and there are benchmarks for testing multimodal models on their ability to do so. Context length is the major limitation but longer videos can be processed in small chunks whose descriptions can be composed into larger scenes.

https://github.com/JUNJIE99/MLVU

https://huggingface.co/datasets/OpenGVLab/MVBench

Ovis and Qwen3-VL are examples of models that can work with multiple frames from a video at once to produce both visual and temporal understanding

https://huggingface.co/AIDC-AI/Ovis2.5-9B

https://github.com/QwenLM/Qwen3-VL

▲

cmcaleer 2 hours ago | parent | prev [-]

You’re moving the goalposts. Gary Marcus’ proposal was being able to ask: Who are the characters? What are their conflicts and motivations? etc.

Which is a relatively trivial task for a current LLM.

	▲	daveguy an hour ago \| parent [-]
		Gary Marcus' proposal was about a novel, and not a codebase. I think GP's point is that motivations require analysis outside the context window, which LLMs are essentially incapable of doing.

▲

colechristensen 3 hours ago | parent | prev [-]

>Can AI actually do this? This looks like a nice benchmark for complex language processing, since a complete novel takes up a whole lot of context (consider War and Peace or The Count of Monte Cristo)

Yes, you just break the book down by chapters or whatever conveniently fits in the context window to produce summaries such that all of the chapter summaries can fit in one context window.

You could also do something with a multi-pass strategy where you come up with a collection of ideas on the first pass and then look back with search to refine and prove/disprove them.

Of course for novels which existed before the time of training an LLM will already contain trained information about so having it "read" classic works like The Count of Monte Cristo and answer questions about it would be a bit of an unfair pass of the test because models will be expected to have been trained on large volumes of existing text analysis on that book.

>reliably answer questions about plot, character, conflicts, motivations

LLMs can already do this automatically with my code in a sizable project (you know what I mean), it seems pretty simple to get them to do it with a book.