Douglas B. Books
What Kind of AI Is That? · read online

Part One — The Kinds

Part One plate — the first washes

Part One plate — the first washes. Art by Douglas John Brownfield.

PART ONE — THE KINDS

---

Chapter 1 — Large Language Models: The Talkers

A large language model is a program that predicts text. Given a stretch of words, it produces the words most likely to come next, then the next, then the next, until the answer is complete. That is the whole mechanism. Everything these models do — answering questions, drafting letters, summarizing reports, writing code, holding a conversation — is that single operation, carried out at a scale where prediction becomes fluent.

The model learns to predict by training. During training it is shown an enormous body of text — books, articles, documentation, conversation, code — with parts hidden, and it is adjusted, over and over, until its guesses about the hidden parts are reliably good. What it stores from that process is not a library of sentences to look up. It stores patterns: grammar, facts, styles, the shapes of arguments and stories and programs, and the statistical relationships among all of them. When the model answers a question it was never specifically taught, it is assembling those patterns into new text. This is why a language model can be right about things in general and wrong about specifics in particular: it knows how answers of a given type usually go, which is a different thing from knowing.

Nearly every model in this chapter is built on one architecture, the transformer, introduced in a 2017 research paper. The transformer's central device is called attention: before producing each new piece of text, the model weighs which earlier pieces matter most to what comes next, and draws on those. Attention is what lets a model keep track of a subject across a long paragraph, resolve a pronoun to the right name, or hold the thread of a contract clause from page one to page forty. The designs differ in detail from maker to maker, but a reader who understands that much — prediction, trained on text, attention holding the context — understands what kind of machine a large language model is.

"Large" refers to the model's parameters, the internal numbers set during training. Current flagships carry hundreds of billions of them, and the largest now pass a trillion. Many of the biggest models use a design called mixture of experts, in which only a fraction of the parameters are active for any given piece of text: Mistral's current flagship has one trillion parameters in total and uses 52 billion at a time, and DeepSeek's uses 49 billion of its 1.6 trillion. Total size describes what the model holds; active size is closer to what an answer costs.

What this type of model is for is, in practice, almost any work that arrives as words and leaves as words. It drafts, edits, summarizes, translates, explains, and answers. It writes and repairs computer code — code is treated in this book as the talkers at a trade, not a separate kind, because the models doing it are language models trained heavily on code. It is also the engine inside most products that appear to do something else: when a service answers in prose, a large language model is almost certainly doing the talking, whatever else is attached to it. Later chapters cover the models that work in other media, and Part Three covers the layer of tools and loops that lets a talker act. This chapter's kind is the foundation the rest of the map stands on.

A reader can recognize a large language model by its inputs and outputs. Text goes in; text comes out. It is reached through a chat window or through an API, the connection other software uses. It has no native grip on the world beyond its training and whatever a given product feeds it — no memory between separate conversations unless the product supplies one, and no ability to check a fact unless the product connects it to a source. Two properties that once defined separate shelves have been absorbed into this kind, and they get chapters of their own because the story matters: today's flagships reason, working a problem in steps before answering (Chapter 2), and the leading ones natively accept images alongside text, and in some cases audio and video as well (Chapter 6). But the base type — the thing being reasoned with and seen through — is the talker described here.

Families and tiers

No maker sells a single language model. Each sells a family: a set of models of different sizes and speeds, organized in tiers. The top tier is the most capable and the most expensive to run. The tiers below trade some capability for speed and price, down to small, fast models built for volume. A family shares a name and a generation number; the tier name says where a model sits inside it.

The tier names are each maker's own, and there is no standard across vendors. In November 2026, OpenAI's GPT-6 family runs, from the top, Astra, then Sol, Terra, and Luna. Under OpenAI's current naming scheme, the number marks the generation and the tier name marks the model's place in it, with each tier updated on its own schedule. Anthropic's ladder, from the top, is Fable, Opus, Sonnet, and Haiku; Anthropic's own documentation names Opus the recommended default for most work, with the tier above it — Fable, new in 2026 — as its most capable widely released model. Alibaba's Qwen line marks its flagship tier with the word Max. The practical lesson is Part Two's subject, but it belongs to this chapter too: a tier name means something only inside its own family, and a model's full name — maker, family, generation, tier — is the smallest unit that actually identifies it.

Open weights and closed weights

One property cuts across every kind in this book, and it appears first here: whether a model's weights — the trained parameters themselves — can be downloaded. Closed models are available only through the maker's own products and API. Open-weight models can be downloaded and run on other hardware, under a license, and the license is part of the fact: this book names it, because "open" covers arrangements that differ a great deal.

The three largest American lines are closed. OpenAI's GPT models and Anthropic's Claude models are available only through those companies' APIs and products, and Google's Gemini flagships are likewise closed. The open shelf, as of this snapshot, is led from outside the United States. DeepSeek's V4 series is released under the MIT license, among the least restrictive licenses in use. Alibaba released the weights of its flagship Qwen3.8 model — a Max-class model, opened for the first time — under a custom Qwen license, while a smaller model in the same generation, Qwen3.8-27B, is under the Apache 2.0 license. Mistral, in France, published its flagship's weights under its own Mistral Ecosystem Open-Weight License, a modified form of the MIT license under which running the model as a hosted service for others requires a separate commercial arrangement. One further caution belongs here: a downloaded model and the maker's hosted product of the same name are not always identical — Alibaba states that its hosted Qwen3.8-Max is its open model with additional features added.

The November 2026 shelf

OpenAI's current line is the GPT-6 family — Astra at the top, with GPT-6.1 Sol, GPT-6.1 Terra, and GPT-6 Luna below it — and the previous generation remains in service alongside it, with GPT-5.6 Sol still a documented flagship-tier model. The scheduled retirements tell the same story of turnover: the GPT-5 family is set to retire in December 2026, and the GPT-4o snapshots follow in January 2027.

Anthropic's shelf is the Claude 5.5 generation under a new top tier: Fable 5.1, released in September 2026, is the most capable model the company offers to the general market, with Opus 5.5 as its recommended default, then Sonnet 5.5 and Haiku 5.5. Anthropic also documents a gated counterpart, Mythos 5.1 — the same underlying model as Fable 5.1 with different safeguards — restricted to vetted organizations under a trusted-access program.

Google's flagship is Gemini 4 Argon, generally available since October 2026 across Google's API, cloud, search, and Gemini app. It natively accepts text, image, video, audio, and PDF input — the widest set among the flagships in this chapter. A second, stronger model, Gemini 4 Carbon, has been announced but is not generally available: as of this snapshot it is in pre-release testing with the United States government and trusted partners, a government-first release pattern that is new for Google's main line.

Meta requires two sentences, because its position changed in 2026. Its open line is still Llama: Llama 4 Scout and Llama 4 Maverick, released in April 2025, remain the downloadable flagships, offered under Meta's own Llama Community License, which carries conditions — including a cap tied to monthly users — that standard open-source licenses do not. No Llama 5 has shipped. Meta's frontier, meanwhile, moved to a new family named Muse, developed by Meta Superintelligence Labs: Muse Spark, released in April 2026, is a natively multimodal model with closed weights, available through Meta's products rather than as a download. Open-weight models have since appeared under the Muse name as well, among them Muse Glimmer, a 30-billion-parameter model released under the Apache 2.0 license.

Mistral's flagship is Mistral Large 4: one trillion parameters, 52 billion active, natively multimodal, with its weights published in October 2026 under the license described above. DeepSeek's line is the V4 family — V4-Pro, generally available since August 2026, and the smaller V4.1-Flash — with open weights under the MIT license; DeepSeek's earlier, separate reasoning line was retired from its API in July 2026 and its capability folded into V4, which is Chapter 2's story. Alibaba's Qwen shipped four generations during 2026 and arrived at Qwen3.8, with the hosted Qwen3.8-Max at the top and, beside it, the first open release of a Max-class model's weights.

The pattern across the shelf is worth stating plainly, because it is the kind's current shape. The closed flagships are American; the largest downloadable models now come from China and Europe. Every flagship reasons, and most take images as readily as text. And the shelf itself is perishable: several of the models named here will be replaced within the year. The kind — a trained predictor of text, working by attention over context — is what endures, and it is the kind the rest of this book builds on.

On the shelf, November 2026

>

- OpenAI (closed) — GPT-6 family: Astra, 6.1 Sol, Terra, Luna. GPT-5.6 Sol still in service.

- Anthropic (closed) — Claude Fable 5.1, Opus 5.5, Sonnet 5.5, Haiku 5.5. Mythos 5.1 gated to vetted organizations.

- Google (closed) — Gemini 4 Argon, generally available. Gemini 4 Carbon announced, in gated pre-release testing.

- Meta (split) — Llama 4 Scout and Maverick, downloadable under the Llama Community License. Muse family at the frontier, mostly closed; Muse Glimmer open under Apache 2.0.

- Mistral (open weights) — Mistral Large 4, 1T parameters / 52B active, under the Mistral Ecosystem Open-Weight License.

- DeepSeek (open weights) — V4-Pro and V4.1-Flash, under the MIT license.

- Alibaba Qwen (both) — Qwen3.8-Max hosted; Qwen3.8-2.4T-A95B weights released under a custom Qwen license; Qwen3.8-27B under Apache 2.0.

---

Chapter 2 — Reasoning Models: The Kind That Became a Setting

This chapter covers a kind that no longer exists as a kind. Two years ago, reasoning models were a separate shelf: distinct names, distinct prices, picked for distinct jobs. By November 2026, every major maker had folded reasoning into its flagship talkers, where it now lives as a mode — a setting, with a dial. The chapter stays in the book for two reasons. Reasoning was a real type, with a real difference in how the models worked. And its melting is the clearest case in this book of how kinds behave: a kind can be genuine, matter for a while, and still end as a feature of the kind next to it.

What they were

A plain large language model answers in one pass. It reads the question and writes the answer in a single forward run, the way a person blurts out a reply. That works well for most things. It works badly for problems that have to be carried in steps — a proof, a budget, a piece of code with moving parts — where the first sentence out is rarely the right place to start.

Reasoning models were trained to do the carrying first. Before answering, the model works through the problem in steps: it lays out sub-problems, tries an approach, checks it, backs up when a step fails, and only then writes the final answer. The working is usually hidden from the reader, or shown as a summary. What the reader gets is the answer at the end of it.

The trade behind this has a name worth knowing: test-time compute. An ordinary model spends nearly all of its computing power in training, before it ever meets you; answering is the cheap part. A reasoning model spends computing power at answer time as well — the longer it is allowed to think, the more work it does on your particular problem. That is why these models were slower, why they cost more per answer, and why, on hard multi-step problems, they were often plainly better. You were not paying for a smarter memory. You were paying for time spent on the question.

Why they stood apart

In 2024 and 2025, reasoning was sold as a separate product line, not a setting. OpenAI kept its o-series — o1, o3, and their smaller variants — beside the GPT chat models, and pointed customers at one or the other depending on the job. DeepSeek shipped a pair: V3 as the general model and R1 as the reasoner, and R1's release in early 2025 was a genuine event — an open-weight reasoning model, MIT-licensed, that matched the closed labs' best. Anthropic offered reasoning as a named feature, Claude's "extended thinking." Google had Deep Think. A buyer in 2025 faced a real choice between two types of model, and the choice mattered: pick the talker for speed and price, the reasoner for problems with steps.

That is what made reasoning a kind in the first place. It was not a size or a brand tier. It was a different way of spending compute, trained into the model, and sold under its own names.

What happened

The capability did not disappear. The shelf did. The record, by maker:

OpenAI wound the o-series down. On October 23, 2026, o1, o1-pro, o3-mini, and o4-mini were shut down; o3 and o3-pro are scheduled to follow on December 11, 2026. In every case the listed replacement is a current GPT model with a reasoning setting. In the current GPT line, reasoning is a control — reasoning.effort, running from none through low, medium, and high to still higher settings on the top models.

DeepSeek retired R1 from its own API on July 24, 2026, days after its V4 line reached general release, and folded the capability into V4's thinking modes. V4-Pro offers three depths of thinking, on by default. R1 itself is not gone: its MIT-licensed weights remain downloadable, and people run it themselves. What ended was R1 as a separate hosted product. No R2 has been publicly released as of this snapshot.

Anthropic stopped treating thinking as a feature with a name. In the current Claude models, thinking is built in and on by default, with effort levels — the current defaults sit at medium on some models and high on others, and the user or the software can turn the dial up or down.

The pattern repeats at the smaller scale. Mistral's flagship offers reasoning as levels in the API, not as a separate model. Google describes its current flagship, Gemini 4 Argon, as "a frontier reasoning model," which is the melt stated in the maker's own words: the flagship is the reasoner now. Deep Think survives as a named mode, still presented by DeepMind on a Gemini 3.1 base.

What to take from the melt

First, learn the dial. Wherever you use a current flagship — OpenAI's, Anthropic's, DeepSeek's, Google's — there is an effort setting, whether or not the app in front of you shows it. It may be labeled effort, thinking, or a depth choice; it may be a fixed default chosen for you. The question "which model should I use for this?" has quietly become "how hard should this model think about this?" Those are different questions, and the second one is usually the one that decides the quality of the answer.

Second, match the thinking to the problem. High effort earns its cost on work with steps that depend on each other: mathematics, code that has to run, plans with constraints, arguments that have to hold together, and checking — asking a model to verify its own or another's work. It earns little on lookups, summaries, first drafts, and anything where a quick answer can be corrected in the next exchange. Paying for maximum thinking on a simple question buys delay, not quality.

Third, notice that the dial is already set. Because thinking is now on by default at the major makers, a reader can be paying for reasoning — in time and in tokens — without ever having chosen it. The defaults are sensible for general use, but they are choices made by the maker, not by you. For cheap, high-volume work, the low and none settings exist on purpose: a flagship told not to think is still a flagship, and it answers at talker speed. Knowing the dial exists is what turns a default into a decision.

Fourth, count the cost correctly. Thinking is billed and timed like output, because it is output — the model writes its working before it writes your answer, and you wait for both. A long think on a hard problem can take minutes and cost several times what the same model's quick answer costs. That is a fair trade when the problem is hard. It is waste when it is not.

And last, the general lesson, which the rest of this book will use again: when you meet a new model name, ask whether it names a kind — a different way of working, trained in — or a mode, a tier, or a moment in a lineup. Reasoning spent two years as a kind and ended as a dial. The dial is worth more than the shelf ever was. But you have to know it is there.

On the shelf, November 2026

>

There is no reasoning shelf. Reasoning lives inside the flagship talkers, as a mode with an effort dial:

>

- OpenAI — the GPT line, with reasoning.effort from none to maximum settings; the o-series shut down October 23, 2026, with o3 scheduled to follow December 11, 2026.

- DeepSeek — V4-Pro and V4.1-Flash, thinking on by default, three depths on the Pro model; open weights, MIT license. R1's weights remain downloadable; R1 left the hosted API July 24, 2026.

- Anthropic — Claude models think by default, with effort levels; no separate reasoning product.

- Mistral — Large 4, reasoning offered as API levels rather than a separate model.

- Google — Gemini 4 Argon, billed by Google as a frontier reasoning model; Deep Think remains a named mode on a Gemini 3.1 base.

---

Chapter 3 — Image Models

An image model takes words in and gives a picture back. You describe what you want — a lighthouse at dawn, a product photograph of a red chair, a cartoon fox in a spacesuit — and the model produces an image of it. Most also work in the other direction on existing pictures: give one an image plus an instruction, and it returns an edited version, with an object removed, a background replaced, or a style changed. Of all the types in this book, this is the kind most people have already used, often without knowing its name.

What an image model is

Most image models of the last several years were built on a technique called diffusion. During training, the model is shown millions of images alongside text describing them, while those images are gradually buried in random noise, step by step, until nothing is left but static. The model learns to run that process in reverse. When you give it a prompt, it starts from a field of pure noise and removes the noise in stages, guided at every stage by your words, until a picture that matches the description emerges. Think of it as sculpting: the block is static, and the prompt tells the model what to carve away. Newer models vary the recipe — some generate the image in patches or in sequence, and some, like Google's current line, are built inside a general multimodal model rather than standing alone as a dedicated diffusion system. The techniques differ. The kind is defined by the exchange: words in, image out.

What it is for

The everyday uses are practical. Illustration for articles, books, and presentations. Product photographs and advertising drafts made without a camera or a studio. Concept art and storyboards — a quick look at an idea before anyone spends money on the real version. Design work where words have to appear inside the picture, such as posters, signs, and logos, which the better current models handle reliably after years of garbled lettering. And editing: extending a photograph beyond its edges, removing a stranger from a holiday picture, restyling an image you already own. This kind also does a great deal of quiet work inside other products, generating the pictures that talkers and apps hand back as part of a larger job.

One behavior follows from how this type works, and it surprises people the first time: the same prompt rarely gives the same picture twice. Because generation starts from random noise, each run carves a different image out of the static, and two people's identical prompts will differ as well. Skilled users treat this as part of the tool rather than a fault. They generate several versions, keep the one that works, and refine from there — or use the model's editing features to push a near-miss the rest of the way. A related habit separates the current models from their ancestors: keeping the same character, product, or style recognizable across a series of images, so a set of pictures can belong to one story or one brand. That consistency, more than raw picture quality, is what the recent generations have been competing on.

How to recognize one

The recognition test is the exchange. If the main thing going in is a text description and the main thing coming out is a picture, you are looking at an image model, whatever the app around it is called. Editing features do not change the classification: an image plus an instruction in, an edited image out, is still this kind. The settings give it away too. Image models ask about size and aspect ratio, about style, and about how closely to follow the prompt — they do not carry on a conversation. If a service writes you three paragraphs about your picture instead of giving you the picture, a talker is doing the talking and an image model is working behind it. One more habit is worth knowing: the same name can cover several different models at once, sold in tiers. What matters is which tier you actually used, and when.

The November 2026 shelf

Two of the most famous names in this kind are gone, and the shelf is clearer for knowing it.

Midjourney remains the closed, hosted service it has been, and its current default model is V8.2, in place since July 24, 2026, with an updated editing model in testing behind it. Midjourney has never released weights; you use its models on its service or not at all.

OpenAI's image line is where the first famous name used to be. DALL·E was retired from OpenAI's API on May 12, 2026, and the name no longer exists as a product. Its replacement is the GPT Image family, and by November 2026 that line is already on GPT Image 2.5 — the first GPT Image version itself shut down in October 2026. “DALL·E” survives as a household word for the whole kind. It is no longer a model you can use.

Google's shelf turned over the same way. Imagen, Google's dedicated image line, shut down — Google's own migration documentation set the final shutdown on the Gemini API for August 17, 2026. The successor is a Gemini-native family nicknamed “Nano Banana”: Nano Banana 2 and Nano Banana Pro, image models built inside the Gemini line rather than standing apart from it. Google's own naming churned during 2026, and the book reports the nickname because that is what the models are called in Google's documentation and in use. The pattern to notice: at both OpenAI and Google, the separate image brand was absorbed into the maker's main model family.

Ideogram split in two during 2026, and the split is the useful lesson. Ideogram 4.0, released in June 2026, is the company's first open-weight model — downloadable, and known for design work and accurate text inside images. Ideogram 4.5, released in September 2026, is the newer flagship, and it is closed: hosted only, including its precise-editing features. Same company, same year, two different arrangements. Which Ideogram a person means depends on which side of the split they are standing on.

FLUX, from Black Forest Labs, changed shape. In 2025, FLUX was the name most people meant by an open-weights image model. By November 2026, FLUX is a broader family — its newest generation is reported as one foundation spanning image, video, and audio — and its newest image model is closed. Open weights survive in the family's Klein tier and in FLUX.2 [dev], the established downloadable model. The name stayed; the arrangement underneath it moved.

Stable Diffusion, from Stability AI, is the shelf's quiet entry. Its current family is still SD 3.5, released in October 2024 — no new base image model has followed in over two years, and the company has directed its recent work toward audio. The weights remain available and widely used. The famous open image model of 2022 to 2024 is alive; it is no longer where the frontier of this kind moves.

Open and closed, with the licenses named

This kind shows why this book names licenses instead of just saying “open.” Closed here means hosted only: Midjourney, GPT Image, Google's Nano Banana models, Ideogram 4.5, and FLUX's newest image model are all used on the maker's service, under the maker's terms, and can be changed or retired on the maker's schedule — as DALL·E and Imagen were. Open weights means the model file can be downloaded and run on your own hardware, but the license decides what you may do with it, and in this kind the licenses differ in ways that matter. Ideogram 4.0's weights are under the Ideogram Non-Commercial Model Agreement: free to download and use non-commercially, with a paid license required for commercial self-hosting. (The Apache 2.0 license attached to Ideogram's release covers its inference code, not the weights — the two are often confused.) FLUX.2 [dev] is under the FLUX Non-Commercial License. Within the FLUX Klein tier, terms differ by size: the FLUX.2 Klein 4B model is Apache 2.0, while the 9B is non-commercial. Stable Diffusion's SD 3.5 weights are under the Stability Community License. Two models can both be called “open” and permit entirely different things. Read the license, not the label.

On the shelf, November 2026

>

- Midjourney V8.2 — closed, hosted; the current default since July 2026.

- OpenAI GPT Image 2.5 — closed; DALL·E retired May 12, 2026, and the GPT Image line replaced it.

- Google “Nano Banana” (Nano Banana 2, Nano Banana Pro) — closed, Gemini-native; Imagen shut down August 17, 2026.

- Ideogram 4.0 — open weights under the Ideogram Non-Commercial Model Agreement. Ideogram 4.5 — closed flagship.

- FLUX — newest image model closed; open weights in the Klein tier (terms vary by size) and FLUX.2 [dev] under the FLUX Non-Commercial License.

- Stable Diffusion SD 3.5 — open weights under the Stability Community License; still the latest base model, unchanged since October 2024.

---

Chapter 4 — Video Models

A video model generates moving pictures. You describe a shot in words — or hand it a still image and ask for motion — and it returns a few seconds of footage: a camera pushing in, a person crossing a room, rain on a window. The kind exists because a single image is often not the thing people actually want. They want the moment, not the frame.

What separates this type from the image models of the previous chapter is time. An image model has to get one frame right. A video model has to get every frame right and keep them consistent with each other — the same face, the same coat, the same light from the first frame to the last, while things move. That consistency problem is the technical heart of the kind. It is why video models arrived later than the other kinds, why they cost more to run, and why they still work in short clips rather than long scenes. Most current models produce footage measured in seconds. A finished minute of video is normally many clips, generated one at a time and edited together.

What it is for. Advertising shots and product clips. Storyboards and previsualization for film and television. Social-media video. Background and stock-style footage. A quick prototype of a scene that would be expensive to shoot. And, increasingly, editing: taking footage that already exists and changing it — removing an object, restyling a shot, extending what the camera captured. Generation and editing are the two halves of the kind, and some vendors sell them as separate models.

How you recognize one. Four signs:

The limits you will meet. Because consistency across frames is the hard part, the failures of this type are consistency failures. Hands and faces drift. Text rendered inside a scene garbles as the camera moves. Objects appear, vanish, or change shape between frames; physical motion — pouring water, a bouncing ball — can look plausible at a glance and wrong on a second look. Length is the other limit: a model asked for one continuous minute will usually do worse than the same model asked for six ten-second shots. Working practice in this kind follows from those facts. Generate short. Generate more than you need and keep the best takes. Give the model reference images when a person or product has to stay recognizable, and expect a human edit at the end. None of this is a defect of one brand; it is the current shape of the type, and it is the honest baseline against which each new release should be measured.

The shelf, November 2026. Five working names and one empty chair. With one partial exception, everything on this shelf is closed: rented from the maker through an app or an API, not downloaded and run on your own machine.

Runway is the specialist that stayed a specialist. Its current flagship is Gen-4.5, released in December 2025 — text- and image-to-video, picture only, with no native audio. Its editing half is a separate model, Aleph 2.0. One company, one kind, two models: the plainest illustration on the shelf of the generation/editing split described above.

Kling, from Kuaishou, is the fastest-moving name here — two full generations within a year. Kling 3.0 shipped in early 2026; Kling 4.0 became fully available in October 2026, with clips of up to thirty seconds and sound included. Where Runway separates its functions, Kling keeps adding them.

Luma fields the Ray family; its current flagship is Ray3.2, released in 2026, silent like Runway's models. Luma matters to this chapter for a second reason. In 2026 it rebuilt its product, Dream Machine, so that it now hosts other companies' models — Google's and Kuaishou's among them — alongside its own Ray line. One of the makers on this shelf has also become a storefront for its rivals. That is a distribution change, not a new kind, but a reader comparing tools will meet it and should know what it is.

Google keeps two names in this kind. Veo 3.1 is the current Veo. But Google now steers its users to a different model — Gemini Omni Flash — as the default video model in its API, and Omni has replaced Veo inside the Gemini app. No Veo 4 has been announced. The pattern will be familiar from the image chapter: a standalone media brand giving way to a model built natively into the maker's main line.

Pika covers both halves of the open-and-closed question in one company. Its own current model is Pika 2.5, closed like the rest of this shelf. Its notable 2026 release, though, is Pika H3 Flex: Pika took MiniMax's open H3 model, post-trained it for speed, and in October 2026 released the result as open weights under the MiniMax H3 Community License. That license is why this book names licenses instead of just saying "open": it excludes commercial use in the United States, the European Union, the United Kingdom, and South Korea. H3 Flex is the one entry on this shelf you can download — and its terms are a worked example of reading what "open" actually permits before you build on it. (The FLUX family, covered with the image models in Chapter 3, has likewise grown a video member; this kind is where it will be met in practice.)

Sora is the empty chair. Sora was OpenAI's video model and, through 2024 and 2025, the most famous name in this kind. It no longer exists. OpenAI announced its discontinuation in March 2026. The Sora app and website shut down on April 26, 2026. The Sora models and the Videos API were removed on September 24, 2026, with no replacement. As of November 2026, OpenAI offers no video generation product at all.

What the Sora exit teaches. Almost every model in this chapter is a rental. A rented model comes with the maker's roadmap and the maker's freedom to end the product, and Sora's users are the standing example: people who built workflows, habits, and in some cases businesses on the most prominent video model available had a few months' notice to move them or lose them. Nothing about the kind failed. Video generation is more capable and more widely available in November 2026 than it was when Sora launched. What ended was one company's product, on that company's schedule.

The practical conclusions are plain. Keep your own copies of the things that are yours — the prompts, the reference images, the finished clips — rather than leaving them inside one vendor's app. Learn the kind, not just the tool: the skill of describing a shot transfers to every model on this shelf, while familiarity with one interface transfers to none of them. And treat moving your work between vendors as a normal, recurring cost of using this type of model, not as an emergency. A name is a lease. A kind is a category. This book sorts by the category.

On the shelf, November 2026

>

- Runway Gen-4.5 — generation; Aleph 2.0 — editing. Closed.

- Kling 4.0 (Kuaishou) — the shelf's longest clips, with sound. Closed.

- Luma Ray3.2 — also sold from a storefront that hosts its rivals. Closed.

- Google Veo 3.1 — current Veo; Gemini Omni Flash is now the default. Closed.

- Pika 2.5 — the in-house line. Closed. Pika H3 Flex — open weights under the MiniMax H3 Community License, territory-restricted.

- Sora (OpenAI) — discontinued in 2026; no replacement. The chair is empty.

---

Chapter 5 — Audio Models

Every other kind in this part of the book works in text or pictures. Audio models work in sound, and sound runs in more than one direction. That is the first thing to understand about this kind: it is really three types sharing one shelf.

The first type listens. Sound goes in, text comes out — a recording of a meeting becomes a transcript. The second type speaks. Text goes in, a voice comes out — a written page becomes narration. The third type makes music. A description goes in, and a finished song comes out, instruments and vocals included.

These are different jobs, done by different models, sold by different companies. A model that transcribes well does not speak well, and neither one writes songs. When someone says "audio AI," the useful question is: which direction?

Ears in: transcription

A transcription model takes recorded speech and returns written words. It is the least glamorous of the three types and probably the most used. Interviews, lectures, meetings, phone calls, podcasts — anywhere people talk and someone later needs the text, a model of this type is doing the work.

The work these models do is narrower than it sounds. A transcription model does not understand a conversation the way a language model does; it matches sound to words, adds punctuation and capital letters, and, in the better versions, marks where one speaker stops and another begins. Judged on accuracy, the differences between good ones are small and mostly show up in hard cases: names, accents, technical vocabulary, and several people talking at once in a noisy room.

The standard example is Whisper, from OpenAI. Whisper was released as open weights, and the open release is the interesting part of its story. The downloadable models — large-v3 and its faster sibling, large-v3-turbo — are unchanged and remain free for anyone to run under the MIT license, one of the most permissive licenses there is. If you downloaded Whisper two years ago, the copy on your machine today is still current, still legal to use, and still entirely yours.

OpenAI's hosted service has moved on. In August 2026 the company deprecated the API models built on Whisper — whisper-1 and the transcribe models in the GPT-4o line — and set their shutdown for February 2027. The default transcription model in OpenAI's API is now called gpt-transcribe, a closed model from the GPT family. It is not a new version of Whisper. It is a different kind of model doing the same job, rented by the minute instead of owned.

Both facts belong in the book because together they teach something general. The name "Whisper" now refers to two different things: a finished, open artifact that anyone can keep, and a retired endpoint in a company's catalogue. The hosted service had a lifespan. The open weights do not — they last exactly as long as people care to run them. This pattern, an open release outliving the product it came from, appears in several kinds in this book. Audio is where it is easiest to see.

Voices out: speech synthesis

A speech model does the reverse of transcription. Text goes in; a spoken performance comes out. The modern versions do more than read words aloud in order. They handle pacing, emphasis, and emotion, and they can hold a consistent voice across a book-length narration or switch between speakers in a dialogue.

The leading name on this shelf is ElevenLabs. Its flagship model is Eleven v3, which became generally available in early 2026. Eleven v3 works in more than seventy languages, accepts directions inside the text itself — markers that tell the voice to whisper, to laugh, to pause — and handles conversations with several distinct speakers. It is built for quality rather than speed, and it is not the model ElevenLabs offers for live conversation.

That job belongs to a second model, Flash v2.5, which answers in roughly seventy-five milliseconds — fast enough that a person talking to it does not feel the delay. A third, Multilingual v2, remains the workhorse for long-form narration, the audiobook end of the trade. This division is typical of the kind: one model tuned to sound its best, another tuned to answer at once, a third tuned to read steadily for hours, and the buyer picks according to the work. ElevenLabs' older Turbo models, which once filled the fast role, are deprecated, with Flash named as their replacement.

ElevenLabs is a closed model. There are no weights to download; the voice exists only inside the company's service. It is also worth noticing, as a sign of where the kind is heading, that the same company now sells a transcription model and a music model alongside its voices. The three types on this shelf are separate kinds of model, but nothing stops one company from fielding all three.

Music

A music model takes a description — a genre, a mood, sometimes lyrics — and returns a complete recording: melody, arrangement, instruments, and usually a sung vocal. Of the three types in this chapter, it is the newest, and it is the one whose recent history has been decided as much in legal offices as in laboratories.

The leading name is Suno. The line moved quickly — v5 arrived in September 2025, and v5.5 followed in March 2026 with voice and custom-model features. Its current model is v6, released on September 9, 2026, together with two smaller variants, v6-wild and v6-mini. On the day v6 launched, Suno retired every older model it had — versions 4 through 5.5 stopped being available at once, although songs already made with them remain in their owners' libraries. That clean sweep is unusual, and it has a reason: v6 is Suno's first generation built together with partners from the music industry, including Warner Music Group, after Suno settled the lawsuits the record companies had brought against it. The model on the shelf in November 2026 is, in part, a product of those settlements.

The second name on the shelf shows the other way the same pressure can end. Udio was, for a time, Suno's closest rival. Its current model is still v1.5 Allegro, released in March 2025. Nothing newer has shipped. Udio also settled with the labels — Universal Music Group in October 2025, Warner a month later — and on the day the Universal settlement was announced, downloads were switched off across the whole platform: no audio files, no video, no stems. They remain off. The terms now limit what users make to personal, non-commercial use, and the fully licensed platform Udio promised for 2026 had not launched as of this snapshot, with no date published.

So the music shelf in November 2026 holds one model that was rebuilt with the industry's participation and one that is waiting, intact but unable to hand its users their own songs. Neither outcome was decided by the quality of the models. It is a fair lesson in reading any new model: the technology is only part of what determines whether a kind thrives. Rights, licenses, and settlements are part of the shelf too.

Reading a new audio model, then, starts with direction. Ears in, voices out, or music — the three types are not interchangeable, and a launch announcement that does not say which one it means has not told you much. After direction comes the question this book asks of every kind: open or closed, and under what license? Whisper's weights can be downloaded and kept; ElevenLabs' voices and Suno's songs exist only inside their services, on whatever terms those services set — and the music shelf shows what can happen to the work when the terms change.

On the shelf, November 2026

>

- Ears in (transcription): Whisper — open weights, unchanged, MIT license; a finished artifact anyone can run. OpenAI's hosted Whisper endpoints are deprecated, replaced in the API by the closed gpt-transcribe.

- Voices out (speech synthesis): ElevenLabs — Eleven v3 flagship for quality, Flash v2.5 for real-time conversation; the older Turbo models deprecated. Closed; service only.

- Music: Suno v6, released September 9, 2026, with all earlier versions retired the same day; built with music-industry partners after its label settlements. Udio remains on v1.5 (March 2025), downloads disabled since its October 2025 settlement, its licensed relaunch not yet shipped.

>

Three directions, three types — and one name, Whisper, that now means both a model you can own and a service that has already ended.

---

Chapter 6 — Multimodal Models: The Kind Becoming a Property

A multimodal model is one model that works in more than one medium natively. It can take text, images, audio, or video as input — not through attachments or plug-ins, but as part of what the model itself was built and trained to handle. The word that does the work in that sentence is natively.

The alternative is assembly. For years, a system that seemed to see and hear was usually several single-medium models wired together: a speech recognizer turned talk into a transcript, a language model read the transcript and wrote a reply, and a voice model read the reply aloud. Systems like that work, and many are still in use. But each handoff is a translation, and translations lose things — tone of voice, emphasis, the pause before an answer. The language model in the middle never heard the question; it read a report of the question. A native multimodal model has no middleman. The same model that reasons about the words also processes the picture or the recording, so it can use what is in each medium when it thinks about the others.

That distinction is why multimodal models get a chapter of their own in this book — and the chapter comes with a warning label, because this kind is in the middle of disappearing into the others.

The omni era

For most of the field's short history, models were single-medium by default. Language models handled text. Image models handled pictures. A model that natively combined media was the exception, and it looked like a new type of thing.

The model that made the point was OpenAI's GPT-4o, released in 2024. The "o" stood for "omni." One model took text, image, and audio input and could answer in text or in speech, holding the media together in a single system rather than passing them between specialists. Other makers fielded their own native multimodal models, and for a time "multimodal" — or "omni" — named a shelf of its own: a kind you chose on purpose, distinct from the plain talkers.

The state of it, November 2026

The shelf has emptied, and it emptied upward: the flagships absorbed the kind.

The clearest evidence is the fate of the model that defined it. GPT-4o is being retired. One of its dated snapshots was shut down on October 23, 2026, and the remaining snapshots are scheduled to retire on January 14, 2027, with GPT-6 Luna named as the replacement. A maker does not retire the model that founded a category unless the category has been folded into the main line — and it has been. The current GPT models take text and image input as a matter of course. There is no separate omni line to choose; the property comes with the flagship.

Google's Gemini 4 Argon, the current Gemini flagship, shows how far the absorption has gone. It natively accepts five types of input — text, image, video, audio, and PDF documents — in a single model, the deepest native set among the flagships surveyed for this book. Anthropic's current Claude models all accept text and image input. Mistral's flagship, Mistral Large 4, is described by its maker as natively multimodal, and its weights are published under the Mistral Ecosystem Open-Weight License. Meta's Muse Spark is a natively multimodal reasoning model.

The property has also spread downward. DeepSeek's V4.1-Flash is its speed tier, not its flagship, and it gained native visual understanding in 2026. Alibaba's Qwen3.8-27B is a mid-size open model — its weights are published under the Apache 2.0 license — and it takes image and video input. When a capability reaches the cheaper and smaller members of a family, it has stopped being a specialty. It is part of what a model is now expected to do.

Stated plainly: in November 2026, a frontier model that handles only text is the exception. Multimodality at the top of the market is no longer a kind of model. It is a property the leading models have.

What still differs

A property can be standard and still vary. Three differences still separate one multimodal model from another, and they are the differences worth checking.

The first is which senses. The input lists are not the same. Gemini 4 Argon takes video and audio natively; the current Claude line takes text and image, not sound or motion. "Multimodal" on two labels can mean two different sets of media.

The second is input against output. Most of today's multimodal flagships take more in than they give out. They accept images, and in some cases audio and video, but they answer in text. The founding model was broader on this point: GPT-4o could answer in speech as well as text. Its successors, as a group, take more media in than it did and send less out. Producing a picture, a voice, or a moving image is still mostly the work of the specialist types covered in the image, video, and audio chapters — or of a sibling model offered alongside the flagship in the same product. A model that can look at a photograph cannot necessarily draw one. When a label says multimodal, it is almost always describing the model's eyes and ears, not its hands.

The third is depth. Native handling is a spectrum. At the deep end, one model holds the media in a shared internal form, so a question about a diagram can draw on the diagram and the caption and the surrounding text at once. At the shallow end, media are converted to text summaries early and the model reasons over the summaries — closer to the old assembly line, wearing a native label. The deep end shows in work like computer use: OpenAI documents its GPT-6.1 Terra tier as handling visual reasoning and computer use — reading a screen as an image and acting on what it sees — which is only possible when the seeing is inside the model doing the reasoning. The label alone does not say where on that spectrum a given model sits; the maker's documentation, and testing on material like your own, does.

The lesson of a melting kind

This is the second chapter in this book about a kind that stopped standing on its own — reasoning did it first, becoming a mode of the talkers. Multimodal is doing it more slowly and more completely: not a mode, but a property, spreading until it belongs to nearly every model that matters.

The practical lesson is about labels. When a kind melts into a property, the word stops sorting. It still tells the truth — the model really does handle several media — but it no longer tells you which shelf a model belongs on, because the answer is all of them. The questions that sort are the narrower ones: Which media go in? Which media come out? How deeply are they joined? A reader who asks those three questions of any new model will know more about it than a reader who stops at the word "multimodal."

That is where this book leaves the word: accurate, and no longer load-bearing.

On the shelf, November 2026

---

Chapter 7 — Small and On-Device Models: The Pocket Kind

Every other kind in this book is sorted by what a model does — talk, reason, draw, listen. This kind is sorted by where a model lives. A small model is one that runs on the device in your hand or on your desk: on the phone itself, not in a data center the request has to travel to. That single fact — where the work happens — produces a cluster of properties that no other kind can match, and it is why small is a kind of its own and not merely a size.

Start with what the location buys. Privacy: if the model runs on the device, the words, photos, and recordings it works on never have to leave the device. Offline operation: a model on the phone works in a tunnel, on a plane, in a dead zone — anywhere the phone works, because there is no server to reach. Speed of response: there is no round trip across a network, so answers start immediately. Cost: there is no per-call charge, because there is no call. The company that shipped the model does not meter your use of it, and using it a thousand times costs nothing beyond the battery.

Those four properties are the kind. A model earns its place on this shelf not by being under some parameter count but by being built to live under those conditions: limited memory, limited power, no connection assumed. A shrunken copy of a data-center model that still phones home for every answer is not this kind. A model designed from the start to work alone, in the pocket, is.

How a small model stays capable

The obvious question is what gets lost. A model with a few billion parameters cannot hold what a model with a trillion holds, and the honest answer is that small models know less and reason less deeply than the flagships of the earlier chapters. The work of the last several years has gone into narrowing that gap by three main routes.

The first route is better training material. Microsoft's Phi line is the standing example: its models are trained on carefully curated, textbook-quality data, on the theory that a small model fed well-organized material learns more per parameter than one fed the raw internet. The second route is distillation — a large model, the teacher, generates answers and worked examples, and a small model, the student, is trained on the teacher's output rather than on raw text. The student does not learn everything the teacher knows; it learns the teacher's way of answering, in a fraction of the space. The third route is sparsity, and it has produced the most striking design on the current shelf.

A sparse model does not use all of itself for every request. It carries a large total of parameters but activates only a small subset for any given question — the parts relevant to that question. Apple's current on-device line is the worked example. Its larger phone model, AFM 3 Core Advanced, carries 20 billion parameters in total but wakes only 1 to 4 billion of them per request. The memory the phone must hold is the full 20 billion; the computation the phone must perform is the small fraction. That is how a model of that total size can live on a phone at all, and it is why "how big is it" has become an ambiguous question for this type: the stored size and the working size are now two different numbers.

On the shelf, November 2026

Five lines make up the current shelf, and they do not move at the same pace.

Microsoft Phi is the research line that defined the kind: small language models built on data quality. The current generation is still Phi-4 — no Phi-5 has been announced. The family includes the 14B Phi-4, the 3.8B Phi-4-mini, multimodal and reasoning members, and its newest general member, Phi-4-reasoning-vision-15B (March 2026), a 15B model that sees images and decides for itself when to think step by step and when to answer directly. The weights are published under the MIT license.

Google Gemma is Google's downloadable family, built from the same research as the closed Gemini line. Gemma 4 arrived in April 2026: Effective 2B and Effective 4B sizes built for phones and edge hardware — natively multimodal, and the two small sizes take audio input as well — alongside larger 26B and 31B members. The change worth recording is the license. Gemma 1 through 3 shipped under Google's custom Gemma Terms of Use; Gemma 4 ships under Apache 2.0, the standard license, with no custom terms layered on top.

Alibaba Qwen runs the fastest-moving open family, and its small shelf is a ladder: the Qwen3.5 small series — 0.8B, 2B, 4B, and 9B, released in March 2026 under Apache 2.0, natively multimodal like the rest of that generation. Qwen shipped two further generations during 2026, but their new sizes start at 27B, so the March small series remains the on-device line. (Qwen's largest tiers — the hosted Max, Plus, and Flash services — are a different matter: those are API-only and belong with the talkers of Chapter 1.)

Apple Foundation Models are the kind in its purest commercial form: models that exist only inside a product, never downloaded, never sold separately. Apple's third generation of the family runs in two on-device tiers — AFM 3 Core, a 3B model, and the sparse AFM 3 Core Advanced described above. Where the work is too big for the device, Apple's Private Cloud Compute servers take the request; the on-device shelf is these two. One fact about their making belongs in the record: Apple states that this generation was custom-built in collaboration with Google. The weights are closed and unpublished; app developers reach the on-device model only through Apple's framework.

Meta Llama, at small sizes, is the line that stood still. Meta's small on-device offering remains the Llama 3.2 1B and 3B models of September 2024, published under the Llama Community License — a custom license, permissive for most uses but not a standard open-source license. No new small Llama appeared in 2025 or 2026, and Meta's newest frontier work has gone to a closed line instead. The 2024 models still work, and they are still the shelf entry; the record is simply that this seat has not been updated while the others moved.

What this kind is chosen for

Readers meet this type most often without choosing it: the model inside a phone's built-in assistant features, summarization, writing help, and photo tools is a small on-device model, picked by the manufacturer for the four properties above. Where a person or a company does the choosing, the reasons are the same four. Choose the pocket kind when the material is private and should not travel; when the work must happen with no connection; when the answer is needed instantly, thousands of times a day, at no per-call cost; or when the job is narrow and constant — sorting, summarizing, transcribing, drafting — rather than open-ended. Choose a flagship from the earlier chapters when the question is hard, the knowledge required is wide, or the reasoning is long. The two kinds are not rivals. Most devices now run both: the small model on the spot, the large one a network away, and the craft is in knowing which of the two a given task is actually asking for.

On the shelf, November 2026

- Microsoft Phi-4 line — including Phi-4-mini 3.8B and Phi-4-reasoning-vision-15B (Mar 2026). Open weights, MIT license.

- Google Gemma 4 — Effective 2B / Effective 4B for phones; 26B and 31B above them (Apr 2026). Open weights, Apache 2.0.

- Alibaba Qwen3.5 small series — 0.8B / 2B / 4B / 9B (Mar 2026). Open weights, Apache 2.0.

- Apple Foundation Models (3rd generation) — AFM 3 Core 3B; AFM 3 Core Advanced, 20B sparse activating 1–4B. Closed; ships inside Apple products only.

- Meta Llama 3.2 1B / 3B — September 2024; still Meta's small line. Open weights, Llama Community License.

---

Chapter 8 — Embedding and Retrieval Models: The Librarians

Every other kind in this book answers you. The talkers write, the image models draw, the audio models speak and listen. This type does none of that. An embedding model reads a piece of text — a sentence, a paragraph, a whole document — and hands back a list of numbers. That is all it produces, and it is enough to run most of the finding in modern software. If the talkers are the voices of AI, this is the index: quiet, invisible, and underneath nearly everything.

An embedding is a translation of meaning into position. The model converts a passage into a long list of numbers — hundreds or thousands of them — that together mark a point in an abstract space. The training behind it arranges that space so that passages with similar meanings land near each other and passages about different things land far apart. "How do I return a package?" and "Where is my parcel?" use almost no words in common, but they mean nearly the same thing, and a good model of this kind puts their number lists close together. "Return" in the sense of a tax return lands somewhere else entirely. Once meaning has an address, closeness becomes arithmetic: software can compare two lists in a blink and say how near they sit.

That one ability — measuring nearness of meaning — turns out to be the load-bearing piece in several systems a reader uses every day.

It is what makes search understand intent instead of matching keywords. A keyword search finds the documents that contain your words; a search built on embeddings finds the documents that mean what you meant, including the ones that never use your words at all.

It is what powers recommendations of the "more like this" kind: the article, product, or song nearest to the one you are looking at, measured in meaning rather than in tags somebody typed in.

And it is the first half of the arrangement called RAG — retrieval-augmented generation — which is how many talkers are made to answer from a specific body of material. Before the language model writes a word, a model of this type searches the library, ranks what it finds by nearness to the question, and fetches the few passages worth reading. The talker then answers from those pages instead of from memory alone. When a chatbot answers accurately about a company's own documents, the librarian did the finding and the talker did the talking. The two types are almost always found working as a pair, and only one of them gets the credit in conversation.

The same geometry does quieter work as well. A pile of documents can be grouped by topic simply by clustering the points that sit near each other. Near-duplicate pages reveal themselves as points that land almost on top of one another. Incoming items — support tickets, applications, reports — can be routed by which established category their position sits closest to. None of this requires the model to say anything; the arrangement of the points is the result.

Recognizing this kind is mostly a matter of noticing what it never does. It never talks back. You will not chat with an embedding model, and no product puts one in front of you as a personality. Text goes in, numbers come out; the numbers are used to rank, sort, group, and retrieve, and then some other part of the system speaks. If a tool can pull the three most relevant pages out of a million in less than a second, a model of this type did that work, whatever the tool is called.

Judging these models is correspondingly plain. They are not rated on eloquence but on retrieval quality — whether the truly relevant items land at the top of the ranking — and the field keeps running comparisons for exactly that. The type also changes more slowly than the talkers. Its flagship names stayed put through 2026 while other categories turned over around them; the news in this kind was consolidation, not replacement.

The shelf, as of November 2026, looks like this.

OpenAI's pair, text-embedding-3-small and text-embedding-3-large, remains the default for much of the industry — closed models, reached through OpenAI's API. The small one represents each passage with 1,536 numbers, the large with 3,072, and both allow the list to be shortened when a coarser position is good enough. OpenAI shipped no new embedding model in 2026; the pair stays in place because it is the low-friction choice, even as rivals have taken the lead on quality in recent comparisons.

The clearest of those rivals is Voyage AI, which is no longer an independent company: MongoDB acquired it in February 2025, and its models now sit inside MongoDB's Atlas vector-search stack as "Voyage AI by MongoDB." The current family is the Voyage 4 series, active since January 15, 2026 by MongoDB's own lifecycle listing: voyage-4-large at the top, then voyage-4 and voyage-4-lite, all closed and API-served, all sharing a single embedding space so the model that positions a question and the model that positions documents can be mixed without the geometry breaking. Domain members followed — voyage-code-4 for code, voyage-context-4 for long material. The whole Voyage 3 series has been moved to Legacy status: still served, no longer the line being built on. One member of the new family breaks the closed pattern — voyage-4-nano is released as open weights under the Apache 2.0 license.

Google's entry, Gemini Embedding 2, also belongs on this shelf. It is a closed model, and by October 2026 published comparisons were placing it at or near the top of the retrieval rankings, alongside the Voyage 4 family.

The open flagship is still BGE-M3, from BAAI, released under the MIT license. It is a modest model by the standards of this book — 568 million parameters — that does three retrieval styles in one body: dense matching by overall meaning, sparse matching that still respects individual words, and multi-vector matching that compares passage piece by piece. It handles more than a hundred languages. Its November 2026 status is best stated plainly: alive, downloadable, widely deployed — and quiet. BAAI has shipped no new base embedding model in over a year, and the open-weights energy in this kind has lately sat with newer lines from other makers. BGE-M3 remains the standard open choice largely because, in its niche, nothing has replaced it.

That quiet is the honest summary of the whole kind. While famous generative names rose and died in a single year, the librarians kept their shelves: the same default pair at OpenAI, one acquisition and one generational step at Voyage, a new Google entry at the top of the comparisons, and a 2024 open champion still holding its place. Readers looking for the type that will still be recognizable in the next edition of this book should start here.

On the shelf, November 2026

---

Chapter 9 — The Frontier Shelf: Kinds Still Forming

Every other chapter in this part of the book describes a kind that has settled. The models in those chapters can be bought, downloaded, compared, and argued about on the strength of a record. This chapter covers the kinds that have not settled yet. They are real — working models, from serious laboratories, that a reader can in some cases try today — but they are few, young, and still changing shape. They get a shelf of their own, and the shelf is labelled honestly: what follows is the state of two forming kinds as of November 2026, and parts of it will be out of date before any other chapter in this book.

Two types of model stand on the shelf. The first is the world model. A world model simulates an environment: not a picture of a place, and not a film of one, but a working model of how a place behaves, which holds together while someone acts inside it. The test that separates a world model from a video model is interaction. A video model produces a sequence of frames and the sequence is finished. A world model responds — turn left and the corridor is there, push an object and it moves as objects move — and it keeps the world consistent while the actions continue.

The second type is the robot-action model, usually called a VLA, for vision-language-action. Its inputs are what a robot has: camera images and an instruction in ordinary words. Its output is not text or pictures but physical action — motor commands that move a real machine. A talker answers a question; a VLA picks up the cup. These models are the current answer to an old problem in robotics: instead of writing a program for every task a robot might be given, train one model on many robots doing many tasks, and let it generalise.

World models

The clearest statement of the kind is Google DeepMind's Genie line. Genie 4, announced on 27 October 2026 — a week before this book's snapshot — generates an explorable world in real time: the user moves through it with ordinary controls while the model invents the surroundings frame by frame and keeps them coherent. Genie 4 is the first real-time world model in the line to generate its audio natively — music, ambient sound, and spoken dialogue from characters inside the world arrive with the picture rather than being added to it. Prompts in natural language can steer a simulation while it runs, and a Gemini model works inside the loop as a guide, watching the scene and keeping longer sequences of events on track. Genie 4 is closed: no weights are published, and at the snapshot it is a research preview reaching trusted testers first.

World Labs, the company founded by Fei-Fei Li, takes the second approach: persistence. Its Marble model, launched in November 2025, turns text, images, video, or a rough layout into a three-dimensional environment that can be explored and exported — as data a game engine or simulator can use, not just as footage. A newer model, Atlas, entered early access on 1 September 2026 and is intended to power future versions of Marble. Both are closed, hosted products; the worlds are downloadable, the models are not. The company's circumstances are also part of the record: on 28 September 2026 AMD announced a definitive agreement to acquire World Labs for approximately $8.2 billion in stock. The acquisition had not closed at the snapshot date. The most prominent independent world-model laboratory is, at the time of writing, in the process of becoming part of a chip company.

NVIDIA's Cosmos family is the third approach, and the open one. Cosmos models are built for physical AI — robotics and autonomous vehicles — and are meant to reason about the physical world, simulate it, and produce synthetic training data and robot actions. Cosmos 3, released in May 2026, is a single family on one architecture with two parts: a reasoner that takes text and vision in and puts text out, and a generator that takes text, vision, audio, and action in and puts vision, audio, and action out. It ships in three sizes — Super at 64 billion parameters, Nano at 16 billion, and Edge at 4 billion, the last intended to run on the robot hardware itself. Cosmos 3 is open: NVIDIA releases the source code and the model weights under the OpenMDW-1.1 license.

Meta's V-JEPA 2 is the deliberate counter-argument. It generates nothing. Trained on over a million hours of video, it learns to predict what happens next in an abstract representation of a scene — an understanding of the world without a rendered picture of it. Given a small amount of robot data on top of that training, it can plan a robot arm's actions in environments it has never seen, without task-specific training. V-JEPA 2 was released in June 2025, and Meta published both the code and the model checkpoints for research and commercial use. Whether a world model must generate a world, or only predict one, is the live disagreement inside this kind, and both positions are held by laboratories with the resources to test them.

Robot-action models

DeepMind's Gemini Robotics 2, released on 30 July 2026, is a suite of three models: a planner (Gemini Robotics ER 2), the VLA itself, and a smaller on-device version that runs on the robot. The generation's step forward is scope. The 2025 models in this line manipulated objects with arms, on tables. Gemini Robotics 2 is the first in the family to control a whole humanoid body — balance, posture, and movement, in DeepMind's phrase "from feet to fingertips" — demonstrated on a partner company's humanoid robot. One model checkpoint drives more than one kind of robot body. The suite is closed and reaches early-access partners only.

Physical Intelligence, the best-known independent laboratory in this type, shows the open-to-closed pattern arriving in robotics. Its earlier models — π0, π0-FAST, and π0.5 — are open: their base checkpoints can be downloaded from the company's openpi project, whose code is under the Apache 2.0 license. π0.5, from April 2025, added generalisation to places the robot had never been. The newer models, *π\0.6 (November 2025) and π0.7** (April 2026), are not in openpi. They were announced as research and, from October 2026, are offered through the company's Pi API beta. The older generation stays downloadable; the frontier of the line sits behind an API. Readers of the earlier chapters will recognise the shape of that arrangement.

NVIDIA's GR00T line is the open entry on this side of the shelf. GR00T models are cross-embodiment foundation models for humanoid robots: language, images, and the robot's own body state go in, motor actions come out, and the base model is then post-trained onto a specific machine. The current downloadable release is GR00T N1.7, a 3-billion-parameter model whose weights are open under the NVIDIA Open Model License Agreement.

Where the shelves meet

The reason these two types share a chapter is that they are converging. GR00T N1.7's vision-language backbone is a Cosmos reasoner — a world-model component doing the seeing and thinking inside a robot-action model. Cosmos 3, from the other direction, takes action among its inputs and produces action among its outputs: a world model that already reaches into the VLA's job. One stack is forming in which a model simulates the world, plans in it, and acts in it. If that stack settles, a future edition of this book may give it one chapter with one name. It has not settled yet, and this book does not name kinds before they exist.

A closing word of calibration. The other chapters of this book describe kinds stable enough that their November 2026 shelves will still be recognisable in a year. This chapter carries no such expectation. Genie 4 was one week old at the snapshot; Atlas was two months into early access; the World Labs acquisition was announced and unclosed. The frontier shelf is where the map is drawn in pencil. It is included because a reader who meets these words — world model, VLA, physical AI — in the wild should be able to place them: what they take in, what they put out, and how young they still are.

On the shelf, November 2026

>

World models — Genie 4 (Google DeepMind; closed, research preview; first real-time world model with native audio and dialogue). Marble and Atlas (World Labs; closed; persistent, exportable 3D worlds; AMD acquisition of World Labs announced at ~$8.2B in stock, not yet closed). Cosmos 3 (NVIDIA; open, OpenMDW-1.1 license; 64B / 16B / 4B). V-JEPA 2 (Meta; open code and checkpoints; predictive, non-generative).

>

Robot-action models (VLA) — Gemini Robotics 2 (Google DeepMind; closed, early-access partners; whole-body humanoid control). π0, π0-FAST, π0.5 (Physical Intelligence; open checkpoints; π\*0.6 and π0.7 API-only). GR00T N1.7 (NVIDIA; open weights, NVIDIA Open Model License).

---

← IntroductionContentsPart Two — How to Read a New Model →