Learn AI Series (#137) - Foundation Models

scipio

Published: 25 Jul 2026 › Updated: 25 Jul 2026 Learn AI Series (#137) - Foundation Models

Learn AI Series (#137) - Foundation Models

What will I learn

What actually makes a model "foundational" -- and why that word is doing a lot more work than it looks;
the common pre-training recipe that turns out to be nearly identical across text, vision, audio and code;
emergent capabilities: the unsettling fact that some abilities only switch on above a certain scale, and nobody put them there on purpose;
the four adaptation methods (prompting, RAG, fine-tuning, distillation) and a decision rule for picking between them without burning money;
foundation models beyond text -- vision, audio, code, science, even robots;
what the whole ecosystem means for you, a practitioner who almost certainly should NOT be training one of these from scratch ;-)

Requirements

A working modern computer running macOS, Windows or Ubuntu;
Python 3.10+ with PyTorch and (optionally) the transformers library installed (pip install torch transformers) -- the code here is illustrative, so nothing needs a GPU;
You've been through the LLM arc, especially #57-60 (Language Modeling through Training LLMs), #62 (Prompt Engineering), #64-65 (RAG) and #74 (the Hugging Face ecosystem). We lean on all of them today.

Difficulty

Beginner

Curriculum (of the `Learn AI Series`):

Learn AI Series (#137) - Foundation Models

At the end of last episode's mini project I left you tugging on a loose thread. We had just wired together an entire production platform -- registry, trainer, server, monitor, A/B router -- and every single model flowing through it was one we trained ourselves, from our own data, from scratch. And I said: that is more and more NOT how it goes anymore. The model you serve today is increasingly a giant that somebody else pretrained on half the internet, which you merely adapt to your little corner of the problem space.

Today we pull on that thread properly. This is a concepts episode -- less "type this and watch it run", more "here is the shape of the thing everyone in AI is actually building on top of in 2026". Having said that, there is still code, because I refuse to teach an idea you cannot poke at with your own fingers ;-)

Let me set the scene with a bit of history, because the shift is genuinely recent and genuinely huge.

The world before, and the world after

Before roughly 2018, every AI task had its own model. Sentiment analysis? Train a classifier on labelled reviews. Machine translation? Train a seq2seq model (episode #50) on parallel corpora. Object detection? Train a detector on bounding-box annotations. Each task, each dataset, each model -- starting from scratch, or from ImageNet pre-training if you were lucky. Every problem was its own little island, and you rebuilt the boat every time.

Then something shifted, and it shifted fast. Researchers noticed that if you train ONE model on a massive pile of data with a stupidly simple self-supervised objective -- predict the next word, fill in the blanks, match images to their captions -- the resulting model could be pointed at an enormous range of downstream tasks with barely any extra training. Often with nothing more than a prompt and zero examples.

These are foundation models: models trained on broad data at scale that then serve as the shared base for many different applications. The term itself was coined by Stanford's HAI Center in 2021, and I'd argue it names the single most important change in how AI is practiced in the last decade. Not a new architecture -- we already had transformers (episodes #52-53). A new relationship between a model and the tasks it serves.

What actually makes a model "foundational"

Here is a trap worth avoiding: not every large model is a foundation model. "Big" is not the qualifier. The distinction lives in how the model is used. A foundation model has three properties, and you need all three.

Trained on broad data. Not task-specific data. GPT was trained on general internet text, not on customer-service transcripts. CLIP was trained on image-caption pairs scraped from the web, not on labelled medical scans. DINOv2 was trained on 142 million curated images spanning hundreds of categories. That breadth is what gives the model general knowledge that transfers -- knowledge it can lend to a task it never saw during training.

Trained at scale. There is a minimum scale below which the "foundational" magic simply does not appear. A 100M-parameter language model is a perfectly useful thing, but it does not do in-context learning. A 7B model does. The exact threshold wobbles depending on task and modality, but the pattern is rock solid: certain capabilities only switch on once the model is large enough AND has seen enough data. Scale is not a nice-to-have here -- it is load-bearing.

Adapted, not retrained. This is the economic heart of the whole paradigm. The expensive pre-training happens ONCE. Adaptation -- through prompting, fine-tuning, or RAG -- is cheap by comparison. That amortization is the reason the whole thing is viable. Training a frontier model can cost well north of $100M. Adapting one for your specific task might cost anywhere from $100 to a few thousand. You are renting the giant's brain by the hour in stead of raising your own from birth.

Let me put that three-part definition into code you can actually run, because the "add a tiny head to a giant base" pattern is the pattern you'll reach for over and over.

# The foundation model paradigm in code
import torch
from transformers import AutoModel, AutoTokenizer

# Step 1: Load a foundation model (pre-trained, general-purpose)
model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
base_model = AutoModel.from_pretrained(model_name)

# This model knows about language in general.
# It cost a fortune and weeks of GPU time to train.
# We downloaded it in about 30 seconds.

# Step 2: Adapt it for a specific task with a tiny addition
class SentimentClassifier(torch.nn.Module):
    def __init__(self, base_model, n_classes=2):
        super().__init__()
        self.base = base_model
        self.classifier = torch.nn.Linear(base_model.config.hidden_size, n_classes)
        # Freeze the base -- we only train the little head on top
        for param in self.base.parameters():
            param.requires_grad = False

    def forward(self, input_ids, attention_mask):
        outputs = self.base(input_ids=input_ids, attention_mask=attention_mask)
        cls_output = outputs.last_hidden_state[:, 0, :]  # the [CLS] token
        return self.classifier(cls_output)

classifier = SentimentClassifier(base_model)

trainable = sum(p.numel() for p in classifier.parameters() if p.requires_grad)
total = sum(p.numel() for p in classifier.parameters())
print(f"Training {trainable:,} of {total:,} parameters ({trainable/total:.2%})")

Run that and you'll see something like Training 1,538 of 109,483,778 parameters (0.00%). Look at that ratio for a second and let it sink in. You are standing on 110 million pre-trained parameters that somebody else paid for, and you are only nudging fifteen hundred of your own. THAT is the foundation model paradigm in one print statement -- the giant does the heavy lifting, you supply a thin sliver of task-specific glue.

The common recipe: how these things get made

Here is the part that surprised me the most when I first internalised it. Foundation models across wildly different modalities -- text, images, audio, protein sequences -- are built with an almost identical recipe. Four steps, over and over.

Step 1: Collect massive data. For language: web crawls, books, code, conversations -- trillions of tokens. For vision: hundreds of millions of images, often carrying captions or surrounding web context. For audio: hundreds of thousands of hours of speech and sound.

Step 2: Choose a self-supervised objective. This is the clever bit. The objective has to be cheap to label -- ideally the data labels itself, so you never pay a human annotator. For language: predict the next token (GPT-style, episode #57) or fill in masked tokens (BERT-style, episode #59). For vision: reconstruct masked image patches (masked auto-encoders), or match images to text (CLIP). For audio: predict masked audio segments (wav2vec 2.0). Nobody labelled any of this by hand -- the data supervises itself.

Step 3: Train a transformer, at scale. The transformer has become the default across every modality precisely because it scales so predictably. The scaling laws we met in episode #60 hold across language, vision AND audio: throw more compute at it and the loss keeps dropping along a smooth curve, over many orders of magnitude. That predictability is why companies are willing to spend nine figures -- they can forecast, roughly, what they'll get.

Step 4: Evaluate emergent capabilities. Test the finished model on tasks it was NEVER explicitly trained for. Can a language model do arithmetic? Translate between language pairs it never saw aligned? Can a vision model recognise an object it never got a labelled example of? Increasingly the answer is "yes" -- and that brings us to the strangest property of the whole business.

Let me sketch that self-supervised idea concretely, because "the data labels itself" sounds like hand-waving until you see it.

# Why self-supervision scales: the labels come for free.
# Next-token prediction turns ONE sentence into many training pairs.

text = "foundation models are trained on broad data at scale"
tokens = text.split()

# Every prefix -> next-word pair is a free labelled example
pairs = [(" ".join(tokens[:i]), tokens[i]) for i in range(1, len(tokens))]
for context, target in pairs:
    print(f"predict {target!r:12} from -> {context!r}")

print(f"\n1 sentence produced {len(pairs)} training examples, zero human labels.")

No annotation budget, no labelling team, no bottleneck. A single sentence coughs up a fistful of supervised examples entirely on its own -- and the whole internet is made of sentences. Multiply that by trillions of tokens and you see where the "scale" in "scale" comes from.

Emergent capabilities: the abilities nobody put there

When GPT-2 (1.5B parameters) landed in 2019, it wrote coherent-ish text but could not reliably do few-shot classification. GPT-3 (175B parameters) could do few-shot classification, simple arithmetic AND code generation. No architectural change between them worth mentioning -- just more parameters and more data. The capabilities came along for the ride.

This is what researchers mean by emergent capabilities: abilities that appear at scale without ever being explicitly trained in. Nobody wrote an arithmetic module. The model learned to add as a byproduct of predicting the next token across billions of lines of text that happened to contain sums. Nobody designed a few-shot learning algorithm into GPT-3 either -- it worked out how to use examples in its prompt entirely by itself.

Some of the notable ones:

In-context learning: learning from examples placed in the prompt, without updating a single weight;
Chain-of-thought reasoning: cracking multi-step problems by generating intermediate reasoning steps out loud;
Cross-lingual transfer: performing a task in a language barely present in the training data;
Instruction following: reading a plain-English instruction and just... doing it (this is what instruction tuning in episode #61 sharpened);
Tool use: learning to call a calculator, an API or a search engine from a handful of demonstrations (the backbone of the agents we built in #67-68).

The existence of emergence has a real, practical sting to it. You cannot always predict what a bigger model will be able to do by extrapolating from a smaller one. Some capabilities have sharp thresholds -- absent below a certain scale, reliable above it, with very little warning in between. That makes planning research genuinely hard: you sometimes only find out what you built after you've built it. A little unsettling, honestly. But also the reason this field feels alive.

Let me make the threshold idea tangible. Below is a rough, illustrative map of when certain abilities tend to "switch on" as you climb the parameter ladder -- not exact science, but the shape is real.

# Illustrative capability thresholds by scale. The point is the SHARPNESS,
# not the exact numbers -- abilities appear, they don't fade in gradually.
capability_onset = {
    "coherent text":            0.1,   # ~100M params: fluent-ish, shallow
    "few-shot classification": 10.0,   # ~10B: learns from prompt examples
    "simple arithmetic":       30.0,   # emerges, was never taught
    "chain-of-thought":        60.0,   # multi-step reasoning unlocks
    "reliable tool use":      100.0,   # calls calculators/APIs from demos
}

def abilities_at(params_billions):
    return [name for name, onset in capability_onset.items()
            if params_billions >= onset]

for scale in [0.1, 7, 30, 175]:
    got = abilities_at(scale)
    print(f"{scale:>6}B params -> {', '.join(got) or 'barely coherent'}")

Run it and you watch abilities blink into existence as the scale crosses each threshold -- a 0.1B model is "barely coherent", a 175B model has the lot. Nobody added arithmetic at 30B on purpose. It just... showed up, a free side effect of predicting the next token over enough mathematical text. THAT is emergence, and it is why the labs keep building bigger even when they cannot fully say in advance what they'll get.

Here is in-context learning in its natural habitat -- a prompt, no fine-tuning, no gradient step in sight.

# In-context learning: teach the task in the prompt itself, no training.
prompt_template = """Classify the sentiment of each review.

Review: "This product exceeded my expectations in every way."
Sentiment: positive

Review: "Terrible quality, broke after one day."
Sentiment: negative

Review: "It's okay, nothing special but gets the job done."
Sentiment: neutral

Review: "{new_review}"
Sentiment:"""

new = "Honestly the best purchase I have made all year."
print(prompt_template.format(new_review=new))

# The model was NEVER fine-tuned for sentiment. It infers the task purely
# from the three worked examples above -- pattern-matching at inference time.
# That is in-context learning: an emergent capability of scale.

Feed that to any capable modern LLM and it answers "positive" without anyone ever having trained it on a sentiment dataset. It read the pattern from three examples and generalised. For a model that is, mechanically, only predicting the next token, that is a genuinely spooky thing to watch.

Adaptation methods: choosing the right lever

Right -- you've got a foundation model. How do you bend it to YOUR task? The options span a wide range of cost and effort, and picking wrong is how budgets quietly evaporate.

Prompting (zero-shot / few-shot). No training whatsoever. Just describe the task, or drop a few examples in the prompt, exactly like the block above. Best when: your task is fairly general, you have little or no labelled data, and the base model is already capable enough. Cost: zero training, you pay per inference. We covered the craft of this in episode #62.

Retrieval-Augmented Generation (RAG). Bolt the model's knowledge onto retrieved documents at inference time. Best when: you need up-to-date or domain-specific knowledge that was never in the training data (last week's prices, your company's internal wiki). Cost: building and maintaining a retrieval index, plus a touch more latency per call. Episodes #64-65 went deep on this.

Fine-tuning (full or parameter-efficient). Actually update the model's weights on your task-specific data. Best when: your task needs specialised behaviour that no prompt can coax out, or you're squeezing for maximum accuracy. Cost: real training cost and labelled data. LoRA and QLoRA (episode #69) drag this down to something you can run on a single consumer GPU.

Distillation. Train a small model to mimic the big one's behaviour. Best when: you love the big model's quality but cannot stomach its inference bill. Cost: you pay to train the student, but the resulting model is far cheaper to serve forever after. We met the technique back in episode #120.

Let me hand you a rough decision rule as runnable code. It is not gospel -- it is the mental checklist I run through, made explicit.

# A decision rule for picking an adaptation method.
def choose_adaptation(task_specificity, data_availability, budget, latency_req):
    """
    task_specificity: how far is your task from the model's general skills?
    data_availability: 'none' | 'documents' | 'some' | 'plenty'
    budget: 'low' | 'high'   -- training budget you can stomach
    latency_req: 'relaxed' | 'strict'
    """
    if latency_req == "strict":
        return "distillation", "Train a smaller student to hit the latency bar"

    if task_specificity == "low" and data_availability == "none":
        return "prompting", "Zero-shot or few-shot with examples in the prompt"

    if data_availability == "documents":
        return "RAG", "Augment the model with retrieved domain knowledge"

    if task_specificity == "high" and data_availability == "plenty" and budget == "high":
        return "full fine-tuning", "Update all weights on your domain data"

    if task_specificity in ("medium", "high") and data_availability in ("some", "plenty"):
        return "LoRA fine-tuning", "Parameter-efficient tuning -- cheap, usually enough"

    return "prompting", "Start with the cheapest baseline, escalate only if it falls short"

for case in [
    ("low", "none", "low", "relaxed"),
    ("low", "documents", "low", "relaxed"),
    ("high", "plenty", "high", "relaxed"),
    ("medium", "some", "low", "relaxed"),
]:
    method, why = choose_adaptation(*case)
    print(f"{case} -> {method}: {why}")

The practical advice underneath all that branching is dead simple: ALWAYS start with the cheapest method (prompting), measure it honestly, and escalate only when the numbers demand it. Fine-tuning a 7B model to chase the last 3% of accuracy, when a well-crafted prompt already gets you 95% of the way there, is a lovely way to set money on fire. I've watched teams do it. Don't be that team ;-)

To make the cost gap concrete -- because "cheap" and "expensive" are lazy words -- here is a back-of-the-envelope comparison in the spirit of episode #134:

# Rough order-of-magnitude costs. Illustrative, not a quote.
options = {
    "pretrain from scratch": 50_000_000,   # you, building a foundation model. don't.
    "full fine-tune (7B)":       3_000,    # all weights, your own GPUs for days
    "LoRA fine-tune (7B)":         150,    # a few adapter matrices, one GPU, hours
    "RAG (index + serve)":          40,    # embedding your docs + a vector store
    "prompting only":                0,    # you pay per call, nothing up front
}
for name, cost in options.items():
    bar = "#" * min(40, len(str(cost)) * 4)
    print(f"{name:24} ~${cost:>10,}  {bar}")

Look at the spread -- eight orders of magnitude between the top and bottom row. Almost nobody reading this should ever be on that top row. The entire point of foundation models is that the top row already happened, somebody else paid for it, and your job is to pick wisely among the cheap rows underneath.

Foundation models beyond text

The paradigm was born in NLP (BERT, GPT) but it has spread to basically every modality that has enough data lying around. A quick tour:

Vision. DINOv2 (Meta) learns visual representations through self-supervised learning on 142M images, and the features it produces work across classification, segmentation, depth estimation and retrieval -- without ever being trained on a single labelled vision task. SAM (Segment Anything) was trained on 11M images carrying a billion masks, and it will segment any object in any image from a click or a text prompt.

Audio. Whisper (OpenAI) was trained on 680,000 hours of multilingual audio, and it handles accents, background noise and technical jargon that specialised speech systems choke on -- it even translates as a side effect. MusicGen and AudioLDM generate music and sound from text descriptions. We touched this world back in the audio arc (#92-101).

Code. CodeLlama, StarCoder, DeepSeek-Coder and friends were trained on hundreds of billions of tokens of source code. They generate, explain, debug and refactor across dozens of languages, and they've become genuinely load-bearing tools for working developers.

Science. AlphaFold was trained on protein sequence-and-structure data to predict 3D protein folds, a problem that stumped biology for FIFTY years. ESM learns protein representations from raw sequences alone. These are foundation models for biology, and they are visibly accelerating drug discovery.

Robotics. RT-2 (Google) trained on BOTH web data and robot demonstrations, so a robot can follow a natural-language instruction for a manipulation task it was never specifically trained on -- the web knowledge transfers into physical reasoning. That still slightly blows my mind.

Notice the recipe never changes: broad data, a self-supervised objective, a big transformer, scale. Swap the data type and you swap the modality -- the machine underneath is the same machine.

# The uniform interface: same three-line ritual, wildly different modalities.
# (Pseudo-loads -- illustrative of the shared pattern, not a benchmark.)
foundation_zoo = {
    "text":     "bert-base-uncased        # language understanding",
    "vision":   "facebook/dinov2-base      # general visual features",
    "audio":    "openai/whisper-base        # speech -> text, 90+ languages",
    "code":     "bigcode/starcoder          # code generation + completion",
    "multimodal":"openai/clip-vit-base-patch32  # text <-> image alignment",
}
for modality, model_id in foundation_zoo.items():
    print(f"{modality:11} load('{model_id.split()[0]}')")
    # from_pretrained(...) -- one call, then adapt. Same shape every time.

That uniformity is not a coincidence, and it is not a small thing. It means the mental model you built for text foundation models transfers, more or less intact, to vision and audio and code. Learn the pattern ONCE, apply it everywhere -- which, come to think of it, is exactly what the models themselves are doing too ;-)

The ecosystem, and what it means for you

Foundation models did not just change the models -- they restructured the whole industry into layers:

Foundation model providers (OpenAI, Anthropic, Google, Meta, Mistral and a handful of others) pour billions into pre-training. This is a pure scale game -- only organisations with enormous compute can even sit at this table.

Adaptation and fine-tuning shops take open-weight foundation models and specialise them for a specific industry or task. This is where smaller companies -- and individuals like you -- can actually compete, because the giant is a free (or cheap) starting point.

Application developers build products on top, via APIs or open-weight models. They compete on user experience, data integration and domain expertise -- NOT on model architecture, which they mostly treat as a utility.

Infrastructure providers (cloud, GPU rental, inference optimisation) sell the picks and shovels to everyone above.

For practitioners -- which is exactly who this series has always been for -- the takeaways are blunt and worth tattooing somewhere visible:

You almost certainly do NOT need to train a foundation model. Take an existing one and adapt it. The top row of that cost chart is not your row.
Understanding how these things work -- which is the whole point of the last 136 episodes -- makes you dramatically better at adapting them. The people who treat the model as an opaque magic box get worse results than the people who know what's inside.
Your differentiator is now data, domain expertise and application design. If you own unique data and genuinely understand your users, you have an edge that no amount of raw compute buys.
The field moves brutally fast. A model that was state-of-the-art six months ago may already be beaten by an open-weight model you can run on your own laptop. Staying current is part of the job now, not an optional extra.

# What a modern AI project actually looks like in 2026.
project_architecture = {
    "foundation_model": "open-weight LLM (Llama 3, Mistral, Qwen, ...)",
    "adaptation":       "LoRA fine-tune on domain data + RAG for fresh knowledge",
    "serving":          "vLLM for efficient inference (episode #121)",
    "monitoring":       "track output quality, latency, cost (episode #123)",
    "evaluation":       "domain benchmarks + human review (episode #73)",
    "total_training_cost": "$100 - $5,000  (fine-tuning ONLY, not pre-training)",
    "inference_cost":      "$0.001 - 0.01 per query (self-hosted)",
    "time_to_prototype":   "days, not months",
}
for k, v in project_architecture.items():
    print(f"{k:22}: {v}")

Hold that dictionary up next to 2018, when building a competitive NLP system meant collecting labelled data, hand-designing an architecture, training from scratch, and praying it generalised. Foundation models compressed that cycle from months down to days. That compression -- not any one clever model -- is the paradigm shift. It is why one developer with taste and a good dataset can now ship something that used to need a research lab.

Exercises

Before the next episode, get your hands dirty. Three tasks, climbing in difficulty:

Count the giant. Load bert-base-uncased (or any small model your machine can handle) with transformers and print its total parameter count. Then freeze the whole base and attach a fresh torch.nn.Linear head for a 3-class problem. Print the ratio of trainable-to-total parameters, exactly like the code above -- and write one sentence explaining, in your own words, why that ratio is the whole selling point.
Few-shot by hand. Take the in-context learning prompt from this episode and adapt it to a DIFFERENT task -- say, classifying a short sentence as "question", "command" or "statement". Give it three worked examples, then a fourth to classify. Do NOT fine-tune anything. Note what happens when you drop from three examples to one, then to zero -- where does the behaviour start to wobble?
Write the decision rule for a real case. Pick an actual task you'd like to solve (summarising your notes, tagging support tickets, whatever). Run it through the choose_adaptation function with honest inputs for specificity, data, budget and latency. Then argue -- in a short paragraph -- whether you agree with what it recommends, and what single fact about your situation would flip the answer to a more expensive method. The reasoning matters more than the code here.

We'll open the next episode with full solutions, as always.

Quick recap

Foundation models are pre-trained on broad data at scale, then adapted for specific tasks -- the expensive training happens once, adaptation is cheap by comparison;
the recipe is nearly universal across modalities: massive diverse data + a self-supervised objective + a transformer + scale;
emergent capabilities (in-context learning, chain-of-thought, cross-lingual transfer) switch on at scale without ever being explicitly trained -- the single most surprising property of the whole business;
adaptation methods run from zero-cost prompting through RAG and LoRA to full fine-tuning and distillation -- always start cheap and escalate only when the numbers demand it;
foundation models now exist for text, vision, audio, code, science AND robotics -- the paradigm has colonised every modality with enough data;
the ecosystem has split into layers: providers, adaptation shops, application builders and infrastructure -- and each layer competes on something different;
for you, the practitioner, the edge is now data, domain expertise and application quality -- NOT model architecture.

And here is the thread for next time. We keep saying "text OR vision OR audio", one modality at a time. But the most interesting foundation models increasingly refuse to pick a lane -- they read an image and a paragraph and a snippet of audio TOGETHER, in one shared representation, and reason across all of it at once. That blending is where a lot of the field is heading, and it is where we head next ;-)

Thanks for reading -- now go count some parameters and feel, with your own eyes, how little of the giant you actually have to touch. De groeten! ;-)

@scipio

Leave Learn AI Series (#137) - Foundation Models to:

Written by scipio

Does it matter who's right, or who's left?

Best Posts From scipio

We have not curated any of scipio's posts yet. But you can encourage our curation team to review posts by visiting them regularly and by referring other readers. Because we give priority to frequently read content.

Learn AI Series (#137) - Foundation Models