Learn AI Series (#150) - Emerging Frontiers

scipio

Published: 07 Aug 2026 › Updated: 07 Aug 2026 Learn AI Series (#150) - Emerging Frontiers

Learn AI Series (#150) - Emerging Frontiers

What will I learn

AI video generation -- how systems like Sora extend diffusion from a single image to a whole sequence of frames, and the spacetime-transformer trick that makes it affordable;
text-to-3D -- how Score Distillation Sampling turns a plain 2D image model into a supervisor for building 3D objects you can drop into a game engine;
quantum machine learning -- what the theory actually promises, and an honest read on whether any of it matters to you this decade;
neuromorphic computing -- spiking neural networks and why brain-style hardware sips power where a GPU gulps it;
Mixture of Experts (MoE) -- how a model can carry hundreds of billions of parameters while only paying for a handful of them per token;
test-time compute scaling -- the paradigm I think matters most right now: spending compute at inference, not just at training, and letting the hard problems think longer.

Requirements

A working modern computer running macOS, Windows or Ubuntu;
Python 3.10+ with PyTorch and NumPy installed (pip install torch numpy) -- and I mean it this time, because a couple of these snippets actually build small networks you can poke at on a plain CPU;
You have read #84-85 (Diffusion Models) and #137 (Foundation Models), it helps to have #142 (Reasoning and Planning) fresh, and we open by settling the three homework problems from #149.

Difficulty

Beginner

Curriculum (of the `Learn AI Series`):

Learn AI Series (#150) - Emerging Frontiers

150 episodes. One hundred and fifty. When I started this series I honestly did not know if I would make it past the linear-regression episodes without half of you wandering off, and here we still are ;-)

I ended #149 by promising you the most speculative episode we would do -- the one where I lift my eyes off the workbench and point at the horizon. So that is today. Everything up to now has been "here is a thing that works, here is the code, go build it." Today is different. Today is "here is a thing that MIGHT work, here is why smart people are excited, and here is my honest guess about whether you should care yet." I will be blunt about which of these I think is real and which is a lovely research paper wearing a business plan as a costume.

Having said that -- we settle last week's homework first, same house rules as always. No skipping ahead ;-)

Solutions to episode #149's exercises

#149 was the ethics-in-practice episode, and all three tasks were about turning a fairness principle into something a machine can actually check.

Exercise 1 -- Add predictive parity to the auditor. Extend BiasAuditor so each group also reports precision, add a predictive_parity_gap, and have the report flag the attribute with the widest gap.

import torch

class BiasAuditorPlus:
    """The #149 auditor, now with predictive parity (precision) per group."""
    def __init__(self, model, protected_attributes):
        self.model = model
        self.protected_attributes = protected_attributes

    def audit(self, X, y_true, group_labels):
        self.model.eval()
        with torch.no_grad():
            y_pred = self.model(X).argmax(dim=1)

        results = {}
        for attr in self.protected_attributes:
            groups = group_labels[attr]
            group_metrics = {}
            for g in groups.unique():
                mask = (groups == g)
                gp, gt = y_pred[mask], y_true[mask]
                tp = ((gp == 1) & (gt == 1)).sum().float()
                fp = ((gp == 1) & (gt == 0)).sum().float()
                fn = ((gp == 0) & (gt == 1)).sum().float()
                group_metrics[g.item()] = {
                    'positive_rate': (gp == 1).float().mean().item(),
                    'tpr': (tp / (tp + fn + 1e-8)).item(),
                    # NEW: precision == predictive parity per group
                    'precision': (tp / (tp + fp + 1e-8)).item(),
                }
            precisions = [m['precision'] for m in group_metrics.values()]
            results[attr] = {
                'group_metrics': group_metrics,
                'predictive_parity_gap': max(precisions) - min(precisions),
            }
        return results

    def print_report(self, results):
        worst = max(results, key=lambda a: results[a]['predictive_parity_gap'])
        for attr, data in results.items():
            flag = "  <-- WIDEST GAP" if attr == worst else ""
            print(f"=== {attr} (pred-parity gap "
                  f"{data['predictive_parity_gap']:.3f}){flag} ===")
            for g, m in data['group_metrics'].items():
                print(f"  group {g}: pos_rate={m['positive_rate']:.3f}, "
                      f"TPR={m['tpr']:.3f}, precision={m['precision']:.3f}")

Precision is the number the person on the receiving end of a decision feels most directly -- "when this thing flagged me, how often was it right?" -- so for a medical-triage model I would put predictive parity first, because a patient who gets told "you are fine" trusts that answer with their life, and that trust had better be worth the same across groups.

Exercise 2 -- Enforce parity and pay for it. Take parity_vs_precision from #149 and, in stead of one shared selection rate, use a per-group threshold so each group hits the SAME positive rate. Confirm demographic parity now holds exactly, then report the precision gap that stubbornly refuses to close.

import numpy as np

def parity_enforced(base_rate_a=0.30, base_rate_b=0.10, target_rate=0.20):
    """Per-group thresholds => demographic parity by construction.
    Then we look at precision and watch the impossibility result bite."""
    rng = np.random.default_rng(0)
    precisions = {}
    for name, rate in (("A", base_rate_a), ("B", base_rate_b)):
        y = (rng.random(100_000) < rate).astype(int)          # who is truly positive
        score = y + rng.normal(0, 1.0, size=y.shape)          # a noisy, honest score
        thresh = np.quantile(score, 1 - target_rate)          # THIS group's own cutoff
        pred = (score >= thresh).astype(int)
        tp = int(((pred == 1) & (y == 1)).sum())
        precisions[name] = tp / max(int(pred.sum()), 1)
        print(f"group {name}: positive_rate={pred.mean():.3f}  "
              f"precision={precisions[name]:.2f}")
    gap = max(precisions.values()) - min(precisions.values())
    print(f"demographic parity holds (both ~{target_rate:.0%}); "
          f"precision gap stays at {gap:.2f}")

parity_enforced()

Both groups now get flagged at the same rate -- demographic parity satisfied exactly, by construction -- and yet precision refuses to match, because the group with the higher true base rate simply has more real positives sitting above any given cutoff. That is not a bug you can engineer away with a better threshold; it is the Kleinberg impossibility result from #149 showing up in ten lines of NumPy, telling you that equalizing one fairness spends another.

Exercise 3 -- Write a model-card linter. Given a model_card dict, write check_card(card) that verifies every required key is present, that performance.by_demographic reports at least two groups, and that ethical_considerations is non-empty -- printing exactly what is missing.

def check_card(card):
    """A machine-checkable model card. Prints exactly what is missing."""
    required = ["model_name", "model_type", "intended_use", "training_data",
                "performance", "limitations", "ethical_considerations"]
    problems = [f"missing key: {k}" for k in required if k not in card]

    by_demo = card.get("performance", {}).get("by_demographic", {})
    if len(by_demo) < 2:
        problems.append("performance.by_demographic needs >= 2 groups")
    if not card.get("ethical_considerations"):
        problems.append("ethical_considerations is empty")

    if problems:
        print("Card FAILED:")
        for p in problems:
            print(f"  - {p}")
    else:
        print("Card OK: all required fields present and populated.")
    return not problems

# a card missing its ethics section and with only one demographic group
check_card({
    "model_name": "LoanApproval-v2.3",
    "model_type": "XGBoost",
    "intended_use": "pre-screening for MANUAL review",
    "training_data": {"size": "2.4M"},
    "performance": {"by_demographic": {"group_w": {"auc": 0.89}}},
    "limitations": ["thin credit files"],
    "ethical_considerations": [],
})

A machine-checkable card beats a beautifully-written PDF because the PDF gets read once, at launch, by the one person who already agreed with it -- while check_card runs in your CI pipeline on every retrain and fails the build the day someone quietly ships a model with no ethics section. Documentation that a computer can enforce is documentation that survives contact with a deadline ;-)

Right -- homework settled. Now let us go stand at the edge of the map.

AI video generation

Video generation today is roughly where image generation was two years ago: improving frighteningly fast, not yet reliable, and about to rearrange entire industries whether they like it or not.

The architecture behind systems like Sora extends the diffusion models we built in #84-85 from 2D into 3D. In stead of denoising a single image, the model denoises a whole stack of frames at once. The key piece is the spacetime transformer -- a transformer that chews on patches drawn from both the spatial dimensions (height, width) AND the time dimension. The first job is cutting a video into those spacetime patches:

import torch
import torch.nn as nn

class SpacetimePatchEmbed(nn.Module):
    """Cut a video into spacetime patches for a video transformer."""
    def __init__(self, patch_size=16, temporal_patch=4,
                 in_channels=3, embed_dim=768):
        super().__init__()
        # A single 3D convolution does spatial AND temporal patching at once.
        self.proj = nn.Conv3d(
            in_channels, embed_dim,
            kernel_size=(temporal_patch, patch_size, patch_size),
            stride=(temporal_patch, patch_size, patch_size),
        )

    def forward(self, video):
        # video: (batch, channels, frames, height, width)
        x = self.proj(video)              # (B, embed_dim, T', H', W')
        x = x.flatten(2).transpose(1, 2)  # (B, n_patches, embed_dim)
        return x

Now the expensive question: how does attention work over all those patches? Full 3D attention over every patch against every other patch is quadratic in the TOTAL number of patches, which for even a short clip is ruinous. The trick that makes it affordable is factored attention -- do spatial attention within each frame first, then temporal attention across frames at the same position:

class VideoTransformerBlock(nn.Module):
    """Factored spacetime attention: spatial first, then temporal."""
    def __init__(self, embed_dim=768, n_heads=12):
        super().__init__()
        self.spatial_attn = nn.MultiheadAttention(embed_dim, n_heads,
                                                  batch_first=True)
        self.temporal_attn = nn.MultiheadAttention(embed_dim, n_heads,
                                                   batch_first=True)
        self.norm_s = nn.LayerNorm(embed_dim)
        self.norm_t = nn.LayerNorm(embed_dim)
        self.mlp = nn.Sequential(
            nn.Linear(embed_dim, embed_dim * 4), nn.GELU(),
            nn.Linear(embed_dim * 4, embed_dim),
        )
        self.norm_m = nn.LayerNorm(embed_dim)

    def forward(self, x, n_temporal, n_spatial):
        B, N, D = x.shape
        # Spatial: group patches by time step, attend within each frame
        xs = x.reshape(B * n_temporal, n_spatial, D)
        a, _ = self.spatial_attn(xs, xs, xs)
        x = x + self.norm_s(a.reshape(B, N, D))
        # Temporal: group patches by position, attend across frames
        xt = (x.reshape(B, n_temporal, n_spatial, D)
                .permute(0, 2, 1, 3).reshape(B * n_spatial, n_temporal, D))
        a, _ = self.temporal_attn(xt, xt, xt)
        x = x + self.norm_t(a.reshape(B, n_spatial, n_temporal, D)
                             .permute(0, 2, 1, 3).reshape(B, N, D))
        return x + self.norm_m(self.mlp(x))

Many modern systems wrap this in DiT (Diffusion Transformer) blocks, where the conditioning -- your text prompt, the current denoising timestep -- modulates the network through adaptive layer normalization, exactly the flavour of conditioning we saw in the image-diffusion episodes.

What is NOT solved: temporal consistency (a dog should not grow a fifth leg between frame 12 and frame 40), physics (things should fall, collide, and pour like they mean it), and long-range coherence (a 60-second clip has to tell one story, not sixty). People are closing these gaps fast, but do not let a cherry-picked demo reel convince you they are closed. They are not, yet.

Text-to-3D

Generating a 3D object from a sentence sounds like it should need a giant 3D model trained on giant 3D datasets -- which barely exist. The clever escape hatch is Score Distillation Sampling (SDS): use a pretrained 2D image diffusion model as a critic, and let it teach a 3D thing to look correct.

The loop is genuinely elegant. Render your 3D object from a random camera angle, hand the 2D render to the diffusion model, and ask "does this look like the prompt?" Use its gradient to nudge the 3D representation, then rotate to a new angle and repeat -- thousands of times. Because you keep changing the camera, the only way the object can satisfy the critic from EVERY angle is to actually become a coherent 3D shape.

The 3D representation itself varies: Neural Radiance Fields (NeRFs, which we touched in #87), the newer and much faster 3D Gaussian Splatting, or plain old explicit meshes. The choice is downstream-driven -- games want meshes, visual effects might take NeRFs, real-time apps love Gaussian splats. This is not sci-fi; game-asset creation, product visualisation, architectural previews and virtual try-on are being built on text-to-3D pipelines right now, today.

Quantum machine learning

I am going to be direct, because you deserve it: quantum machine learning (QML) is, for practical purposes, mostly hype at the moment. There. I said it. But the theory is genuinely interesting, so let me give you the honest version rather than either the breathless one or the dismissive one.

The premise is seductive. A quantum computer can represent and manipulate an exponentially large state space using superposition and entanglement, and a handful of ML problems -- high-dimensional distributions, certain nasty optimisation landscapes, specific kernel computations -- might one day get a real speedup from that. The current workhorse idea is the variational quantum circuit (VQC): a parameterised quantum circuit that behaves like a trainable model, with the parameters tuned by good old classical gradient descent.

import torch

class QuantumClassifier:
    """CONCEPTUAL hybrid quantum-classical classifier.
    Not runnable without a quantum simulator (PennyLane / Qiskit)."""
    def __init__(self, n_qubits=4, n_layers=3):
        self.n_qubits = n_qubits
        # trainable rotation angles for the quantum gates
        self.params = torch.randn(n_layers, n_qubits, 3, requires_grad=True)

    def quantum_circuit(self, x, params):
        # 1. ENCODE:  map classical features -> qubit rotations
        # 2. PROCESS: apply layers of parameterised rotations + entangling gates
        # 3. MEASURE: collapse the state; measurement probabilities ARE the output
        raise NotImplementedError("needs quantum hardware or a simulator")

    def forward(self, x):
        return self.quantum_circuit(x, self.params)

Notice I left the circuit unimplemented on purpose -- that is not laziness, it is the honest state of the field for someone on a laptop. The blunt status: today's quantum machines are too noisy and too small to beat classical ML on any task you actually have. The theoretical advantages assume fault-tolerant machines with millions of clean qubits; we have hundreds of noisy ones. Might that change in 5-10 years? Maybe. Might it not? Also maybe. Keep half an eye on it, read a survey once a year -- but do NOT build your career on it this decade. If someone sells you a "quantum AI" product in 2026, keep your hand on your wallet ;-)

Neuromorphic computing

Conventional chips march to a synchronous clock and do floating-point arithmetic. Your brain does neither -- it fires asynchronous spikes between neurons and burns about 20 watts doing it. Neuromorphic computing builds hardware that works more like the brain, and its software counterpart is the spiking neural network (SNN).

In stead of the smooth continuous activations we have used all series, an SNN passes around binary spikes over time. The canonical unit is the Leaky Integrate-and-Fire neuron -- it accumulates incoming current, leaks a bit each step, and fires a spike when it crosses a threshold:

import torch
import torch.nn as nn

class LIFNeuron(nn.Module):
    """Leaky Integrate-and-Fire neuron -- one timestep at a time."""
    def __init__(self, tau=0.9, threshold=1.0):
        super().__init__()
        self.tau = tau            # leak: how much membrane potential survives
        self.threshold = threshold

    def forward(self, input_current, membrane=None):
        if membrane is None:
            membrane = torch.zeros_like(input_current)
        membrane = self.tau * membrane + input_current    # leak + integrate
        spikes = (membrane >= self.threshold).float()      # fire
        membrane = membrane - spikes * self.threshold      # reset what fired
        return spikes, membrane

Stack those into a network and you process the input over many timesteps, then classify by counting output spikes:

class SpikingNetwork(nn.Module):
    """A small SNN for classification -- classify by spike count."""
    def __init__(self, input_dim=784, hidden=256, n_classes=10, n_steps=25):
        super().__init__()
        self.n_steps, self.n_classes = n_steps, n_classes
        self.fc1, self.lif1 = nn.Linear(input_dim, hidden, bias=False), LIFNeuron()
        self.fc2, self.lif2 = nn.Linear(hidden, n_classes, bias=False), LIFNeuron()

    def forward(self, x):
        # rate-code the input: brighter pixel -> more likely to spike each step
        trains = (torch.rand(self.n_steps, *x.shape) < x.unsqueeze(0)).float()
        mem1 = mem2 = None
        out = torch.zeros(x.size(0), self.n_classes)
        for t in range(self.n_steps):
            s1, mem1 = self.lif1(self.fc1(trains[t]), mem1)
            s2, mem2 = self.lif2(self.fc2(s1), mem2)
            out += s2
        return out

The upside is real: spikes are binary, so neuromorphic silicon can be astonishingly frugal -- Intel's Loihi chip runs certain workloads at orders of magnitude less power than a GPU. The downside is also real: spikes are non-differentiable (the same wall we hit with binary activations, worked around with surrogate gradients), and the tooling and pretrained-model ecosystem is still thin. Where does it shine? Edge AI (#122) -- always-on, low-power sensing like keyword spotting, gesture detection and anomaly monitoring, where a milliwatt matters more than the last percent of accuracy. Niche, but a genuinely valuable niche.

Mixture of Experts (MoE)

MoE is not new -- the idea dates to the 1990s -- but it is having a full-blown renaissance because it answers one of the sharpest questions in modern AI: how do you make a model bigger WITHOUT making every forward pass proportionally more expensive?

The answer: in stead of one giant feedforward block, keep many smaller expert networks plus a learned router that picks which experts handle each token. Only a few experts fire per token, so your total parameter count can be enormous while the compute per token stays modest.

import torch
import torch.nn as nn

class MoELayer(nn.Module):
    """Mixture of Experts with top-k routing."""
    def __init__(self, input_dim=512, expert_dim=2048, n_experts=8, top_k=2):
        super().__init__()
        self.n_experts, self.top_k = n_experts, top_k
        self.router = nn.Linear(input_dim, n_experts)
        self.experts = nn.ModuleList([
            nn.Sequential(nn.Linear(input_dim, expert_dim), nn.GELU(),
                          nn.Linear(expert_dim, input_dim))
            for _ in range(n_experts)
        ])

    def forward(self, x):
        B, S, D = x.shape
        probs = torch.softmax(self.router(x), dim=-1)          # (B, S, n_experts)
        top_p, top_i = probs.topk(self.top_k, dim=-1)          # pick k experts / token
        top_p = top_p / top_p.sum(dim=-1, keepdim=True)        # renormalise
        out = torch.zeros_like(x)
        for k in range(self.top_k):
            idx, w = top_i[..., k], top_p[..., k:k+1]
            for e in range(self.n_experts):
                mask = (idx == e)
                if mask.any():
                    out[mask] += w[mask] * self.experts[e](x[mask])
        return out

Mixtral 8x7B is the classic example: about 46.7B total parameters but only ~13B active per forward pass (2 of 8 experts per layer), which buys it the quality of a much larger dense model at a fraction of the inference bill. The catches are engineering, not theory: load balancing (stop the router from lazily always picking the same two experts -- you add an auxiliary loss to spread the load) and communication overhead (when experts live on different GPUs, tokens have to travel). Both are solved well enough that MoE now sits inside a lot of frontier systems you have already used, whether the marketing told you or not.

Test-time compute scaling

If you make me pick the ONE idea on this page that already matters most in practice, it is this one -- and it is the least flashy of the bunch, which is exactly why I trust it.

For most of this series, scaling meant training-time compute: bigger model, more data, longer run. Test-time compute scaling adds a second dial. The insight (which ties straight back to reasoning in #142) is dead simple: harder problems deserve more thinking. So in stead of one forward pass and out, let the model chew -- generate several reasoning paths, check its own intermediate steps, search over candidate solutions, and keep the best one. Here is the cheapest honest version, best-of-N with a verifier:

def best_of_n(model, verifier, prompt, n=8):
    """Spend more compute at INFERENCE: sample n answers, keep the best-scored one.
    `model` returns a candidate; `verifier` scores how good a candidate is."""
    candidates = [model(prompt) for _ in range(n)]      # n forward passes, not 1
    scored = [(verifier(prompt, c), c) for c in candidates]
    scored.sort(key=lambda pair: pair[0], reverse=True)
    best_score, best = scored[0]
    print(f"sampled {n} candidates; best verifier score = {best_score:.2f}")
    return best

That is the whole trick in five lines: trade inference cost for quality. Systems like o1 and o3 do a far more sophisticated version of this, sometimes spending 10-100x the compute of a plain forward pass on a single hard query -- and the accuracy you get back follows a scaling law, just like training compute does. For a practitioner this quietly rewrites the economics. In stead of paying once to train a bigger model and eating that cost on every query forever, you can train a smaller model and spend extra compute ONLY on the queries that are actually hard. Easy questions stay cheap; hard questions get the thinking they need. That flexibility is why I think this is the frontier that changes your day job first.

Exercises

Get your hands dirty before we start the final stretch of this series. Three tasks, climbing in difficulty:

Router entropy for MoE. Add a method to MoELayer that, given a batch, returns how often each expert was selected across all tokens, plus the entropy of that distribution. One sentence on why a HIGH entropy is what you want, and what a collapsing (low-entropy) router would mean for all those parameters you paid for.
A leaky-neuron dial. Take LIFNeuron and sweep tau from 0.1 to 0.99 while feeding it a constant input current. Plot or print how many timesteps pass between spikes at each tau, and write two sentences on what the membrane time constant is really controlling -- and why a spike-count classifier cares.
Budgeted best-of-N. Extend best_of_n so it takes a compute_budget and an easy/hard label per prompt, spending n=2 on easy prompts and n=16 on hard ones, and report the average verifier score per dollar of compute versus a flat n=8 baseline. Two sentences connecting what you find back to why test-time scaling beats "just train a bigger model" on a mixed workload.

We open the next episode with full solutions, exactly like always.

The important bits

video generation extends diffusion into spacetime, and factored spatial-then-temporal attention is what keeps it from being computationally impossible -- temporal consistency, physics and long-range coherence are the parts still not solved;
text-to-3D sidesteps the lack of 3D data by using Score Distillation Sampling: a pretrained 2D image model critiques renders from thousands of angles until the 3D object looks right from everywhere;
quantum ML is theoretically interesting but practically unproven -- today's noisy, tiny machines cannot beat classical ML on real work, so watch it but do NOT bet your decade on it;
neuromorphic computing with spiking networks buys extreme energy efficiency for always-on edge sensing, at the cost of harder training and thin tooling -- a small niche, but a real one;
Mixture of Experts lets a model hold enormous parameter counts while paying for only a few experts per token, with load balancing and communication as the engineering price of admission;
test-time compute scaling is the emerging paradigm I would bet on first: spend variable compute at inference based on how hard the problem is, and beat a bigger model at lower average cost.

That is the horizon as I read it in the summer of 2026 -- part solid ground, part shifting sand, and I have tried to tell you honestly which is which. Next time we come back down off the mountain and put our hands to work: we take everything this series has taught and build something that is not a toy, something that actually MATTERS. Bring your editor and a pot of coffee ;-)

Bedankt for reading all one hundred and fifty of these with me -- now go pick the one frontier above that made your eyes light up, and read one real paper on it this week. Tot de volgende keer! ;-)

@scipio

Leave Learn AI Series (#150) - Emerging Frontiers to:

Written by scipio

Does it matter who's right, or who's left?

Best Posts From scipio

We have not curated any of scipio's posts yet. But you can encourage our curation team to review posts by visiting them regularly and by referring other readers. Because we give priority to frequently read content.

Learn AI Series (#150) - Emerging Frontiers