Learn AI Series (#151) - Mini Project: Building Something That Matters

scipio

Published: 08 Aug 2026 › Updated: 08 Aug 2026 Learn AI Series (#151) - Mini Project: Building Something That Matters

Learn AI Series (#151) - Mini Project: Building Something That Matters

What will I learn

choosing a project that matters -- a blunt little scoring framework for telling a real problem from a toy demo before you waste a weekend on it;
boring architecture on purpose -- wiring a pretrained vision encoder into an accessibility tool that describes images for people who cannot see them;
the data pipeline nobody brags about -- provenance, licensing and deduplication, the unglamorous part that decides whether your project can ever be open-sourced;
serving it to actual humans -- a tiny FastAPI surface that a screen reader can talk to;
measuring impact, not accuracy -- why repeat usage tells you more than any F1 score ever will;
open-sourcing and writing it up honestly -- because code on a git host is not the same thing as a project someone can use.

Requirements

A working modern computer running macOS, Windows or Ubuntu;
Python 3.10+ with PyTorch, torchvision and FastAPI installed (pip install torch torchvision fastapi pillow) -- the vision encoder downloads pretrained weights the first time you run it, so do that on a connection you do not hate;
Familiarity with the whole series, because this one borrows from nearly every arc -- vision (#77-91), language models (#57-76), production (#117-136) -- and we open, as always, by settling last week's homework from #150.

Difficulty

Beginner

Curriculum (of the `Learn AI Series`):

Learn AI Series (#151) - Mini Project: Building Something That Matters

We have built a mini project at the end of nearly every arc. Crypto regime prediction (#21), a full ML pipeline (#36), a transformer from scratch (#56), an AI assistant (#76), a visual system (#91), a voice controller (#101), a game-playing agent (#116), a production platform (#136). Each one pulled together the skills of its arc. This one pulls together the whole series -- and it does it with a different goal in mind.

Because here is the trap I want to steer you around today. With 150 episodes of AI knowledge behind you, the temptation is to build the most technically impressive thing you can. Resist that. The gap between a "cool demo" and a "useful tool" is exactly where most AI projects quietly die, and it is not a gap you close with a bigger model. You close it by caring about a specific person. Having said that -- we settle last week's homework first. House rules ;-)

Solutions to episode #150's exercises

#150 was the emerging-frontiers episode, and the three tasks poked at the mechanisms behind MoE, spiking neurons, and test-time compute.

Exercise 1 -- Router entropy for MoE. Add a method to MoELayer that returns how often each expert is selected across a batch, plus the entropy of that distribution.

import torch

def expert_usage(moe_layer, x):
    """How often each expert fires across a batch, plus routing entropy.
    HIGH entropy => the router spreads load evenly across experts, which
    is what you want. LOW entropy => collapse: a couple of experts do all
    the work and the rest are dead weight you paid for but never use."""
    with torch.no_grad():
        probs = torch.softmax(moe_layer.router(x), dim=-1)   # (B, S, n_experts)
        _, top_i = probs.topk(moe_layer.top_k, dim=-1)        # chosen experts
        counts = torch.bincount(top_i.flatten(),
                                minlength=moe_layer.n_experts).float()
        frac = counts / counts.sum()                          # selection frequency
        nz = frac[frac > 0]                                   # avoid log(0)
        entropy = -(nz * nz.log2()).sum().item()              # bits
    balanced_max = torch.log2(torch.tensor(float(moe_layer.n_experts)))
    return frac.tolist(), entropy, balanced_max.item()

A perfectly balanced router over 8 experts hits log2(8) = 3 bits of entropy, and that is the number you are chasing. When entropy collapses toward zero, the router has fallen in love with one or two experts and is ignoring the rest -- which means you are carrying the memory cost of eight expert networks while only ever running two. That is the whole reason production MoE training bolts on a load-balancing auxiliary loss (#150): entropy is not a nice-to-have metric here, it is the difference between the parameter count you paid for and the one you actually get to use.

Exercise 2 -- A leaky-neuron dial. Take LIFNeuron and sweep tau from 0.1 to 0.99 with a constant input current, printing how many timesteps pass between spikes.

def spike_interval(tau, current=0.6, threshold=1.0, steps=1000):
    """Feed a constant current into a Leaky Integrate-and-Fire neuron and
    measure the average gap (in timesteps) between its spikes."""
    membrane = 0.0
    spike_times = []
    for t in range(steps):
        membrane = tau * membrane + current       # leak, then integrate
        if membrane >= threshold:
            spike_times.append(t)
            membrane -= threshold                  # reset what fired
    if len(spike_times) < 2:
        return float('inf')                        # never reached threshold
    gaps = [b - a for a, b in zip(spike_times, spike_times[1:])]
    return sum(gaps) / len(gaps)

for tau in (0.1, 0.5, 0.9, 0.99):
    print(f"tau={tau:.2f}  mean steps between spikes = {spike_interval(tau)}")

The membrane time constant tau controls how long the neuron REMEMBERS its recent input. A leaky neuron (low tau) forgets almost immediately, so on a weak constant drive it can settle below threshold and never fire at all -- its steady state sits at current / (1 - tau), and if that is under the threshold, no spike ever comes. Crank tau up and the neuron integrates over a longer window, the membrane climbs higher, and spikes come faster and closer together. A spike-count classifier cares deeply about this, because tau is quietly setting how much temporal context each neuron folds into its decision.

Exercise 3 -- Budgeted best-of-N. Extend best_of_n so it spends n=2 on easy prompts and n=16 on hard ones, and compare verifier score per unit of compute against a flat n=8 baseline.

def budgeted_best_of_n(model, verifier, prompts, is_hard,
                       easy_n=2, hard_n=16, flat_n=8):
    """Spend compute where it actually pays: little on easy prompts, a lot
    on hard ones. Compare score-per-compute against a flat n=8 baseline."""
    def run(prompt, n):
        best = max(verifier(prompt, model(prompt)) for _ in range(n))
        return best, n                             # (score, compute spent)

    smart_score = smart_cost = flat_score = flat_cost = 0.0
    for prompt, hard in zip(prompts, is_hard):
        s, c = run(prompt, hard_n if hard else easy_n)
        smart_score += s; smart_cost += c
        s, c = run(prompt, flat_n)
        flat_score += s; flat_cost += c

    print(f"budgeted: score/compute = {smart_score / smart_cost:.4f} "
          f"(total compute {smart_cost:.0f})")
    print(f"flat n=8: score/compute = {flat_score / flat_cost:.4f} "
          f"(total compute {flat_cost:.0f})")

On a mixed workload the budgeted policy wins on score-per-compute, and it wins for a boring reason: easy prompts get solved on the first or second sample, so paying for eight of them is pure waste, while the genuinely hard prompts are exactly where extra samples buy real accuracy. That is the whole argument for test-time compute scaling from #150 in one function -- match the thinking to the difficulty, in stead of paying the hard-problem price on every single query forever.

Right -- homework settled. Now let us build something a real person could actually use.

Choosing a project that matters

The best projects solve a specific problem for a specific group of people. Not "people who want AI" -- that is not a project, that is a mood. So before a single line of model code, I run every idea through three blunt tests.

The specificity test. Can you name who benefits in one sentence? "Visually impaired users who want to know what is in the photos their friends post" is specific. "Users who need image understanding" is not. If you cannot picture one actual human, you are building for nobody.

The data reality check. Does the data exist, or can you realistically make it? Brilliant ideas die when there is no training data, and "someone will label it" is not a plan -- ask who, and why they would bother.

The baseline test. What do people do today without your tool? If the honest answer is "nothing, because no decent solution exists," you may have something. If the answer is "they use a thing that already works fine," you need to be honest with yourself about whether AI actually helps here or just looks impressive on a slide.

Let us make that scoreable, because a number you have to write down is harder to lie to yourself about than a feeling:

# Project selection framework - score your idea honestly, 0 to 3 per row.

project_criteria = {
    "specific_beneficiary": "Can you name the exact person/group who benefits?",
    "data_availability":    "Does training data exist or can you make it?",
    "current_alternative":  "How badly do people cope without this today?",
    "technical_feasibility":"Can you build a useful version on hardware you have?",
    "measurable_impact":    "Can you measure whether it actually helps?",
}

# Example: an accessibility tool that describes images for blind users.
scores = {
    "specific_beneficiary": 3,    # blind and low-vision users, very concrete
    "data_availability": 3,       # public captioning datasets exist (COCO, etc.)
    "current_alternative": 2,     # some alt-text, mostly missing or useless
    "technical_feasibility": 3,   # pretrained encoders, runs on a plain CPU
    "measurable_impact": 3,       # repeat usage + user feedback are trackable
}
total = sum(scores.values())
print(f"Project score: {total}/15")
# 14/15 - a strong candidate, so that is what we build.

That is the project for today: an accessibility image describer. It fuses computer vision (#77-91), language models (#57-76) and multimodal understanding (#75, #138) into something that directly improves a person's daily experience. It scores well not because it is clever, but because it is specific, the data exists, and I can measure whether anyone comes back.

Architecture: keep it boring where possible

The single most important production lesson of this series -- hammered home in #117 and again in #121 -- is to use the simplest architecture that solves the user's problem. No hero engineering. For our describer, that is a pretrained vision encoder feeding a projection into a language head:

import torch
import torch.nn as nn
from torchvision import transforms, models


class ImageDescriber:
    """Accessible image description system.

    Pipeline:
    1. Image encoder (pretrained ViT, #54) -> visual features
    2. Feature projection -> language embedding space
    3. Language model -> a natural-language description
    4. Post-processing -> accessibility-optimised output
    """
    def __init__(self, device='cpu'):
        self.device = device
        self.transform = transforms.Compose([
            transforms.Resize(224),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225]),
        ])
        # Visual encoder: ViT-B/16 pretrained, classification head removed.
        self.visual_encoder = models.vit_b_16(
            weights=models.ViT_B_16_Weights.DEFAULT)
        self.visual_encoder.heads = nn.Identity()
        self.visual_encoder.eval().to(device)
        # Project 768-dim visual features into the text embedding space.
        self.projection = nn.Sequential(
            nn.Linear(768, 512), nn.GELU(), nn.Linear(512, 512),
        ).to(device)

    def encode_image(self, image):
        pixels = self.transform(image).unsqueeze(0).to(self.device)
        with torch.no_grad():
            return self.visual_encoder(pixels)

    def describe(self, image, detail_level='standard'):
        """Describe an image. detail_level: 'brief' | 'standard' | 'detailed'."""
        features = self.encode_image(image)
        projected = self.projection(features)
        described = self._generate_description(projected, detail_level)
        return self._format_for_accessibility(described, detail_level)

    def _generate_description(self, visual_features, detail_level):
        # In production this feeds a vision-language head (BLIP-2, LLaVA, ...).
        # We return a structured stub so the pipeline is fully wired and testable.
        return {
            'main_subject': '', 'scene_description': '', 'text_detected': '',
            'colors_dominant': [], 'people_count': 0,
        }

    def _format_for_accessibility(self, d, detail_level):
        """Assemble output tuned for a screen reader."""
        parts = [d['main_subject']]
        if detail_level in ('standard', 'detailed'):
            parts.append(d['scene_description'])
            if d['text_detected']:
                parts.append(f"Text in image: {d['text_detected']}")
        if detail_level == 'detailed':
            if d['people_count']:
                parts.append(f"People visible: {d['people_count']}")
            if d['colors_dominant']:
                parts.append("Dominant colors: "
                             + ', '.join(d['colors_dominant'][:3]))
        return ' '.join(p for p in parts if p)

Notice the three detail levels. That is not a technical feature, it is an accessibility feature. A brief line works for quickly scrolling a feed; a standard description works for understanding a post; a detailed one works when someone wants to really examine an image. The USER decides how much information they get -- and that little bit of respect for the person on the other end is exactly what turns a demo into a tool.

The data pipeline nobody brags about

You would think the model is the hard part. It is not. The data pipeline is -- and specifically, knowing where every image came from and what you are allowed to do with it. This is where the ethics of #35 stop being a lecture and become code:

import hashlib
import json
from pathlib import Path
from datetime import datetime


class DataPipeline:
    """Data collection with provenance, so the project stays open-sourceable.

    Principles from #118: version everything, validate inputs, track origin.
    """
    def __init__(self, data_dir='./data'):
        self.data_dir = Path(data_dir)
        self.data_dir.mkdir(parents=True, exist_ok=True)
        self.manifest = self.data_dir / 'manifest.jsonl'

    def ingest_image(self, image_path, source, license_info):
        """Add an image with full provenance. Returns its content id, or
        None if we have already seen this exact file."""
        image_path = Path(image_path)
        if not image_path.exists():
            raise FileNotFoundError(f"Image not found: {image_path}")
        content_hash = hashlib.sha256(image_path.read_bytes()).hexdigest()[:16]
        if self._is_duplicate(content_hash):
            return None
        record = {
            'id': content_hash,
            'original_path': str(image_path),
            'source': source,
            'license': license_info,
            'ingested_at': datetime.now().isoformat(),
        }
        with open(self.manifest, 'a') as f:
            f.write(json.dumps(record) + '\n')
        return content_hash

    def _is_duplicate(self, content_hash):
        if not self.manifest.exists():
            return False
        with open(self.manifest) as f:
            return any(json.loads(line)['id'] == content_hash for line in f)

    def validate_dataset(self):
        """Data-quality gate: missing files, missing licenses (#118)."""
        if not self.manifest.exists():
            return ['No manifest found']
        issues, records = [], []
        with open(self.manifest) as f:
            for i, line in enumerate(f):
                try:
                    records.append(json.loads(line))
                except json.JSONDecodeError:
                    issues.append(f"Corrupt record at line {i}")
        for r in records:
            if not Path(r['original_path']).exists():
                issues.append(f"Missing file: {r['original_path']}")
        unlicensed = [r for r in records if not r.get('license')]
        if unlicensed:
            issues.append(f"{len(unlicensed)} images without a license")
        return issues or ['All checks passed']

Every image carries its origin: where it came from, what license it has, when it landed. This is not bureaucracy -- it is the difference between a project you can proudly open-source and one that eats a takedown notice the week it gets popular. The content hash also gives you free deduplication, so the same picture scraped twice does not quietly triple its weight in your training set.

Serving it to actual humans

A model in a notebook helps nobody. To reach a screen reader it needs a plain HTTP surface (#121), and small is a feature here:

from fastapi import FastAPI, UploadFile, File, Query
from PIL import Image
import io

app = FastAPI(title="Image Describer API",
              description="Accessible image descriptions for blind users")
describer = None


@app.on_event("startup")
def load_model():
    global describer
    describer = ImageDescriber(device='cpu')


@app.post("/describe")
async def describe_image(
    file: UploadFile = File(...),
    detail: str = Query('standard', regex='^(brief|standard|detailed)$'),
):
    contents = await file.read()
    image = Image.open(io.BytesIO(contents)).convert('RGB')
    return {
        'description': describer.describe(image, detail_level=detail),
        'detail_level': detail,
        'model_version': '0.1.0',
    }


@app.get("/health")
def health_check():
    return {'status': 'ok', 'model_loaded': describer is not None}

One health check, one describe endpoint, detail level as a query parameter. That model_version in the response looks trivial but earns its keep the day you ship a new model and need to know whether your outputs changed under you (#123). Boring, deliberate, easy to monitor.

Measuring impact, not accuracy

This is where most projects stop -- they print a model accuracy and call it a success. But accuracy is not impact. A model can be 94% accurate and help absolutely no one. What you actually want to know is whether real people use the thing, and whether they come BACK:

class ImpactTracker:
    """Track usage patterns that actually indicate value, not just accuracy."""
    def __init__(self, log_path='./logs/impact.jsonl'):
        self.log_path = Path(log_path)
        self.log_path.parent.mkdir(parents=True, exist_ok=True)

    def log_request(self, request_id, detail_level, latency_ms, desc_length):
        record = {
            'request_id': request_id,
            'timestamp': datetime.now().isoformat(),
            'detail_level': detail_level,
            'latency_ms': latency_ms,
            'description_length': desc_length,
        }
        with open(self.log_path, 'a') as f:
            f.write(json.dumps(record) + '\n')

    def compute_metrics(self):
        records = [json.loads(l) for l in open(self.log_path)]
        if not records:
            return {}
        return {
            'total_requests': len(records),                       # do they use it?
            'active_days': len({r['timestamp'][:10] for r in records}),  # come back?
            'detail_distribution': {
                lvl: sum(1 for r in records if r['detail_level'] == lvl)
                for lvl in ('brief', 'standard', 'detailed')
            },
            'median_latency_ms': sorted(
                r['latency_ms'] for r in records)[len(records) // 2],
            'avg_description_length':
                sum(r['description_length'] for r in records) / len(records),
        }

The key signal buried in there is active_days -- repeat usage. If someone tries your tool once and never returns, it was a curiosity, not a help. If they use it every day, you have built something that matters to them, and no confusion matrix will ever tell you that. Track it from day one, because you cannot go back and measure engagement you never logged.

Open-sourcing so people can actually use it

Code sitting on a git host is not open source in any way that counts. Open source means a stranger can find it, understand it, and run it without emailing you. That takes a shape:

project-root/
|-- README.md          # what it does, who it is for, how to run it in 5 min
|-- LICENSE            # MIT / Apache 2.0 - a real, permissive license
|-- CONTRIBUTING.md    # how to help, even if nobody has yet
|-- requirements.txt   # pinned dependencies
|-- Dockerfile         # one-command deployment
|-- src/
|   |-- model.py       # ImageDescriber
|   |-- api.py         # FastAPI server
|   |-- data.py        # DataPipeline
|   |-- metrics.py     # ImpactTracker
|-- tests/             # does it produce output, respond, and validate?
|-- docs/              # how it works, and how well (with numbers)
|-- examples/          # a minimal working example + sample images

The README is the most important file in the whole tree. If a reader cannot figure out what the project does and get it running within five minutes, they close the tab and never come back -- and that is just as true for a repo with zero stars as one with ten thousand. Write the README first, honestly, for the specific person from your specificity test.

Writing about it honestly

This series has been one long exercise in sharing knowledge, so your project deserves the same. A good writeup states the problem in HUMAN terms ("blind users cannot tell what is in a photo," not "multimodal embedding projection"), admits why existing solutions fall short, explains your approach at two altitudes (deep enough for a practitioner, clear enough for everyone else), reports honest results including what did NOT work, and ends with the lowest-friction way to try it. The technical world has an oversupply of polished press releases and an undersupply of honest writeups about building real things. Be the honest one.

The reality check

Let me be straight, because you have earned straight after 150 of these. Most AI projects -- including plenty built by experienced people -- never make a real impact. They work technically but find no users, or find users but do not truly help them, or help a handful but cannot sustain themselves. That is not discouraging, it is just normal, and knowing it up front is armour, not poison.

The point of building something that matters is not that you are guaranteed to succeed. It is that the attempt teaches you things no tutorial ever can: about real users, about deployment, about maintenance, about the long cold distance between "it runs on my laptop" and "it works for someone who genuinely needs it." Every episode in this series -- the math, the code, the architectures -- was building tools. This episode is about what you finally build WITH them.

We are nearly at the end of the road now. In the last handful of episodes we step back and look at the whole picture: the day-to-day kit a working practitioner actually reaches for, how a model becomes a product people pay for, how to read the research so it does not read you, and how all 150-plus pieces we have assembled fit into one coherent stack. Bring the project you just scoped -- we are going to talk about how to actually finish it.

The important bits

pick by specificity and data, not by cleverness -- name the one person who benefits and confirm the data exists before you write any model code;
keep the architecture boring -- a pretrained encoder plus a small projection beats a hero design, because the simplest thing that solves the user's problem wins;
the data pipeline is the real work -- provenance, licensing and deduplication are what make a project sustainable and open-sourceable, not an afterthought;
measure impact, not accuracy -- repeat usage (active days) tells you whether you built something that matters in a way no model metric can;
open-sourcing is documentation, examples and a five-minute README -- code alone helps nobody, and a good writeup is honest about what did not work;
most projects do not change the world, and that is fine -- the attempt teaches you the distance between a demo and a tool, which is the one lesson this whole series was quietly building toward.

Thanks for building this far with me -- now go score one real idea against that framework and see if it survives. De groeten! ;-)

@scipio

Leave Learn AI Series (#151) - Mini Project: Building Something That Matters to:

Written by scipio

Does it matter who's right, or who's left?

Best Posts From scipio

We have not curated any of scipio's posts yet. But you can encourage our curation team to review posts by visiting them regularly and by referring other readers. Because we give priority to frequently read content.

Learn AI Series (#151) - Mini Project: Building Something That Matters