A proactive cache system for my CACR tracker

When I first introduced the Cuban Assets Control Regulations Tracker, I described its core mission: to make the text of 31 CFR 515—and its historical changes—easier to navigate and understand. However, over the last few days, my focus has shifted toward the "deep infrastructure," specifically working on a caching layer. Every user request still depended heavily on the eCFR API, very reliable but anyway outside my control. I wanted to change that.

I have some history with caching systems because I obtained my title of Informatics Sciences Engineer—back in 2011—with a solution of this kind for an Internet filtering system. Thus, I have implemented a proactive caching strategy for this last tracker. I leveraged Claude, DeepSeek, and Grok for intensive code generation while I managed data flow, architecture, endpoint testing, and data integrity.

The Data Model: Immutable Snapshots

Instead of a complex "current vs. historical" logic, I decided to treat every version of a section as an immutable historical reference point. The "current" version is simply the most recent snapshot that is stored on disk. To ensure the integrity of this data, there is a persistence module that organizes the file system hierarchically, ensuring that even if the external API fails, our local "source of truth" remains intact:

const CACHE_DIR = path.join(__dirname, 'cache');
const CACHE_VERSIONS_DIR = path.join(CACHE_DIR, 'versions');

function sectionVersionPath(section, date) {
    return path.join(CACHE_VERSIONS_DIR, `${section}__${date}.json`);
}

How each snapshot is obtained and stored

On the first run, the server downloads the entire Part 515 in a single call and fragments every section in memory. Each fragment is written to disk under its version date:

async function bulkPopulateAllSections() {
    const todayDate = await getEcfrUpToDateDate();
    const versionsResp = await axios.get(CFR_VERSIONS_URL, { timeout: 30000 });
    const sectionMap = extractLatestVersionPerSection(versionsResp.data);

    const fullUrl = `${ECFR_API_BASE}/full/${todayDate}/title-31.xml?subtitle=B&chapter=V&part=515`;
    const fullResp = await axios.get(fullUrl, { timeout: 60000 });

    const $ = cheerio.load(fullResp.data, { xmlMode: true, decodeEntities: false });
    $('DIV8').each((i, el) => {
        const sectionId = $(el).attr('N');
        if (!sectionId || $(el).attr('TYPE') !== 'SECTION') return;
        const versionDate = sectionMap[sectionId] || todayDate;
        const sectionXml = '\n' + $.xml(el);
        writeJsonSafe(sectionVersionPath(sectionId, versionDate), {
            xml: sectionXml, versionDate, fetchedAt: Date.now()
        });
        cacheMeta.sectionVersions[sectionId] = versionDate;
    });
}

When a user asks for an older historical version, the endpoint always looks at disk first. Only on a true miss does it reach out to eCFR, store the new snapshot, and then serve it:

app.get('/api/cfr-cached/:date/:section', async (req, res) => {
    const { date, section } = req.params;

    // 1) Exact snapshot already on disk → immediate HIT
    let cached = readJsonSafe(sectionVersionPath(section, date), null);
    if (cached && cached.xml) {
        res.set('X-Cache', 'HIT');
        return res.send(cached.xml);
    }

    // 2) Request for “today” or the known latest → serve the newest local copy
    const latestKnown = getLatestCachedDate(section);
    if (latestKnown && (date === latestKnown || date === getEasternDateString())) {
        cached = readJsonSafe(sectionVersionPath(section, latestKnown), null);
        if (cached && cached.xml) {
            res.set('X-Cache', 'HIT-LATEST');
            return res.send(cached.xml);
        }
    }

    // 3) True MISS → fetch from eCFR, persist, then serve
    const url = `${ECFR_API_BASE}/full/${date}/title-31.xml?subtitle=B&chapter=V&part=515§ion=${section}`;
    const resp = await axios.get(url, { timeout: 20000 });
    writeJsonSafe(sectionVersionPath(section, date), {
        xml: resp.data, versionDate: date, fetchedAt: Date.now()
    });
    if (!cacheMeta.sectionVersions[section] || date > cacheMeta.sectionVersions[section]) {
        cacheMeta.sectionVersions[section] = date;
        saveMeta();
    }
    res.set('X-Cache', 'MISS-FETCHED');
    return res.send(resp.data);
});

Historical snapshots are never deleted. Once a date has been requested (or populated), it stays on disk forever as an immutable reference point.

The system “clock” and federal business days

I have introduced an automatic update process that checks at 10:00 a.m. every business day in the United States, excluding holidays—which are calculated dynamically for each year according to the federal standard—whether a new version of any section exists. If it finds something, the server downloads the new version, and from this local source it will be served dynamically to the user.

The calculation of U.S. federal holidays (per 5 U.S.C. 6103) looks like this:

function federalHolidaysForYear(year) {
    const set = new Set();
    const add = (d) => { set.add(d.toISOString().slice(0, 10)); };

    add(observedFederalHoliday(year, 0, 1));           // New Year's Day
    add(nthWeekdayOfMonth(year, 0, 1, 3));              // MLK Day
    add(nthWeekdayOfMonth(year, 4, 1, -1));             // Memorial Day
    // ... other dynamically calculated holidays
    return set;
}

Efficiency in the initial population: the Mutex pattern

To avoid saturating the official APIs, the bulk population described above runs under a logical mutex so that only one population operation can be active at a time:

let _bulkPopulateInFlight = null;

function runBulkPopulateOnce() {
    if (!_bulkPopulateInFlight) {
        _bulkPopulateInFlight = bulkPopulateAllSections().finally(() => {
            _bulkPopulateInFlight = null;
        });
    }
    return _bulkPopulateInFlight;
}

Resilience and Federal Register slices (1963–1993)

I also proactively resolve the extraction of links to PDF files with the slices of the Federal Register editions from the 1963–1993 period that contain amendments to the CACR. There are 33 unique references of this type that are already linked with their specific URL; the server never has to resolve these links again. Failed resolutions are placed in a retry queue that attempts recovery every three minutes.

The resolution logic only persists real issue_slice / archives URLs:

const frSliceCache = new Map(Object.entries(readJsonSafe(FR_SLICES_PATH, {})));
function persistFrSlices() {
    writeJsonSafe(FR_SLICES_PATH, Object.fromEntries(frSliceCache));
}

const pendingFrRetries = new Map();
const FR_RETRY_INTERVAL_MS = 3 * 60 * 1000;

function scheduleFrRetry(vol, page) {
    const key = `${vol}-${page}`;
    if (frSliceCache.has(key) || pendingFrRetries.has(key)) return;
    pendingFrRetries.set(key, {
        vol, page,
        nextAttemptAt: Date.now() + FR_RETRY_INTERVAL_MS
    });
    ensureFrRetryLoop();
}

async function resolveFrSliceSilent(vol, page) {
    const cacheKey = `${vol}-${page}`;
    if (frSliceCache.has(cacheKey)) return frSliceCache.get(cacheKey);

    const volNum = parseInt(vol, 10);
    if (!isNaN(volNum) && volNum >= 59) {
        const url = `https://www.govinfo.gov/link/fr/${vol}/${page}`;
        frSliceCache.set(cacheKey, url); persistFrSlices();
        return url;
    }

    try {
        const targetUrl = `https://www.federalregister.gov/citation/${vol}-FR-${page}`;
        const response = await axios.get(targetUrl, {
            headers: { 'User-Agent': 'Mozilla/5.0 CubanAssetsTracker/1.0' },
            timeout: 10000
        });
        const $ = cheerio.load(response.data);
        let pdfUrl = $('a[href*="issue_slice"]').attr('href') ||
                     $('ul.fr-list a[href$=".pdf"]').attr('href') ||
                     $('a[href*="archives.federalregister.gov"]').attr('href');
        if (pdfUrl) {
            if (!pdfUrl.startsWith('http')) {
                pdfUrl = new URL(pdfUrl, 'https://www.federalregister.gov').href;
            }
            frSliceCache.set(cacheKey, pdfUrl); persistFrSlices();
            return pdfUrl;
        }
    } catch (e) {
        console.warn(`⚠️ [FR SILENT] ${vol} FR ${page}:`, e.message);
    }
    return null;
}

After populating or refreshing sections, the server scans the cached XML for FR citations of volumes below 59 and resolves them in the background:

const FR_CITE_RE = /\b(\d{1,3})\s*FR\s*(\d{1,6})\b/gi;

function extractFrCitationsFromXml(xml) {
    const found = new Set();
    let m;
    const re = new RegExp(FR_CITE_RE.source, 'gi');
    while ((m = re.exec(xml)) !== null) {
        const vol = parseInt(m[1], 10);
        if (!isNaN(vol) && vol < 59) found.add(`${vol}-${m[2]}`);
    }
    return found;
}

async function proactiveFrSliceScan() {
    // ... scan every cached section XML, collect unresolved citations,
    // resolve them silently, and schedule retries for any that fail
}

In this way, the system’s ability to respond to interruptions in the API service is very high, and it will be even higher if I promote the automatic download of all versions available in eCFR. What would still be missing, then, is to cache copies of the documents themselves, which would allow us to have all this rich historical material “at home.” But, for now, I really feel satisfied.