A (future) challenge for language models to show it is AGI
(image generated with Gemini)
The X AI parallel universe
On X, there is quite a big community of people who seem to be living in a parallel universe. A parallel universe where language models, specifically the closed cloud models from Anthropic and OpenAI, are so advanced that they are able to write and deploy entire production grade applications with hardly any user interaction., A parallel universe where frontier models keep getting exponentially better with each version and each new model. Where LLMs have reached the holy grail of being Artificial General Intelligence (AGI), or will in the next week, and where Artificial Super Intelligence (ASI) is going to be undeniable before the end of the year. A Universe where people actually manage to produce products with Claude Code and Codex, working with AI as if they are managers rather than engineers.
In the universe where I live, companies with the same names exist. Their language models and harnesses also happen to have the exact same names as in the parallel universe, only the language models are just language models, the harnesses are just harnesses that are rather disconnected from the harnesses needed in day to day software engineering, and improvement in core models has not only been just incremental since last summer, the progress increments have been growing smaller with each release while a major price of this progress has been increased drift that is actually starting to show the signs that it might in fact be exponential.
The AI universe that I live in instead
In my own universe, closed cloud models have grown increasingly less usable for software engineering purposes since GPT 5 and Sonnet 4 were released, and while I've been fitting language models into a more professional workflow, local open models like the Chinese qwen3-coder:30b, the American starcoder2:15b and the French codestral:22b, have proven increasingly more suitable for an increasing percentage of my and the teams I've worked with their software engineering workflows than any of the cloud frontier models that are lauded so extensively in the alternate universe. The combination of multiple smaller models, routing, templating, and emerging yet still primitive per team Domain Specific Languages that tie these together in stable industrially robust agentic constellations are showing way more promising being dumb than even a short glance at Fable reveals the best of the best of current frontier models is capable of for >95% of daily programming needs. Generation? If it’s scaffolding, boilerplate or refactoring, use qwen3 and codestral. Running a template that is more likely to regurgitate? Downgrade to a 15b starcoder so you don’t need to worry about due diligence. Code reviews? Run it through a few different 70b sized models if you have the hardware for that, or branch out to the ollama cloud.
Testing prompts on new models
And why do I think frontier progress has been incremental at best? Well, when the local models are not cutting it for specific tasks, I experiment with the frontier models. Sometimes they do manage it, but many times they don’t. I keep my best failing prompts, or improve them and duplicate them to be more specific about the gap in the currently best models, and I keep them in a private repo so new models can’t overfit on them. Then when a new model comes out, I use my monthly allowance of tokens on exactly these prompts and evaluate the differences in results. So far, starting last year september, the little progress I have seen has at no time been sufficient to warrant automating it in my workflow, not even if drift wouldn’t have been so big a problem as it is.
My Merg-E efforts and AI
Meanwhile, in my spare time I have been working on my own programming language. An opinionated least authority dataflow centric language aimed at usage with Web 3.0 blockchains, and all of the notions of AGI for any of the cloud models have been proving themselves as completely disconnected from reality, at least in my universe. And this is exactly where I want to write about today: A challenge for the people in the alternate universe where language model development has been exponential and AGI has already been achieved. A challenge about something that AGI should be able to do: Work with the language spec of a novel programming language with its own unique paradigms and idioms that is absent from the training data.
To get back down to earth, I don't think right now anyone really thinks we have reached AGI, what is good, because I'm still working on the first testing version of my programming language Merg-E, so the full challenge can't be done yet right now. But I want to write down the challenge now and get some discussion going.
The 'what' of the AGI challenge
The what is simple: Port gnuchess to ”idiomatic” Merg-E v0.4 and use Lumbra’s GigaBase to test the port’s behaviour and performance against the original with a wide range of random entries from the database, including recent updates.
The 'how' of the AGI challenge
The old saying goes “You Can Write FORTRAN in any Language”, and not doing that is what the how is about. Merg-E isn’t a C-ish or a Lisp-ish language in any way, and many aspects of Merg-E are either very uncommon or at least uncommon in combination with other programming languages. Merg-E is a language that aims to be parallelism friendly but in a non-committing way. You express that you are OK with things being parallel and it is up to the runtime to decide if it will. Merg-E has no classes or structs, it has dataframes, tensors and DAGs instead. Shared mutables are there only for special usecases, for the most parts data is either immutable, or mutable data is moved, not shared or copied. Merg-E is a least authority language with only two keywords and a set of huge trees that define the language, but because of the least authority and two keywords design, most parts of a program work only with a few small branches of these trees, making the very language syntax very much scoped because of internal name resolution. The language mandated strict two spaces Ratliff style indentation, with as only exception a special pre-lexing multi-line operator. Merg-E has a rich numeric typesystem with strict overflow-safe numbers that by default promote to wider and wider types unless you are very explicit about when that isn’t needed. From the 0,4 version of the language, Merg-E has no raw float support anymore because of type safety for physics. Also from v0.4 it adds tensors with potential parallel processing and symmetry based type-hint performance hacks that should prove useful in many matrix oriented programs. It is the furthest away from a c-ish or lisp-ish language you can get.
It combines:
- novel syntax
- novel semantics
- novel paradigms
- extremely limited training exposure
- a non-trivial implementation target
and the how is about using all of this in the most idiomatic way, adhering to least authority, giving the runtime the opportunity to run things in parallel and if at all possible for the chess challenge, what I believe it should be, making use of tensors where it makes sense.
A bit of reality: the delay.
Right now, I hope everyone who isn’t living in an actual parallel universe realizes that the challenge presented in this post would currently be impossible, that the models, including the currently overhyped Fable, are nowhere near the actual general intelligence that would be needed to complete it. What is good because Merg-E isn’t ready either right now. If anyone feels Fable plus their top agentic orchestration harness is up to the task, please try to find another programming language with a tiny public codebase available in it, with a language spec the model can read, a novel syntax, semantics and paradigms that can be a non-trivial implementation target.
The challenge will come most to it’s right if the nidhogg-s (as described in this post is ready before a serious AGI model becomes available that could possibly complete the challenge. From my universe it seems likely the later will take at least until 2035 if not longer, so maybe even fucanglong might be available by then and someone (maybe me) could do the challenge by hand, just to show it is possible, publishing the zero knowledge proof of the existence of a manually coded Merg-E port, so we could assert the solution exist without exposing language model training to it. But I’m getting ahead of myself, for now I think this (delayed) challenge is one that one day could show we have or haven’t reached AGI yet. See which universe is actually real.
Leave A (future) challenge for language models to show it is AGI to:
Read more #merg-e posts
Best Posts From Rob J Meijer
We have not curated any of pibara's posts yet. But you can encourage our curation team to review posts by visiting them regularly and by referring other readers. Because we give priority to frequently read content.
More Posts From Rob J Meijer
- HF-JSON 2.0: A type-rich textual data exchange format
- Version 0.3 of the Merg-E language specification : The $threshold keyword and logging.
- RVC Seed-coding: A prototype/experiment vibe-coding alternative ?
- Version 0.3 of the Merg-E language specification : The merge parts of Merg-E
- A (future) challenge for language models to show it is AGI
- Version 0.3 of the Merg-E language specification : Flow control and exceptions.
- AI for software engineering. About ASICs for AI, reengaging with syntax, embracing provenance, RVC, model-in-a-box and what to think about the AI IPOs.
- K-IX (Part-I : chapter 1)
- Version 0.4 of the Merg-E language specification : Deprecating float/complex for rquantity/cquantity for full dimensional type-safety.
- Version 0.4 of the Merg-E language specification : Tensors and tensor literals