Blog · · 13 min read

Building a 3D map of a Minecraft server with an AI agent, and the GPU bugs we found

mcatlas turns a backup of a Minecraft world into a 3D map that runs in a browser, phones included. I built it in three weeks with an AI coding agent that wrote the code, ran it on real phones and measured its own work. This is how that went, and the GPU bugs it ran into on the way.

Open the map →

Tilted 3D view of the server's spawn: rivers, forests, a cherry grove, snowy peaks and labels for stations and places
Spawn on the server. Every label, road, tree and building here was read from the world files.

The map is of a Minecraft server whose players built towns, farms and railways, and in the Nether a metro: long tunnels floored with ice, where a boat moves very fast. Nobody had a map of any of it, so I built one.

The rule I set myself early on: nothing on the map is drawn by hand. Every station, building, field and railway comes from a rule that reads blocks and signs. When the server sends a new backup, a script rebuilds the whole map in about two minutes, and a friend’s server got its own map from the same code.

Map size
~23 × 25 km
Region files read
1,143
Metro stations
34
Rebuild from a backup
~2 min

The page is three.js on WebGL 2, built with Vite and served as static files. A set of Node scripts reads the world’s region files and writes the data: height and colour images cut into tiles like a web map (512-pixel tiles at 4, 8, 16… blocks per pixel), plus binary files for trees and buildings. The page loads only the tiles around the view, so its memory doesn’t grow with the world.

The rest of this post is how it was made, and the parts that were harder than they looked.

Built by one person and an agent

I didn’t type this map’s code. Nearly all of it, about 11,600 lines of browser JavaScript and GLSL and 4,100 lines of Node tools, was written by Claude Code, an AI coding agent running on my Mac Studio. My part was the part an agent can’t do for me: deciding what the map should be, and judging what it looked like.

Calendar time
3 weeks
Agent sessions
46
Lines of code
~15,600
Test & bench scripts
117

What surprised me is how far “write the code” reaches now. The agent didn’t stop at the editor:

  • It read the game’s files itself. It wrote a parser for Minecraft’s binary region format, wrote the rules that find stations, buildings, fields and railways, and audited its own output: a script lists every station and the rule that found it. When no rule could explain a place, it asked me instead of special-casing it.
  • It looked at its own work. It flew the camera to fixed views in headless Chrome, took screenshots and diffed them pixel by pixel, to prove that a speed-up hadn’t changed the picture.
  • It profiled the GPU, and learned the hardware’s habits. It timed frames with GPU timer queries, noticed that the GPU flips between fast and slow clock states between runs, and from then on compared versions by alternating them inside one page.
  • It drove real phones over USB. A Pixel 8 through adb and Chrome’s DevTools protocol, sampling the GPU process’s memory every second. An iPhone 17 Pro through safaridriver, with real touch drags. It found on its own that the iPhone drops its GPU clock about 15 seconds after a page loads, so every phone measurement now waits 20 seconds first.
  • It fetched its own tools. When a cheap Samsung couldn’t load the map and no Samsung was plugged in, it got Arm’s Mali Offline Compiler and measured the shaders for three Mali GPUs on the Mac.
  • It delegated. Players write signs in English and Russian; the map shows both. The agent split 1,263 signs between subagents to translate.
  • It shipped. It built every server’s site, uploaded it to Cloudflare Pages and added the DNS record.
Diagram: Claude Code in the middle, directed by me, driving the world backup parser, headless Chrome, Safari, Arm's Mali compiler, a Pixel 8, an iPhone 17 Pro, translation subagents and the deploy
Everything the agent ran by itself. I set goals and judged the results.

Between sessions it keeps notes: a folder of short memory files saying what was measured, what I turned down and why, and which traps to avoid. Each new session starts from them. That is how a lesson like “exact data needs alpha 255” is still applied a week later, in a different file, without me repeating it.

My part was rules and taste. A few of the calls that were mine:

  • “A station is a sign with a name, a portal and the ice road nearby.” And: no hand edits, ever. If the map is wrong, the rule is wrong.
  • For the ground up close, the agent rendered four looks side by side. I picked the soft wash, and it deleted the other three. Cliffs went the same way: four hatching styles, I picked thin broken pen lines.
  • To save GPU time on phones it offered to draw the sea at half size and the page at a lower resolution. I said no, it looks worse. It went looking for savings that don’t show: a cheaper noise function took 16% off the frame with the picture unchanged.

Every bug below was found and fixed in that loop. I noticed something wrong, or a phone crashed; the agent reproduced it, measured it and proposed a fix I could check by eye.

The Nether metro, found by rule

In Minecraft one block in the Nether is eight in the overworld, so a 1 km tunnel there saves 8 km of walking. The server’s players dug long straight tunnels at height 70, floored them with ice and put portals at the ends. That network is the metro, and stations were the hard part. No file says “this is a station”; players just put up signs.

My definition: a station is a sign with a name, with a nether portal and the ice road nearby. On the actual signs that shows up in three ways, and the builder tries them in this order:

Three sign patterns: a circle sign with the station dot, a list of direction signs where the one without an arrow is the station, and a bare name next to a portal on the track
The three sign patterns that make a station, and how many of the 34 each one found.

Anything within 24 blocks of a station already found counts as station furniture (maps, notes, direction boards), not a new station. Signs with the same name within 40 blocks are one station. Players write in English and Russian, and a bilingual sign such as ПЛИТНЯКОВАЯ | DEEPSLATE tells the builder the two names are the same place. That merges names across signs and gives the map its English/Russian switch.

When a station shows up wrong, I don’t fix the station; I fix the rule. A hand edit would be lost with the next backup, and it wouldn’t help the next server.

The Nether view: ice-road lines in different colours meeting at a hub, with station labels and the tunnels players dug around them
The Nether view. Each line takes its colour from the blocks players lined it with; the brown areas are what they dug and built around the tracks.

Ink, paper and a soft wash

I wanted it to look like an atlas, not like the game. The scene is drawn into an offscreen target and then composited:

  • Ink lines wherever depth jumps: tree and crag silhouettes, cliff lips. They go on the nearer side only, so they hug the object in front.
  • The sea is a plane, solved per pixel. Nearly all Minecraft water sits at one height, so for each pixel the view ray meets the plane and the scene’s depth says what lies under it. That gives the true depth of water over every point of the sea bed, so its colour, clarity and foam follow the land exactly at any zoom.
  • A soft wash up close. Minecraft’s block grid looks cheap when you zoom in. I tried round blobs per block, flat painted regions and painted regions with ink outlines. The winner was the simplest: colour blurred over a few blocks, baked per tile.
  • Cliffs as pen lines, every three blocks down the fall line, broken by noise and darker in shade.
  • A faint paper grain and vignette on top.
Close-up of mountains near the Khazad'dum station: cliffs drawn as vertical pen lines, snow on the tops, a lake with trees
Cliffs drawn as pen lines, around a station players dug into a mountain.

Safari wiped my data through the alpha channel

The map’s data travels as images. A WebP is a good container for a height map: four bytes per pixel, compressed losslessly, decoded natively by the browser. So I packed data into all four channels, alpha included. The “soft” map, for example, kept the explored mask in red and the distance to the shore in alpha, which was 0 far inland.

Then I opened the map in Safari, and 30% of the explored world was covered in fog of war.

Safari (26.3 on the Mac) decodes images premultiplied: it multiplies R, G and B by alpha, stores them as bytes, then divides alpha back out when WebGL asks for the pixels. It does this even with createImageBitmap(img, { premultiplyAlpha: 'none' }). Where alpha is 0, the colour comes back as 0. Where alpha is small, it comes back off by a step or more.

Diagram: a texel with alpha 255 survives exactly, with alpha 140 its green and blue come back one lower, with alpha 0 all colour is lost
The same texel through Safari’s decoder at three alpha values.

Chrome doesn’t do this, and neither does Playwright’s headless WebKit, so every automated test had passed. To see it, the agent drove real Safari from AppleScript (do JavaScript works even with Safari in the background) and read the decoded pixels back through WebGL.

It came back a second time, in a quieter form. Height maps marked filled-in texels with alpha 140 and lava with 230. Safari moved green and blue by ±1 on about half of them, which put whole chunks of ground a block too high or too low. They showed up as glowing squares with a ring of foam round them.

The fix is a rule about where data may live:

// R  bit 0: ground + 64, high bit · bit 6: lava · bit 7: filled in
// G  ground + 64, low byte
// B  water depth (7 bits) | frozen << 7
// A  255 explored, 0 not. Nothing else, ever.
export function packHeight(ground, depth, frozen, { lava = false, filled = false } = {}) {
  const h = ground + 64;
  return [(h >> 8) | (lava ? 64 : 0) | (filled ? 128 : 0), h & 255,
          Math.min(127, depth) | (frozen ? 128 : 0), 255];
}

For the blurred soft map, where ±1 doesn’t matter, alpha became 192 + 2 × distance, clamped to 128–255. Above 128 the round trip is off by one step at most.

If an image holds exact values (heights, ids, flags), keep its alpha at 255. Test it in real Safari, not in a headless WebKit.

A vertex shader that needed 4 GB to compile

On a Pixel 8 the tab died while loading, with Chrome’s “Aw, Snap!” page. The agent sampled the GPU process’s memory with adb shell dumpsys meminfo once a second during the load. Its native heap, which is where the driver’s shader compiler lives, went past 4 GB before Android’s low-memory killer stopped it. Then it compiled the ground shader on the phone again and again, each time with one function emptied out, until it saw where the memory went.

The culprit was the ground’s vertex shader. GPU compilers inline every function call, and mine looked up the ground height through a chain of small helpers. Each looked harmless; multiplied out, the compiler was working through dozens of copies of the same bilinear lookup.

Call tree of the old vertex shader multiplying to 54 to 72 inlined copies of a lookup, and a bar chart: over 4 GB of compiler memory before, 0.36 GB after
How three height lookups became 54–72 inlined copies, and what that cost Mali’s compiler.

The fix was to stop computing those heights in the shader. They change only when new tiles arrive, so the page now bakes them into a texture (uOv) once, and the vertex shader reads one texel per vertex. The ground compiles with a peak of 0.36 GB, and the picture is the same.

A week later a cheap Samsung got stuck on the loading screen. This time no Samsung was plugged in, and the agent didn’t need one: Arm’s free Mali Offline Compiler (malioc) reports a shader’s compile memory on a Mac, close to what the Pixel’s driver used. It found two more offenders, a 1.1 GB shader for ruins and a 0.8 GB one for the ink pass. Two surprises:

  • Memory grows much faster than code. One more inlined height lookup took a shader from 0.38 GB to 1.1 GB.
  • Turning repeated calls into a loop helped in vertex shaders and made fragment shaders worse on Mali (0.23 → 0.41 GB). I now pick per stage, and only for Android, because the loop versions cost 0.5–2% of frame time on Apple GPUs.

Desktop GPUs will compile almost anything. If a phone tab crashes on load, measure the compiler before you blame your textures.

Sixteen texture units, all taken

The ground shader blends a lot of sources: the base map, two levels of close-up detail, baked light, trees, roads, fields, railways. One day I added four more textures and the ground vanished, with the console repeating Trying to use 17 texture units while this GPU supports only 16.

Apple GPUs allow 16 textures per shader stage, and three.js counts the vertex and fragment shaders’ textures together against that number. So a texture read only in the vertex shader still takes a slot.

Grid of the 16 textures the ground shader uses, with the slots freed by packing highlighted, and a dashed 17th slot that fails
The ground program today: 16 of 16. The orange slots were freed by packing.

Making room meant packing, never adding:

  • The lava flag moved into a spare channel of the water texture.
  • A colour palette got folded into the texture that used it.
  • The trees’ leaf colour and canopy slope became two layers of one array texture.
  • The Mali fix above replaced two height maps with one baked texture, which freed a slot by accident.

Know your sampler count. Before adding a texture to a shader that’s near the limit, free one first.

Pale dashed lines from fwidth

I kept seeing “seams”: pale dashed lines, one or two pixels wide, round every pond and across cliff faces. They followed boundaries you couldn’t see. The agent switched the ground shader’s features off one at a time with URL flags, screenshotting each, until the dashes disappeared.

Both came from fwidth, the usual tool for anti-aliasing in a shader. The GPU shades pixels in 2×2 quads, and fwidth(v) is how much v changes between a pixel and its neighbours in the quad. Use it as the width of a smoothstep and you get an edge one pixel soft at any zoom. Unless v jumps.

A row of pixel values that jump from -1 to 0.03 where a field starts, and the fwidth bars: a spike of 1.03 at the jump before the fix, small everywhere after
Where the field starts, fwidth sees a jump of about 1 and widens the edge about 30 times.

Around ponds, the sand edge took its width from fwidth of an “is water” field. Out on dry grass, where that field isn’t computed, it was set to −1. Where the field started, a pixel at −1 sat in the same quad as one at 0.03, so fwidth was about 1 and the sand edge blurred into a pale ring. On top of that the code sat inside an if. Derivatives inside a branch that some pixels of the quad skip are undefined, and on Chrome over Metal the width still came from the pixels that skipped it.

// before (simplified): a sentinel of -1, and derivatives inside a branch
if (wv >= 0.) beach = smoothstep(.18 - fwidth(wv), .18 + fwidth(wv), wv);

// after: no branch, and the width from the field floored at 0
float w = max(fwidth(max(wv, 0.)), 1e-3);
float beach = wv >= 0. ? smoothstep(.18 - w, .18 + w, wv) : 0.;

The cliffs had the same bug in another form. Their pen lines run in one of twelve fixed directions, and where the slope turned from one direction to the next, fwidth of the line position spiked and faded the strokes out along a dashed seam. Now the line width comes from dFdx/dFdy of the world position projected onto the direction, and the two directions either side are cross-faded instead of snapped.

Close-up of a river between cliffs: sand beaches blend into the water with no dashed lines, and the cliffs are drawn with pen strokes
After the fix: a clean sand edge round the water, and unbroken pen lines on the cliffs.

A thin dashed line that follows an invisible boundary usually means a derivative of a value that jumps (a sentinel, a snapped angle), or a derivative taken inside an if.

A sphere test that drew everything

On an iPhone 17 Pro the map lagged when zoomed in and tilted. The agent wrapped WebGL’s draw calls in a script to count triangles per pass. At a 400 m view the ground drew 1.3 million triangles, while about 150,000 were on screen.

The ground near the camera is a patch of chunks, 112 blocks each. Their heights come from textures on the GPU, so the CPU doesn’t know how high a chunk is and gives each one bounds from y −400 to 900. three.js culls meshes with a bounding sphere, and the sphere around a tall thin box is huge: a radius of about 655 blocks for a chunk 112 wide. The camera sat inside many of those spheres, and every chunk passed.

Side view of tall thin chunk bounds and the huge sphere around one of them reaching into the camera's view; sphere test passes 111 of 111 chunks, box test 17
Same chunks, two tests: the sphere passes all 111, the box passes 17.

The fix is a few lines: turn off three.js’s culling for these meshes and test each chunk’s box in code.

for (const c of chunks) {
  box.min.set(c.x, -400, c.z);
  box.max.set(c.x + 112, 900, c.z + 112);
  if (frustum.intersectsBox(box)) visible.push(c);
}

GPU time at a phone-sized view went from 7.0 to 4.3 ms. The perspective camera had a bigger version of the same problem: tilted, the horizon is in the frame, so the whole world was in the frustum (697 draw calls, 4.9 M triangles). Far chunks now use coarser grids by distance.

Count what you draw. Frustum culling with spheres is fine for round things and useless for tall thin ones.

What I’d do from day one

  • Test on a real iPhone and a cheap Android early. Chrome on a desktop forgives everything. Safari and Mali each broke the map in ways no desktop browser showed.
  • Give the agent eyes and numbers. An agent can only judge what it can measure. A ?fps flag that shows draw calls and triangles, a switch to turn off each pass, scripts that fly to fixed views, screenshot them and time the GPU: once those existed, it could check its own work and found more than guessing ever did.
  • Plug the devices in. A phone on a USB cable is a test the agent can run a hundred times without asking.
  • Keep the data honest. Building the map only from rules made every rebuild boring, in the good way.
The whole explored world from high above, tilted: coasts, deserts, a red badlands area and station labels, with unexplored land as white clouds
The whole explored world. White is fog of war: land nobody has visited yet.

How this post was made

The same way. I asked for a write-up of these bugs. The agent read its notes and the code, flew the map to the views you see above and took the screenshots, asked the running page which 16 textures the ground shader binds so the diagram would be exact, drew the diagrams as SVG, added a blog to this site and deployed it. The first draft went from my request to this page in under an hour.

The map is live at map.pomisoft.dev. Try the Nether tab, the time-of-day switch, and zooming into a cliff. If it breaks on your device, I’d like to hear about it: hello@pomisoft.dev.