Please Save This For Me

I'll come back for you, love, I promise to.

This week I've been working on saving and loading the world as you move through it. Until now, the world has been procedurally generated as needed. You could make changes to the world, but if you left and came back, the terrain would be re-generated.

A simplest way to handle saving and loading is to divide the world into parts and save/load each part to a file. However, this can incur a lot of hidden overhead. File reads/writes are very expensive. They're pretty much the most expensive thing a computer program can do short of accessing the internet.

So I decided to (once again) take a lesson from Minecraft. I originally implemented a format that was more or less identical to the Region file format used by Minecraft. After doing all that work, I suddenly realized that the Region format wouldn't work for me, and there's one important difference between my project and Minecraft that makes this a problem: Minecraft has a fixed height limit.

My project, on the other hand, does not. The game world extends infinitely in all directions, including up and down. The reason this is a problem is because the Region file format stores world chunks in vertical columns. This is fine in Minecraft since moving up or down can never trigger loading/saving. This is not the case in my project. If I used the Region format, I would have to store twice as much data when the player is straddling a vertical boundary.

So I decided to modify the format to suite my needs. Instead of working with columns of the world, I'm working with large cubes. I chose a size that was large enough to take advantage of compression (using run length encoding), but small enough not to incur a huge memory overhead. I ended up settling on 64 x 64 x 64 block areas.

Unfortunately, this change created a problem. In the Region format you're unlikely to get an entire column compressed down to significantly less than 4096 bytes. This number is significant because most modern computers read from hard drives 4096 bytes at a time (it's more efficient to have a few large reads than a lot of small ones). Region stores its data in a way that minimizes the number of reads/writes.

When you store the world as large cubes, you end up having cubes of solid air or rock. After compression, these only take up a few bytes. Basically, this would mean using 4096 bytes for something that requires less than 10. This wouldn't be so bad if these sort of regions were few and far between, but they actually make up the majority of the game world. My solution to this problem was to store multiple small chunks in a single 4096 byte area. Large chunks are still stored the original way. The end result isn't quite as elegant as the Region format, but it did, inadvertently, result is a larger maximum chunk size (255 MiB vs Minecraft's 1).

I finished all the code for this, but I still have to fix a few bugs before I can say it's done.