Doug Binks - Juliette Foucaut - 25 Feb 2021
Our Avoyd Voxel Editor has come a long way since the last devlog post about it in 2018, and having just released a significant update which enhances the Minecraft import pipeline this seems like as good a time as any to talk about it once again.
Minecraft maps Greenfield city and Cuirassé DSD-401 by MrBatou imported and combined together in Avoyd's Voxel Editor.
As many of you know, in addition to making the Avoyd game free to download, we make the Avoyd Voxel Editor which is bundled with it. As well as being the tool we use to build everything in the game, the editor can import Minecraft maps, MagicalVoxel files, Slab .vox files, heightmaps, and export to Wavefront .obj vertex data (for use as general 3D data in tools like Blender 3D). So it's a usable general voxel tool designed to work with large voxel models/worlds.
In october 2020 we started to see one of our early Minecraft import videos featuring Westeroscraft's King's Landing gain a large (for us) number of views, all down to a Lazy Assassin Youtube video "10 SHOCKING THINGS THAT ARE MADE IN MINECRAFT! 😱😱".
I decided to make an updated video with some new content, and set off to download a new map I'd heard of - Greenfield City - and try it out. This turned out to be in a new Minecraft map format which I didn't support in the enki Minecraft Import library enkiMI, so a small step turned into a journey down a long and winding road. The views along the way were, however, pretty sweet.
Video showing the result of our work: large Minecraft maps loaded into Avoyd, featuring
Music: 'Just a Suggestion' by nervous_testpilot
Version 1.13 of Minecraft Java Edition introduced a new format for the base voxel data which had previously been stored as a simple 3D array of bytes per chunk section (a chunk section is a 16x16x16 volume), with optional extra arrays for extended data. The new format has a palette of string named block types per chunk section, and then a variable width array of bitstream data indexing the palette (with the format changed in 1.16 so that voxels were stored on 64bit boundaries for performance reasons). Additionally, block types no longer have an integer id, and are known by their namespaced string name. Currently the enkiMI library only supports a default set of Minecraft block types, and returns a byte id per block. In future, I hope to have time to expand this to allow the full palette blocks of any map to be read.
Whilst I was testing the importer changes, I became frustrated with the performance of loading large Minecraft maps. Whilst most of the testing could be done with portions of the data, I had found some cases where the region files (individual files of data in the region folder of a Minecraft map) had variations on the tagged format I was expecting, so it was important to test against a wide range of data. The easiest solution was to move the file processing to a multithreaded operation by threading the loading of each region file, which I did using enkiTS, my open source permissively licensed C and C++ Task Scheduler for creating parallel programs.
Since Avoyd uses a single octree to represent each model, care needs to be taken with the multithreaded construction. Since inserting one octree into another is a fast operation for my octree, even whilst deduplicating the data (I use a reference counted directed acyclic graph sparse voxel octree), I build small octrees from the Minecraft data on multiple threads and then insert them on one thread only. The canonical approach to synchronization with enkiTS is to avoid it entirely and use a combination of 'pinned tasks' (tasks which run on only the assigned thread) and dependencies, but in this case a simple lock proved the easiest approach and profiling showed no contention even on a 36 thread CPU. On my 18 core 36 thread CPU the Minecraft map importer was over 16x faster than before.
Meanwhile, we updated the default Minecraft material palette to complete the range of material colours. Avoyd does not use textures, but the materials used in Avoyd match the Minecraft default materials as closely as possible by basing their colour on Minecraft textures averages.
We also added appropriate Metallic and Smoothness parameters for improved specular lighting, something which really helps bring the loaded maps to light. The materials affected are: water, snow, ice, vegetation, metals, ores, gems, glass and glazed terracottas. Note that there is currently no emissive lighting in Avoyd.
Overall we're really happy with the changes to the Minecraft import pipeline so we decided to submit Avoyd Voxel Editor as a mod on PlanetMinecraft
If this is your first time using Avoyd's voxel editor, the tips below should help you get started.
Don't hesitate to contact us if you have questions.
Using Avoyd's Edit tool to do a spherical cut and paste of part of the Greenfield map (click to enlarge)