This Week In Veloren 78

This week has seen lots of progress from lots of developers. @Sharp gives us some perspective into refactoring the world dimension constant.

- AngelOnFira, TWiV Editor

Contributor Work

Thanks to this week's contributors, @xMAC94x, @imbris, @yusdacra, @Sam, @Snowram, @lausek, @Pfau, @Slipped, and @T-Dark!

@Ellinia has been working on some SFX with help from @Eden. @lausek worked on German translations and refactoring auth. @lobster has been cleaning up the number of draw calls for particles and added an option to disable them completely. @Pfau was working on adding a UI for the group system implemented by @imbris. He was also working on a social window overhaul. @zesterer has been working on caves.

@Sam added a new secondary attack to the bow. They are also completing work on diverse weapon stats. @Capucho threw together a small site to display info about the official Veloren server. @Felixader finished a format spec draft to follow when writing quests. @XVar integrated imgui-rs into Veloren as a test for flexible debug UIs. It's still in the air if it will be fully integrated.

@T-Dark worked on the character alias substring filter. This will help server owners ban certain works from character names. @Sharp did a lot of work to support arbitrarily sized maps. @Snowram helped with the latest animation rework. @Gemu reworked all of the small quadruped animal models. @Songtronix figured out where the auth 500 error came from. @Capucho has been revamping the server cli.

Refactoring WORLD_SIZE by @Sharp

For at least as long as I've been part of Veloren, we've had a global variable called WORLD_SIZE that determined the size of the map at compile time. Initially, it was only used in a handful of places localized to WorldSim, however, mostly due to me, we ended up using it all over the place. In my branch, it was even starting to creep into the client. This was due to generating the map directly on the client, rather than just using an image (for complicated reasons I don't want to go into).

Obviously, that was not good, since a client couldn't even host a map of a different size from the server. And it also meant that hosting maps of different sizes from the default required code changes and recompiles. No test server could keep a different-from-default-size map, which a lot of people wanted. It wasn't just laziness that was stopping me from doing this refactor, though. We also have several invariants on the world map size. For example, each chunk dimension has to be a power of 2, the product of the chunk dimensions must fit in a usize, each block dimension must fit in an i32, each chunk dimension must fit in a u16, no chunk dimension can exceed 2^14, and so on.

And, since doing arithmetic around WORLD_SIZE is on the critical path for a lot of performance-sensitive stuff, I wanted to make sure that if we switched to using dynamic values, we did it in a way that we could (in theory) recover the performance we lost due to missed compiler optimizations.

So, my solution was to create a new type, MapSizeLg, storing the number of bits for each dimension. The type verifies at construction time that all invariants are satisfied, so for any instance of that type we can freely assume all of those invariants. We can also encode those as optimized methods and functions that operate only on values of that type, enforcing that code practices good hygiene and doesn't accidentally mix non-world-map-size values in.

Not only that, since we verify the invariants on construction, in the future if we need to we can use unsafe LLVM assertions or integer arithmetic in order to let the compiler perform the same optimizations it would have in the constant case (outside of stuff like constant folding of course). So this design both simplifies a lot of existing code (making it much more robust to issues like overflows, since we have consistent methods we call for given operations), and ensures that we can claw back any performance we lose.

Ultimately it was a lot of work--the diff was about 2.5k lines--but it made the next step easy: updating our map format. This is actually the first time we've updated the format since its inception, but it was built from the get-go to support forward migrations, so the process was pretty straightforward. For maps without dimension information, we make an educated guess at the actual dimensions. We do this by insisting that the map dimensions be the square of a power of 2 in order to perform the migration, and if they are we assume the map is actually square with that dimension.

For new maps, we directly save the real dimension information. We also include a "continent scaling hack" that's used as a factor to our world generation and is usually based on world size, to help keep outputs consistent with the initial map that was generated without sacrificing flexibility. Because the migrations occur automatically and in memory, no changes to any existing files are required. Just like before, you can set a map in your server_settings.ron, but unlike before this will work for maps of arbitrary size with no code changes.

In the end, we have zero remaining instances of WORLD_SIZE, and only one use of its replacement. The replacement is a default map size for use in the initial generation, which can be updated to produce maps of different size. In the future, when generation is more stable, we will provide hooks so that modifying code to set this value, along with other generation parameters, is not needed to produce a new map.

Finally, since I thought it was cool: we exploit Rust's new ability to panic in constants to make sure that the default map size respects the invariants we mentioned, at compile time. If our validation function fails when initializing the constant, we will panic! This is the kind of extra assurance that makes me love Rust 🙂

In fact, the refactor itself did not (appear to) introduce any new bugs at runtime! Only two runtime issues surfaced, and both of them were due to writing the migration too hastily. For me, this really illustrates why Rust is such a powerful tool for collaborative development.

First, Rust's design makes abusing global variables difficult, pushing coders towards a much cleaner design. In a language with a pervasive global mutable state, I might have been tempted to try to just make the world size updatable somehow, but that would've been a big mistake.

Secondly, being able to perform large-scale refactors on a widely shared codebase without introducing significant bugs is really important for avoiding tech debt, especially on a fast-moving project like Veloren. I've used a lot of languages in my career, and Rust is the only one where I can routinely perform a large refactor without breaking tons of existing code. While some have IDEs that make refactoring faster than it is in Rust (for now, anyway), in none of them has refactoring been as "safe" as it is in Rust.

This is not an accident. From day one, Rust's mission has been to provide strong mechanisms for enforcing strict isolation boundaries; mechanisms powerful enough to actually enforce the invariants that occur in large, real-world programming projects. Ownership, eschewing exceptions and global state, explicit type annotations at function boundaries, mutex poisoning, explicit casts, floats lacking Ord, and so on are all aimed squarely at this target. It's not just about type safety, it's about working as hard as humanly possible to avoid footguns. And it only works because the whole community buys into it.

So, while refactoring may not be the most "sexy" kind of programming out there, for me it's where Rust really shines through 🙂