Three Weeks in Veloren 237

These weeks, we look at partial downloads in Airshipper.

- Christof, TWiV editor

Contributor Work

Thanks to last three weeks' contributors @coffee-compiler (most of these merge requests), @Sam, @muphblu, @Tomodachi94 and @xMAC94x.

Special thanks to @xMAC94x for reviewing and merging most of these patches!🏆

The merged work includes to recommend openGL instead of DX11 in panic message, arthropod animation restructuring, unification of module and binary file names, translation script fixes, localize item amount, single player server name and invites, fix modular weapons crafting instructions text flow, localize skill window texts, fixing black minimap, localization of pseudo recipes, fix UI flickering in crafting, a translatable message on give_item, fixing a crash on server list removal, a repair UI rework, fix wrong button highlight in interface settings, preventing empty server list entries, removal of unused translation keys, correcting the NixOS Wiki link and partial archive upgrades in airshipper.

Ongoing (unmerged) work is happening on new sword animations, potion rework, glider courses, a winit update, repair UI overhaul, a chat clear command, new translations, gamepad bindings, updated contribution guidelines, riposte, hydra, and mud/swamp.

Translation work

No translations have been merged in these weeks.

Development discussions

@Isse is working on Hydra and @unii created a video

@unii posted interesting videos of how to combat a mindflayer more safely with a shield.

@zesterer found procedural running on YouTube and is tempted to implement this in Veloren

Airshipper Partial Download by @xMAC94x

Ever wondered why Steam updates are so fast and Airshipper requires you to redownload 390MB each week? That's because Steam has differential updates, they know what files don't need to change and only update what is necessary. Veloren on the other hand didn't have this feature, our updates were meant to re-download the complete game and replace the old folder with the new one.

This will change with the next Airshipper version 0.12. We will enable differential updates by default with a fallback to full updates. This will increase download speed, and also enable us to validate game files for corruption. The following article will deep dive into what improvements we can expect and how the new differential updates work.

Improvements

Veloren's differential updates, work on a file-by-file basis, which means that if a file changes (either slightly, or completely) it needs to be re-downloaded. To understand the benefits, we need to understand Veloren's files and how they change. Inside our Veloren installs folder, you will find 2 executables: veloren-server-cli.exe and veloren-voxygen.exe. Being ca, 180MB and 250MB in size. Compared to other games, in Veloren everything is compiled into a single executable, so we don't have any .dll files at all. That means, each week, those files will be updated, and thus, will require a re-download.

But wait, 180MB and 250MB is already bigger than the 390MB, how does that work? During download, the files are ZIP compressed, and luckily our binaries can be compressed down to around 100MB in ZIP.

So what are the other 290MB? ASSETS! In fact, 250MB out of it are audio files, because audio is already encoded space-efficient, it cannot be further compressed by ZIP. This is where differential updates shine, as most of the large assets, will rarely change, we can keep them, and only download the files that change. So you can expect updates in the future to be much smaller, around ~150MB instead of 390MB!

How does it work?

The old download method had a local config file where it stored the local version, it did a lookup to Veloren's servers and compared both versions. When there was a diff, it downloaded a single ZIP file (with all of Veloren's content) unzipped that and updated the local config. The new algorithm does the differential updates on the same ZIP, but instead of downloading the complete zip, it only downloads part of the Zip file. We make use of 3 technologies here:

• Zip files store a List of all contained files at the end, so we can peek into it, without downloading the whole file
• Within Zip, each file is compressed separately, which means, we can extract a single file, without reading other files
• HTTP allows using the RANGE header, which will instruct a server only to send a partial file

Our new algorithm does a lot of work now:

• It peeks into the Veloren Update Zip, and tries to get a list of all files, there is a checksum for each file.
• It verifies the checksum of those files locally and generates a list of updated files
• It only downloads those files, with the help of RANGE requests

There is a drawback with this method, we need to do multiple HTTP calls, and many small calls take longer than a single large call. That's why there exists some fallback logic, in case too many files change at once, Airshipper will fall back and re-download the whole zip. Another drawback is that we need to read all local files to get their CRC checksum, however, with modern SSDs, this is quite fast (~2s) and gives us free, integrity checks.

So when Airshipper v0.12 will be released? Unfortunately, it will still take a few weeks to prepare everything, but we hope to bring this awesome feature to you, to save you a few GB of monthly downloads :)

This developer blog is based on information collected in #blog-content, where game contributors are always invited to communicate their updates or simply post a link to a different channel.