This Week In Veloren 103

This week, we see lots of new art coming into Veloren. Skill trees have been merged. @Christof continues his weekly writeups on economic simulation. @Yakei talks about new improvements to CI.

- AngelOnFira, TWiV Editor

Contributor Work

Thanks to this week's contributors, @zesterer, @ccgauche, @XVar, @Pfau, @imbris, @SWilliamsGames, @xMAC94x, @James, and @ubruntu!

@Rotsuoy and @Sarra_Kitty worked together to get some new items into the game. @Sam has been going through a lot of code review for skill trees. There are new bugs and issues to resolve now that it's been released. @James figured out what was causing NPCs to spam chat messages when fleeing and fixed it. @Songtronix worked on getting partial downloads working.

The Florabite by @Rotsuoy

I'm a new contributor and for my first big contribution I started tackling items that didn't have in-game models. I started working on models for critters as well, mostly variants of existing monsters like a green dragon and forest drake. I've also started working on a new monster, the Florabite, designed by my 9 year old daughter who also loves Veloren! So you can definitely say that multiple generations of people already adore the game!

Trade Between Sites with @Christof

Starting from last week's state of reading professions/raw materials/products from a .ron file, I added uncontrolled land as a raw material and trading good (fief/tenure). I also added and included different types of guards for different types of ground/water (e.g. a Naval Guard needs more potions, but less Armor) which then produce Farmland, Quarry, FishingGround... In turn, this enables normal workers to access resources there, producing the raw materials for their food.

Having different types of Guards enabled the town's control board to (re-)assign tasks according to changing needs. This resulted in much more people living in the same city, due to a smarter usage of the same natural resources. Also, I designed a concept for trading between sites that take local prices on both sides into account. Merchants plan their wares according to the information from the last trip and local stocks and prices, then at the distant market updated local prices and stock amounts apply, so merchants will come back with new information and potentially a different amount than planned for.

Coin will act as a generic ware (without other use) when direct exchange is not desirable. Unfortunately, both the planning and the auction will involve non-linear multi-dimensional (one dimension per ware) optimization.

CI Improvements With @Yakei

For my first real contribution, I decided to work on the CI. Pipelines can take as long as an hour, which is definitely too long. The current system makes use of a cache that is manually built into the Docker image. This has several problems, the main one being the cache gets outdated fast when it comes to Veloren's rapid development.

I decided to leverage GitLab CI's cache system to remove this factor. This system is fairly new, or at least not as stable as we would all like it to be. In fact, some of the main features I've used to reduce build time are still broken on the last LTS version, which means I'm using a release candidate version that was built about a week ago.

Once I overcame this first hurdle, I found out the average time spent creating and extracting the cache for each step was very significant. This is due to the default Go ZIP library used by the runner. I swapped this out to fastzip with a feature flag and found negligible overhead, compared to entire multiple minutes before.

The new workflow is the following:

• The Docker image is created when dependencies are changed (either by adding, removing, or upgrading them).
• Each step, the cache is extracted to the compilation folder.
• We iterate through the container cache and link each of the dependencies. This requires some overhead, but is needed as this step will always be called after the extraction of GitLab's cache. This means we cannot link an entire folder.
• The normal step is now executed. Cargo will now take care of only changed source files, and compilation completes much faster.
• When finished, the new cache is created if changes were detected. This workflow means only Veloren's modified source files will be compiled on CI.

While these improvements have mostly been completed, we have not yet decided on the cache policy that we will use. Recent talks with @xMAC94x and @XVar leads me to think we would introduce a cache only created on master and used on every branch, along with a special cache for Tarpaulin and benchmark. This policy ensures a decent max storage size on cache, as well as reducing again overhead on the cache by not creating a new cache on branches. We would also change benchmark and/or coverage behavior to manually trigger on branches as this is the main cause of time spent in pipelines.

A lot of questions aren't completely answered for now (the correct git clean policy to adopt as we use fetch instead of clone, how to reduce link creation overhead, etc.), but I think the new CI should be ready either this week or next.

I'm already seeing build times reduced by around 5 to 15 minutes. Hopefully we can stabilize runs to around 5 - 8 minutes each if we change benchmark + coverage to manually trigger on branch which is probably as good as we can get on this matter. My next goal will be to reduce the number of dependencies that Veloren uses and improve linking time.