This Week In Veloren 106

This week, we see lots of progress on different meta tasks. The GitHub mirror is on its way to working, and Torvus is more stable than ever. @zesterer is experimenting with giant trees, and @LunarEclipse made some changes to the AUR packages. @Sam has been working on improvements to weapon states.

- AngelOnFira, TWiV Editor

Contributor Work

Thanks to this week's contributors, @Sam, @Snowram, @Sarra_Kitty, @AngelOnFira, @SWilliamsGames, @Imbris, @PhillippMevbenkamp, @James, @Ambien, and @azymohliad!

@zesterer has been playing around with giant trees. Hopefully, they'll become sites containing villagers or even small towns in time. Lots of work yet to be done to get to that stage though. They're 100% procedurally generated, thanks to @ccgauche's initial work developing the original tree generator code.

@Gemu is still working on basic one-handed weapons, and made a hydra concept model. @Pascal fixed a bug where the hammer leap with a low attack speed hammer would lead to a huge leap that caused fall damage even when on a flat surface. @Pascal also changed the damage and duration integers to floats throughout the ability & combat system to reduce typecasts.

@James worked on a functional behavior tree setup that will allow for better modularization of the agent AI code. @DaforLynx created a system that varies cricket chirps with temperature.

AUR Improvements by @LunarEclipse

This week, I made some improvements to the AUR packages:

• veloren-bin actually works now. I had to write wrappers for the game and the server to make them store user data in the system-specific folders instead of alongside the executable. (When installed as a system package, the directory with the executable is read-only so the game just crashed before changing that).
• Veloren and veloren-git can now be compiled in a clean chroot. I tracked down all the needed compile-time dependencies, so the builds should never fail due to those missing.
• All the packages now use the .desktop, .metainfo.xml, and the icon from the game assets instead of custom ones. I actually had no idea those existed until someone pointed it out in the comments.

Weapons Changes with @Sam

I've been changing inventory manipulations that touch equipped slots (e.g. your weapons and armor) to go through control actions instead of control events. What this means is that it'll be a bit more limited in when one can change their loadout, no more swapping weapons from the hotbar while you're actively swinging a sword. Practically though, there should be very little difference observable as currently, you'll still be able to freely modify your loadout when not actively attacking. Eventually, this feature could also be extended to add a duration to changing loadout slots while wielding a weapon or something similar.

Torvus Update by @xMAC94x

Short Torvus stability update; thanks to Dumbeldor's metrics additions for Torvus we now can track operation easily. The change is now for 7 days operational. Our uptime is a whopping 99,98%! Though we see some errors on Friday between 2:00 UTC and 3:30 UTC. having those metrics is extremely important to get reliable data and analyse whether something doesn't work.

Grafana dashboard (the units are in operations per second)

The Continued Mirror Issues by @AngelOnFira

This week, I spend some time re-examining our GitHub mirror problems. The mirror had been outdated for several months; I last pushed it manually in November 2020. Historically, the mirror has worked. This stopped somewhere around the beginning of 2020 I think. Before then, GitLab would periodically send mirror requests as a built-in feature to our GitLab repo. This stopped once we went over our monthly limit for LFS files being sent to GitHub.

Git is a tool that is meant for non-binary files. The version control method it uses keeps track of line differences through each commit. This means that if you navigate to any commit in our tree, it would have to build that commit from all of the previous changes since the beginning of a repo.

On the other hand, binary files can't have recorded "line changes". Think of a PNG image, or 3D model file. They're stored in a way that changes drastically when small changes are made. There isn't a trivial way of breaking down the changes with the file formats we use for our assets.

This problem is often only seen in game development projects that use Git. In the AAA industry, other version control tools are used instead. One of the largest is Perforce, which takes a very different approach to version control. Since its use case is for very large games with massive assets, its solution doesn't work the best for other software projects.

So if you are a game that uses Git/GitHub/GitLab as your version control systems, you have to take a different approach. There are a few different ways around this. GitHub allows you to "check in" files up to 100 MB. This means that as long as we have no single asset files that are larger that 100 MB, we can just use Git like normal, and not care about the fact that any time one of these binary files change, it has to update the entire file in Git. Ignorance is bliss.

However, this accrues technical debt. Each time we add a file, it gets stored permanently in the Git history. So after checking in asset files, and updating them over and over again, we eventually have a lot of files in the history. This means whenever someone clones the repo, they would have to download all of these files that have ever existed in the repo. Currently, this would mean around a 500 MB download for our entire repo.

The alternative to the approach of checking in binary files is to use a tool called Git LFS, or Git Large File Storage. This will instead store each binary file as a "pointer" to an external server, and store the assets there. Whenever you check out a commit, if there are any pointers in the file tree, it will replace them with downloaded files from the LFS server. This means that you don't have to have all of the asset files throughout the history of the project, but instead only the ones for the commit you're on. It saves data for developers, but also from the Git servers that are hosting the repo.

GitLab gives us unlimited bandwidth for LFS files, but GitHub only gives us 2 GB per month. This means that not only will we go against GitHub's limit as we mirror our repo, but anyone who clones the repo from there will tick our limit. I sent an email to GitHub asking if we could have the limit removed from the project, but it didn't get much useful feedback.

So now we seem to be at an impasse. We can't just get the LFS files to redirect to the GitLab server that we're hosting our files on, since GitHub forces them to be internal. Also, there is really not a lot of documentation around this. The use case seems to be small enough that it's not worth GitHub's time to look into, even for open source projects.

However, I did find a dirty solution to get around this. I manually set up a GitHub Action that clones the repo from GitLab, and uninstalls LFS from it. It then checks in all of the LFS files as normal Git files, and force pushes it to the master branch of the GitHub repo. This means that anyone cloning from GitHub will have to download the full repo with all of the assets, but at least we have an updated repo!

Here is what the GitHub Action looks like. I currently have it delete any GitHub Action workflows, as a bot isn't allowed to push them to the master branch. Each run takes around 6 minutes, but because GitHub Actions are free for open source projects, we can run it once an hour without any worry.

steps:
  - name: Install git-filter-repo
    run: git clone https://github.com/newren/git-filter-repo.git;
      ./git-filter-repo/git-filter-repo --help

  - name: Clone GitLab repo
    run: git lfs install;
      git clone https://gitlab.com/veloren/veloren.git;

  - name: Uninstall LFS
    run: cd veloren;
      git lfs uninstall;
      git lfs migrate export --include="*" --everything;

  - name: Remove workflow files
    run: cd veloren;
      ../git-filter-repo/git-filter-repo --invert-paths --path .github --force;

  - name: Push changes
  uses: ad-m/github-push-action@master
    with:
        github_token: ${{ secrets.GITHUB_TOKEN }}
        branch: master
        force: true
        tags: true
        directory: veloren

There is still more work to be done so that each job can properly run once an hour, but this is currently the best solution I could find to the problem. I do hope that GitHub works on making Git LFS more accessible to open source projects, as it's quite a necessity for open source games. GitHub has done well in recent years to support open source projects, but this alone is one of the largest reasons Veloren prefers GitLab to GitHub.