Update: LZMA map compression

[Kavawuvi 7,026]

Yesterday, I made map compression a thing, and I am considering implementing it into Chimera. Here's a mini-FAQ on it:

 

What compression algorithm are you using?
LZMA

 

What compression ratios can be achieved?
Compression ratios vary from map to map. Generally, uncompressed data compresses better than compressed data. For example, model data and tag data generally compress better than DXT texture and OGG sound data with OGG sound data being the least compressible.

 

For example, the Refined Campaign is a collection of 10 singleplayer maps. These maps, when combined, take up a large amount of data: 2.649 GiB. When compressed, they are reduced to 1.003 GiB, achieving a compression ratio of 35.27%.

 

How are loading times impacted?
Loading times vary depending on the size of the compressed map, but it more depends on the specifications of your PC. Faster storage and more CPU threads will drastically improve performance. Judging by the results of the recent poll, most people have at least four threads.

 

How do I make a compressed map?
A tool is in development to turn existing Halo cache files into compressed files. Not all maps can be compressed, however. Maps compiled using os_tool, most protected maps, and maps with embedded Chimera scripts are examples of maps that cannot be compressed.

 

How long does it take to compress a map?
While LZMA decompression is fast, LZMA compression is much slower and can be fairly memory-intensive. Compressing Refined's d40.map took 20.591 seconds with RAM usage of around 4 GiB. Reducing RAM usage without affecting the output file is possible, but it requires limiting the thread count, further increasing compression time.

 

Can compressed maps be used with Refinery?
Currently, no. The tool that compresses maps can decompress maps, but those maps will still not work with Refinery. Making it work would most likely be trivial, however, if anyone is up to the task.

 

[Kavawuvi 7,026]

Here are some more questions that were asked about map compression:

 

Does it use LZMA or LZMA2?
LZMA2 is not actually a compression algorithm but a container format that can use LZMA. However, if you were to strip the cache file header and the table of blocks from a compressed map, then it could possibly be decompressed using tools like xz.

 

Does it compress in blocks or is it one stream of data?
That depends on the uncompressed size. Generally, smaller blocks result in a worse compression ratio. If the uncompressed size is less than 8 MiB, then it will not do it in blocks. There isn't any significant performance benefit to separating such small files into blocks, while doing so would result in worse compression ratios.

 

If anyone has any more questions, feel free to ask them here.

 

Also, I'm considering dropping Open Sauce support. There are several reasons why I'd want to do this, but here are the main reasons:

 

 - It is difficult to use Open Sauce on Linux with Wine, and this makes it difficult for me to develop Chimera while testing Open Sauce.
 - Chimera's widescreen mod and Open Sauce's widescreen mod do not play well, and I'd rather not upset people by disabling the widescreen mod if Open Sauce is detected.
 - Open Sauce has a map downloader, and I absolutely do not want to make a workaround to support it while still having my own map downloader.
 - Chimera's map downloader does not support Open Sauce maps, anyway, and I have no way of knowing if the server is hosting such a map.

 

Essentially, this means that Chimera's features will not care if Open Sauce is installed. However, like HAC2, if you encounter any problems, I won't try to fix it and will tell you to uninstall Open Sauce or Chimera, instead.

[Kavawuvi 7,026]

Here is another table. This one is for the entire Refined campaign.

 

 

Here are the important points:

• On average, compression ratio is under 40%. This reduces the total size of the Refined campaign on disk from around 2.65 GiB to around 1 GiB.
• Decompression on 8+ threads provides the best decompression times. It is worth noting that times and scaling can be improved further, but it will come at the cost of compression ratio.
  • Even on CPUs that don't have 8 physical cores but have at least 8 threads (e.g. desktop Intel Core i7 CPUs, Ryzen 5 CPUs, etc.), SMT/HT can still provide improved throughput.
• Decompression on 4 threads provides worse (longer) decompression times compared to 8+ threads.
  • CPUs that are dual core but have four threads (e.g. pre-Cannon Lake Intel Core i3 CPUs, Kaby Lake and Cannon Lake Pentiums, etc.) will perform worse than true quad cores but may still get better times than those that have only two threads.
• Even when locked to 2 threads, decompression usually takes under 4 seconds.
  • Note that PCs with CPUs that actually have only 2 threads will probably perform worse due to having low clock speeds, slow memory, etc. Also, the operating system may be doing tasks in the background, consuming CPU time.