Invader

[Kavawuvi 7,026]

Invader is a work-in-progress, open source, cross-platform toolkit for creating Halo: Combat Evolved maps. There are a number of tools that come with Invader:

• invader-archive - CLI program for creating archives of all of the tags required to build cache files
• invader-bitmap - CLI program for generating bitmap tags
• invader-bludgeon - CLI program for backhanding uncooperative tags
• invader-build - CLI program for building cache files from scenario tags
• invader-compare - CLI program for comparing sets of tags and showing differences
• invader-convert - CLI program for converting between different tag types - highly useful for Xbox map porting
• invader-dependency - CLI program for listing the dependencies of a given tag OR the tags that depend on a given tag
• invader-edit - CLI program for scripting tag editing operations
• invader-edit-qt - GUI program for editing tags
• invader-extract - CLI program for extracting tags from a map
• invader-font - CLI program for generating font tags
• invader-index - CLI program for generating a list of all of the tags in a cache file or resource map (useful for invader-bitmap)
• invader-info - CLI program for getting metadata of a cache file
• invader-model - CLI program for creating gbxmodel/model tags
• invader-recover - CLI program for recovering source data from tags
• invader-refactor - CLI program for renaming tags (or directories of tags) without breaking references
• invader-resource - CLI program for generating resource map files (i.e. bitmaps.map, sounds.map, loc.map)
• invader-script - CLI program for compiling scripts.
• invader-sound - CLI program for generating sound tags
• invader-string - CLI program for generating string list tags
• invader-strip - CLI program for cleaning up tags
• Six Shooter (Windows only) - GUI frontend for Invader
• ... and (hopefully) more programs to come!

You may be wondering, why am I taking time to replace the Halo Editing Kit, something that supposedly already works fine? I'm glad you asked (or I asked?)! The Halo Editing Kit...

• ...is closed source. This means that you cannot make changes to it or add functionality without resorting to modifying the .exe file directly. Also, information has been obfuscated away through compilation. Invader is open source.
• ...is unsupported. Since it's closed source, you cannot rely on the developers to issue any updates to fix problems with the program. If they never update it, then it is the final version you get. An ideal program should never need updated, but the HEK is very far from ideal. Because Invader is open source, anyone may fork and support Invader at any time.
• ...is limited. Since it's closed source, you have to modify the .exe file directly in order to make changes to it. If you don't, you're limited to building 384 MiB cache files. Also, singleplayer maps are tied to the resource maps you built them with, thus users must have the same exact bitmaps.map and sounds.map you used to ensure the correct assets are displayed/played. Invader does away with most of Halo's arbitrary limits, even the 384 MiB cache file size limit (it's 4 GiB now!).
• ...was made for older PCs. As robust as Windows's backwards compatibility may be, even it has limitations, especially with Win32 GUI programs like Guerilla and Sapien. Invader runs natively on 64-bit x86-based PCs and takes advantage of modern hardware to boot.
• ...is slow. This is due to thousands of unnecessary checks as well as the program, itself, not being compiled with optimizations. Invader is considerably faster than tool.exe!
• ...only guarantees to work on Windows. Not everyone uses Windows, and Wine compatibility on Linux is, at best, a mixed bag especially in regards to the GUI-based applications Sapien and Guerilla. Invader natively runs on both Windows and Linux without having to worry about Wine.

tl;dr: You can't guarantee the Halo Editing Kit will continue to work indefinitely, and it doesn't meet all of our needs anymore. Invader is here to fix that.

 

I recommend reading this post for more information on why it is important that the Halo Editing Kit should be replaced:

 

Site: https://invader.opencarnage.net/

Source code: https://github.com/SnowyMouse/invader

Builds: https://invader.opencarnage.net/builds/nightly/download-latest.html

 

Original version of this post (for posterity):

Spoiler

Invader is a work-in-progress, open source tag data compilation tool. What this does is it converts a scenario tag in a tag directory into a cache file. While I can't guarantee anything will come of this, I'm giving this a try at the very least.

 

Why do this?

I'm bored and I want to work on something other than Chimera for a change.

 

How long has this been worked on?

Since 2018-07-25

 

What could this (theoretically) do?

There are several uses for a map compiler.

 

Building (your) maps

While the obvious use is building a map, many argue that when you compile a map with tool.exe, then the resulting map belongs to Microsoft. Often, these people will cite the End User License Agreement and tool.exe specifically, despite the fact that neither the Halo Custom Edition EULA nor Halo Editing Kit EULA indicate anything about tool.exe. Regardless, by using an open source tool such as Invader, you can be sure that your map file is yours because I'm not a huge asshole.

 

Distributing and loading (your) maps via tag folders

By only distributing only your tags when distributing your map, you can save space, as clients and servers will already have all of the Halo Editing Kit tags installed. Also, you do not distribute any Bungie/Microsoft created assets by doing this - only your content. This can also help with organization, too. Rather than replacing the tag files in one tags folder, you simply override them in a second tags folder that's dedicated to your map.

 

Larger cache files

Cache files are limited to 384 MiB (402653184 bytes) of data. Any more than this and Halo will refuse to load the map. As a result, tool.exe may not allow such large map files to even be generated. In maps with large amounts of internalized assets such as custom bitmaps and sounds or large amounts of BSPs, it can be very easy to run into limitations.

 

A potential new cache file format

Halo limits the amount of tag data that can be loaded at once to 23 MiB (24117248 bytes). This includes all tag paths, the tag array, the tag data header, all non-BSP tag data, as well as the currently loaded BSP tag. Normally this isn't a problem for most multiplayer maps, but many large singleplayer maps with heavily complex geometry can get constrained by this. By compiling the map to a format that supports loading BSP tag data to load into a space separate from tag data, you could make bigger BSPs as well as use more tag data.

 

There are some hurdles to such an idea. For one, Halo CE would need to be modified to support this format, and this would require changing how BSPs are loaded into memory. Also, Sapien and tool.exe will have to either be modified or replaced to support the new format.

 

Where is the sauce?

On GitHub: https://github.com/Kavawuvi/Invader

 

Roll the screenshots

BSPs and shaders working (after being imported to a map via Eschaton)

 

Loading the map (almost) works. Models are broken and the game crashes after a few seconds, but it lived long enough for me to take a screenshot.

 

Please note that Invader is a work in progress. Using it would be nice, but it is limited. With invader-build, stock multiplayer maps are very close to perfect, while singleplayer maps and maps with scripts work but may have issues with AI, scripts, and detail object collection tags. Also, invader-bitmap cannot process sprites, yet. Lastly, Invader is untested for macOS. As such, macOS is not supported, but it will most likely work just as well as it does on Linux if built correctly.

---

 

Here is a FAQ composing of some of the questions I've received:

 

What is Invader?
Invader is an open source replacement of various tools in the Halo Editing Kit.

 

What license does Invader fall under?
GNU General Public License version 3. Note that, for numerous reasons, this is specifically version 3 and NOT "version 3 or later" like what many GPL-licensed projects do, so if a GPL version 4 comes out, then you cannot use this project under that license.

 

Where do I get Windows builds?
https://invader.opencarnage.net/builds/nightly/download-latest.html 

 

Where do I get Linux builds?
Currently you must build them from source. If you are on Arch Linux, you can use the package on the AUR to help you do this.

 

Is Invader finished?
No.

 

Where do I get the source code?
https://github.com/Kavawuvi/invader

 

What operating system can run Invader?
There are probably dozens of operating systems that can run Invader in some form since it's open source, but it's been at least tested on Arch Linux which is what I use to develop it.

 

How do I compile Invader?
I've written some instructions on the readme.

 

What HEK functionality can Invader do right now?
tool.exe build-cache-file, tool.exe windows-font, tool.exe bitmap

 

Is there a tutorial on how to use Invader?
No.

 

Is there a 32-bit version of Invader?
No.

 

How can I contribute to the project?

I recommend reading https://github.com/Kavawuvi/invader/blob/master/CONTRIBUTING.md

Basically, there are several ways you can contribute:

• One of the most important things you can do right now is test Invader and report issues at https://github.com/Kavawuvi/invader/issues or here on this topic.
• You can also contribute directly to the project by fixing issues listed there. Simply fork the repository, fix the issue, and submit a pull request for me to review, ensuring you adhere to the standards I set in the CONTRIBUTING.md file. If I don't approve your changes, I will leave a comment explaining why.

[Kavawuvi 7,026]

Models work now. Here's the cyborg model imported from an Invader-built map into the one map that everyone mods.

 

 

Though I guess model animations aren't as finished as I thought they were. I tried importing the Invader built animation tag and... well... this happened.

 

 

 

He's so happy, he doesn't want to leave the warthog even when you press the action button.

 

 

Fixed

[Kavawuvi 7,026]

Invader can now create (barely) playable map files.

 

 

You can download this map here:

 

Be sure to load with map_name unless you like exception errors.

[Kavawuvi 7,026]

Stuff looks a lot better than what it did before. It's still not playable in multiplayer, the HUD is still totally broken, vehicles are screwy, and plenty of other stuff doesn't work.

 

However, crouching works, running around is now normal, fall damage is normal, and first person animations look correct.

 

 

You can download this map here:

 

Once again, use map_name to load the map. Do not load it any other way or else your game will crash.

 

Note: To save space, every time I post a tutorial.7z file, I will purge the older tutorial.7z file. I am not saving these, so do not ask me for a copy.

[Kavawuvi 7,026]

I did more work on Invader on and off in the past week.

 

 

HUD works, effects work, but particles and lights are broken. Also, some particles appear to freeze the game.

 

Yet again, use map_name to load this one. Do not load it any other way or the game will crash.

 

Download here:

[Takka 13,377]

Not the particles!

[ShikuTeshi 1,299]

So would this be like modding in some other games?
In Halo we have to put everything we want into a map scenario and compile it all into a single file.
Where as most other games can just get a mod as an add-on and tell the game/map to use said files?

[Kavawuvi 7,026]

13 hours ago, ShikuTeshi said:

So would this be like modding in some other games?
In Halo we have to put everything we want into a map scenario and compile it all into a single file.
Where as most other games can just get a mod as an add-on and tell the game/map to use said files?

Something like this was similar to an idea I had, actually. My idea was being able to distribute a .zip or .7z of all of the tag files necessary to compile the map (or maps if this is a map pack), and you'd include a .json file inside of the root of this archive to determine the maps in the archive.

 

There are quite a few benefits to this approach:

• You only distribute a single archive of the tags for your map(s) rather than individual cache files.
   
• You only send your tags and not the tags included with the game, potentially saving space when distributing maps, especially multiple maps that are in the same tags folder Also, your maps are invariably and indisputably yours, because you do not include any assets from the original game with them, nor are they being run through tool.exe when being built.
   
• A single archive files means you send each tag only once rather than the same tag for each map file, saving space if distributing a map pack.
   

However, there are plenty of downsides and obstacles:

• Map loading time is increased in order to build the necessary cache file(s) when they're needed.
  • This can mostly be solved by caching the map files, only rebuilding them if a tags folder they depend on is changed, but this negates any space saving benefits.
     
• Everyone will need the Halo Editing Kit tags folder present, particularly one that matches their language, otherwise you'll have issues such as Spanish speaking players seeing only English assets as well as, potentially, mismatched tag data.
  • Including the entire HEK can be easily done and the EULA allows it to be distributed in whole, but it will make the download size larger.
  • I don't know where you can obtain a non-English HEK, but it probably wouldn't be that much effort to rip the installed bitmaps.map, sounds.map, and loc.map files and distribute only the English HEK.
     
• Invader may produce files slightly different to the files made by tool.exe, and revisions to Invader may produce slightly different files from previous revisions. Either will result in not being able to join multiplayer servers with the slight differences.
  • Removing the check or just checking tags rather than the resulting maps could also solve this problem, but would you want to risk issues to occur if the user has a different map file?
     
• It's less effort to rip your tags because they're already in HEK format, ready to be used. Although map protection is a futile endeavor, some map makers may still want it as an option with reasons such as for distributing betas or for preventing modifications and preserving the original mapper's intent.
  • Simply accepting the fact that your map will be ripped regardless of if, how, and/or why you protect it will solve this issue for you.
     
• Sapien, Guerilla, and tool.exe all assume a single tags directory. Therefore, for this approach to be cohesive for both mappers and players, all of these tools will likely need to be modified and/or replaced.
  • For Guerilla, MEK replaces it through Mozzarella, and this open source alternative could simply be modified, if needed. I'm sure MosesofEgypt would be happy to accommodate Invader's ability to use multiple tags folders.
  • Sapien has no known, complete replacement. This tool is required for compiling Halo scripts as well as populating scenario tags.
  • tool.exe has no known, complete replacement, but I'm sure MosesofEgypt is working on replacing a lot of its functionality.

[Kavawuvi 7,026]

More work has been done on Invader recently. I ran into a snag a couple months ago and decided to take a break from the project. This snag was the game freezing with high CPU usage (an infinite loop possibly?) when shooting a surface with a weapon which I had suspected was something particle related, because the issue only started to occur when particles were being dealt with.

 

After picking it back up a couple hours ago, I am pleased to announce that it no longer does this. Particles are now done, for the most part.

 

If you want to download the resulting tutorial.map file, here: 

 

As always, use map_load tutorial to load this. Do not load from multiplayer. The game will still crash

[Kavawuvi 7,026]

As of yesterday morning, I got Invader to build maps that work with multiplayer without crashing as well as mostly functioning vehicles. Therefore, you may use the in-game menu to load the map now. Note, however, that the menu tags are not in the map, so to exit, you will need to use console commands.

 

 

You can download the resulting tutorial.map file here:

 

 

In other news, Invader's source code works on the Raspberry Pi with no modification required. I will not support the pi or anything like that, but it was a neat test, nonetheless.