Invader

[Kavawuvi 7,026]

I've added threading to invader-sound. This will allow you to take advantage of modern, multi-threaded CPUs when resampling multiple sounds or encoding lots of sounds or split permutations.

 

Here's an example graph:

 

 

I included tool.exe as a baseline. Here are some notes about it:

• This test is being done on the latest version of Wine. When running the same tests on Windows 10, I've found it to be, on average, about 5% slower to compile sounds on Windows 10. Not really a significant discrepancy, and it does ultimately make tool.exe look better, but it's an unfair comparison nonetheless.
• tool.exe was not able to do entire albums. It'd fail to do some tracks, and the massive number of permutations due to being split exceeded its maximum permutation count of 256.

Edit: Fixed an issue with invader-sound -j 1 being more than 4x slower than four threads.

[Kavawuvi 7,026]

Some news today!

 

I've reworked some bitmap-related things, including some details about mipmap generation and error handling. This isn't going to affect most bitmaps, but accuracy is accuracy!

 

The GitHub repository has also been moved to https://github.com/SnowyMouse/invader. If you currently have the master branch of the Invader repository checked out using git, you can switch to the new upstream URL by running these commands:

git remote set-url origin https://github.com/SnowyMouse/invader.git
git pull

People have requested tutorials for using Invader on the Invader Discord, so I've made the first tutorial: how to create a bitmap tag. You can find this at https://github.com/SnowyMouse/invader/wiki/Creating-a-bitmap.

 

I'm planning on posting more there, later, such as a tutorial on how to make a map (after some prerequisite tools are finished). I probably won't post any of them here, as most people on this forum don't really use Invader.

 

[Kavawuvi 7,026]

I've been working on a new model compiler. It'll be a while before we see it, but nevertheless, I have plans for it!

 

I've also done some more work on the wiki!

 

The bitmap tutorial has been updated with information regarding cube maps and bitmap formats.

 

I've also added a new tutorial for creating sounds. The sound tag tutorial discusses:

• Setting up your data folder structure
• Sound classes
• Split permutations
• Audio formats
• Resampling
• Remixing
• Sound tag generation
• Pitch ranges

I've yet to see a sound tag tutorial quite as in-depth as mine, so check it out at https://github.com/SnowyMouse/invader/wiki/Creating-a-sound

 

More tutorials are on the way.

[Kavawuvi 7,026]

I haven't posted an update in a couple months, but that doesn't mean I haven't been working on things all this time!

 

I added mcc-custom as a build target for invader-build. This builds a Custom Edition map compressed with MCC-style compression and it generates a .fmeta file in the same directory.

 

A number of small changes have been made:

• invader-build now automatically swizzled/deswizzles bitmaps based on the target engine.
• invader-build now overwrites the leading width as tool.exe does (as such you cannot edit this field anymore)
• invader-build now defaults a number of values it didn't before
• invader-build now allows for heat loss per second greater than 1
• invader-bitmap has been refactored, and a lot of stuff has been rewritten. This was to fix a few edge cases for certain color plates.
• invader-bludgeon --all now has multithreading support and uses your CPU thread count by default
• invader-sound now uses your CPU thread count by default

 

For more information regarding MCC support, see this post: 

 

[Kavawuvi 7,026]

The nightly builds of Invader no longer have any dlls in it! Oh, and this is even with Six Shooter. This does mean that the sizes of the exe files are a lot larger (about 240 MiB) but the folder's certainly a LOT cleaner. I might add libinvader.dll (Invader's core functionality) which would cut about 180 MiB off of the file size, though this may result in user error when updating Invader, namely with mismatched .exe files and .dlls. A few people have suggested I add an installer, though I dunno if that's a great idea at this point.

 

Also, I've fixed some issues with postprocessing with invader-bitmap, notably with blurring and sharpness. It's PRETTY close to tool.exe's output (although you should probably just sharpen/blur in whatever image editor you use anyway). Speaking of which, I've decided to let invader-bitmap produce .bitmap tags with dithering after all. In fact, the whole -x (--extended) stuff has been dropped altogether!

 

And... wow, the year is almost over, so I'm looking into maybe making a post soon about the year in review, as Invader has improved so much across the year 2020!

 

Anyway, I'm working on the following stuff right now:

• Accurate Xbox map tag extraction
• Improving invader-build's accuracy
• More features for Six Shooter
• Xbox map compilation

And, goodness, there are quite a lot of things I have planned for 2021 that I'd love to do. One thing in particular I'd like to maybe try working on is a scenario editor. It MIGHT just make more than three people want to test Invader. MIGHT.

 

Um... probably not, though...

[Sunstriker7 698]

I bet it actually would.

Edited December 26, 2020 by Sunstriker7

[Kavawuvi 7,026]

Xbox map support is coming along pretty well.

 

 

...And it's done. Version 0.40 is now out! I'm going to start writing a year-in-review sort of post tomorrow.

[Kavawuvi 7,026]

So here's that 2020 year-in-review post.

 

2020 may have been a pretty bad year with the world, but it was actually a really good year for Invader, not just thanks to our efforts but also your efforts with testing as well as the donations I've received that have helped... a lot. And, of course, Tucker933 for generously hosting Invader on his own dime. I am truly thankful for you people helping to make this project's goals a reality.

 

invader-sound: Sound tag generation

 

Invader now has sound tag generation which is faster than tool.exe, uses newer Ogg Vorbis libraries (fewer vulnerabilities), and no longer requires you to pirate legitimately salvage the Xbox ADPCM codec that came from the Xbox development SDK (many thanks to MosesofEgypt and David Brynart for making that possible). 

 

It can also resample and mixdown/mixup audio, and it supports FLACs as input in addition to uncompressed WAVs. You can even use 8-bit and 24-bit integer audio as input, too!

 

This tool was technically introduced at the end of 2019, but I’m including it since it’s reasonably close enough to 2020 for it to warrant going on this list.

 

invader-bludgeon: Tag fixing

 

Invader now has a tool for fixing common issues with tags as a result of being improperly extracted or edited with other tools. Did you know that some Bungie and HEK tags have some invalid values, too? This is likely from a result of numerous changes in the tag definitions which Halo (sometimes) ignores.

 

invader-refactor, invader-dependency: Tag dependency manipulation

 

Invader now has a tool for changing references to a tag or tag class to another, allowing you to rename tags without breaking dependencies. You no longer have to do the tedious task of building a map ‘til it works or building a map, renaming the tag in the map, and extracting the tags (which have no doubt incurred some data loss as a result). This tool also goes well with porting maps to Xbox, since the Xbox version has better stock tags available in a lot of cases.

 

Invader also has a tool for querying all tags that depend on a tag (or all tags a tag depends on).

 

invader-extract: Better tag extraction

 

Invader has a tag extraction program that focuses on making accurate, reproducible tags. It was introduced in late November of 2019, and it has received a number of improvements, resulting in faster, more accurate tag extraction, including support for Xbox tag extraction. It’s worth noting that unlike Arsenic, tags it extracts do not need to be deswizzled or anything like that. They are immediately ready for use in Halo PC or back to the Xbox version with invader-build’s new Xbox map building feature.

 

invader-build: Better error checking

 

Here’s a requested feature: better error checking. Invader has highly useful error checking that utterly destroys tool.exe’s error checking. Here are a couple of errors for example:

 

Invalid vectors (caused by improperly generated or edited tags)

invader-build: Tell you the exact value that’s broken as well as the exact tag path; stop building
tool.exe: Complain about a bad vector for some model; dereferences the model and makes a completely invalid map

 

Invalid lightmap bitmap (caused by improperly generated or edited tags)

invader-build: Complain about the invalid lightmaps; stop building
tool.exe: Exception error with some worthless register information; stop building

 

This helps it to make it a great tool for both experienced map developers and beginners looking into finding an easy game to mod, as it provides more helpful error information that tool.exe while also finding even more crashing issues than tool.exe ever could. We’ve fixed several maps thanks to this, and I plan to further enhance this feature as I continue to work on Invader.

 

This feature has received a lot of praise. Of course, it’s also received some criticism due to it resulting in a lot of seemingly fine maps not being allowed to build because of actual errors that can cause problems with their map and generally just being heavy in error checking in general without any option to turn it off. I hear you! Unfortunately the fabric of reality destroyed itself when you used a bitmap sequence that didn’t have enough reticles for your zoom levels, so I'm not able to help you. Sorry!

 

invader-edit-qt: Tag editing

 

invader-edit-qt is a tag editor that is not only cross-platform but it is much faster than Guerilla. It is capable of opening the largest BSP tags in the game within only a few milliseconds, making it the fastest tag editor for Halo PC by far.

 

 

 

 

For example, on my Arch Linux box, d40c takes around three seconds to load on Guerilla, but it takes around one third of a second on invader-edit-qt, being about 7x faster on average. On my Windows 10 machine, Guerilla is even slower, taking around four seconds, yet invader-edit-qt is still around half a second – about 5x faster on average.

 

It also features sound playback, bitmap previewing, and a dark theme for Windows 10 users who have the feature enabled in Windows 10’s personalization settings.

 

invader-build: Xbox map building

 

Saving the best for last, Invader now builds maps for the original Xbox version of the game, and it is just as accurate at building those as it is at building PC maps. For most tags, you can also use original Halo Editing Kit tags without any sort of annoying tag importation or extraction work, though you may need to some conversions using the new invader-convert and invader-refactor tools, namely with various gbxmodel and shader_transparent_chicago_extended tags.

 

Arsenic had a good life generating completely broken Xbox maps under an annoying, convoluted process, but now you can add or remove it from your PC for good.

 

 

--

 

Honestly, this post just highlights the major details, but far more went into Invader, including a number of critical bug fixes and improvements towards Invader's accuracy. I wrote a whole changelog if you want a more raw list of everything: https://github.com/SnowyMouse/invader/blob/master/CHANGELOG.md

 

Again, many things to the people who have supported Invader through testing and donations. This would not have been possible without your help! I hope 2021 is another fantastic year for this project, as I have many exciting plans for this.

 

Thank you!

[Enclusion 163]

55 minutes ago, Kavawuvi said:

Spoiler

 

So here's that 2020 year-in-review post.

 

2020 may have been a pretty bad year with the world, but it was actually a really good year for Invader, not just thanks to our efforts but also your efforts with testing as well as the donations I've received that have helped... a lot. And, of course, Tucker933 for generously hosting Invader on his own dime. I am truly thankful for you people helping to make this project's goals a reality.

 

invader-sound: Sound tag generation

 

Invader now has sound tag generation which is faster than tool.exe, uses newer Ogg Vorbis libraries (fewer vulnerabilities), and no longer requires you to pirate legitimately salvage the Xbox ADPCM codec that came from the Xbox development SDK (many thanks to MosesofEgypt and David Brynart for making that possible). 

 

It can also resample and mixdown/mixup audio, and it supports FLACs as input in addition to uncompressed WAVs. You can even use 8-bit and 24-bit integer audio as input, too!

 

This tool was technically introduced at the end of 2019, but I’m including it since it’s reasonably close enough to 2020 for it to warrant going on this list.

 

invader-bludgeon: Tag fixing

 

Invader now has a tool for fixing common issues with tags as a result of being improperly extracted or edited with other tools. Did you know that some Bungie and HEK tags have some invalid values, too? This is likely from a result of numerous changes in the tag definitions which Halo (sometimes) ignores.

 

invader-refactor, invader-dependency: Tag dependency manipulation

 

Invader now has a tool for changing references to a tag or tag class to another, allowing you to rename tags without breaking dependencies. You no longer have to do the tedious task of building a map ‘til it works or building a map, renaming the tag in the map, and extracting the tags (which have no doubt incurred some data loss as a result). This tool also goes well with porting maps to Xbox, since the Xbox version has better stock tags available in a lot of cases.

 

Invader also has a tool for querying all tags that depend on a tag (or all tags a tag depends on).

 

invader-extract: Better tag extraction

 

Invader has a tag extraction program that focuses on making accurate, reproducible tags. It was introduced in late November of 2019, and it has received a number of improvements, resulting in faster, more accurate tag extraction, including support for Xbox tag extraction. It’s worth noting that unlike Arsenic, tags it extracts do not need to be deswizzled or anything like that. They are immediately ready for use in Halo PC or back to the Xbox version with invader-build’s new Xbox map building feature.

 

invader-build: Better error checking

 

Here’s a requested feature: better error checking. Invader has highly useful error checking that utterly destroys tool.exe’s error checking. Here are a couple of errors for example:

 

Invalid vectors (caused by improperly generated or edited tags)

invader-build: Tell you the exact value that’s broken as well as the exact tag path; stop building
tool.exe: Complain about a bad vector for some model; dereferences the model and makes a completely invalid map

 

Invalid lightmap bitmap (caused by improperly generated or edited tags)

invader-build: Complain about the invalid lightmaps; stop building
tool.exe: Exception error with some worthless register information; stop building

 

This helps it to make it a great tool for both experienced map developers and beginners looking into finding an easy game to mod, as it provides more helpful error information that tool.exe while also finding even more crashing issues than tool.exe ever could. We’ve fixed several maps thanks to this, and I plan to further enhance this feature as I continue to work on Invader.

 

This feature has received a lot of praise. Of course, it’s also received some criticism due to it resulting in a lot of seemingly fine maps not being allowed to build because of actual errors that can cause problems with their map and generally just being heavy in error checking in general without any option to turn it off. I hear you! Unfortunately the fabric of reality destroyed itself when you used a bitmap sequence that didn’t have enough reticles for your zoom levels, so I'm not able to help you. Sorry!

 

invader-edit-qt: Tag editing

 

invader-edit-qt is a tag editor that is not only cross-platform but it is much faster than Guerilla. It is capable of opening the largest BSP tags in the game within only a few milliseconds, making it the fastest tag editor for Halo PC by far.

 

 

 

 

For example, on my Arch Linux box, d40c takes around three seconds to load on Guerilla, but it takes around one third of a second on invader-edit-qt, being about 7x faster on average. On my Windows 10 machine, Guerilla is even slower, taking around four seconds, yet invader-edit-qt is still around half a second – about 5x faster on average.

 

It also features sound playback, bitmap previewing, and a dark theme for Windows 10 users who have the feature enabled in Windows 10’s personalization settings.

 

invader-build: Xbox map building

 

Saving the best for last, Invader now builds maps for the original Xbox version of the game, and it is just as accurate at building those as it is at building PC maps. For most tags, you can also use original Halo Editing Kit tags without any sort of annoying tag importation or extraction work, though you may need to some conversions using the new invader-convert and invader-refactor tools, namely with various gbxmodel and shader_transparent_chicago_extended tags.

 

Arsenic had a good life generating completely broken Xbox maps under an annoying, convoluted process, but now you can add or remove it from your PC for good.

 

 

--

 

Honestly, this post just highlights the major details, but far more went into Invader, including a number of critical bug fixes and improvements towards Invader's accuracy. I wrote a whole changelog if you want a more raw list of everything: https://github.com/SnowyMouse/invader/blob/master/CHANGELOG.md

 

Again, many things to the people who have supported Invader through testing and donations. This would not have been possible without your help! I hope 2021 is another fantastic year for this project, as I have many exciting plans for this.

 

Thank you!

 

 

Love invader, great work as always! Excited to see what magical things are coming in the future :-)

[Kavawuvi 7,026]

I've added a new command line tag editor to Invader, invader-edit.

 

You aren't supposed to use this directly for modding (that's what invader-edit-qt is for!). This tool is primarily for scripting, and it's been used a lot for various projects, including Refined.

 

 

You can even use this to make your own tools. For example, here is a tool I wrote which can determine if any given point is inside a BSP. It's a little slow (takes about half a second to load large BSPs), but it gets the job done. This can be very useful!