Invader

[Shelly Herself 221]

Quote

Extra options that do not preserve themselves in the tag data leads to an issue where bitmaps could end up being changed inadvertently, for example if you used a script to run through the data directory and rebuild every bitmap it finds.

Honestly? Tough shit imo.

[Kavawuvi 7,026]

On 9/26/2019 at 6:55 PM, gbMichelle said:

If you used a better compression scheme that looked better that would also not save in the tag. If you can recreate what you made with your tools using vanilla tool, what is the point of your tool?

 

To be clear, compatibility with legacy is good, but I don't think we should be limiting ourselves to that all the time. The whole draw for this project for me is to be able to do stuff that I've been manually purposefully avoiding tool to accomplish, but automated. Edit: Even if it means adding an extra switch to my command.

To answer your first question, here is what invader-bitmap does that tool.exe doesn't do:

• invader-bitmap can use png, bmp, and tga files in addition to tif/tiff
• invader-bitmap can let you create non-2D texture bitmaps as well as bitmaps with any available data format without you ever having to leave the command line interface
• invader-bitmap can let you overwrite various other bitmap settings like bump height and mipmap count without you ever having to leave the command line interface
• invader-bitmap can let you specify any data folder and any tags folder rather than using the current directory
• invader-bitmap can create bitmaps that exceed 16 MiB

As for your other part of the comment, every single parameter used to generate a bitmap tag is stored in the bitmap tag. By doing this, you don't need to re-enter every single parameter every time you re-generate the bitmap tag. This is how tags fundamentally work, and deviating from this puts us in a situation where some settings get saved and some settings don't, negating the point of using tags in the first place.

 

The main goal is that anything made with tool.exe that is re-made in invader-bitmap should function identically. If it doesn't, I consider this a bug that needs fixed.

 

However, the opposite doesn't need to be true: something made with Invader does not necessarily have to work with tool.exe. For example, tool.exe will not process bitmaps that exceed 16 MiB. A 16 MiB limitation more-or-less limits you to these resolutions:

 

With mipmaps:

• 4096x4096 - DXT1
• 4096x2048 - DXT3/DXT5
• 2048x2048 - 16-bit
• 2048x1024 - 32-bit

Without mipmaps:

• 8192x4096 - DXT1
• 4096x4096 - DXT3/DXT5
• 4096x2048 - 16-bit
• 2048x2048 - 32-bit

 

However, invader-bitmap doesn't have such arbitrary limits. You can make 4096x4096 32-bit bitmaps, and Invader won't have any issue with them provided they are otherwise valid.

[Kavawuvi 7,026]

Invader is now 0.13.0. It just so happens that this spooky version number happened on a spooky month. Here are the changes since 0.12.0:

 

Changed

• invader-build: Changed how stubbed tags are created so tag extractors won't try to extract them
• invader-bitmap: Errors if the tag contains uppercase characters in its path
• invader-build: Errors if any non-stubbed tags contain uppercase characters in their paths
• invader-font: Errors if the tag contains uppercase characters in its path
• invader-string: Errors if the tag contains uppercase characters in its path

Fixed

• invader-build: Fixed not fixing the render bounding radius if it was less than the bounding radius but non-zero.
• invader-build: Fixed not setting the weight value for color change permutations in objects.
• invader-bitmap: Fixed detail fade factor so it matches tool.exe's detail fade factor more closely if not exactly.
• invader-build: Fixed certain sound permutation file offsets not being correctly marked as internal; this should fix some sounds that sounded fine when built with tool.exe but sounded corrupt when built with invader-build

Removed

• invader-bitmap: Removed --filter-blur and --filter-sharpen. Tags that have these values set will still have the filter(s) applied. However, for newer tags, you should use an image editor, as you will get similar or better results.
• invader-bitmap: Removed --sprite-spacing and used tool.exe's broken functionality, instead.
• invader-bitmap: Removed the ability to set sprite budgets below 32 or above 512.

View the full changelog at https://github.com/Kavawuvi/Invader/blob/master/CHANGELOG.md

 

From what I've seen, people prefer these less frequent but longer update announcements, so I'll keep doing them like this since it's less work for me. The changelog and GitHub repo, however, will be updated as changes are made. So, if you need to read about the more recent changes, then be sure to check that from time to time.

 

As always, you can get the latest build at https://invader.opencarnage.net/builds/nightly/download-latest.html. These are automatically compiled and uploaded via a script on a local machine. These snapshots are built from the latest commit (at the time of building) on the Git repository which may include unreleased features, so these announcements can and probably will get outdated fairly quickly.

 

See you next month!

[Kavawuvi 7,026]

Here's a monthly update for the progress made in October. I hope everyone had a good Halloween last night.

 

Invader is now 0.15.2, and there have been a number of changes, including the new invader-compress program. View this topic for more information.

 

Added

• invader-archive: Now accepts filesystem paths for the tag if -P is passed
• invader-bitmap: Now accepts filesystem paths for the image input if -P is passed
• invader-build: Now accepts filesystem paths for the scenario tag if -P is passed
• invader-dependency: Now accepts filesystem paths for the tag if -P is passed
• invader-font: Now accepts filesystem paths for the the TTF file if -P is passed
• invader-string: Now accepts filesystem paths for the text file if -P is passed
• invader-crc: Now prints a warning to stderr if the CRC32 in the cache file header is wrong
• invader-crc: Now errors with a useful error message if a resource map is loaded with it
• invader-build: Halo Demo / Trial maps can now be built (-g demo)
• invader-crc: Halo Demo / Trial maps can now be used
• invader-build: --no-indexed-tags was renamed to --no-external-tags
• invader-build: Building retail and demo maps now uses the resource maps
• invader-build: The gain modifier for sound\sfx\impulse\ting\ting.sound is now changed to 0.2 if building for retail or demo and 1.0 otherwise
• invader-build: Now shows the target engine of the map
• invader-compress - Compresses cache files using the Zstandard algorithm
• invader-resource: Added --retail / -R which will allow you to build retail maps
• invader-build: Copying TagString data now checks for string overflow
• invader-crc: Works with maps compressed with invader-compress
• invader-indexer: Works with maps compressed with invader-compress

Changed

• invader-build: The diagnostic messages now show tags using external raw data when building a retail or demo map
• invader-dependency: Now uses -R instead of -r for reverse dependencies
• invader-dependency: Now uses -r instead of -R for recursive dependencies
• invader-build: Bitmaps and sound data are now shown separately next to their combined total in the diagnostic output
• invader-build: The number of tags that could be cached/indexed is now shown
• invader-archive: Now handles exceptions instead of calling abort() while either compiling or parsing cache files
• invader-build: No longer errors when orphaned model nodes exist
• invader-bitmap: Made detail fade factor closer to tool.exe's output - still needs more work but it's remarkably close

Fixed

• invader-build: Fixed decals with null references crashing the game
• invader-bitmap: Fully implemented and fixed 3D textures
• invader-dependency: Fixed an issue where it didn't include a null terminator, causing some tags to fail to open
• invader-archive: Fixed an issue where some stock HEK .gbxmodel tags tried (and failed) to be archived as .model tags
• invader-bitmap: Fixed an issue where sprites spanning the entire width wouldn't be detected
• invader-archive: Fixed an issue where the root tag would have its full system path included when using -s

As always, you can get the latest build at https://invader.opencarnage.net/builds/nightly/download-latest.html or view the changelog at https://github.com/Kavawuvi/invader/blob/master/CHANGELOG.md.

 

If anyone has any comments, suggestions, or bug reports, feel free to reply here. Otherwise, see you next month! Stay warm.

[Kavawuvi 7,026]

A mini-update: I converted the tag definitions to the more parsable .json format. This won't significantly impact Invader's performance since these definitions will be transpiled back into C++ headers. However, it will allow me to tag parsers.

 

As a test, I converted them into Eschaton plugins. I use Eschaton for viewing tag data and finding offsets (which for building cache files is VERY useful), and I've found Sparky's plugins to be fairly inaccurate in some cases. Since my definitions are basically used to build cache files, I figured these would actually be more complete and more useful for my purposes. Here's a screenshot of Eschaton with them:

 

 

As usual, I will post an update to Invader next month. To give you an idea of the size of the update, this month is only half over and there are already a number of changes already logged in the changelog. These changes over 80 commits with about 40 more on the way in a separate branch, and there have been several version bumps.

[Kavawuvi 7,026]

Here's a update on the month of November! Invader is now 0.20.2.

 

Not much has changed, besides adding a new tag extractor and a new meta data viewer as well as fixing a lot of issues, of course.

 

Added

• New tool: invader-info - Displays meta data of a map file but can also show specific data (-T): build, compressed, compression-ratio, crc32, crc32-mismatched, dirty, engine, map-type-scenario-scenario-path, stub-count, tag-count, tags
• New tool: invader-extract - Extracts tags from cache files
• Added a new tag parser
• invader-build: Added -N to rename scenarios when building
• invader-build: Added -c for compress (forging CRC now uses -C)
• invader-build: Strips default data from compressed animations now
• invader-font: Added support for .otf files

Changed

• Help menus have been rewritten
• All tag definitions have been converted to .json format
• invader-bitmap: Changed -m to -M to avoid confusion with --maps
• invader-bitmap: The default bump height is now 0.026
• invader-build: Shows map type when building
• invader-build: Uses new indexer format (previous index files will not work)
• invader-build: 32 byte strings are now zeroed out on copy
• invader-build: The stub count is now shown
• invader-build: Index parsing is now case insensitive
• invader-compress: Strings are now zeroed out before copy
• invader-compress: The default compresson level is now 19
• invader-indexer: Changed index format to simply list tag paths with extensions

Fixed

• invader-bitmap: Fixed -P not working with non-.tif files
• invader-build: Fixed indexed sound data taking up more space than needed in the tag file; this will reduce tag space usage a little more
• invader-build: Fixed -g not erroring if an invalid engine was given
• invader-build: Fixed sound looping tags' gain being set to 0
• invader-build: Fixed some values in shader_transparent_plasma being set to 0
• invader-build: Fixed biped A In, B In, C In, D In being set to 0
• invader-build: Fixed some objects' flags being set to 0
• invader-build: Fixed an undefined behavior issue with firing positions
• invader-build: Fixed light tag durations not being properly multiplied by 30
• invader-build: Fixed some sound tags not working as intended
• invader-build: Fixed lens flare rotation scale not being converted properly
• invader-build: Fixed setting mouth data and subtitle data to 0 size
• invader-build: Fixed incorrect footstep sounds being used if not using an MEK-extracted BSP tag

Removed

• Removed invader-crc. invader-info now fulfills the same exact purpose
• invader-compress: Levels above 19 were removed; they are pointless and needlessly use extra memory

Another feature added post-0.20.2 was colored diagnostic text output. I'm still figuring out how I'll do this for Windows. Basically, here are some screenshots:

 

 

 

Lastly, I posted this earlier today on the Halo CE Reclaimers Discord, but basically, I'm considering removing dithering and nearest-neighbor mipmapping from invader-bitmap. Here's the whole spiel from the Discord I gave:

[12:18 PM] Kavawuvi: So, next version of Invader, I'm considering removing dithering as well as nearest-neighbor mipmapping from invader-bitmap. Any thoughts?
[12:20 PM] Kavawuvi: The problem is that the original point of having these bitmap tags having all of this data was so that it provided parameters for the bitmap tool (tool.exe) to generate the bitmap data.
[12:21 PM] Kavawuvi: Previously, in order to satisfy this, I added more flags to specify how to generate bitmap data, and this included dithering and mipmap settings.
[12:22 PM] Kavawuvi: The problem with doing this is that there is no way to know if a tag has these flags or not, since tags lack a versioning system once they're in the map file. For all I know, MCC uses the data where these flags are located for other purposes.
[12:25 PM] Kavawuvi: Considering hardly anyone actually uses Invader let alone invader-bitmap, these features probably won't be missed anyway, so having them in here is more a liability than an asset in my opinion.

Anyway, as always, you can download Windows builds from the Nightly Builds page. https://invader.opencarnage.net/builds/nightly/download-latest.html

 

Stay tuned until next month for the next update!

[Kavawuvi 7,026]

Here's an update for the month of December! Normally I post these at the start of the next month, but I felt like not doing that this one time because it's my project and I can (mostly) do whatever I want with it.

 

This update is a bit smaller than last update, but it's also just as important as it introduces a new tool: invader-sound.

 

As the name suggests, this tool allows you to generate sound tags. You can either opt for 16-bit PCM (you will need either the latest builds of HAC2 or Chimera to enable these in Halo!), or you can use lossy compression through Ogg Vorbis.

 

Xbox ADPCM is supported, too, and you do NOT need to pirate obtain the original codec to use Xbox ADPCM with invader-sound, but note that ADPCM is worse, compression ratio and quality wise, than Vorbis. If you're unsure as to what to use, check out the readme.

 

Added

• New tool: invader-sound Generates sound tags
• Added colors for various diagnostic messages depending on the user's terminal
• invader-info: Added `-T external-tags` which indicates the number of tags that use external resource maps, including indexed tags
• invader-info: Added `-T external-bitmaps` which indicates the number of bitmap tags that use external resource maps, including indexed tags
• invader-info: Added `-T external-loc` which indicates the number of loc tags that use external resource maps, including indexed tags
• invader-info: Added `-T external-sounds` which indicates the number of sound tags that use external resource maps, including indexed tags
• invader-info: Added `-T external-bitmap-indices` which indicates the number of externally indexed bitmap tags
• invader-info: Added `-T external-loc-indices` which indicates the number of externally indexed loc tags
• invader-info: Added `-T external-sound-indices` which indicates the number of externally indexed sound tags
• invader-info: Added `-T languages` which lists all languages valid for the map separated with spaces
• invader-info: Added `-T external-pointers` which indicates whether the map uses external pointers (sometimes occurs if built from tool.exe due to a bug with indexing raw data)
• invader-info: Added `-T tags-external-pointers` which list tags that have external pointers that were not indexed

Changed

• invader-extract: Tag IDs are now set to FFFFFFFF when extracted. This will make extracted tags more reproducible independent of the map being extracted.
• invader-info: Now outputs some details in color on some terminals

Fixed

• invader-build: Fixed an issue with some old device_light_fixture tags being treated as device_control tags. This generally did not result in gameplay issues, but it may have resulted in a prompt to "activate" it when the player was close to the object.
• invader-build: Fixed converting radians to degrees with the lens_flare rotation scale
• invader-extract: Fixed converting degrees to radians with the lens_flare rotation scale

 

As always, the latest build can be retrieved from https://invader.opencarnage.net/builds/nightly/download-latest.html

[Kavawuvi 7,026]

Another month. Another update. Here is January's update, and it's a big one, notable invader-strip being added as well as much more diagnostic messages from invader-build. In fact, invader-build has been mostly rewritten, and it can now catch many crash-prone errors tool.exe could never catch, plus it's a lot simpler to build stock maps now, since you no longer need to index/CRC32 those manually.

 

 

Also, Invader is now 0.24.2, and here are the detailed changes since the last update:

 

Added

• New tool: invader-strip - Strips extra hidden data from tags.
• invader-build: Now shows tag space usage on map build and errors if exceeded
• invader-build: Added a CRC32 check on tag load. This will drastically increase build time, but it will still be much faster than tool.exe. You can turn this off by turning off pedantic warnings (-H) or reset the CRC32 with invader-strip.
• invader-info: Added -T tags-external-bitmap-indices which lists all bitmap tags in a Custom Edition map that use an external index
• invader-info: Added -T tags-external-indices which lists all tags in a Custom Edition map that use an external index
• invader-info: Added -T tags-external-loc-indices which lists all tags in a Custom Edition map that aren't sounds or bitmaps but use an external index
• invader-info: Added -T tags-external-sound-indices which lists all sound tags in a Custom Edition map that use an external index
• invader-build: Minor warnings are now considered "pedantic" and can be hidden with -H. You should probably still fix these warnings, as you may be getting an undesirable result.
• invader-build: Added warnings for non-power-of-two bitmaps being incorrectly set as power-of-two and vice versa.
• invader-build: Stock multiplayer maps for demo, retail, and Custom Edition are now automatically indexed to prevent crashing when joining with modified maps.
• invader-build: Stock multiplayer maps for Custom Edition now have their CRC32 automatically forged to whatever the stock CRC32 is to allow users to join other servers with modified maps.
• invader: Added color support for Windows
• invader-build: Added -O which can reduce tag space usage
• invader-build: Added numerous warning and error checks which can be used to spot and report problems with maps
• invader-build: Added more helpful text output

Changed

• invader: Node count is now capped to a maximum of 255
• invader-build: Adds markers in the same order as tool.exe
• invader-build: Node indices are now checked
• invader-extract: Now recalculates detail node count on extraction
• invader-extract: Extracts markers in the reverse order that tool.exe adds them
• invader-extract: Removes the blend_shared_normals flag on extraction to prevent generational loss (since tool.exe may change the normals again!)
• invader-dependency: -r and -R no longer show tag errors
• invader-extract: -r no longer shows tag errors
• invader-build: Detail node counts for gbxmodels are now calculated.
• invader-sound: Permutations are now alphabetized. For split sounds, only the "actual" permutations are alphabetized.
• invader-sound: Warns if indexed damage effect or object tags are stubbed out
• invader-sound: Disabled the creation of split dialogue. This does not work well with the game, anyway.
• invader-sound: Older sound tag data is now copied over.
• invader-build: Specifying the target engine is now required when building a map.
• invader-build: If building a Custom Edition map with a stock scenario name, the following values are set:
      vehicles\rwarthog\rwarthog_gun autoaim angle: 6 degrees
      vehicles\rwarthog\rwarthog_gun deviation angle: 12 degrees
      vehicles\banshee\banshee bolt stun: 0
      vehicles\banshee\banshee bolt max stun: 0
      vehicles\banshee\banshee bolt stun time: 0
      vehicles\ghost\ghost bolt stun: 0
      vehicles\ghost\ghost bolt max stun: 0
      vehicles\ghost\ghost bolt stun time: 0
• invader-build: If building a retail map with a stock scenario name, the following values are set:
      vehicles\rwarthog\rwarthog_gun autoaim angle: 1 degree
      vehicles\rwarthog\rwarthog_gun deviation angle: 1 degree
      vehicles\banshee\banshee bolt stun: 1
      vehicles\banshee\banshee bolt max stun: 1
      vehicles\banshee\banshee bolt stun time: 0.15
      vehicles\ghost\ghost bolt stun: 1
      vehicles\ghost\ghost bolt max stun: 1
      vehicles\ghost\ghost bolt stun time: 0.15
• invader-build: If a value is changed automatically due to the map being a singleplayer or stock map, a warning is now displayed.
• invader-build: Improved the accuracy of building
• invader-bitmap: Now uses the new parser
• invader-font: Now uses the new parser
• invader-string: Now uses the new parser
• invader-string: Changed "utf-16" to "unicode"
• invader-extract: Tag IDs are now set to FFFFFFFF when extracted. This will make extracted tags more reproducible independent of the map being extracted.
• invader-extract: The maps folder the map is in is used if no maps folder is supplied

Fixed

• invader-build: Fixed ejection port recovery rate being calculated as 1 for some tags
• invader-build: Fixed "zoner" gbxmodel tags not having the correct detail node counts
• invader-extract: Fixed change color permutation weights being extracted incorrectly. You will need to re-extract your tags!!!
• invader-build: Fixed change color permutation weights not being converted to cutoffs.
• invader-build: Fixed a bug where negative-scale weapon HUD interface values were NOT defaulted to 1
• invader-build: Fixed various maps with decals crashing
• invader-build: Fixed some weapon HUD interface tags not compiling due to finding an error when there was none
• invader-build: Fixed some various division-by-zero issues
• invader-extract: Fixed some values being extracted when unnecessary
• invader-extract: Fixed an issue where certain singleplayer tags' values were not restored to the original HEK values such as pistol.weapon's error angle
• invader-extract: Fixed not clearing do_not_cull flag
• invader-build: Fixed defaulting height/width scales for interface numbers in weapon HUD interfaces to 1
• invader-build: Fixed decals not being put in the correct cluster on some maps
• invader-build: Fixed camera shake wobble period not being converted to ticks
• invader-build: Fixed resource bitmap tags with different bitmap data count matching when they shouldn't when building for Halo Custom Edition
• invader-extract: Fixed camera shake wobble period not being converted from ticks
• invader-resource: Fixed a segmentation fault when building resource maps
• invader-build: Warning for non-power-of-two was fixed (it warned only if it was an interface bitmap, not only if it was NOT an interface bitmap)
• invader-sound: Fixed std::terminate being called on an invalid WAV being used.
• invader-sound: Fixed not loading .wav files with fmt subchunks that are larger than standard.
• invader-extract: Fixed recursive extraction causing segmentation faults with some tags
• invader-build: Fixed an out-of-bounds issue resulting in undefined behavior
• invader-build: Improved the accuracy of encounter firing position location
• invader-build: Fixed various light tags not working correctly
• invader-build: Fixed sound_looping gain not being defaulted to 1
• invader: Fixed an easy-to-reproduce segmentation fault crash on Windows.

As always, the latest build can be retrieved from https://invader.opencarnage.net/builds/nightly/download-latest.html 

[Kavawuvi 7,026]

Here's a mini-update!

 

Let's start with the new tool: invader-refactor. This took allows you to rename tags and re-reference the tags without having to build a cache file and re-extract. We feel that tag extraction should only make sense if you don't have access to the original tags. This tool is also quite fast, taking only 21-22 seconds on my PC to move the entire Gearbox tag set's "levels\test" directory to a more logically-named "levels\multiplayer" directory, and the maps still build, too!

 

Also, I am working on a new GUI-based tag editor. This tag editor will be based on the Qt framework and is made to take advantage of Invader's multiple-tag directory support while also providing fast, native support for Linux - a first for a GUI HEK tag editor.

 

 

As always, a full update will be posted on the start of the next month, but you can get invader-refactor right now either by compiling Invader from source or by using the Windows nightly builds. I recommend compiling from source, personally, as you may get a slightly more up-to-date version than what is available on the nightly builds, but I imagine many people don't care to do that. So, it's merely an option.

[Kavawuvi 7,026]

I made a web page for Invader, hosting courtesy of @Tucker933.

 

Here's the page: https://invader.opencarnage.net/

 

Also, the nightly builds page got a style update to match it: https://invader.opencarnage.net/builds/nightly/download-latest.html