Posted September 26, 2019 Quote Extra options that do not preserve themselves in the tag data leads to an issue where bitmaps could end up being changed inadvertently, for example if you used a script to run through the data directory and rebuild every bitmap it finds. Honestly? Tough shit imo. Share this post Link to post Share on other sites
Posted September 28, 2019 On 9/26/2019 at 6:55 PM, gbMichelle said: If you used a better compression scheme that looked better that would also not save in the tag. If you can recreate what you made with your tools using vanilla tool, what is the point of your tool? To be clear, compatibility with legacy is good, but I don't think we should be limiting ourselves to that all the time. The whole draw for this project for me is to be able to do stuff that I've been manually purposefully avoiding tool to accomplish, but automated. Edit: Even if it means adding an extra switch to my command. To answer your first question, here is what invader-bitmap does that tool.exe doesn't do: invader-bitmap can use png, bmp, and tga files in addition to tif/tiff invader-bitmap can let you create non-2D texture bitmaps as well as bitmaps with any available data format without you ever having to leave the command line interface invader-bitmap can let you overwrite various other bitmap settings like bump height and mipmap count without you ever having to leave the command line interface invader-bitmap can let you specify any data folder and any tags folder rather than using the current directory invader-bitmap can create bitmaps that exceed 16 MiB As for your other part of the comment, every single parameter used to generate a bitmap tag is stored in the bitmap tag. By doing this, you don't need to re-enter every single parameter every time you re-generate the bitmap tag. This is how tags fundamentally work, and deviating from this puts us in a situation where some settings get saved and some settings don't, negating the point of using tags in the first place. The main goal is that anything made with tool.exe that is re-made in invader-bitmap should function identically. If it doesn't, I consider this a bug that needs fixed. However, the opposite doesn't need to be true: something made with Invader does not necessarily have to work with tool.exe. For example, tool.exe will not process bitmaps that exceed 16 MiB. A 16 MiB limitation more-or-less limits you to these resolutions: With mipmaps: 4096x4096 - DXT1 4096x2048 - DXT3/DXT5 2048x2048 - 16-bit 2048x1024 - 32-bit Without mipmaps: 8192x4096 - DXT1 4096x4096 - DXT3/DXT5 4096x2048 - 16-bit 2048x2048 - 32-bit However, invader-bitmap doesn't have such arbitrary limits. You can make 4096x4096 32-bit bitmaps, and Invader won't have any issue with them provided they are otherwise valid. Vaporeon, DSalimander and Sunstriker7 like this Share this post Link to post Share on other sites
Posted October 5, 2019 Invader is now 0.13.0. It just so happens that this spooky version number happened on a spooky month. Here are the changes since 0.12.0: Changed invader-build: Changed how stubbed tags are created so tag extractors won't try to extract them invader-bitmap: Errors if the tag contains uppercase characters in its path invader-build: Errors if any non-stubbed tags contain uppercase characters in their paths invader-font: Errors if the tag contains uppercase characters in its path invader-string: Errors if the tag contains uppercase characters in its path Fixed invader-build: Fixed not fixing the render bounding radius if it was less than the bounding radius but non-zero. invader-build: Fixed not setting the weight value for color change permutations in objects. invader-bitmap: Fixed detail fade factor so it matches tool.exe's detail fade factor more closely if not exactly. invader-build: Fixed certain sound permutation file offsets not being correctly marked as internal; this should fix some sounds that sounded fine when built with tool.exe but sounded corrupt when built with invader-build Removed invader-bitmap: Removed --filter-blur and --filter-sharpen. Tags that have these values set will still have the filter(s) applied. However, for newer tags, you should use an image editor, as you will get similar or better results. invader-bitmap: Removed --sprite-spacing and used tool.exe's broken functionality, instead. invader-bitmap: Removed the ability to set sprite budgets below 32 or above 512. View the full changelog at https://github.com/Kavawuvi/Invader/blob/master/CHANGELOG.md From what I've seen, people prefer these less frequent but longer update announcements, so I'll keep doing them like this since it's less work for me. The changelog and GitHub repo, however, will be updated as changes are made. So, if you need to read about the more recent changes, then be sure to check that from time to time. As always, you can get the latest build at https://invader.opencarnage.net/builds/nightly/download-latest.html. These are automatically compiled and uploaded via a script on a local machine. These snapshots are built from the latest commit (at the time of building) on the Git repository which may include unreleased features, so these announcements can and probably will get outdated fairly quickly. See you next month! Takka, ST34MF0X, Sunstriker7 and 1 other like this Share this post Link to post Share on other sites
Posted November 1, 2019 Here's a monthly update for the progress made in October. I hope everyone had a good Halloween last night. Invader is now 0.15.2, and there have been a number of changes, including the new invader-compress program. View this topic for more information. Added invader-archive: Now accepts filesystem paths for the tag if -P is passed invader-bitmap: Now accepts filesystem paths for the image input if -P is passed invader-build: Now accepts filesystem paths for the scenario tag if -P is passed invader-dependency: Now accepts filesystem paths for the tag if -P is passed invader-font: Now accepts filesystem paths for the the TTF file if -P is passed invader-string: Now accepts filesystem paths for the text file if -P is passed invader-crc: Now prints a warning to stderr if the CRC32 in the cache file header is wrong invader-crc: Now errors with a useful error message if a resource map is loaded with it invader-build: Halo Demo / Trial maps can now be built (-g demo) invader-crc: Halo Demo / Trial maps can now be used invader-build: --no-indexed-tags was renamed to --no-external-tags invader-build: Building retail and demo maps now uses the resource maps invader-build: The gain modifier for sound\sfx\impulse\ting\ting.sound is now changed to 0.2 if building for retail or demo and 1.0 otherwise invader-build: Now shows the target engine of the map invader-compress - Compresses cache files using the Zstandard algorithm invader-resource: Added --retail / -R which will allow you to build retail maps invader-build: Copying TagString data now checks for string overflow invader-crc: Works with maps compressed with invader-compress invader-indexer: Works with maps compressed with invader-compress Changed invader-build: The diagnostic messages now show tags using external raw data when building a retail or demo map invader-dependency: Now uses -R instead of -r for reverse dependencies invader-dependency: Now uses -r instead of -R for recursive dependencies invader-build: Bitmaps and sound data are now shown separately next to their combined total in the diagnostic output invader-build: The number of tags that could be cached/indexed is now shown invader-archive: Now handles exceptions instead of calling abort() while either compiling or parsing cache files invader-build: No longer errors when orphaned model nodes exist invader-bitmap: Made detail fade factor closer to tool.exe's output - still needs more work but it's remarkably close Fixed invader-build: Fixed decals with null references crashing the game invader-bitmap: Fully implemented and fixed 3D textures invader-dependency: Fixed an issue where it didn't include a null terminator, causing some tags to fail to open invader-archive: Fixed an issue where some stock HEK .gbxmodel tags tried (and failed) to be archived as .model tags invader-bitmap: Fixed an issue where sprites spanning the entire width wouldn't be detected invader-archive: Fixed an issue where the root tag would have its full system path included when using -s As always, you can get the latest build at https://invader.opencarnage.net/builds/nightly/download-latest.html or view the changelog at https://github.com/Kavawuvi/invader/blob/master/CHANGELOG.md. If anyone has any comments, suggestions, or bug reports, feel free to reply here. Otherwise, see you next month! Stay warm. Sunstriker7, (SBB) Storm and ST34MF0X like this Share this post Link to post Share on other sites
Posted November 17, 2019 A mini-update: I converted the tag definitions to the more parsable .json format. This won't significantly impact Invader's performance since these definitions will be transpiled back into C++ headers. However, it will allow me to tag parsers. As a test, I converted them into Eschaton plugins. I use Eschaton for viewing tag data and finding offsets (which for building cache files is VERY useful), and I've found Sparky's plugins to be fairly inaccurate in some cases. Since my definitions are basically used to build cache files, I figured these would actually be more complete and more useful for my purposes. Here's a screenshot of Eschaton with them: As usual, I will post an update to Invader next month. To give you an idea of the size of the update, this month is only half over and there are already a number of changes already logged in the changelog. These changes over 80 commits with about 40 more on the way in a separate branch, and there have been several version bumps. Sunstriker7, Vuthakral and Takka like this Share this post Link to post Share on other sites
Posted December 1, 2019 Here's a update on the month of November! Invader is now 0.20.2. Not much has changed, besides adding a new tag extractor and a new meta data viewer as well as fixing a lot of issues, of course. Added New tool: invader-info - Displays meta data of a map file but can also show specific data (-T): build, compressed, compression-ratio, crc32, crc32-mismatched, dirty, engine, map-type-scenario-scenario-path, stub-count, tag-count, tags New tool: invader-extract - Extracts tags from cache files Added a new tag parser invader-build: Added -N to rename scenarios when building invader-build: Added -c for compress (forging CRC now uses -C) invader-build: Strips default data from compressed animations now invader-font: Added support for .otf files Changed Help menus have been rewritten All tag definitions have been converted to .json format invader-bitmap: Changed -m to -M to avoid confusion with --maps invader-bitmap: The default bump height is now 0.026 invader-build: Shows map type when building invader-build: Uses new indexer format (previous index files will not work) invader-build: 32 byte strings are now zeroed out on copy invader-build: The stub count is now shown invader-build: Index parsing is now case insensitive invader-compress: Strings are now zeroed out before copy invader-compress: The default compresson level is now 19 invader-indexer: Changed index format to simply list tag paths with extensions Fixed invader-bitmap: Fixed -P not working with non-.tif files invader-build: Fixed indexed sound data taking up more space than needed in the tag file; this will reduce tag space usage a little more invader-build: Fixed -g not erroring if an invalid engine was given invader-build: Fixed sound looping tags' gain being set to 0 invader-build: Fixed some values in shader_transparent_plasma being set to 0 invader-build: Fixed biped A In, B In, C In, D In being set to 0 invader-build: Fixed some objects' flags being set to 0 invader-build: Fixed an undefined behavior issue with firing positions invader-build: Fixed light tag durations not being properly multiplied by 30 invader-build: Fixed some sound tags not working as intended invader-build: Fixed lens flare rotation scale not being converted properly invader-build: Fixed setting mouth data and subtitle data to 0 size invader-build: Fixed incorrect footstep sounds being used if not using an MEK-extracted BSP tag Removed Removed invader-crc. invader-info now fulfills the same exact purpose invader-compress: Levels above 19 were removed; they are pointless and needlessly use extra memory Another feature added post-0.20.2 was colored diagnostic text output. I'm still figuring out how I'll do this for Windows. Basically, here are some screenshots: Lastly, I posted this earlier today on the Halo CE Reclaimers Discord, but basically, I'm considering removing dithering and nearest-neighbor mipmapping from invader-bitmap. Here's the whole spiel from the Discord I gave: [12:18 PM] Kavawuvi: So, next version of Invader, I'm considering removing dithering as well as nearest-neighbor mipmapping from invader-bitmap. Any thoughts? [12:20 PM] Kavawuvi: The problem is that the original point of having these bitmap tags having all of this data was so that it provided parameters for the bitmap tool (tool.exe) to generate the bitmap data. [12:21 PM] Kavawuvi: Previously, in order to satisfy this, I added more flags to specify how to generate bitmap data, and this included dithering and mipmap settings. [12:22 PM] Kavawuvi: The problem with doing this is that there is no way to know if a tag has these flags or not, since tags lack a versioning system once they're in the map file. For all I know, MCC uses the data where these flags are located for other purposes. [12:25 PM] Kavawuvi: Considering hardly anyone actually uses Invader let alone invader-bitmap, these features probably won't be missed anyway, so having them in here is more a liability than an asset in my opinion. Anyway, as always, you can download Windows builds from the Nightly Builds page. https://invader.opencarnage.net/builds/nightly/download-latest.html Stay tuned until next month for the next update! Takka, Sunstriker7 and ST34MF0X like this Share this post Link to post Share on other sites
Posted December 27, 2019 Here's an update for the month of December! Normally I post these at the start of the next month, but I felt like not doing that this one time because it's my project and I can (mostly) do whatever I want with it. This update is a bit smaller than last update, but it's also just as important as it introduces a new tool: invader-sound. As the name suggests, this tool allows you to generate sound tags. You can either opt for 16-bit PCM (you will need either the latest builds of HAC2 or Chimera to enable these in Halo!), or you can use lossy compression through Ogg Vorbis. Xbox ADPCM is supported, too, and you do NOT need to pirate obtain the original codec to use Xbox ADPCM with invader-sound, but note that ADPCM is worse, compression ratio and quality wise, than Vorbis. If you're unsure as to what to use, check out the readme. Added New tool: invader-sound Generates sound tags Added colors for various diagnostic messages depending on the user's terminal invader-info: Added `-T external-tags` which indicates the number of tags that use external resource maps, including indexed tags invader-info: Added `-T external-bitmaps` which indicates the number of bitmap tags that use external resource maps, including indexed tags invader-info: Added `-T external-loc` which indicates the number of loc tags that use external resource maps, including indexed tags invader-info: Added `-T external-sounds` which indicates the number of sound tags that use external resource maps, including indexed tags invader-info: Added `-T external-bitmap-indices` which indicates the number of externally indexed bitmap tags invader-info: Added `-T external-loc-indices` which indicates the number of externally indexed loc tags invader-info: Added `-T external-sound-indices` which indicates the number of externally indexed sound tags invader-info: Added `-T languages` which lists all languages valid for the map separated with spaces invader-info: Added `-T external-pointers` which indicates whether the map uses external pointers (sometimes occurs if built from tool.exe due to a bug with indexing raw data) invader-info: Added `-T tags-external-pointers` which list tags that have external pointers that were not indexed Changed invader-extract: Tag IDs are now set to FFFFFFFF when extracted. This will make extracted tags more reproducible independent of the map being extracted. invader-info: Now outputs some details in color on some terminals Fixed invader-build: Fixed an issue with some old device_light_fixture tags being treated as device_control tags. This generally did not result in gameplay issues, but it may have resulted in a prompt to "activate" it when the player was close to the object. invader-build: Fixed converting radians to degrees with the lens_flare rotation scale invader-extract: Fixed converting degrees to radians with the lens_flare rotation scale As always, the latest build can be retrieved from https://invader.opencarnage.net/builds/nightly/download-latest.html (SBB) Storm, Sunstriker7, ST34MF0X and 1 other like this Share this post Link to post Share on other sites
Posted February 2, 2020 Another month. Another update. Here is January's update, and it's a big one, notable invader-strip being added as well as much more diagnostic messages from invader-build. In fact, invader-build has been mostly rewritten, and it can now catch many crash-prone errors tool.exe could never catch, plus it's a lot simpler to build stock maps now, since you no longer need to index/CRC32 those manually. Also, Invader is now 0.24.2, and here are the detailed changes since the last update: Added New tool: invader-strip - Strips extra hidden data from tags. invader-build: Now shows tag space usage on map build and errors if exceeded invader-build: Added a CRC32 check on tag load. This will drastically increase build time, but it will still be much faster than tool.exe. You can turn this off by turning off pedantic warnings (-H) or reset the CRC32 with invader-strip. invader-info: Added -T tags-external-bitmap-indices which lists all bitmap tags in a Custom Edition map that use an external index invader-info: Added -T tags-external-indices which lists all tags in a Custom Edition map that use an external index invader-info: Added -T tags-external-loc-indices which lists all tags in a Custom Edition map that aren't sounds or bitmaps but use an external index invader-info: Added -T tags-external-sound-indices which lists all sound tags in a Custom Edition map that use an external index invader-build: Minor warnings are now considered "pedantic" and can be hidden with -H. You should probably still fix these warnings, as you may be getting an undesirable result. invader-build: Added warnings for non-power-of-two bitmaps being incorrectly set as power-of-two and vice versa. invader-build: Stock multiplayer maps for demo, retail, and Custom Edition are now automatically indexed to prevent crashing when joining with modified maps. invader-build: Stock multiplayer maps for Custom Edition now have their CRC32 automatically forged to whatever the stock CRC32 is to allow users to join other servers with modified maps. invader: Added color support for Windows invader-build: Added -O which can reduce tag space usage invader-build: Added numerous warning and error checks which can be used to spot and report problems with maps invader-build: Added more helpful text output Changed invader: Node count is now capped to a maximum of 255 invader-build: Adds markers in the same order as tool.exe invader-build: Node indices are now checked invader-extract: Now recalculates detail node count on extraction invader-extract: Extracts markers in the reverse order that tool.exe adds them invader-extract: Removes the blend_shared_normals flag on extraction to prevent generational loss (since tool.exe may change the normals again!) invader-dependency: -r and -R no longer show tag errors invader-extract: -r no longer shows tag errors invader-build: Detail node counts for gbxmodels are now calculated. invader-sound: Permutations are now alphabetized. For split sounds, only the "actual" permutations are alphabetized. invader-sound: Warns if indexed damage effect or object tags are stubbed out invader-sound: Disabled the creation of split dialogue. This does not work well with the game, anyway. invader-sound: Older sound tag data is now copied over. invader-build: Specifying the target engine is now required when building a map. invader-build: If building a Custom Edition map with a stock scenario name, the following values are set: vehicles\rwarthog\rwarthog_gun autoaim angle: 6 degrees vehicles\rwarthog\rwarthog_gun deviation angle: 12 degrees vehicles\banshee\banshee bolt stun: 0 vehicles\banshee\banshee bolt max stun: 0 vehicles\banshee\banshee bolt stun time: 0 vehicles\ghost\ghost bolt stun: 0 vehicles\ghost\ghost bolt max stun: 0 vehicles\ghost\ghost bolt stun time: 0 invader-build: If building a retail map with a stock scenario name, the following values are set: vehicles\rwarthog\rwarthog_gun autoaim angle: 1 degree vehicles\rwarthog\rwarthog_gun deviation angle: 1 degree vehicles\banshee\banshee bolt stun: 1 vehicles\banshee\banshee bolt max stun: 1 vehicles\banshee\banshee bolt stun time: 0.15 vehicles\ghost\ghost bolt stun: 1 vehicles\ghost\ghost bolt max stun: 1 vehicles\ghost\ghost bolt stun time: 0.15 invader-build: If a value is changed automatically due to the map being a singleplayer or stock map, a warning is now displayed. invader-build: Improved the accuracy of building invader-bitmap: Now uses the new parser invader-font: Now uses the new parser invader-string: Now uses the new parser invader-string: Changed "utf-16" to "unicode" invader-extract: Tag IDs are now set to FFFFFFFF when extracted. This will make extracted tags more reproducible independent of the map being extracted. invader-extract: The maps folder the map is in is used if no maps folder is supplied Fixed invader-build: Fixed ejection port recovery rate being calculated as 1 for some tags invader-build: Fixed "zoner" gbxmodel tags not having the correct detail node counts invader-extract: Fixed change color permutation weights being extracted incorrectly. You will need to re-extract your tags!!! invader-build: Fixed change color permutation weights not being converted to cutoffs. invader-build: Fixed a bug where negative-scale weapon HUD interface values were NOT defaulted to 1 invader-build: Fixed various maps with decals crashing invader-build: Fixed some weapon HUD interface tags not compiling due to finding an error when there was none invader-build: Fixed some various division-by-zero issues invader-extract: Fixed some values being extracted when unnecessary invader-extract: Fixed an issue where certain singleplayer tags' values were not restored to the original HEK values such as pistol.weapon's error angle invader-extract: Fixed not clearing do_not_cull flag invader-build: Fixed defaulting height/width scales for interface numbers in weapon HUD interfaces to 1 invader-build: Fixed decals not being put in the correct cluster on some maps invader-build: Fixed camera shake wobble period not being converted to ticks invader-build: Fixed resource bitmap tags with different bitmap data count matching when they shouldn't when building for Halo Custom Edition invader-extract: Fixed camera shake wobble period not being converted from ticks invader-resource: Fixed a segmentation fault when building resource maps invader-build: Warning for non-power-of-two was fixed (it warned only if it was an interface bitmap, not only if it was NOT an interface bitmap) invader-sound: Fixed std::terminate being called on an invalid WAV being used. invader-sound: Fixed not loading .wav files with fmt subchunks that are larger than standard. invader-extract: Fixed recursive extraction causing segmentation faults with some tags invader-build: Fixed an out-of-bounds issue resulting in undefined behavior invader-build: Improved the accuracy of encounter firing position location invader-build: Fixed various light tags not working correctly invader-build: Fixed sound_looping gain not being defaulted to 1 invader: Fixed an easy-to-reproduce segmentation fault crash on Windows. As always, the latest build can be retrieved from https://invader.opencarnage.net/builds/nightly/download-latest.html ST34MF0X, (SBB) Storm, Sunstriker7 and 2 others like this Share this post Link to post Share on other sites
Posted February 12, 2020 Here's a mini-update! Let's start with the new tool: invader-refactor. This took allows you to rename tags and re-reference the tags without having to build a cache file and re-extract. We feel that tag extraction should only make sense if you don't have access to the original tags. This tool is also quite fast, taking only 21-22 seconds on my PC to move the entire Gearbox tag set's "levels\test" directory to a more logically-named "levels\multiplayer" directory, and the maps still build, too! Also, I am working on a new GUI-based tag editor. This tag editor will be based on the Qt framework and is made to take advantage of Invader's multiple-tag directory support while also providing fast, native support for Linux - a first for a GUI HEK tag editor. As always, a full update will be posted on the start of the next month, but you can get invader-refactor right now either by compiling Invader from source or by using the Windows nightly builds. I recommend compiling from source, personally, as you may get a slightly more up-to-date version than what is available on the nightly builds, but I imagine many people don't care to do that. So, it's merely an option. Takka, Vaporeon, Sunstriker7 and 2 others like this Share this post Link to post Share on other sites
Posted February 14, 2020 I made a web page for Invader, hosting courtesy of @Tucker933. Here's the page: https://invader.opencarnage.net/ Also, the nightly builds page got a style update to match it: https://invader.opencarnage.net/builds/nightly/download-latest.html Takka, ST34MF0X and Sunstriker7 like this Share this post Link to post Share on other sites