Conservancy - automated CE to MCC converting

[Zatarita 407]

Conservancy is a bat file I threw together to streamline the steps required to port a classic map to anniversary using invader and the anniversary HEK.

This requires invader:
https://invader.opencarnage.net/builds/nightly/download-latest.html

Usage is simple.

First extract invader and Conservancy into your Chelan_1 folder.
Then you place all the maps you want converted into the "classic maps" folder


The bat file will process each of the entries inside of the classic maps folder and use invader-extract to extract the data from the map. After each map has been extracted, it will run invader-bludgeon, and invader-strip on all the tags.

Once this is done, it will extract the scenario name from the map using invader-info, and rebuild the cache file using tool. Once all of this has been finished, all the successful conversions will be built into your maps folder. 

 

This is practically the same process as the tutorial I made in the tutorials subforum, just automated with a bat file. Do note it is slightly inefficient bludgeoning, and stripping EVERY tag every time; however, for sake of automation the process has been simplified; that is a sacrifice needed to be made for ease of use.

Do note if there were any issues with extraction/stripping/bludgeoning a manual approach may be required, to which I recommend following the tutorial here:

Please post any issues. I have tested on multiple maps, and it seems to run as expected c:
Happy modding!

 

Conservancy.zip

[Takka 13,376]

Most excellent. =D

[Sunstriker7 698]

When @Kavawuvi first began working on invader a bunch of noobs complained that it was only a bunch of CLI tools with no gui wrapper.

 

Well, this is an excellent demonstration of why Kava did it that way and I'm glad so see that decision so roundly vindicated.

 

Sick process though bro.

[Zatarita 407]

59 minutes ago, Sunstriker7 said:

When @Kavawuvi first began working on invader a bunch of noobs complained that it was only a bunch of CLI tools with no gui wrapper.

 

Well, this is an excellent demonstration of why Kava did it that way and I'm glad so see that decision so roundly vindicated.

 

Sick process though bro.

Yep c:

I think it comes down to how each generation views computers.

I was born on the cusp of DOS, but as I was getting my experience in my field I also used Linux. I think a lot of people only know graphic interferace. They don't realize how powerful scripting can be. In fact modern windows is shite compared to Linux in that matter. Since windows dominates, a lot of people never learned how to leverage scripting.

 

Not to mention, if you make a strong backend, and it's useful enough, people will create a front end. Like tool+ or osoyoos

[Kavawuvi 7,026]

Goodness, this is really cool! Seeing people writing scripts that use my tools is an amazing feeling, so thank you for that!

 

There are a couple things I'd recommend doing for some of these maps to take advantage of the shiny new fixes released in MCC:

• Refactor .shader_transparent_chicago_extended and .shader_transparent_chicago to .shader_transparent_generic. Unlike Halo PC as released by Gearbox, the renderer in MCC supports all of the original graphical effects from the original Xbox version, such as the original teleporter and jackal shield shaders. Therefore, any maps that use the stock Gearbox shaders that could use the original stock Halo shaders should use them. This will drastically improve the graphics in a number of areas. Also, the jackal shields will look incorrect if you use the Gearbox shaders since Gearbox changed this bitmap to work with their own workaround (to support fixed function GPUs in 2003), where 343 Industries changed the bitmap tag to be the original, reverting those workarounds.
• Refactor .shader_transparent_chicago_extended to .shader_transparent_chicago. Gearbox converted these tags in a lossy way that results in inaccuracies when baking lightmaps, so if the stock chicago shaders are available, they should be used. This is not a huge deal for porting aside from slightly smaller cache file sizes (so, the visuals won't change), but it is best to have the original data in the maps whenever possible just in case.

As for porting, itself, there are a few gotcha's that might be important to keep in mind in case you encounter these issues:

• Some tags may need changed, since 343 Industries changed multipurposes and HUDs to use the original channel order. So, if a custom shader_model tag references a stock multipurpose bitmap or a custom HUD interface references a stock meter bitmap, you will need to set the respective flag in those tags. I think Coldsnap does this for the warthogs?
• It's also worth noting that there are also a few Refined-specific workarounds to watch out for that can't be automated. I mention these because some maps use the Custom Edition version of the Refined tagset which includes workarounds that are either unnecessary, worse quality, or they break entirely. In most cases, provided you aren't overwriting existing stock MCC CEA tags, these last points won't cause any issue (besides having these workarounds loose in your tags folder), but just in case, these are something to keep in mind:
  • Refined uses a water shader for some of the glass textures since glass is completely broken on Halo Custom Edition. This will look wrong, especially in oblique angles, but it was the closest thing to approximate the glass shaders on Custom Edition. Glass is fixed on MCC CEA however.
  • Refined and other maps using Jesse's HUD have workarounds for the heat meter that involve manually coloring the meter. Since MCC CEA repurposes the colors for something else (and it does the heat meters correctly now!), they will look wrong.
  • Refined and other maps based on Refined's tagset uses large animated textures (I call them "GIF" shaders) to replicate some Xbox effects. These are woefully inaccurate in comparison to the real thing and break the D3D9 spec due to being extremely high resolution (e.g. 16K to account for all of the frames), and they bloat map size very significantly.
• tool.exe recompiles the scripts from the data folder, thus if your map has any scripts and you did not invader-recover them, your map won't have scripts. This behavior is different from the original tools. Most maps don't use scripts, but the ones that do may not work correctly anymore. There are also a few changes to scripting you may want to watch out for:
  • A few scripting commands originally added in Halo Custom Edition were reimplemented in MCC as stubbed-out commands. This was done so the script will run, but a few Easter eggs won't work since these commands do nothing on MCC (though it's better than crashing!).
  • The netcode is a bit different. Hog syncing scripts won't work. Fortunately, 343 Industries added a built-in host check, though you'd need to edit the scripts yourself.
  • Obviously the original menus are no longer accessible, so things like changing the time of day on Coldsnap won't be possible.

For many maps (maps that use a stock tagset or use custom tags that don't reference stock bitmaps) the above list won't be an issue, but it might be worth mentioning in a guide somewhere... some day.

 

Overall, this is a really handy tool you've made. Excellent work!

 

7 hours ago, Sunstriker7 said:

When @Kavawuvi first began working on invader a bunch of noobs complained that it was only a bunch of CLI tools with no gui wrapper.

 

Well, this is an excellent demonstration of why Kava did it that way and I'm glad so see that decision so roundly vindicated.

 

Sick process though bro.

Aah thanks!! I actually made my own GUI frontend for Invader called "Six Shooter".

 

I still use the command line tools directly, as I find typing to be more efficient than clicking in a lot of cases, but some people work better with GUIs I guess!

[Zatarita 407]

On 10/18/2021 at 1:21 PM, Kavawuvi said:

There are a couple things I'd recommend doing for some of these maps to take advantage of the shiny new fixes released in MCC:

• Refactor .shader_transparent_chicago_extended and .shader_transparent_chicago to .shader_transparent_generic. Unlike Halo PC as released by Gearbox, the renderer in MCC supports all of the original graphical effects from the original Xbox version, such as the original teleporter and jackal shield shaders. Therefore, any maps that use the stock Gearbox shaders that could use the original stock Halo shaders should use them. This will drastically improve the graphics in a number of areas. Also, the jackal shields will look incorrect if you use the Gearbox shaders since Gearbox changed this bitmap to work with their own workaround (to support fixed function GPUs in 2003), where 343 Industries changed the bitmap tag to be the original, reverting those workarounds.
• Refactor .shader_transparent_chicago_extended to .shader_transparent_chicago. Gearbox converted these tags in a lossy way that results in inaccuracies when baking lightmaps, so if the stock chicago shaders are available, they should be used. This is not a huge deal for porting aside from slightly smaller cache file sizes (so, the visuals won't change), but it is best to have the original data in the maps whenever possible just in case.

Never thought of that! That's an easy fix too. I can definetly add that.

 

On 10/18/2021 at 1:21 PM, Kavawuvi said:

As for porting, itself, there are a few gotcha's that might be important to keep in mind in case you encounter these issues:

• Some tags may need changed, since 343 Industries changed multipurposes and HUDs to use the original channel order. So, if a custom shader_model tag references a stock multipurpose bitmap or a custom HUD interface references a stock meter bitmap, you will need to set the respective flag in those tags. I think Coldsnap does this for the warthogs?
• It's also worth noting that there are also a few Refined-specific workarounds to watch out for that can't be automated. I mention these because some maps use the Custom Edition version of the Refined tagset which includes workarounds that are either unnecessary, worse quality, or they break entirely. In most cases, provided you aren't overwriting existing stock MCC CEA tags, these last points won't cause any issue (besides having these workarounds loose in your tags folder), but just in case, these are something to keep in mind:
  • Refined uses a water shader for some of the glass textures since glass is completely broken on Halo Custom Edition. This will look wrong, especially in oblique angles, but it was the closest thing to approximate the glass shaders on Custom Edition. Glass is fixed on MCC CEA however.
  • Refined and other maps using Jesse's HUD have workarounds for the heat meter that involve manually coloring the meter. Since MCC CEA repurposes the colors for something else (and it does the heat meters correctly now!), they will look wrong.
  • Refined and other maps based on Refined's tagset uses large animated textures (I call them "GIF" shaders) to replicate some Xbox effects. These are woefully inaccurate in comparison to the real thing and break the D3D9 spec due to being extremely high resolution (e.g. 16K to account for all of the frames), and they bloat map size very significantly.

Sadly specialty stuff like this will likely need to be addressed manually. I will bring that up in the manual conversion tutorial though.

 

Also isn't the new shader settings for the multipurpose default off? So classic tags that use the old system should default to the CE style. Xbox requires manual setting

 

Edit: I understand what you mean now.

I keep messing things up with my edits x.x gotta get used to this moderator thing. Sorry lol.

The bitmaps will be mismatched. That might require a more nuanced approach, but I'll have to run my brain on it for a bit

 

On 10/18/2021 at 1:21 PM, Kavawuvi said:

• tool.exe recompiles the scripts from the data folder, thus if your map has any scripts and you did not invader-recover them, your map won't have scripts. This behavior is different from the original tools. Most maps don't use scripts, but the ones that do may not work correctly anymore. There are also a few changes to scripting you may want to watch out for:

This is a good point too, I didn't know invader could extract scripts. That's an easy fix too!

 

As far as the forbidden script commands that might be a bit more difficult. That could go into the manual conversion tutorial as well.

 

I'll update the changes when I have a free night