Halo map file structure (Revision 2.1.2)

[Kavawuvi 7,026]

I've written a list of offsets that I use for my own applications and scripts that may be useful for others. This is considerably more information than what was available in the Eschaton era and should be enough to write a basic parser. However, it does not include any tag data, so you will need to refer to Eschaton plugins for this information.

 

This includes information on both cache files and resource maps for both the retail version of the game and Custom Edition and a small amount of information for the Xbox version of the game. More information may be added in a future date.

 

Source code: https://gist.github.com/SnowyMouse/39168bddd597549038a35d78aee39513

Download (PDF): halo-map-structure-2.1.2.pdf

 

Older versions:

 

Spoiler

REVISION 1.X

 

Google Docs:  https://docs.google.com/document/d/1hRcf26PHoDmpRkJV1fQLt69tlsjYuvt1mqMQPHij7iA/edit#

Revision 1.3.1: Halo’s Map Files-1.3.1.pdf

Revision 1.3: Halo’s Map Files-1.3.pdf

Revision 1.2: Halo’s Map Files.pdf

 

[Takka 13,377]

Added to the shortlist. =]

[DeadHamster 954]

Quote

This is the “old” way that’s used by older apps including Eschaton before this stuff was actually properly mapped out. It involved magic and other magnificent, wonderful nonsense, but it’s pretty interesting.

FTFY <3

[Kavawuvi 7,026]

On 2/8/2017 at 7:35 PM, DeadHamster said:

FTFY  <3

I cast the Smokescreen Map Protector cantrip. Until 002 releases another deprotector, my map now has resistance against ripping attacks dealt by HaloMD users and Halo CE map makers.

 

But seriously, the old method of parsing maps that involved magic and centaurs was kind of silly. Now that all this stuff is out in the open for anyone to use, they have some basic building blocks for creating their new tools while being somewhat immune to some silly map protections that I pulled out of my ass when I was learning this stuff.

[DeadHamster 954]

13 hours ago, 002 said:

I cast the Smokescreen Map Protector cantrip. Until 002 releases another deprotector, my map now has resistance against ripping attacks dealt by HaloMD users and Halo CE map makers.

 

But seriously, the old method of parsing maps that involved magic and centaurs was kind of silly. Now that all this stuff is out in the open for anyone to use, they have some basic building blocks for creating their new tools while being somewhat immune to some silly map protections that I pulled out of my ass when I was learning this stuff.

 

Oh it was absurd, absolutely. But it was wonderful all the same. Learning Map Magic from Modzy and Alt is one of my fondest memories.

 

 

[Kavawuvi 7,026]

1 hour ago, DeadHamster said:

Oh it was absurd, absolutely. But it was wonderful all the same. Learning Map Magic from Modzy and Alt is one of my fondest memories.

I mainly learned map parsing from a dead Mac Gaming Mods topic.

 

http://macgamingmods.com/forum/viewtopic.php?f=55&t=8171

 

I then compared the headers of the demo version to the full version of Halo and explored some of the other offsets that was here. It was fun learning how to do these things, but then I got into seeing how much I can screw with the map yet still have it work. As a result, my map protector is the most powerful map protector ever made and cannot be fully broken with even Deathstar (consequently this is why I have not released it on Open Carnage). This is how I learned that the scenario tag does NOT have to be the first tag, nor does it have to have scnr in the tag class.

 

None of any of my tools made in the past few years will use a "magic" but rather directly access the tag data block and subtract 0x40440000 from addresses to get an offset. In essence, the 0x40440000 constant would be what is considered the "magic" rather than some number calculated out of the program's butthole.

[DeadHamster 954]

On 2/9/2017 at 2:20 PM, 002 said:

None of any of my tools made in the past few years will use a "magic" but rather directly access the tag data block and subtract 0x40440000 from addresses to get an offset. In essence, the 0x40440000 constant would be what is considered the "magic" rather than some number calculated out of the program's butthole.

 

 

That is actually what a good number of programs did, so you are actually using map magic. The only difference is you don't call it map magic

 

I prefered to calculate it by going to index offset, read the....well I know it as map magic. I read the map magic from there, subtract 40 for the header and the indexoffset;

 

    br.position=16
    dim indexoffset as int32
    indexoffset=br.readint32
    
    br.position=indexoffset
    
    Dim MapMagic As Int32 = br.ReadInt32 - 40 - IndexOffset

 

But many programs and sources I've read had a variable called map magic that they "hardcoded" to 0x40440000, just as you would. 

 

Dim MapMagic as Int32=0x40440000

 

I always felt, at the time, that was an improper shortcut that could lead to problems. But, as you point out, if somebody were to play with that value it'd throw off any software that calculates it like I do above, hence "protecting" the mapfile. 

 

Everything old is new again, as they say.

[Kavawuvi 7,026]

4 hours ago, DeadHamster said:

That is actually what a good number of programs did, so you are actually using map magic. The only difference is you don't call it map magic

 

I prefered to calculate it by going to index offset, read the....well I know it as map magic. I read the map magic from there, subtract 40 for the header and the indexoffset;

 

But many programs and sources I've read had a variable called map magic that they "hardcoded" to 0x40440000, just as you would. 

 

I always felt, at the time, that was an improper shortcut that could lead to problems. But, as you point out, if somebody were to play with that value it'd throw off any software that calculates it like I do above, hence "protecting" the mapfile. 

 

Everything old is new again, as they say.

The "magic" you're talking about is merely the address to where the tag data is located in Halo's memory. I hardcode it because there is no reliable way to calculate it, and you shouldn't "calculate" it anyway if it's never going to change unless you use a different engine (i.e. Xbox). Which if you use a different engine, you're probably going to parse the map differently, anyway.

 

The point of the "magic" is to convert an address to an offset. Simply subtracting 0x40440000 from the address (the address to the beginning of the tag data block) then adding the offset of the tag data block on the file will give you the offset to where the pointer points to in the file. The address 0x40440000 is not map-specific but engine specific.

 

As for the value 0x40440028 which is at the beginning of what you call an "index" that you subtract 40 from, that's just a pointer to the tag array. It's read as any other pointer, similar to a reflexive. Sure you could try reading it and subtract 40, but you'll get 0x40440000 anyway, and if you don't, then that means the tag array has been moved, and your program will not be able to read the map if you use whatever else you got.

 

Lastly, the value offset 0x4 from the header of the tag data block is the scenario tag ID. While on most maps, the scenario tag ID will be the first tag, Halo's engine doesn't require it. Simply modifying this will change which tag is the scenario tag.

 

Most of the old stuff does still apply, though I typically treat the tag data block separately from the map data rather than converting anything directly to a file offset.

[Kavawuvi 7,026]

I've updated the PDF with revision 2.0.

 

This is a complete rewrite of it, and it no longer uses Google Docs, instead using LaTeX. You can view information about that here: https://en.wikipedia.org/wiki/LaTeX

 

This rewrite contains more information as well as existing information being updated. Here is a sample page:

 

 

Here's what it looked like before:

 

I will leave the older versions up for now, but I do not recommend anyone use them as they contain old or outdated information.

[Zatarita 407]

On 2/9/2017 at 10:04 PM, Kavawuvi said:

The "magic" you're talking about is merely the address to where the tag data is located in Halo's memory. I hardcode it because there is no reliable way to calculate it, and you shouldn't "calculate" it anyway if it's never going to change unless you use a different engine (i.e. Xbox). Which if you use a different engine, you're probably going to parse the map differently, anyway.

 

The point of the "magic" is to convert an address to an offset. Simply subtracting 0x40440000 from the address (the address to the beginning of the tag data block) then adding the offset of the tag data block on the file will give you the offset to where the pointer points to in the file. The address 0x40440000 is not map-specific but engine specific.

 

As for the value 0x40440028 which is at the beginning of what you call an "index" that you subtract 40 from, that's just a pointer to the tag array. It's read as any other pointer, similar to a reflexive. Sure you could try reading it and subtract 40, but you'll get 0x40440000 anyway, and if you don't, then that means the tag array has been moved, and your program will not be able to read the map if you use whatever else you got.

 

Lastly, the value offset 0x4 from the header of the tag data block is the scenario tag ID. While on most maps, the scenario tag ID will be the first tag, Halo's engine doesn't require it. Simply modifying this will change which tag is the scenario tag.

 

Most of the old stuff does still apply, though I typically treat the tag data block separately from the map data rather than converting anything directly to a file offset.

DeadHamster is correct technically. A Magic value is just a hard coded number by definition. In this case it can be calculated; however, you would need to do it from within the program and know all the other memory allocations that have been done.

I see where you are coming from though. You're looking at it as actually memory. Magic is just an offset to a preallocated chunk of ram that everything gets "placement new"-ed into.
This is all semantics though

Awesome work on the write up though c: I definitely learned something and I've been studying it for a bit.

Edited May 18, 2021 by Zatarita