Zatarita

The s3dpak format - Rewrite

Introduction
    Hello! My name is Zatarita. I'm the current dev for SeK. I felt I should share some of the information I have acquired while researching the engine. Pieces of this information was acquired through reverse engineering the files, combing through a disassembly, and even other games saber has produced around the same time. The S3dpak file is an archive file similar to a .zip, .rar, or .tar.gz ( for the linux bros ). Currently, not everything is known about the files contained within the s3dpaks, but the structure of the s3dpak as a whole has been mapped out completely. ( for this generation's saber engine at least ).
    I will periodically review this post to add updated information as new discoveries are made; however, all information present here is correct to my current understanding. Some things may be proven incorrect. All that aside, I'm fairly confident in my findings; I have developed working mod tools using this understanding. I hope to present to you enough information to confidently navigate a s3dpak from the lowest level. By the end, you should theoretically understand how to manually extract data from an s3dpak given you had the need.


The S3dpak

 

    As previously stated the s3dpak is an archive format. This means it's a file that contains other files. Some s3dpaks are general purpose, as they aren't linked to any specific level, and others are level specific. There is one primary difference between s3dpak and other archive formats. That is s3dpaks are specialized for use inside the Saber engine. It only takes specific "file types" if you will. These are assets required for the level; level geometry, models, sounds, textures, etc. This is called the format of the data. There are only a handful of available options, and some of them are just remnants of the saber engine; completely unused by Halo. 
    The s3dpaks are compressed using CEA compression. ( For more information on compression formats see appendix a. ) This means in order to see the contents the file needs to be decompressed. There are a number of tools available to decompress the files.( see appendix a ) Once decompressed, if we open up the file in a hex editor we get our first glimpse into a s3dpak. The byte order of all saber files, including the s3dpak, is little endian. ( Byte Ordering )

 

 

The Header


unknown.png          unknown.png          
Fig 1a. Overview File                                                                                                                                                                    Fig 1. Overview Data                           

 

    The header is the beginning section of a file. It typically presents a high level view into a file similar to a glossary does in a book. For the s3dpak; it breaks the file into entries and raw data. Where each entry contains file metadata, or information about the file, and a pointer to where to find the data in the raw data. At the beginning of any s3dpak there is an entry count telling us how many entries are contained within this file. Following this is a list of entries. Everything else in the file is raw data used by the entries.

    Referencing the graph above we can get an idea of how the header helps guide you through the file and gives you an idea of what you're looking at. The text in yellow is metadata about the entry. Things like what the filename is and what format the data is. Using the entry's offset and size, we can get data from the file. 

 

unknown.png                                    unknown.png
            Fig 2a. Header Hex                                                                                                                                                             Fig 2b. s3dpak hex map             

      

    That's rather abstract, so let's try taking a look at the actual data. On the left we have the raw data for a10.s3dpak. On the right we have the same data mapped to colors. Some colors may be hard to see; however, I used those colors on unimportant data as you will see. The first four bytes of a s3dpak will always be how many entries are in the file. The first four bytes in this case are 07 04 00 00; although, Since the file is little endian we reverse the order and get 00 00 04 07. After we drop the leading zeros we get 0x407. This is how many children are present in the file. If this was confusing please see the appendix for an explanation on byte ordering, as endianness will be important for this section. Following that we're going to load each entry left to right top to bottom.

 

White (4 bytes) = first four bytes of the file are the child count
Green (4 bytes) = Offset to the start of the raw data = 0xa68c

Cyan (4 bytes) = Size of the raw data = 0x18624

Yellow (4 bytes) = Number of characters in the name of the filename = 0xC (TexturesInfo)

Purple(n bytes) = Characters that make up the file name. This is easier to see in the string view of the hex editor. Underlined in the same color. = "TexturesInfo"

Orange (4 bytes) = Entry format (see below)

Blue(8 bytes) = unused padding.

 

    Using this information we could very easily extract data from an s3dpak. A few hex editors offer the option to select a block of data. You can just plug and chug the data from the header to select the entire entry. Using the previous example with HxD. I'll start by going to "edit > select block" It will pull up a tool that allows me to select a range. In this case I just plugin in the starting offset 0xA68C, the Size 0x18624, and click ok. ( Do note that HxD does not require the 0x ) It will select the range of data for me, which I can copy and paste into a new document. Upon doing so you have manually extracted data from the s3dpak!

unknown.png
Fig 3. Manual extraction


 

Format

    Now that we've gotten the raw data from the file, how are we supposed to interpret what we're looking at? This is where the format comes in. Format determines how the engine attempts to interpret the data when it loads. Is this data a texture, or is it a model? This is an important distinction. Currently all of these formats haven't been completely mapped out; however, as it stands here is what I know.

 

unknown.png

Fig 4. Supported formats by build.

 

    Not all s3dpaks are equal. Xbox has some features that the pc version lacks. With that comes an exciting chance to make a table, and who can turn that offer away. Above I have shown a table of formats, and their compatibility across platform. In the xbox version of the game the s3dpak is the primary storage file, compared to pc's imeta/ipak system. This means that a lot more functionality needs to be packed in on the xbox version to offer more flexibility. Another thing to note is "multiplayer". The multiplayer section only applies to pc. It appears as though all maps have a bare minimum requirement by the saber engine, and while the saber engine isn't used for multiplayer, these files are still needed to work. I believe most of these files are practically identical from level to level bar a few exceptions anyway. just generic placeholders.

 

    A few of these formats have been worked out, some are pretty basic, some are still being researched.


    Here is what is known now. Any item with a hyperlink will be a link to a post about it if it exists. ( this will be updated with time )

  • [0] SceneData - A high level list of items present in the s3dpak. This document is primarily plain text
  • [1] Data - To be updated at a later time.
  • [2] CacheBlock - To be updated at a later time.
  • [3] Shader - Primarily plaintext shader source. On xbox it is typically compressed using zlib.
  • [4] ShaderCache - To be updated at a later time.
  • [5] TexturesInfo - High level information about the textures contained inside the s3dpak. Isn't xbox only; however, is unused in pc.
  • [6] Textures - Texture data similar to an ipak entry. Contains dds textures.
  • [7] TexturesMips64 - Also texture data similar to ipak entry. Not sure why this one is special.
  • [8] SoundData - To be updated at a later time.
  • [9] Sounds - Unused by any Halo game. Remnant of Saber engine
  • [10] WaveBanks_mem - To be updated at a later time.
  • [11] WaveBanks_strm_file - To be updated at a later time.
  • [12] Templates - Proprietary 3d model format for use in Saber games.
  • [13] VoiceSplines - Unused by any Halo game. Remnant of Saber engine
  • [14] Strings - Wide char text file for misc strings used in the game.
  • [15] Ragdolls - Unused by any Halo game. Remnant of Saber engine.
  • [16] Scene - Level geometry with merged level specific templates. Similar to Blam scenario
  • [17] Hkx - PC only ( not sure if enforced on xbox ) breakable glass definition. Typically paired with a template of the same name
  • [18] Gfx - Flash file, can be decompiled with a Flash Decompiler. it is compressed on xbox using the same method as Shader.
  • [19] TexturesDistanceFile - To be updated at a later time
  • [20] CheckPointTexFile - (Unused by any Halo game.)
  • [21] LoadingScreenGfx - Unused by any Halo game.
  • [22] SceneGrs - * assumption * Scene grass detail object definitions and locations.
  • [23] SceneScr - Unused by any Halo game. Remnant of Saber engine.
  • [24] SceneAnimBin - Unused by any Halo game. Remnant of Saber engine.
  • [25] SceneRain - Defines precipitation ( not just rain ) for a level
  • [26] SceneCDT - To be updated at a later time.
  • [27] SceneSM - To be updated at a later time.
  • [28] SceneSLO - Unused by any Halo game. Remnant of Saber engine.
  • [29] SceneVis - To be updated at a later time.
  • [30] AnimStream - To be updated at a later time.
  • [31] AnimBank - Unused by any Halo game. Remnant of Saber engine.
     

The appendix

Compression Formats

 

Takka and Kavawuvi like this

Specifications:

S3dpak - format - Imeta/ipak - format - Fmeta - format

Programs:

H2a-inflate - SuP

Share this post


Link to post
Share on other sites

Tiddy-bits:

On 8/2/2020 at 6:38 PM, Zatarita said:

Found out today that there are more types for the Xbox version. So it's not compete complete.
Will update with Xbox specifics later this week

Fixed

Updated whole post to reflect new discoveries.
Better organized
Pretty colors

Enclusion, tarikja and Takka like this

Specifications:

S3dpak - format - Imeta/ipak - format - Fmeta - format

Programs:

H2a-inflate - SuP

Share this post


Link to post
Share on other sites
  • Recently Browsing   0 members

    No registered users viewing this page.