SeT team - progress

[Zatarita 407]

So, things have been a bit quiet from my front this past month, but I figured I’d post an update to some of the progress that’s been getting made. When I say “team” I currently mean myself; however, I am looking for people to join. If you’re interested shoot me a dm

 

 onto the updates!

 

SeT had a major revision. I refactored everything to utilize polymorphism a bit more. Hopefully to keep things a bit more organized.

There have been 2 major updates to SeT. One is SEK. Which is a saber editing kit. It’s a hek style set of tools that will be used to modify saber files. this is going to replace SuP. Planned tools are:

 

S-edit - a guerillaesque program used to edit the files inside the saber paks.

 

s-tool - a tool-esque program used to compile new files.

 

 

 

 

P-arc: this will let us pack All the mod files into one file (.s3dpak, .ipak, .imeta, .fmeta, .map). It also lets us send only the modded files, greatly reducing file size.

 

File size comparison (desert hog mod):

 

 

 my hopes is this will simplify modding cea, and make it more user friendly.

 

besides planned programs, my hopes is to create an export/importer with templates to blender.
 

  Coordinates, texture coordinates, and faces have been mapped out for cea. This means I can extract most of the model data; however, this is only for “simple” meshes. I have not found rigging data. Research is still being done. I’m going to attempt to manually build a model into a template to test building soon.

 

 lastly, While the primary focus is for h1, I have been poking around with h2a as well.

I managed to extract the contents of the h2a pck files. I have a lot of templates from there as well.

 I figured out some of the model data for them as well; however, I wasn’t able to figure out texture coordinates, or rigging information for that either.

 

 

 is your interested in helping out, I could really use a hand. I only have ~2 days a week that I can dedicate to this.

[Zatarita 407]

Another update!

 

SeT got updated (and is now in desperate need of some house keeping.) 

 

sadly progress has been slow. I had an infected tooth and only really had the chance to work on this like 4/5 days this month.

Still looking for help c:

 

added:

    Bugs (probably)

    DirectX support

    Importing from dds (still kinda broken; however, it has been pushed)

    dds definitions for the used types

    Converter for 16 bit argb to 32 bit

    A basic beginning to a texture tool

    Compression (which needs to be threaded)

 

 

 imagine is a test of importing custom dds textures.

Edited October 3, 2020 by Zatarita

[Zatarita 407]

More progress made!

Fixed a compression bug, implemented h2a decompression, and compression (still needs to be tested) also added the beginnings of a tool "SMI" an ode to the nightmare fuel days of PMI with HMT. it's for injecting templates into different s3dpaks. The functionality has been added to recursively extract templates, and their coresponding Imeta entries. Making injecting templates as simple as extract, import. It also adds the templates, and the textures to the SceneData. Automating the process almost entirely.

 

Currently SMI only has extracting programmed in; however, it shouldn't be too difficult to finish.

 

This is more of a "proof of concept" than anything. The tool is designed to hold people over until the full release of SeK.

Edited October 8, 2020 by Zatarita

[Zatarita 407]

Some progress has also been made on importing textures to IPAKs. "Tex mod" is another proof of concept program for SeT that will be replaced.

It's implementation is almost complete; however, there are still bugs needing squashed.

 

Keep your eyes peeled. I'm going to try and stream line this due to popular demand. Hopefully I'll have it done by the end of next week. But I tend to set due dates too soon, so let's say the end of the month of I hit too many speed bumps

[Zatarita 407]

Progress update!

The team has been primarily focusing on h2a as of late. We've been able to map out most of the template format; however, there are still a bunch of unknowns. There is two major speed bumps we're stuck on; However, when I post the technical update it might give the community a chance to brainstorm with us, all help is appreciated! c:

Besides this I've been migrating the SeT SDK to c++ to better utilize threading. It'll be built into a dll for anyone who wants to use it to create their own tools.

The python modules will still be worked on as well.

 

I will have a more technical update over the weekend for those interested

[Zatarita 407]

okay! I finally have a bit more time to break down where we're at.

CEA:
Starting off SeT-cpp has been started, and initialized. My c++ is a bit rusty; however, hopefully it'll pay off in the long run.
https://github.com/Zatarita/SeT-CPP
Right now Imeta is only really implemented, and parsing scenedata. still needs a lot of work, and there are no fail safes currently. So it's more for reference's sake right now.

CEA base file types:
For the primary file types associated with CEA, we've pretty much managed to map out 99% of the structures and definitions for the base data types, and we've been moving into understanding the files in the files. The last few unknowns for the imeta was found to be depth, only leaving certain file types in s3dpaks as "uncertain"; however, luckily the names we have for them now will work until more information is found out.

CEA sub-file types:

This is where we've been spending a lot of our time, primarily on my front with templates.
We've been making some progress. though, there's still a lot to go. We've determined the template system is a sentinel system. Each struct is defined as an 2 byte enumeration, and a 4 byte end of block offset. making it layout similarly to a binary tree layout/c structs. This sentinel block structure is used EVERYWHERE, so there are a lot of sentinels to figure out for almost every file inside of s3dpaks. for now, focusing on templates:

 

To better understand what I mean, here is an example assault rifle template.

In this example all the end of block (eob) are colored blue, and each sentinel block has its own color (kinda)

First looking at yellow (0x02e4) this is the sentinel that denotes a template object. So this first sentinel lets the parser know to expect a template type, and it's 0x30cea till the end of block. Technically this entire file is the first block, and every other block is contained in this block. I like to visualize it as
 

template

{

    //0x30cea bytes of SOMETHING
}

now let's expand on this.
in purple we have the sentinel for filename (0x02e5), the end of block for filename (0x23), and the value of filename (LPTAassault_rifle.tpl).
in orange we have an unknown sentinel type (0x0316), the end of block (0x2c), and the value of what 0x0316 holds. 3 bytes of something.

in green we have a textures array (0x0156), the end of block(0xf6), and the value of what 0x156 holds. which in this case is a bit more interesting than the others.

the textures array has a 4 byte count of how large the array is. Followed by more sentinel, end of block pairs. This is all contained within the textures array block. these sentinels are texture names, with end of block again. Going back to our original visualization we can kinda look at it like this:
 

template
{
	char* file_name = "LPTAassault_rifle.tpl"; // 0x02e5
	unknown structure; // 0x0316
	textures[count] = { "lig_ray_cone3", "sc_blue_0", "sfx_arifle_flame_01", "wpn_assault_display",
				"wpn_assault_display_3p_back_01", "wpn_assault_display_3p_digits_01", "wpn_assault_rifle_01"}; // 0x0155
...					//^ 0x0156
}

this is what's been taking us so long to map out the templates and files. A lot of the structures are just one variable, and it's hard to guess what some of them do without some more pieces. It's like a giant sudoku puzzle.

I will type out more in future posts, the snow is overloading my browser though, so I have to type in small bursts and refresh the page

Edited December 8, 2020 by Zatarita

[Zatarita 407]

Migration to c++ is chugging along. Compression is broken for now; however, I have parsing of s3dpaks and imeta down. (See: https://github.com/Zatarita/SeT-CPP) don’t judge my c++ lol

 Decided I want to see what changes the tags have In h2v -> h2a.

 Extracting tags seems to be a bit more difficult for h2 than I thought with existing tools (and I lost my original cd), so I’m just writing my own library for it. It’ll make things easier anyway having access to both engines, I’ll likely do the same for cea as well so I can link together the pieces from blam to saber 

Edited December 10, 2020 by Zatarita

[Zatarita 407]

So, I finally did my first build of what I'm now calling the anniversary SDK or A-sdk for a pck unpacker. It works for everything except shared.pck. I need to write a light weight handler that doesn't load everything into memory.

 

First time writing a dll as well; however...

 

Apparently windows defender thinks it's a Trojan ಠ_ಠ I didn't think my c++ was that bad, but ok

 

Also one step closer to a h2v map parser. I think I'm going to keep going at it.

[Zatarita 407]

Update

I've been thinking a lot about this project, and how I wanted to go about organizing and managing this project. My programming style has drastically changed since I started developing software with a purpose for others to use. SO, I feel I should model my paradigm accordingly. Developers follow a design philosophies for a reason, and it may be time to adopt my own.

The Scope
SeK will officially begin. It will be an ongoing project with updates as new discoveries are made; however, for the first official release all of these requirements must be addressed:

 

SeK must be build as two parts. Functionality, and Design. (library, and ui)

SeK must come with additional tools to meet goals not required in main program. (eg Mod patcher, Integrity checker)

 

SeK should be able to do most of what HEK can do, but with saber files.

SeK should allow editing of blam files to reduce jumping between tools.

SeK should not prioritize editing of blam files, due to the tools that already exist. Instead these definitions should be created after saber definitions.

 

FUNCTIONALITY

SeK must be able to extract data from both generations of anniversary titles. Both blam, and saber. (eg maps->tags | pck->files | s3dpak->files)
SeK must be able to build data for both generations of anniversary titles. likely just saber. (files->pck | files->s3dpak)

    Both of these conditions must do be accomplished in such a way to antiquate my previous tools, and mark them as depreciated.

SeK must be able to update itself - preferably seamlessly.
SeK must be modular - To help with ease of updating, and creation of further add-ons from third parties

    all definitions must be defined using modular system. No definitions are to be hard coded. (Likely runtime dynamic linking, I recently came across this and I like it a lot.)

 

SeK must be cross platform.

SeK must utilize threading and multiproccesssesing. (No more procrastinating Andy. buy a book dammit.)

SeK should be able to access and edit data as both an archive, and as individual parts (E.G. view tag values in map form, and as a tag)

SeK should be able to manage creation of new files, and conversions to usable filetypes (e.g. exporters for 3d modeling programs, texture to ipak entry converter, so on)

DESIGN

SeK must be able to be ran as a GUI or through a CLI.

SeK must be able to be customized to allow for optimization for weaker system.
SeK must be able to present the data for both engines seamlessly in the same window.

SeK must allow natural navigation, and speedy searches.
SeK must allow "project" oriented, "global" oriented, and "map" oriented work flows

    Project must present the data for a mod spanning multiple maps.
    Map must present the data for a mod consisting on a single map.
    Global must be able to present all the data.

SeK must use QT to design the ui.

SeK plugins must supply a QT widget that will present the data it defines.

SeK plugins should be able to utilize generic widgets. (and I should likely create some generic style widgets to be shipped with (tree views, model viewer, bitmap viewer, script view) )

 

SeK should implement a vulkan (with opengl fallback) renderer (for cross compatibility).

 

 

This list is likely incomplete. Im sure as time goes on I'll realize something I need to add; however, I want to keep that at minimum. I feel these rules are vague enough to cover everything, but specific enough to avoid scope creep.

Something else to note.

I plan to incorporate Radium into this project for version two. Radium fell into scope creep. This means it will likely be completely redone to utilize the above rules. Radium will likely start off as the script editor plugin since I've already programmed it's script highlighting and autocomplete. Then it will be expanded to utilize the renderer, then recorded animations.

Edited April 7, 2021 by Zatarita

[DSalimander 213]

Please shift away from CLI.