What happened to OC? - CLOSED Carnage?!
Kavawuvi

Update: New map format being worked on. Thoughts?

13 posts in this topic

Tiddy-bits:

Nice! We'll patch our custom tool to give this a try. Our tagset currently is using 21.92MiB of tag space, so I'll just fill the rest with duplicated objects. I'll come back here to give a report :)

Edited by MrChromed
Spark likes this

Share this post


Link to post
Share on other sites

Here's a question...  and I am not sure if this will make sense...

 

If the tag space was to be increased fromm 23MB to 64MB, would that not put extra demands on the computer system's resources?  I mean, it would mmake sense if it did...

Spark and Kavawuvi like this

Share this post


Link to post
Share on other sites
1 hour ago, Thomas said:

If the tag space was to be increased fromm 23MB to 64MB, would that not put extra demands on the computer system's resources?

 

At the end of the day Halo is still a 32 bit program and is limited by that to a maximum allocation of 4 gigabytes of RAM. Halo also will not load any map file larger than 200-something megabytes, and while I'm sure that can be extended you'll still be limited by the 4GB cap. So basically, unless you have a potato or live in Latin America, the impact will be negligible.

Edited by Sunstriker7
Takka and Kavawuvi like this

Share this post


Link to post
Share on other sites
4 hours ago, Thomas said:

Here's a question...  and I am not sure if this will make sense...

 

If the tag space was to be increased fromm 23MB to 64MB, would that not put extra demands on the computer system's resources?  I mean, it would mmake sense if it did...

 

As @Sunstriker7 stated, modern PCs won't care. 64 - 23 = 41, thus Chimera will increase memory usage by a fixed 41 MiB. Extra RAM usage does NOT directly impact frame rate unless your system is starved of RAM. Most gaming PCs have at least 8 GiB of RAM, with the latest gaming PCs typically having 16 GiB of RAM or more.

 

Not everyone is on a modern PC, so let's look at an old PC. Halo's system requirements are listed here: https://www.systemrequirementslab.com/cyri/requirements/halo-combat-evolved/10234

 

Halo recommends 512 MiB of RAM. We're talking about a PC from 2003, so 512 MiB was what was mid-ranged (perhaps slightly higher) back then. Pretty much every desktop PC sold within the past 10 years has had well over 512 MiB of RAM. 2-4 GiB was pretty common in 2008 with Core 2 Duo CPUs which seem to be pretty commonplace with Latin American players.

 

But let's assume, for a moment, you were on a mid-ranged gaming PC from 2003 and you had 512 MiB of RAM. Would 41 MiB be a lot? 41 MiB / 512 MiB = around 8%. So, that means that if you're on a PC that is so old that you have only half a gig of RAM, an additional 8% of your total RAM will be used. I really don't think this is going to be noticeable if you're just running Halo and maybe a few things in the background and, again, your PC just so happens to be THAT old.

 

Of course, if you're running a dedicated server, it's a slightly different discussion. If you're trying to host Halo on a box with only 512 MiB of RAM, then if the game normally takes up 100 MiB but now has to take up 141 MiB, the number of games you can run simultaneously is reduced from around 4 to now around 3.

 

It's unlikely this will be a big deal for most people, but if you're the type of person that thinks it's a good idea to host about 200 Sniper Gulch servers, you're going to be needing another 8.6 GiB.

 

All-in-all, 41 MiB is a small price to pay for a more than doubling of the tag space.

 

That said, we were considering 48 MiB tag space instead of 64 MiB. This would mean only an extra 25 MiB instead of 41 MiB which might matter if you're trying to run against Halo Combat Evolved's absolute minimum requirement of 128 MiB of RAM. However, those systems probably shouldn't be used for playing even stock Halo.

 

...or games in general.

Takka, ST34MF0X and Sunstriker7 like this

Share this post


Link to post
Share on other sites

In that case, then it is a good job that my dedi has 72GB of RAM ;)...

And that my current PC has 32GB and my new one, being built later this month will have 64GB... ;)

Edited by Thomas

Share this post


Link to post
Share on other sites

# Issues

There are several issues with the way non-vanilla map formats are being created and distributed.

 

1. End-User File Transparency

- Vanilla maps already have this issue. Xbox, Trial, Retail, Custom Edition, and MacDemo engines all use the same filename extensions despite their maps using different, incompatible formats.

- OpenSauce has a .yelo filename extension for maps that have a non-standard format. However, it also uses the .map extension for maps that the engine can run without OpenSauce installed at the cost of the map's OpenSauce features not working. This allows maps to work, but not as the map-maker intended.

- Compressed .map files made for CEA or by Invader for the classic PC ports. Again, file transparency. CEA uses/used a similar file format as Custom Edition, but it compresses the file header, making it incompatible with the engine version indicated in the header.

Invader compresses the map data, but this too creates .map files that are incompatible with any engine that can run .map files out of the box, without Chimera or some other plugin.

 

What are some ways the user can be informed of a map file's prerequisites? A new file name extension, file header changes, et cetera.

How can we display the map header to end-users? A file format API/SDK may be necessary for other developers to add support for this file format to their applications (mod and map managers, server applications, et cetera). The API/SDK could then be wrapped or rewritten for different languages as needed. However, that contribution would be the responsibility of developer-users, not maintainers of the file format.

 

2. Format versioning.

After file transparency is resolved, format versioning would be the next issue.

However, jamming every prerequisite into a filename and extension would be ridiculous. If OpenSauce did this, we would have .yelo, .yelo1, .yelo2, et cetera. Then what if the .yelo format is extended to Halo 2 maps? CE Retail? Trial?

 

At best, the major version of the file format should be included in the filename's extension to indicate compatibility-breaking changes.

In addition to this, I believe this new format needs both a format version and a feature set list (see Feature Flags). This concept is similar to the way DirectX and OpenGL feature support is stated and handled by GPUs and their drivers. They indicate support for both API versions as well as individual feature flags.

Likewise, a map's format version should indicate a standard of core features while feature flags indicate individual features, including potentially non-standard extensions to the format. To be honest, that last part worries me as we can only hope feature flags are supported by publicly shared plugins. Again, See Feature Flags for more info.

 

----

 

# Feature Flags

Metainfo in the file header indicating features required for the map to work as intended.

Note: Feature flags may be redundant if only used to indicate features already indicated by the format version. They would be more useful to indicate non-standard features as well as inform the user of the plugin(s) required to support said features.

 

This should allow for a more extensible map format at the cost of file compatibility transparency to end-users. (See File Transparency above)

 

1. Plugins that are made to work with this proposed file format will first need to be told how to handle a base set of features (those that are part of the file format's specifications), then know how to deal with unsupported/unrecognized feature flags (format extensions).

 

2. One issue that may arise from Feature Flags is the requirement of privately-shared plugins to support a feature while the map file itself is public.

Although these unrecognized, non-standard feature flags could be handled by plugins, it would suck to have maps floating around that require a privately shared plugin to work at all.

 

3. Another issue is how these maps would affect multiplayer map-downloading experiences. How would the client handle an unsupported map without first downloading it? For many rural and developing countries, downloading large files that can't be used can be problematic.

This could be resolved by transferring the map header first, but only if the isn't supports that in the first place

 

4. Lastly, how would someone know which plugin(s) they need to get in order to support a feature flag? See File Transparency.

 

5. How do we differ between standard feature flags and extension feature flags? Perhaps compiler could enforce a naming scheme for feature flags e.g. std_tagspace, ext_PBR

 

6. How do we handle extension feature flags that are beginning to be used as often as standard feature flags? Hopefully, the developer of the extension feature flag would agree to and help integrate it into the standard format specifications

 

These six issues need to be addressed before this feature can be added to this map format's specifications.

Edited by VoidsShadow
tarikja and Sunstriker7 like this

Share this post


Link to post
Share on other sites
17 hours ago, VoidsShadow said:

# Issues

There are several issues with the way non-vanilla map formats are being created and distributed.

 

1. End-User File Transparency

- Vanilla maps already have this issue. Xbox, Trial, Retail, Custom Edition, and MacDemo engines all use the same filename extensions despite their maps using different, incompatible formats.

- OpenSauce has a .yelo filename extension for maps that have a non-standard format. However, it also uses the .map extension for maps that the engine can run without OpenSauce installed at the cost of the map's OpenSauce features not working. This allows maps to work, but not as the map-maker intended.

- Compressed .map files made for CEA or by Invader for the classic PC ports. Again, file transparency. CEA uses/used a similar file format as Custom Edition, but it compresses the file header, making it incompatible with the engine version indicated in the header.

Invader compresses the map data, but this too creates .map files that are incompatible with any engine that can run .map files out of the box, without Chimera or some other plugin.

 

What are some ways the user can be informed of a map file's prerequisites? A new file name extension, file header changes, et cetera.

How can we display the map header to end-users? A file format API/SDK may be necessary for other developers to add support for this file format to their applications (mod and map managers, server applications, et cetera). The API/SDK could then be wrapped or rewritten for different languages as needed. However, that contribution would be the responsibility of developer-users, not maintainers of the file format.

 

2. Format versioning.

After file transparency is resolved, format versioning would be the next issue.

However, jamming every prerequisite into a filename and extension would be ridiculous. If OpenSauce did this, we would have .yelo, .yelo1, .yelo2, et cetera. Then what if the .yelo format is extended to Halo 2 maps? CE Retail? Trial?

 

At best, the major version of the file format should be included in the filename's extension to indicate compatibility-breaking changes.

In addition to this, I believe this new format needs both a format version and a feature set list (see Feature Flags). This concept is similar to the way DirectX and OpenGL feature support is stated and handled by GPUs and their drivers. They indicate support for both API versions as well as individual feature flags.

Likewise, a map's format version should indicate a standard of core features while feature flags indicate individual features, including potentially non-standard extensions to the format. To be honest, that last part worries me as we can only hope feature flags are supported by publicly shared plugins. Again, See Feature Flags for more info.

 

The problem is that .map is such a generic file extension. Tons of programs use files with .map extensions. And the game even has bitmaps.map/sounds.map/loc.map which aren't cache files but a completely different format to the other map files. Yes, changing the header will prevent the stock game from loading it. This is something Chimera does. However, people still want the extension changed - something I'm implementing in a future Chimera update that's in the near future.


It's worth noting that MCC no longer compresses the header, but these maps are still incompatible as they use a different base memory address (thus they will crash the game) despite using version 7.

 

17 hours ago, VoidsShadow said:

# Feature Flags

Metainfo in the file header indicating features required for the map to work as intended.

Note: Feature flags may be redundant if only used to indicate features already indicated by the format version. They would be more useful to indicate non-standard features as well as inform the user of the plugin(s) required to support said features.

 

This should allow for a more extensible map format at the cost of file compatibility transparency to end-users. (See File Transparency above)

 

1. Plugins that are made to work with this proposed file format will first need to be told how to handle a base set of features (those that are part of the file format's specifications), then know how to deal with unsupported/unrecognized feature flags (format extensions).

 

2. One issue that may arise from Feature Flags is the requirement of privately-shared plugins to support a feature while the map file itself is public.

Although these unrecognized, non-standard feature flags could be handled by plugins, it would suck to have maps floating around that require a privately shared plugin to work at all.

 

3. Another issue is how these maps would affect multiplayer map-downloading experiences. How would the client handle an unsupported map without first downloading it? For many rural and developing countries, downloading large files that can't be used can be problematic.

This could be resolved by transferring the map header first, but only if the isn't supports that in the first place

 

4. Lastly, how would someone know which plugin(s) they need to get in order to support a feature flag? See File Transparency.

 

5. How do we differ between standard feature flags and extension feature flags? Perhaps compiler could enforce a naming scheme for feature flags e.g. std_tagspace, ext_PBR

 

6. How do we handle extension feature flags that are beginning to be used as often as standard feature flags? Hopefully, the developer of the extension feature flag would agree to and help integrate it into the standard format specifications

 

These six issues need to be addressed before this feature can be added to this map format's specifications.

We did give this consideration a while ago. The problem isn't so much implementing it on a client but moreso having tools that build maps with these tags. While Invader can build maps, people prefer to use the stock tools to build maps. So to get more people to use this convention, you would have to write a program that patches a .map file with this metadata.

 

Anyway, right now, the map downloader isn't going to download things that aren't incompatible with the client you're using. HAC2 users aren't going to get a map that requires Chimera. Rather, each mod gets their own type of file.

 

As for getting the client to only download just the header, this is something we'd have to figure out with msalerno1965, as he's the guy who runs the repo. He'd have to modify his repo.

 


 

This is a really good writeup and some really good ideas for a new format.

 

However, now there's yet ANOTHER format coming, this time for MCC (version 13). I have no doubt that this will create even more confusion. Not only that, but it has more tag space and support for things that Halo PC doesn't support like monochrome textures. This, too, will be .map.

 

I'm starting to wonder if the new format I was considering will even be worth it at this point.

Share this post


Link to post
Share on other sites
  • Recently Browsing   0 members

    No registered users viewing this page.