Stat tracking API

[Devieth 164]

Today I am launching a very alpha version of a stats api for SAPP servers.  If you run this script it will send back stats of all players (current and past) to the API server at http://api.halopc.com/stats/.

Here is the script for the SAPP server: stats.lua

Setting up the script: If you are on Windows server 2016 or newer, you can set `use_powershell` at the top of the script, all though this will make almost no difference. If you are on older versions of Windows do not change this setting, the script will attempt to download wget.exe to use that instead using commands that are compatible with powershell 2.0.  If you are on wine you will have to download wget.exe and place it next to your haloceded.exe.

Next you will need to download this: json.lua, place this lua script next to your haloceded.exe, this is required.  (The server will try to download this with the current version uploaded on 6/15/2021.  If it fails download it manually from here.)

What data is collected: IP, hash, name, score, kills, assists, deaths, betrays, suicides, weapon stats, time played (in seconds), other gametype stats, and server details.

How is a player tracked: Within the server, hash.  With the API, IP or a custom UUID that be sent with a custom version of Chimera (based on -572) or with a script for Chimera.  UUID is hardware based.
    Note: The script version will open a cmd window for a moment after launching Halo, if this is annoying I recommend using the custom version of chimera.
    Note2: Player IPs and Hashes are not exposed in the API.

Registering your server: Severs must register their IP in order to send stats to the stats API, this can be done at http://api.halopc.com/register. Servers found sending modified player stats will be blacklisted.

Polling stats: Stats can be pulled from the API in two ways, as a Lua table (http://api.halopc.com/stats/raw?lua) or as a Json string (http://api.halopc.com/stats/raw?json)

If there are any issues please let me know, I can be found on the OpenCarnage discord server, feel free to @ me.  Sorry if this post is kind of messy.

Files in post:

uuid.lua

 

json.lua

Edited June 15, 2021 by Devieth
Updated stats.lua (made it so the script can download json.lua)

[Enclusion 163]

Very nice, I will check this out at some point. Is the server source somewhere? 

[Devieth 164]

32 minutes ago, Enclusion said:

Very nice, I will check this out at some point. Is the server source somewhere? 

The api source?  No not at the moment.  While I am able to host it I may keep it private.  Having multiple sources running it would be kind of annoying.  I will also still be adding more when I can (updating the server script or the api so you could search match history by server instead of by player.)

The sapp server script, its non-compiled Lua script.

The custom chimera, no.  It was originally part of an defunct anti-cheat I wrote.  It grabs the HWID of the C: drive.

Edited June 14, 2021 by Devieth
more info

[mouseboyx 88]

Great work, this release solves many problems I've run into while trying to make something similar. I spent my time dillydallying instead of working on the web stats thing that I was working on.  I guess the ideology is different in releasing source for the server code, so I may still have a chance at continuing work, but most likely I'll continue my procrastination.  The method I fell back on for uniquely identifying players only depends on IP+Name if either of those things change it's considered a different player, but the UUID seems like an interesting workaround to the problem of halo being pretty much anonymous.  I was considering consolidating, or having an option to treat "unique" players as the same if they had the same name and if their different IP addresses were relatively on the same network.  It wouldn't be possible for players to see other's UUID, but could they edit the chimera portion to send a pre-fabricated UUID?  The http request for sapp can do requests, although I think limited to the get method, without lagging or doing them on the main thread.  Even if the web server takes 10 seconds or longer to even send back a response there's no lag using the async method and putting part of the code in a timer.  I'm unsure if you already found a way around the lag with the system shell call to powershell or wget.

Edit:

Sorry, I feel like I'm putting up too much criticism of your release, but I'm very curious about the methods you used because of my attempt in creating something similar.

Edited June 15, 2021 by mouseboyx

[Devieth 164]

3 hours ago, mouseboyx said:

I was considering consolidating, or having an option to treat "unique" players as the same if they had the same name and if their different IP addresses were relatively on the same network.  It wouldn't be possible for players to see other's UUID, but could they edit the chimera portion to send a pre-fabricated UUID?  The http request for sapp can do requests, although I think limited to the get method, without lagging or doing them on the main thread.  Even if the web server takes 10 seconds or longer to even send back a response there's no lag using the async method and putting part of the code in a timer.  I'm unsure if you already found a way around the lag with the system shell call to powershell or wget.

UUIID as I said above is based on the HWID of the users C: drive, which is 100% unique between every player.  The UUID pulled from the Lua script for Chimera (and is compatible with the new chimera and older versions) is based on the the Windows system UUID.  Both are hashed then converted into a number on the API.

The lag with powershell and wget is minimal, about a second or so as long as the API responds somehow (either with 200 OK or otherwise, also put timeouts inside of the commands of 5 seconds.)  Also I have the requests go out on `EVENT_GAME_END` so no one is going to be lagging around in-game and it shouldn't drop the connection to anyone.  I was looking into making a ffi module that would be a multi-use http request system (POST, GET, PUT, ect.)

Anyway you could use this API to make a readable webpage so people could look up their stats.  I am not great with html/js so I didn't even bother to try and write that part.

[mouseboyx 88]

1 hour ago, Devieth said:

UUIID as I said above is based on the HWID of the users C: drive, which is 100% unique between every player.  The UUID pulled from the Lua script for Chimera (and is compatible with the new chimera and older versions) is based on the the Windows system UUID.  Both are hashed then converted into a number on the API.

The lag with powershell and wget is minimal, about a second or so as long as the API responds somehow (either with 200 OK or otherwise, also put timeouts inside of the commands of 5 seconds.)  Also I have the requests go out on `EVENT_GAME_END` so no one is going to be lagging around in-game and it shouldn't drop the connection to anyone.  I was looking into making a ffi module that would be a multi-use http request system (POST, GET, PUT, ect.)

Anyway you could use this API to make a readable webpage so people could look up their stats.  I am not great with html/js so I didn't even bother to try and write that part.

Thanks, I think an ffi module that was capable of doing any http method that would not cause lag would be an indispensable asset to any web related integration. 

 

The project that I was working on completely lacked an api portion, but I was thinking about how it would be useful.

 

I'm thinking what if a nefarious group of players decide they want to make up a UUID?  Then concatenate that value through the chimera script, rather than the real UUID from their C: drive, and then they all use that same UUID among them?  There would probably be some logic somewhere on the server side to account for a problem like that.  Also if they all logged in through a vpn while using that shared UUID their ip addresses would more or less look like a single player, except their names would be different if they were all playing on the same server at the same time.  And they could make the UUID follow the same format as one that would be obtained from a drive.  Sorry this is just where my mind goes with this stuff.

 

execute_script('rcon uuid cl_uuid '..'some_uuid_here')

Edit:

I guess when the end report is known by the webserver and it encounters any duplicate uuid from different players, it could flag that uuid for abuse, and/or toss that information out.  I guess the most they could do at that point would be to work in shifts, especially if 2 games going on 2 different servers had the same uuid at the same time and that could also be detected.

Edited June 15, 2021 by mouseboyx

[Devieth 164]

2 hours ago, mouseboyx said:

Thanks, I think an ffi module that was capable of doing any http method that would not cause lag would be an indispensable asset to any web related integration. 

 

The project that I was working on completely lacked an api portion, but I was thinking about how it would be useful.

 

I'm thinking what if a nefarious group of players decide they want to make up a UUID?  Then concatenate that value through the chimera script, rather than the real UUID from their C: drive, and then they all use that same UUID among them?  There would probably be some logic somewhere on the server side to account for a problem like that.  Also if they all logged in through a vpn while using that shared UUID their ip addresses would more or less look like a single player, except their names would be different if they were all playing on the same server at the same time.  And they could make the UUID follow the same format as one that would be obtained from a drive.  Sorry this is just where my mind goes with this stuff.

 

execute_script('rcon uuid cl_uuid '..'some_uuid_here')

Edit:

I guess when the end report is known by the webserver and it encounters any duplicate uuid from different players, it could flag that uuid for abuse, and/or toss that information out.  I guess the most they could do at that point would be to work in shifts, especially if 2 games going on 2 different servers had the same uuid at the same time and that could also be detected.

Yeah Kavawavi made an ffi module, it only has http get.  I would just need one for doing post request.  Using post because it is a better way of sending of the data then having it in the url of a http get request.  Like I said, lag caused by it at the moment is not a huge concern but, the script for the server side is open source so someone could integrate their own 

Also if people set custom UUID's I don't care, and if they share a UUID with another player well that's only going to impact their individual stats.  What I do care about is if servers send false match data, which is why servers have to register to be on the whitelist, and if they send fake data I can move them over to a blacklist.  Right now registering is still a thing but the blacklist is disabled, as this is like alpha stage and I just want to see some data come in first.

Also thinking of this made me remember I forgot to include the json library. Need to update the main post.

[mouseboyx 88]

3 minutes ago, Devieth said:

Yeah Kavawavi made an ffi module, it only has http get.  I would just need one for doing post request.  Using post because it is a better way of sending of the data then having it in the url of a http get request.  Like I said, lag caused by it at the moment is not a huge concern but, the script for the server side is open source so someone could integrate their own 

Also if people set custom UUID's I don't care, and if they share a UUID with another player well that's only going to impact their individual stats.  What I do care about is if servers send false match data, which is why servers have to register to be on the whitelist, and if they send fake data I can move them over to a blacklist.  Right now registering is still a thing but the blacklist is disabled, as this is like alpha stage and I just want to see some data come in first.

Also thinking of this made me remember I forgot to include the json library. Need to update the main post.

I really wish I knew enough to be able to create ffi libraries for lua/sapp.  Sorry for my rambling post earlier, I think having no available source for the webserver side of things started to make me go crazy with questions in my mind of things to look out for.  If I can get elitegameservers.net to upload wget.exe along with its .dll files I would be willing to help send you some info, I've got this server on retail SuicideBomber3 104.192.224.11:2301, that gets a good amount of traffic, it's ffa.  I don't know if elitegameservers.net runs windows for hosting but I really don't know if they allow a shell to pulled up from lua. 

 

If your website has ssh access you can get a free signed ssl certificate and enable https, it would help a bit with securing data, it's the only reason I felt comfortable with sending information through a url rather than post. This is what I used: https://github.com/srvrco/getssl

[Devieth 164]

18 hours ago, mouseboyx said:

I really wish I knew enough to be able to create ffi libraries for lua/sapp.  Sorry for my rambling post earlier, I think having no available source for the webserver side of things started to make me go crazy with questions in my mind of things to look out for.  If I can get elitegameservers.net to upload wget.exe along with its .dll files I would be willing to help send you some info, I've got this server on retail SuicideBomber3 104.192.224.11:2301, that gets a good amount of traffic, it's ffa.  I don't know if elitegameservers.net runs windows for hosting but I really don't know if they allow a shell to pulled up from lua. 

 

If your website has ssh access you can get a free signed ssl certificate and enable https, it would help a bit with securing data, it's the only reason I felt comfortable with sending information through a url rather than post. This is what I used: https://github.com/srvrco/getssl

I can set up ssl but it would take some work.  Not on the plans until things are finished.

Also idk what EGS does, not of fan of them for multiple reasons.  Feel free to register your server on the site and try though.  If its on windows server it will try to download wget and everything, it it doesn't work well, the script will just error out.

[mouseboyx 88]

On 6/16/2021 at 6:34 AM, Devieth said:

I can set up ssl but it would take some work.  Not on the plans until things are finished.

Also idk what EGS does, not of fan of them for multiple reasons.  Feel free to register your server on the site and try though.  If its on windows server it will try to download wget and everything, it it doesn't work well, the script will just error out.

I sent EGS a ticket, and they say the shell exec from sapp lua is no-go.  I was using shared linux vps hosting, but ran into the noisy neighbors effect and got some bad lag so I went back to EGS.  For 1 to 2 servers EGS is fine I think although the complete lack of control over anything is one of the compromises of price and reliability.  EGS added a plugin script to auto-install the sapp http client that Kavawavi made.  I was doing some testing with the client and it can request urls up to around ~8100 characters long, probably 8000 or less to keep things safe.  Would you consider writing in a compatibility layer to use the sapp http request to bring your script to locked down hosting like EGS?