Welcome to Open Carnage

A resource for Halo Custom Edition and MCC modding, with unique means of rewarding content creation and support. Have a wander to see why we're worth the time! - EST. 2012

Sign in to follow this  
Followers 0
mouseboyx

ffi async curl development

Posted (edited)

I want to preface this by saying that I had not written any C++ before I started working on this, and also that I don't know how to manage memory within C++.  I would not recommend using this on your halo server because it could potentially crash it if not used correctly.  This is more like an overview of how an async C function call can work within lua.  The .dll seems to work under the circumstances, but I don't know how everything works within the C++ code, so it definitely shouldn't be used in any production environment.

 

I would not have been able to figure out how to make the dll if it were not for these posts, I used the structure from Discord Webhooks Script dllmain.cpp, and the lua timer structure from the sapp http client, also special thanks to a halo player named dummkopf.

 

Using Visual Studio 2019 on Windows 10, I followed this guide to setup the environment with libcurl to compile the dll https://medium.com/@farhabihelal/how-to-set-up-libcurl-on-visual-studio-2019-a9fdacce6945

 

The dll can only make one request at a time, lua can pass in a url and post data through the request, and the request is completed asynchronously.  I'm fairly confident halo will not lag no matter how long the webserver takes to respond.  The text response is stored in an lua variable so that you can do whatever needs to be done with it.  It sort of performs an http get and post request at the same time where get variables are passed through the query string, and the post data is passed to the 2nd parameter of curl_request().  If any of the C functions are called out of order then it could crash the server, and I haven't figured out how to create curl request instances, thus only one request at a time, then wait, then make the next one.  The dllmain.cpp could be modified from documentation https://curl.se/libcurl/c/libcurl-easy.html to perform any http method, however the compiled dll in the zip only supports http get/post.  There are lots of comments in the sources to help.

 

The curl_test.7z includes:

dllmain.cpp - C++ code that can compile with visual studio 2019

curl_test.dll - dll that I compiled with visual studio 2019 (put this in the same directory as haloded.exe)  I didn't test this on wine or anything other than windows 10

curl_test.lua - example lua script that allows curl_test.dll to be loaded and called

sleep.php - put this at the root directory of a web server with PHP installed (you can install xampp and put it in C:\xampp\htdocs)  This was mainly for testing whether curl was performed using async and to echo back post variables

 

Honestly though if someone has experience writing C/C++ out there this is a decent enough base to create C functions that won't lag the halo server, but the C++ that I cobbled together is not very good due to my lack of understanding of C++.  I guess I'm looking for someone to come and improve this idea, but I might try to figure out a better way of doing it after I learn more.

 

Problems running with wine:

I was testing the dll included in the zip, and it looks like I compiled it in debug mode rather than release mode, also I attempted to ffi load the dll while running haloded.exe through wine and it asks for VCRUNTIME140D.dll and ucrtbased.dll which could not be found it led me to this https://stackoverflow.com/questions/6261034/building-dll-for-lower-version-of-visual-studio which it looks like it will be necessary to learn about the solutions described in this question in order to compile a working dll for wine. 

The VC++ 2010 runtime has a platinum rating on winehq.org, https://appdb.winehq.org/objectManager.php?sClass=application&iId=5766, my assumption is to somehow obtain a legitimate copy of visual studio 2010 or the VC++ 2010 compiler and compile the dll using that.  Then use: winetricks vcrun2010 within the 32bit wine prefix that the halo server is run in.  In addition visual studio 2010 would somehow have to work with libcurl, which hopefully it would.

 

Ignore the grayed out text, if you open the solution properties in visual studio 2019 before writing any code, then specify the platform toolset as the 2015 version, not the 2015 xp version, then the compiled dll will work under wine after running: winetricks vcrun2015 within the 32 bit wine prefix, and make sure to compile in release mode and not debug mode, or wine will throw an error trying to load the dll, I also uploaded the VC++ 2015 version of the dll as culr_test_wine.dll.7z, it has an identical source to the dllmain.cpp, only that it works with wine, and should work on windows as well after installing the VC++ 2015 runtime.  Using winetricks vcrun2015 over ssh may require the ssh -X option to forward x11 applications because there is a graphical popup to accept the Microsoft terms of the redistributable.  I would really like to find a way to compile the dll without the need for any runtime redist, but I haven't figured that out.

 

Other Notes:

If some protection were added with extra bool values within the dllmain.cpp and wrapping some of the code within an if statement so that certain functions could only be called after others had been completed then improper usage of the C functions causing the server to crash could most likely be avoided.

 

There should probably be an if statement within do_curl() in dllmain.cpp so that if you pass an empty string as the 2nd parameter in lua like:

curl_client.curl_request("http://localhost/sleep.php?wait_time=4","")

Then curl should not perform with  any post data or so that post would not be a part of the request, because sending post data to a dynamic web page whether it be PHP or anything, if the page is setup in such a way that you send post data when it doesn't expect any or doesn't need any it could possibly fail on the server side.  I don't have an example of this happening, but it would be a good idea to have it operate like a switch rather than sending blank post data.  In other words, wrap curl_easy_setopt(curl, CURLOPT_POSTFIELDS, post_data); in an if statement so that if post_data is an empty string don't include it in the request.

 

There is no current error reporting or checking, get_curl_status()==true only means that the curl portion of the C++ has completed, if curl was unable to contact the web server there's no real way to tell.  I believe if it failed you'll get back an empty string "" from get_curl_text_response() yet if you were already expecting an empty response from the web server it would be difficult to discern whether there was an error.

 

Queue of requests:

curl_test_queue.7z contains:

curl_test_queue.lua  This is one example of how lua can create a queue for curl_test_wine.dll, an lua table of outgoing requests can be done one after another, where the timer loop is constantly running checking for more data in the table, sending all of it out until there's no more entries in the table.  There's probably also a way to trigger the timer loop rather than to have it constantly running if there's no data to be sent, but I didn't include that in the code.

log_player.php  This appends the data sent from lua to the http server to a file in the same directory where the php script is located called halo_players.txt that contains the name of the player who joined and the current unix timestamp.  It has to be run within a web server compatible with php.

 

Setting custom headers:

Sorry for this messy edited timeline of work, but curl_test_headers.7z works the same as curl_test_wine, however lua is given the opportunity to pass custom headers using the curl rules outlined in this doc https://curl.se/libcurl/c/httpcustomheader.html I believe I also fixed a memory leak in the program, but I don't have any way to know for sure, or don't know how. I also added some protection for the functions just in case they get called out of order.  So this should be considered the "stable version" although relatively "stable" compared to previous attempts.  I've played a little with this code https://gist.github.com/ianklatzco/c033b33757915feaf48fd0caf14b42cb for running curl from bash to communicate with a discord bot, but I vaguely remember there needing to be an initial authorization that has to be done before the bot will accept requests, which may or may not hinder trying to create one in within lua using this dll.

 

Setting custom method:

It's on my todo list to add an lua counterpart to add an option and parameter to change this libcurl function:  curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, <string from lua here>);  so that something like "DELETE", "PATCH", or "PUT" etc... can be used.

 

sapp_curl_1.7z:

I'm happy with the way this turned out and I did some testing with it and included 2 examples of how sapp_curl_1.dll can be used within lua to query the discord API, and in turn instruct a discord bot to do things.

Due to the massive number of options here: https://curl.se/libcurl/c/curl_easy_setopt.html I think it would be wise to stop development.  The dllmain.cpp could easily be restructured to include any of those options, but sapp_curl_1.dll is simple enough to get started with http api requests or sending any http method along with data.  There is a quirk where if you are sending JSON data with sapp_curl_1.dll it may have been a good idea to format that as UTF-8 within the .dll, but it may be possible to do that within lua.  I might try to figure out either method whether lua or cpp to send special characters, but the fix that I've put in the discord lua examples is a hack method that is reliable at changing the data to be acceptable but won't send special characters.  I believe that the post data must be ASCII encoded.  The included examples go into a lot more detail of usage, so I'm not going to put that information here, but you can set custom headers, and set custom request methods, and custom post data, and turn standard output messages on and off. 

 

It looks like attempting to detect the charset given to sapp_curl_1.dll and then convert to utf-8 is a problem that would require quite a bit of work.  I'm unsure how lua is deriving the charset from any data that is passed to it, so it could possibly be any charset.  Making a list of all the ascii characters and making sure that the data you send is one of those characters seems like an easy fix, but no special characters are able to get sent.  If there was a way to force lua to encode a particular string with a certain charset all the time once it's sent to sapp_curl_1.dll it would make life a lot easier, but there are also ways of detecting charsets within a C/C++ program, however there are quite a few that would need to be set up to be detected.  I even went so far as to think that possibly url encoding then url decoding the string, or somewhere in the process of doing that might make this easier to figure out https://help.interfaceware.com/code/details/urlcode-lua send all post data as a url encoded string, then you could decode it once it was passed to the dll, however that seems like a wild goose chase.  I've hit a point where I need to learn more before being able to make improvements: https://stackoverflow.com/questions/16208079/how-to-work-with-utf-8-in-c-conversion-from-other-encodings-to-utf-8  

 

Nevermind the issue above is solved in with this stackoverflow question: https://stackoverflow.com/questions/16624036/how-to-convert-windows-1256-to-utf-8-in-lua it probably works for both halo chat and names chosen in halo.  There is also probably some usage of lua's string.byte() that can be done beforehand to make sure you aren't trying to convert a string with a character represented by a byte valuer greater than 255.  It makes me think about the halo name hack program and I'm wondering if it would inject characters into a player's name that have a byte value greater than 255, I don't know enough about how the halo client/server handles names and chat strings in order to give a solution that will always work even if someone was being naughty with the strings that you can get from get_var(PlayerIndex,"$name") and also the Message parameter from register_callback(cb["EVENT_CHAT"], "OnPlayerChat") in sapp.

 

sapp_curl_2.7z is the current version:

I fixed a dumb mistake by making sure: curl_slist_free_all(chunk); gets called, this probably would have been bad over long periods of use.

 

I did some unorthodox stress testing attempting to see how reliable the dll was.  I did ~500,000 requests at ~250 requests per second with everything being done over localhost, and 99 header rules, and 1000 characters of data per request.  So far as I can tell the memory usage of the machine I was running it on while running with wine did not increase as if there was a leak.  Connecting to the halo server through localhost while this was going on with 1 player it seemed as everything was normal, which seemed kind of comical because haloded.exe was hovering around 20% cpu usage (i5 6600k).

 

Errors:

As of the time of writing this there is no error reporting for curl within the dll, you are at the mercy of the api endpoint to give relevant errors, but an empty string "" will be returned if curl fails for whatever reason.  An example would be the webserver not being able to be contacted because there is no webserver running at the hostname of the url does not exist, curl will return "" after a pre-defined amount time because the webserver did not exist.  This could happen for any number of reasons, like a dns lookup failed, or you aren't connected to the internet, trying to request from exampel.com instead of example.com, etc...

 

The name of the sapp_curl_x.dll is arbitrary, and in order to load whichever version you downloaded:

curl_client = ffi.load("sapp_curl_x")  replace the name within the quotes with the file name of the dll, you can also rename it to anything and use that name instead, and I don't think you have to put .dll on the end in order for lua to load it.

The dll should be put in the same directory that contains the halo server i.e. the same location of sapp.dll and haloded

 

Todo list:

File posting https://curl.se/libcurl/c/postit2.html

curl_slist_free_all(chunk);  needs to be added when custom headers are specified (Done)

 

Reason for making this and future thoughts:

I wanted a portable solution that could contact http api's that was also not going to lag the halo server.  The sapp_http_client is a better option if you only need to make GET requests.

I'm still wondering whether this sapp curl dll would be compatible with a service like elite game servers, I'm hoping / praying that they have a way to use the VC++ 2015 runtime and will accept the .dll if they are asked to upload it to a service.

If services like elite game servers will accept the dll then that opens up some lua programming freedom on EGS for script writers to create interactions with http api's that require more than the GET method.  EGS doesn't care too much if you upload any lua script by yourself and write your own lua scripts for sapp.  But they disallow uploading of .dll files unless you open a ticket with them.  The sapp_http_client has already been added to their list of mods that can be "one click" installed, so I'm hoping this can get on that list too, but I would like someone to review the dllmain.cpp before that happens.  Of course improper use of the dll within lua will crash the server, and it could be warped and twisted into a cpu sucking program if used improperly in lua.  I'm hoping it's treated with respect and not abused, waiting 250ms between initiating and processing a request is just an arbitrary number that I put in the examples, but changing that number is up to the lua programmer to decide.

 

curl_test.7z

curl_test_wine.dll.7z

curl_test_queue.7z

curl_test_headers.7z

sapp_curl_1.7z

sapp_curl_2.7z

Edited by mouseboyx
Clarification; Wine problems; Other notes; Thanks; Queue; Errors; Custom stuff; Character encoding; Reason and Future
Chalwk, Enclusion, Kavawuvi and 1 other like this

Share this post


Link to post
Share on other sites

Tiddy-bits:

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0
  • Recently Browsing   0 members

    No registered users viewing this page.