How to debug encrypted API calls

2016-01-18 15:59:35

I am writing this post in the hopes that someone doing a search will find useful information on debugging encrypted API calls.

How do you debug a request if the conversation is encrypted? This isn't about figuring what version of TLS the server supports. It is about the content of the request not being recognized by the API.

My technique is to intercept the conversation between the browser and the server and try to duplicate a working request. You have to have a functioning example API call to make this work, but in this case we have the API docs to work with. The example ruby code for a bid or offer is here:
https://starfighter.readme.io/docs/getting-started

The mechanics of this request can be investigated with ngrep:
http://sourceforge.net/projects/ngrep/

This is a tool like tcpdump that produces easy to read network captures. Install it like so:

Download it:

[root@sl7-vm ~]# wget
        "http://downloads.sourceforge.net/project/ngrep/ngrep/1.45/ngrep-1.45.tar.bz2?r=http%3A%2F%2Fngrep.sourceforge.net%2Fdownload.html&ts=1452879259&use_mirror=netassist"

unpack it:

[root@sl7-vm ngrep]# tar xvfj ngrep-1.45.tar.bz2

Patch it:

[root@sl7-vm ngrep-1.45]# wget
        http://sourceforge.net/p/ngrep/bugs/_discuss/thread/ee4219d2/db82/attachment/patch

        --2016-01-15 18:38:18--  http://sourceforge.net/p/ngrep/bugs/_discuss/thread/ee4219d2/db82/attachment/patch

        Resolving sourceforge.net (sourceforge.net)... 216.34.181.60

        Connecting to sourceforge.net (sourceforge.net)|216.34.181.60|:80... connected.

        HTTP request sent, awaiting response... 200 OK

        Length: unspecified [application/octet-stream]

        Saving to: ‘patch’

        

        [root@sl7-vm ngrep-1.45]# patch < ./patch

        [root@sl7-vm ngrep-1.45]# autoconf configure.in > configure

Install the prerequisites:
[root@sl7-vm ngrep-1.45]# yum install libpcap-devel

Configure it, build it, and install it:
[root@sl7-vm ngrep-1.45]# ./configure && make && make install

you should see this:

gcc -g -O2 -DLINUX -DHAVE_CONFIG_H  -D_BSD_SOURCE=1 -D__FAVOR_BSD=1  -I. -I/usr/include  -g -c
        ngrep.c

        gcc -g -O2 -DLINUX -DHAVE_CONFIG_H  -D_BSD_SOURCE=1 -D__FAVOR_BSD=1  -L/usr/lib -s -o ngrep ngrep.o 
        regex-0.12/regex.o -lpcap

        ./install-sh -c -m 0755 ngrep  ///bin/ngrep

        ./install-sh -c -m 0644 ngrep.8 ///share/man/man8/ngrep.8

Got to love that autoconf. Three slashes are better than one.

So what do we do with it? This is the tricky part. You are going to need stunnel for this part:

[root@sl7-vm ~]# yum install stunnel

make a certificate:
[root@sl7-vm mitm]# openssl req -batch -new -x509 -days 365 -nodes -out server.pem -keyout server.pem

now create a config file that looks like this:

[root@sl7-vm mitm]# cat stunnel-mitm-proxy.conf

        debug = 3

        #foreground = yes

        pid =

        

        [server]

        client = no

        cert= ./server.pem

        accept = 127.0.0.1:443

        connect = 127.0.0.1:4434

        

        [client]

        client = yes

        accept = 127.0.0.1:4434

        connect = api.stockfighter.io:443

This will create a Man In The Middle proxy server that will permit you to inspect traffic that would otherwise be end-to-end encrypted. In order to use this proxy server you will need to either change the address you post to in your code to 127.0.0.1 or alter the /etc/hosts file like so:

127.0.0.1         api.stockfighter.io

This looks like something I will forget, so I opted to change my code.

You now need to run the proxy as root:
stunnel stunnel-mitm-proxy.conf

you can now start ngrep like so:
ngrep -W byline -d lo port 4434

This will look like nothing is happening. That is fine just start your request in another window:

[starfighter@sl7-vm stockfighter]$ ruby first-trade.rb

You should see something like this:

#########

        T 127.0.0.1:43643 -> 127.0.0.1:4434 [AP]

        POST /ob/api/venues/AIDEX/stocks/LPEI/orders HTTP/1.1.

        X-Starfighter-Authorization: REDACTED.

        Connection: close.

        Host: 127.0.0.1.

        Content-Length: 116.

        Content-Type: application/x-www-form-urlencoded.

        .

        

        ##

        T 127.0.0.1:43643 -> 127.0.0.1:4434 [AP]

        {"account":"REDACTED","venue":"AIDEX","symbol":"LPEI","price":9000,"qty":100,"direction":"buy","orderType":"limit"}

        ##

        T 127.0.0.1:4434 -> 127.0.0.1:43643 [AP]

        HTTP/1.1 200 OK.

        Server: nginx/1.8.0.

        Date: Wed, 20 Jan 2016 14:59:33 GMT.

        Content-Type: application/json.

        Content-Length: 289.

        Connection: keep-alive.

        Strict-Transport-Security: max-age=31536000; includeSubdomains.

        .

        {

          "ok": true,

          "symbol": "LPEI",

          "venue": "AIDEX",

          "direction": "buy",

          "originalQty": 100,

          "qty": 100,

          "price": 9000,

          "orderType": "limit",

          "id": 6681,

          "account": "REDACTED",

          "ts": "2016-01-20T14:59:11.212784703Z",

          "fills": [],

          "totalFilled": 0,

          "open": true

        }

If you are like me, you tried to do the same thing with wget and you saw this:
{"ok":false,"error":"json: cannot unmarshal string into Go value of type main.OrderDesc"}

This puzzled me for a bit, but read on for a solution. When I ran my wget command, using what I thought was the correct syntax from the man page for wget, I got this in the ngrep window:

########

        T 127.0.0.1:43625 -> 127.0.0.1:4434 [AP]

        POST /ob/api/venues/JOGDEX/stocks/PIZ/orders HTTP/1.1.

        User-Agent: Wget/1.14 (linux-gnu).

        Accept: */*.

        Host: 127.0.0.1.

        Connection: Keep-Alive.

        Content-Type: application/x-www-form-urlencoded.

        Content-Length: 94.

        X-Starfighter-Authorization: REDACTED.

        .

        

        ##

        T 127.0.0.1:43625 -> 127.0.0.1:4434 [AP]

        "account=REDACTED&venue=JOGDEX&symbol=PIZ&price=5500&qty=100&direction=buy&orderType=limit"

        ##

        T 127.0.0.1:4434 -> 127.0.0.1:43625 [AP]

        HTTP/1.1 404 Not Found.

        Server: nginx/1.8.0.

        Date: Wed, 20 Jan 2016 14:39:38 GMT.

        Content-Type: text/plain; charset=utf-8.

        Content-Length: 62.

        Connection: keep-alive.

        .

So I set to work altering my code to look like the functional ruby script:



        #export base_url="https://api.stockfighter.io/ob/api"

        export base_url="https://127.0.0.1/ob/api"

        

        # Set up the order

        # must look like this:

        # {"account":"REDACTED","venue":"AIDEX","symbol":"LPEI","price":5500,"qty":25,"direction":"buy","orderType":"limit"}

        

        export
        order='{"account":"'$account'","venue":"'$venue'","symbol":"'$stock'","price":'$price',"qty":'$qty',"direction":"'$direction'","orderType":"'$orderType'"}'


        export poststring="$base_url/venues/$venue/stocks/$stock/orders"

        export content_type="Content-Type: application/json"

        export headers="X-Starfighter-Authorization: $apikey"

        wget --no-check-certificate --header=$content_type --header='X-Starfighter-Authorization: '$apikey'' --post-data=$order
        "$poststring"

That looks pretty gruesome, but it works. This is the result:

 {

          "ok": true,

          "symbol": "LPEI",

          "venue": "AIDEX",

          "direction": "buy",

          "originalQty": 100,

          "qty": 100,

          "price": 9000,

          "orderType": "limit",

          "id": 6681,

          "account": "REDACTED",

          "ts": "2016-01-20T14:59:11.212784703Z",

          "fills": [],

          "totalFilled": 0,

          "open": true

        }

Don't forget to change your code or the /etc/hosts file back.

Posted by StarFighter | Permanent link