(the Last) StockFighter

I am writing this post in the hopes that someone doing a search will find useful information on debugging encrypted API calls.

How do you debug a request if the conversation is encrypted?  This isn't about figuring what version of TLS the server supports.  It is about the content of the request not being recognized by the API.

My technique is to intercept the conversation between the browser and the server and try to duplicate a working request.  You have to have a functioning example API call to make this work, but in this case we have the API docs to work with.  The example ruby code for a bid or offer is here:
https://starfighter.readme.io/docs/getting-started

The mechanics of this request can be investigated with ngrep:
http://sourceforge.net/projects/ngrep/

This is a tool like tcpdump that produces easy to read network captures.  Install it like so:

Download it:

[root@sl7-vm ~]# wget "http://downloads.sourceforge.net/project/ngrep/ngrep/1.45/ngrep-1.45.tar.bz2?r=http%3A%2F%2Fngrep.sourceforge.net%2Fdownload.html&ts=1452879259&use_mirror=netassist"

unpack it:

[root@sl7-vm ngrep]# tar xvfj ngrep-1.45.tar.bz2


Patch it:
[root@sl7-vm ngrep-1.45]# wget http://sourceforge.net/p/ngrep/bugs/_discuss/thread/ee4219d2/db82/attachment/patch
--2016-01-15 18:38:18--  http://sourceforge.net/p/ngrep/bugs/_discuss/thread/ee4219d2/db82/attachment/patch
Resolving sourceforge.net (sourceforge.net)... 216.34.181.60
Connecting to sourceforge.net (sourceforge.net)|216.34.181.60|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘patch’

[root@sl7-vm ngrep-1.45]# patch < ./patch
[root@sl7-vm ngrep-1.45]# autoconf configure.in > configure

Install the prerequisites:
[root@sl7-vm ngrep-1.45]# yum install libpcap-devel

Configure it, build it, and install it:
[root@sl7-vm ngrep-1.45]# ./configure && make && make install

you should see this:

gcc -g -O2 -DLINUX -DHAVE_CONFIG_H  -D_BSD_SOURCE=1 -D__FAVOR_BSD=1  -I. -I/usr/include  -g -c ngrep.c
gcc -g -O2 -DLINUX -DHAVE_CONFIG_H  -D_BSD_SOURCE=1 -D__FAVOR_BSD=1  -L/usr/lib -s -o ngrep ngrep.o  regex-0.12/regex.o -lpcap
./install-sh -c -m 0755 ngrep  ///bin/ngrep
./install-sh -c -m 0644 ngrep.8 ///share/man/man8/ngrep.8

Got to love that autoconf.  Three slashes are better than one.

So what do we do with it?  This is the tricky part.  You are going to need stunnel for this part:
[root@sl7-vm ~]# yum install stunnel

make a certificate:
[root@sl7-vm mitm]# openssl req -batch -new -x509 -days 365 -nodes -out server.pem -keyout server.pem

now create a config file that looks like this:

[root@sl7-vm mitm]# cat stunnel-mitm-proxy.conf
debug = 3
#foreground = yes
pid =

[server]
client = no
cert= ./server.pem
accept = 127.0.0.1:443
connect = 127.0.0.1:4434

[client]
client = yes
accept = 127.0.0.1:4434
connect = api.stockfighter.io:443


This will create a Man In The Middle proxy server that will permit you to inspect traffic that would otherwise be end-to-end encrypted.  In order to use this proxy server you will need to either change the address you post to in your code to 127.0.0.1 or alter the /etc/hosts file like so:
127.0.0.1         api.stockfighter.io

This looks like something I will forget, so I opted to change my code.

You now need to run the proxy as root:
stunnel stunnel-mitm-proxy.conf

you can now start ngrep like so:
ngrep -W byline -d lo port 4434

This will look like nothing is happening.  That is fine just start your request in another window:

[starfighter@sl7-vm stockfighter]$ ruby first-trade.rb

You should see something like this:

#########
T 127.0.0.1:43643 -> 127.0.0.1:4434 [AP]
POST /ob/api/venues/AIDEX/stocks/LPEI/orders HTTP/1.1.
X-Starfighter-Authorization: REDACTED.
Connection: close.
Host: 127.0.0.1.
Content-Length: 116.
Content-Type: application/x-www-form-urlencoded.
.

##
T 127.0.0.1:43643 -> 127.0.0.1:4434 [AP]
{"account":"REDACTED","venue":"AIDEX","symbol":"LPEI","price":9000,"qty":100,"direction":"buy","orderType":"limit"}
##
T 127.0.0.1:4434 -> 127.0.0.1:43643 [AP]
HTTP/1.1 200 OK.
Server: nginx/1.8.0.
Date: Wed, 20 Jan 2016 14:59:33 GMT.
Content-Type: application/json.
Content-Length: 289.
Connection: keep-alive.
Strict-Transport-Security: max-age=31536000; includeSubdomains.
.
{
  "ok": true,
  "symbol": "LPEI",
  "venue": "AIDEX",
  "direction": "buy",
  "originalQty": 100,
  "qty": 100,
  "price": 9000,
  "orderType": "limit",
  "id": 6681,
  "account": "REDACTED",
  "ts": "2016-01-20T14:59:11.212784703Z",
  "fills": [],
  "totalFilled": 0,
  "open": true
}

If you are like me, you tried to do the same thing with wget and you saw this:
 {"ok":false,"error":"json: cannot unmarshal string into Go value of type main.OrderDesc"}

This puzzled me for a bit, but read on for a solution.  When I ran my wget command, using what I thought was the correct syntax  from the man page for wget, I got this in the ngrep window:
########
T 127.0.0.1:43625 -> 127.0.0.1:4434 [AP]
POST /ob/api/venues/JOGDEX/stocks/PIZ/orders HTTP/1.1.
User-Agent: Wget/1.14 (linux-gnu).
Accept: */*.
Host: 127.0.0.1.
Connection: Keep-Alive.
Content-Type: application/x-www-form-urlencoded.
Content-Length: 94.
X-Starfighter-Authorization: REDACTED.
.

##
T 127.0.0.1:43625 -> 127.0.0.1:4434 [AP]
"account=REDACTED&venue=JOGDEX&symbol=PIZ&price=5500&qty=100&direction=buy&orderType=limit"
##
T 127.0.0.1:4434 -> 127.0.0.1:43625 [AP]
HTTP/1.1 404 Not Found.
Server: nginx/1.8.0.
Date: Wed, 20 Jan 2016 14:39:38 GMT.
Content-Type: text/plain; charset=utf-8.
Content-Length: 62.
Connection: keep-alive.
.

So I set to work altering my code to look like the functional ruby script:

#export base_url="https://api.stockfighter.io/ob/api"
export base_url="https://127.0.0.1/ob/api"

# Set up the order
# must look like this:
# {"account":"REDACTED","venue":"AIDEX","symbol":"LPEI","price":5500,"qty":25,"direction":"buy","orderType":"limit"}

export order='{"account":"'$account'","venue":"'$venue'","symbol":"'$stock'","price":'$price',"qty":'$qty',"direction":"'$direction'","orderType":"'$orderType'"}'
export poststring="$base_url/venues/$venue/stocks/$stock/orders"
export content_type="Content-Type: application/json"
export headers="X-Starfighter-Authorization: $apikey"
wget --no-check-certificate --header=$content_type --header='X-Starfighter-Authorization: '$apikey'' --post-data=$order "$poststring"


That looks pretty gruesome, but it works.  This is the result:

 {
  "ok": true,
  "symbol": "LPEI",
  "venue": "AIDEX",
  "direction": "buy",
  "originalQty": 100,
  "qty": 100,
  "price": 9000,
  "orderType": "limit",
  "id": 6681,
  "account": "REDACTED",
  "ts": "2016-01-20T14:59:11.212784703Z",
  "fills": [],
  "totalFilled": 0,
  "open": true
}

Don't forget to change your code or the /etc/hosts file back.

I'm back after a hectic holiday calendar and a bad flu. Fortunately in the intervening four weeks the platform has become more stable so we should be able to make more progress.

I just read an article about what it takes to be a stock broker at one of the biggest brokerage firms on wall street:

http://www.zerohedge.com/news/2016-01-14/so-you-want-be-citadel-trader-here-are-requirements

Note that "human" trading is not even mentioned in the posting:

Quantitative Trading Systems Developer: Undergraduate, Master’s, PhD and Postdoc (New Grads)

Location: Chicago, IL

Are you a natural programmer who loves to optimize and make systems more efficient? Do you have a sophisticated computational skillset – code optimization, systems architecture and library design? Do you know your core languages like you do your native tongue?

At Citadel, Quantitative Trading System Developers are responsible for designing, building, and optimizing our automated trading platform. Developers work closely with our research, simulation and live trading teams. We are seeking top undergraduate, master’s, and PhD students who are entrepreneurial self-starters and enjoy being in a fast-paced and dynamic environment for exciting opportunities in our automated quantitative trading businesses. This opportunity offers excellent exposure to a quantitative trading career path in one of the world’s leading global financial institutions.

If this is what excites you:

• Developing core trading infrastructure
• Designing low-latency, high-throughput trading systems
• Optimizing massive parallel computing platforms.

And this is what you’ve got:

• Advanced training in Computer Science, Computer Engineering, or other related fields
• Extensive programming experience with strong object oriented design skills and fluency in C, C++, or Java
• Expertise with algorithms and data structures
• Demonstrated ability to communicate complex ideas in a clear, concise fashion
• Ability to thrive in a complex, fast-paced, and highly technical environment

Useful:

• Scripting languages (e.g., Linux shell, Python, Perl)
• Analytical packages (e.g., R, Matlab)
• FPGA development and high performance computing
• NoSQL databases (e.g., MongoDB)
• Distributed computing using MapReduce
• Multi-threaded programming
• Network programming

What caught my eye was the fact that not a single one of the 189 participants that have chosen to make their github repositories public have decided to write in R. R programming language is mature and stable and has an active user community. It is listed as one of the "useful" skills in the above job description. Note that the required skills are C, C++, and Java. Not a single one of the participants chose to write in C or C++ and only 13 of 189 or 6.8% chose to write in Java. In addition if you write in Python ( chosen by 15% of the participants ) you are considered to be writing in a "scripting" language. This word "scripting" is a pejorative that means that you are not writing a "real" program. It is somehow not as good as a real honest "man's" programming language. This is a slap in the face to all the Python, Perl and Bash programmer out there who's collective work holds the internet together. The arrogance and dismissive attitude galls me. I have written primarily in Bash for many years and I consider my work to be professional and of tremendous value.

One might conclude that the participants either do not wish to be hired, or think that their skill in other languages will be thought fungible. In light of the fact that the Stockfighter challenge bills itself as a job placement web site, I have to conclude the latter. My experience with human resource departments at large institutions is that they are inflexible and frankly incompetent. They cannot make a determination about the technical skills or conversely the trainability of candidates. In general if you know one language it is pretty easy to pick up another one in a matter of weeks. The current buzz word is "Object Orientation" and the job description mentions only C++ and Java as candidate OO languages. Given the huge academic and industry move to Java as a standard over the last 15 years or so I think it is a given that they are only looking for skill in Java and most likely specifically Hibernate + Spring + Maven or Java EE with all the trimmings. These frameworks take years to learn and genuinely are not fungible skills. Moreover, this type of programming damages your brain making you incapable of writing small efficient programs that can, for instance, trade stocks quickly.

Á Propos of my previous discussion on the stability and maturity of Python; it seems to me that there is an emphasis on Time to Market in the brokerage business rather than maintainability. This is the same pressure I saw in the advertising industry where web sites are frequently made live for a period of a few hours and never used again. My goal here is to write Functional Programs. To a considerable extent these Programming Paradigms are fads that come and go. Industry has latched on to Object Orientation because it allows them to write spaghetti code in a maintainable way. Functional programming by it's very nature produces clean compartmentalized code without excessive loops or branches. I write in R because it is stable and mature and above all because it is a rapid development environment. Functional Programming is a desirable, marketable life skill. I leave you with this excerpt from the Wikipedia page on Programming paradigms:

Parallel Computing

Main article: Parallel computing

Carnegie-Mellon University Professor Robert Harper in March 2011 wrote: "This semester Dan Licata and I are co-teaching a new course on functional programming for first-year prospective CS majors... Object-oriented programming is eliminated entirely from the introductory curriculum, because it is both anti-modular and anti-parallel by its very nature, and hence unsuitable for a modern CS curriculum. A proposed new course on object-oriented design methodology will be offered at the sophomore level for those students who wish to study this topic.