I have been fascinated by the markets already for more than ten years and along the journey have occasionally beaten the market, but more often the market has beaten me. The function of the market is to match the supply and demand to find the price of the day, the hour, or even the millisecond. The current price is an outcome of millions of participants taking part in the market by selling and buying where participants are more increasingly likely to be bots utilizing various means to find an edge.
This will be an article about how to start your algorithmic journey into markets by streaming latest forex prices from cTrader into your own datastore using Quickfix library for Python. I try to keep things as practical as possible. IC Markets offer demo accounts for cTrader which will provide a good starting point. The data is not be tick data and tick data has to be acquired from other sources.
The reason why I decided to write this article is that I had difficulties finding any decent documentation about the usage of Quickfix and especially regarding FIX API implementations. I also perhaps intend to expand this into a set of articles where this data will eventually be used to take trades in the markets.
The reader has to be familiar with Python to be able to follow the tutorial. Quickfix and FIX API also will take time to become familiar, but I think this article is all about trying to give that insight. Also it seems that the Quickfix library will not build in Windows platform, so the reader needs either Linux or OSX to run the project locally. The last requirement is to open a forex account in IC Markets, or any other provider with a FIX API. To my understanding the API implementations may vary slightly between providers and this article applies only to IC Markets.
cTrader FIX API specification here: https://help.ctrader.com/fix/specs/cTraderFixApi_v2.18.1.pdf
If you have any issues with the FIX API, check debugging section at the end of this tutorial.
Create the project
Clone project template from Github: https://github.com/joni-mikkola/forex-quotes-template
Build and install the latest quickfix library with command pip install quickfix. Quickfix is a FIX protocol implementation which we will use to create a socket connection to cTrader server. The installation can take quite long and a lower end computer.
There are two important files here we use for communicating with the counterparty server:
- spec/FIX44.xml contains the specification for cTrader FIX interface and received or sent data has to match this specification
- client.cfg contains configuration for the socket connection
At this point you should log in to your cTrader account, find your FIX API details and fill in Username, Password and SenderCompID fields in the client.cfg file.
Start by creating main.py with following code. We will start then building on top of this.
import asyncio
import logging
import quickfix
import quickfix44 as fix44
from model.event import Event, EventType
from client import QuoteClient
from helpers.fixhelper import FixHelper
from model.logger import setup_logger
__SOH__ = chr(1)
setup_logger('logfix', './_logs/quote_message.log')
logfix = logging.getLogger('logfix')
class FixApp(quickfix.Application):
def __init__(self, config):
super().__init__()
self.settings = quickfix.SessionSettings(config)
self.storefactory = quickfix.FileStoreFactory(self.settings)
self.logfactory = quickfix.FileLogFactory(self.settings)
self.initiator = quickfix.SocketInitiator(
self, self.storefactory, self.settings, self.logfactory)
self.initiator.start()
def onCreate(self, sessionID):
print("onCreate : Session (%s)" % sessionID.toString())
def onLogon(self, sessionID):
self.sessionID = sessionID
print("Successful Logon to session '%s'." % sessionID.toString())
asyncio.run_coroutine_threadsafe(queue.put(Event(EventType.LOGIN)), loop)
def onLogout(self, sessionID):
print("Session (%s) logout !" % sessionID.toString())
def toAdmin(self, message, sessionID):
print("toAdmin")
def fromAdmin(self, message, sessionID):
print("fromAdmin")
def toApp(self, message, sessionID):
print("toApp")
def fromApp(self, message, sessionID):
print("fromApp")
#client = QuoteClient()
queue = asyncio.Queue()
loop = None
async def update():
while True:
event: Event = await queue.get()
if event.type == EventType.LOGIN:
print("successful login")
#client.quote(fixApp.sessionID, '1')
elif event.type == EventType.QUOTE:
print(f"quote {event.value}")
if __name__ == '__main__':
fixApp = FixApp('client.cfg')
loop = asyncio.new_event_loop()
loop.run_until_complete(update())
loop.close()
Notice that the FixApp class implements quickfix.Application’s interfaces. The documentation for each of the interface can be found here: https://www.quickfixj.org/usermanual/2.3.0/usage/application.html. If you run the app, you should get Session (FIX.4.4:{your_id}->cServer) logout ! message. Next we need to add credentials to headers and we’re interested in toAdmin interface.
This callback provides you with a peek at the administrative messages that are being sent from your FIX engine to the counter party. This is normally not useful for an application however it is provided for any logging you may wish to do. Notice that the FIX::Message is not const. This allows you to add fields to an administrative message before it is sent out.
We now add the username and password information to be delivered to cTrader for authentication.
def toAdmin(self, message, sessionID):
header = message.getHeader()
msg_type = quickfix.MsgType()
message.getHeader().getField(msg_type)
if msg_type.getString() is quickfix.MsgType_Logon:
message.setField(quickfix.ResetSeqNumFlag(True))
header.setField(quickfix.TargetSubID(
self.settings.get(sessionID).getString("TargetSubID")))
message.setField(quickfix.Password(
self.settings.get(sessionID).getString("Password")))
message.setField(quickfix.Username(
self.settings.get(sessionID).getString("Username")))When you now run the code, on the console you should see Successful Logon to session ‘FIX.4.4:demo.icmarkets.xxxxxxx->cServer’.
The way Quickfix messages are interpreted and are composed is quite complicated initially, but it will get easier over time. Reading specification XML file while writing the code will make things easier, but more on that later.
Now we have successfully formed connection to our counterparty and it is time to send out market data request command. Check out cTrader FIX documentation and at 5.4.2. Market Data Snapshot/Full Refresh for more details.
Next we should create a new class called QuoteClient with following code:
import quickfix as fix
import quickfix44 as fix44
class QuoteClient():
def quote(self, session_id, symbol):
message = fix.Message()
message.getHeader().setField(fix.BeginString(fix.BeginString_FIX44))
message.getHeader().setField(fix.MsgType(fix.MsgType_MarketDataRequest))
message.setField(fix.MDReqID('1'))
message.setField(fix.SubscriptionRequestType(
fix.SubscriptionRequestType_SNAPSHOT_PLUS_UPDATES))
message.setField(fix.MarketDepth(1))
message.setField(fix.NoMDEntryTypes(2))
message.setField(fix.MDUpdateType(
fix.MDUpdateType_INCREMENTAL_REFRESH))
group = fix44.MarketDataRequest().NoMDEntryTypes()
group.setField(fix.MDEntryType(fix.MDEntryType_BID))
message.addGroup(group)
group.setField(fix.MDEntryType(fix.MDEntryType_OFFER))
message.addGroup(group)
new_pairs = [symbol]
message.setField(fix.NoRelatedSym(len(new_pairs)))
requestSymbol = fix44.MarketDataRequest().NoRelatedSym()
for pair in new_pairs:
requestSymbol.setField(fix.Symbol(pair))
message.addGroup(requestSymbol)
fix.Session.sendToTarget(message, session_id)
The quote method takes session_id and symbol as parameters. The FixApp instance stores the session_id variable once the connection is established, but for symbol value we have to look inside cTrader app.
As the symbol information shows above, GBPJPY’s Symbol ID is 7 and that is what we’ll use for this exercise.
You might also notice in the code, that there is something different going on with NoMDEntryTypes and this is the way groups/arrays are expressed in QuickFIX. This to me seemed impossible to get right just on the basis of the documentation, but if you check the actual FIX44.xml provided in the spec folder, you can search for MarketDataRequest to see how the call should be built.
<message name="MarketDataRequest" msgtype="V" msgcat="app">
<field name="MDReqID" required="Y"/>
<field name="SubscriptionRequestType" required="Y"/>
<field name="MarketDepth" required="Y"/>
<field name="MDUpdateType" required="N"/>
<group name="NoMDEntryTypes" required="Y">
<field name="MDEntryType" required="Y"/>
</group>
<group name="NoRelatedSym" required="Y">
<field name="Symbol" required="Y"/>
</group>
</message>So the FIX44.xml is the actual specification that you should look for, and maybe use the PDF as complementary information. The quality of these FIX API implementation guides probably vary between different forex operators and cannot really comment on that, but the process with cTrader have worked like this.
Now armed with this information, you should already be able to form commands and send them forwards. Until now, we have formed connection to our counterparty successfully, and we can send commands. Next thing for use is to receive reply and proceed storing the information into database.
Once the message is sent (sendToTarget is called), it will proceed through toApp interface and from there to the counterparty. The reply then comes back through the fromApp interface where we need to interpret and ‘decompile’ it.
Next we need to add some code to fromApp method inside main.py.
def fromApp(self, message, sessionID):
msgType = FixHelper.get_header_field_value(quickfix.MsgType(), message)
if msgType is quickfix.MsgType_MarketDataSnapshotFullRefresh:
fix_no_entries = quickfix.NoMDEntries()
message.getField(fix_no_entries)
no_entries = fix_no_entries.getValue()
group = fix44.MarketDataIncrementalRefresh.NoMDEntries()
for i in range(1, no_entries):
message.getGroup(1, group)
price = FixHelper.get_field_value(quickfix.MDEntryPx(), group)
asyncio.run_coroutine_threadsafe(queue.put(Event(EventType.QUOTE, price)), loop)Having ready solution in front of your eyes may deceive you, but again parsing these messages can initially be quite cumbersome. Now as you are becoming quite proficient FIX44.xml specification reader, you can see the type MarketDataSnapshotFullRefresh here, and then intuitively know to look for details from the xml file. Knowing then that the price information is grouped, you can proceed to unpack it.
At this point you should have a working stream providing you price information from the host for your chosen symbols. You should be aware that the socket connection is not running in MainThread and that is why I have used asyncio queue’s so each of the messages from socket connection will be read in MainThread avoiding thread-safety issues.
As we are almost finished with this, I will add few words about debugging which probably is the most important part as you keep on extending this.
Debugging
If you have problems delivering messages or interpreting messages, you need to log the messages. If you want to debug messages coming from the counterparty, you can add the following code in the beginning to fromApp interface:
msg = message.toString().replace(__SOH__, "|")
logfix.info(f"fromApp {msg}")After running the code, you will receive something similar to this:
2022-10-30 14:59:04,164 : fromApp 8=FIX.4.4|9=128|35=W|34=2|49=cServer|50=QUOTE|52=20221030-12:59:04.078|56=demo.icmarkets.xxxxxxx|55=1|268=2|269=0|270=0.99665|269=1|270=0.99666|10=002|All you have to do, is to check out the corresponding values from FIX44.XML file. For example:
35=W
<field number="35" name="MsgType" type="STRING">
<value enum="W" description="MARKET_DATA_SNAPSHOT_FULL_REFRESH"/>With this you should be able to find the culprit for most of the issues you might have. Why I say most, is that you can have wrong FIX44.xml file that doesn’t match your FIX API provider and these will be harder to debug. Also I’ve read that some of the APIs simply doesn’t function as documented so maybe in these situation you have to reach out their helpdesk.