Building a Position Calculator on FIX messages in Flink
This past quarter, we had on our backlog a task to build an example Flink application capable of processing millions of transactions for capital markets. We chose to build a position calculator on top of FIX transaction data. FIX is widely used across the capital markets industry and would be a format participants instantly recognize. Along the way, we answered a different question as well:
Can we build this application in a day?
The developer who picked up the task has worked with large-scale data analytics platforms for decades, and has extensive experience with cloud technologies as well. However, he had not worked with Confluent Cloud previously (only with Confluent Platform). He opted to deploy on AWS MSK/MSF, because that is what he knew. We planned to use Flink SQL, partly to evaluate it’s usefulness for such use cases. After several days of trying to manage data ingestion and SQL execution on MSK and MSF, it was clear that this would be harder than anticipated. In one of our stand-ups, we discussed trying Confluent Cloud Flink instead. By that afternoon, the demo was up and running. Developer feedback on Confluent Cloud Kafka/Flink is that it is easy to use and the IDE helps a user get going and troubleshoot. There are cases where you may opt for a self-managed environment, but for most use cases with a million events a minute, and analytical logic that can be expressed as SQL, Confluent Cloud does seem to be the fastest way to get Flink up and running.
A demo of the Flink position keeper is below along with more information on how it was built.
Position Calculator on Flink SQL
As trading volumes have grown across global markets, legacy order management and reporting systems have struggled to keep up with client demands for advanced features and real time visibility. This leads to client dissatisfaction and loss of market share to competitors with more modern systems capable of processing larger volumes at lower latencies.
This demo provides an example of a simple position-keeping system that can handle very high volumes, built on Confluent Cloud’s (CC) scalable stream processing platform. The same platform can be used to build more complex logic, such as that required for order management and trading systems. The tech stack can process millions of market data or trade events per minute on auto-scaling infrastructure. Lower latencies can be achieved through tuning and dedicated infrastructure. This allows broker dealers to build scalable market-making, algorithmic trading, prime brokerage services, position tickers, allocation and post-trade reporting systems.
Architecture diagram for a simple position calculator application in Flink SQL
FIX Messages
A widely used industry format
Our first step is to take FIX messages and publish them to Kafka as JSON using a python application.
Our design uses sample data from NYSE. Once FIX messages have been transformed into a format native to Kafka/Flink (JSON in this case), they are available for us to view and manipulate using the IDE in Confluent Cloud
Position Calc
Flink excels at filter and aggregation functions on large streams of data. A position is an aggregation on a stream of trade events. The logic for our demo is expressed in simple SQL, and Confluent Cloud provides a convenient IDE and UI to test and deploy statements.
Results
We pass 3 million FIX messages to Kafka and demonstrate how CC Flink scales to handle additional volume and meet latency targets. In our example 3 million messages are processed by Flink SQL within 5 minutes. The cluster auto-scales from 1 CFU to 4 to handle this load.
Confluent Cloud’s state management and auto-scaling capabilities mean your cluster also automatically shrinks in response to lower volume and restarts with state preserved so you pick up from where the system left off. This can we exceptionally efficient for bursty workloads , for example higher transaction activity during the trading day.
Write to us at info@greyker.com to learn more!
