Showcasing the Betfair Exchange at Kafka Summit, London

ks

A few months ago, I had a unique opportunity to represent Betfair at the prestigious Kafka Summit in London where I presented the story behind how the Betfair Exchange evolved technologically from a monolithic system with serious reliability problems that was tightly coupled to an RDBMS, to a modern, event-driven, scalable and reliable platform, that is the backbone of our global business today. The talk mainly focused on how Kafka has played and continues to play a central role in this transformation. It was a fifty minute talk at the end of a long day of presentations and I was the only one standing between the audience and a cocktail party at the time. However, I was both surprised and proud of the turnout at this session, which I was told was the largest the Use cases stream of talks, for the entire day and I thoroughly enjoyed presenting our work to the packed audience. Given the time, the presentation had a deliberately casual tone,  but the technical crux of the presentation was not lost on anyone. After the presentation I had the opportunity to interact with architects and senior developers from various organisations across the globe, who were interested in delving deeper into the technical nitty-gritties of our system and in discussing how they could solve their own problems, using some of the patterns we have used at Betfair.

Here is a brief abstract of the presentation, Confluent hosted video recording, standalone video and the slides from the presentation.

Kafka Summit is the premier event for data architects, engineers, devops professionals, and developers who want to learn about streaming data. It brings the Apache Kafka community together to share best practices, write code, and discuss the future of streaming technologies. I was really privileged to represent Betfair here and showcase what is a truly world class distributed system implementation built around Kafka, more importantly presenting the outcome of painstaking and diligent hard work of the entire Exchange Platform development team, here in London, over a number of years.

If you like to be part of this journey as we continue to evolve the Exchange towards an exciting future, apply here.

Showcasing the Betfair Exchange at Kafka Summit, London

Exchange Streaming – The journey so far

Betfair is the largest on-line betting exchange in the world and has seen tremendous growth in the past few years. One of the fundamental problems that comes with growth is that the technology that drives the exchange has to scale in line with not just current growth rates but also the forecast growth over next X months/years.

What also complicates the matter is that growth is a function of many variables, including but not limited to number of users, jurisdictions, market offerings and the list goes on. So the goal post is constantly moving for us. Whilst it is a massive challenge, in some ways it is also a nice problem to have. As noted in a previous post, the current exchange back-end architecture, whilst still proving its worth everyday, puts some limitations on rapid scale outs and on our ability to deliver new capabilities to maintain the edge in a highly competitive market. We have been slowly migrating the entire back-end architecture to be event driven, and we recently achieved another milestone in this path. A new stream of core exchange activity, that provides the lowest level of transactional activity ticks that occur in the exchange, in near real time, was deployed live after some painstaking work over a few months. This is a significant step forward for us as it essentially opened up options for us to migrate a lot of our internal service applications to source the exchange activity data from the stream, instead of relying on some legacy sources which were highly reliant on a very heavily used database. More significantly, it also allows us to build similar capabilities exposed to our end users. The detailed benefits of this, can probably be discussed in another entry, but this is no doubt a step change for us. Continue reading “Exchange Streaming – The journey so far”

Exchange Streaming – The journey so far