Last summer, we opened vacancies for summer internships at our Porto office, Portugal. Our goal was to introduce students to technologies and development methodologies that we have here at PaddyPowerBetfair and present them the opportunity of working side-by-side with our team on a project that would not be forgotten in a matter of days.
This post focuses on the internship experience of Margarida, from the Faculty of Engineering of the University of Porto, and Pedro, from Polytechnic of Porto School of Engineering, who worked as a team member from June to September 2018.
At PaddyPowerBetfair, we make extensive use of Apache Kafka for building real time streams of data. We also make extensive use of Google Protobuf with our Kafka topics, as they provide a space and time efficient encoding/decoding algorithm of our data model.
As the Protobuf format is binary, it is difficult to manually inspect the content of a topic using a simple consumer (such as the console consumer available with Apache Kafka). How can we inspect (and produce) serialized content to a kafka topic in the most user-friendly way, even to non-technical people? That was exactly the challenge that we proposed to Margarida and Pedro in their first day of internship.
The goal of the internship was to build a tool, named
Kinspektor, that makes it possible to send and receive messages to/from Kafka queues and, if needed, serialize and deserialize said messages. The initial prototype of
Kinspektor has a user-friendly interface that allows us to write/read Protobuf in a human readable format.
The internship had an initial learning phase, not only of the new technologies (such as Apache Kafka and Protobuf) but also of the agile approach, the importance of unit, mock, integration testing, documentation and continuous integration/delivery throughout the development.
After defining the main requirements of the
Kinspektor prototype, we decided to use Scrum to manage this project. Consequently, we defined a set of initial user stories that bootstrapped the development process. The stand-up, planning, grooming and retrospective ceremonies of the Scrum methodology where followed.
The final product is, therefore,
- allows the user to send and receive serialized and plain text messages,
- allows the user to choose an available serialization/deserialization method
- allows the user to specify the Kafka
You can see the main layout of Kinspektor in the following image:
It also saves the messages sent and received in a session, making it possible to export a log of sent/received messages at any time.
It is possible to add new serialization/deserialization methods by providing a
.proto file. This functionality was the most difficult to implement because it requires compilation of the
.proto file into a
java file and, in turn, compile the
java file into a
class file, so that it could be added to the classpath. All of this has to be made at runtime, so that it is possible to use the serialization/deserialization method without restarting the application.
Kinspektor backend is implemented in Java (using Cougar framework) and
SQLite for the persistent layer. For the web frontend, React.
The following diagram shows the overall architecture of the
Kinspektor is split into a frontend, responsible for the presentation layer, and a backend that consumes and produces messages to Kafka queues.
The backend implements an array of serializers which are applied to the human readable format of the Protobufs and an array of deserializers which are applied to the binary serialized messages from the queue.
The SQLite database stores the available serialization and deserialization methods and a list of predefined Kafka hosts. These are configurable by the user. The Message Cache is there to prevent creating Kafka consumers every time the frontend polls for new messages on the queue. Instead, we create a consumer the first time a client polls for a topic and store the incoming messages in a memory cache. This prevents message duplicates as we do not need to manually track offsets (the consumer is always running).
Kinspektor is still a work in progress and we intend to further expand it to the community and turn it into a general use tool to inspect serialized message queues. It can be further developed to include more functionality, independence of the message queue implementation (such as RabbitMQ, ZeroMQ or ActiveMQ), and support other binary formats, such as BSON, Java serialization or MessagePack.
At the end of the internship, we reached out to Margarida and Pedro in order to hear about their experience at PaddyPowerBetfair.
“My overall experience at PaddyPowerBetfair was excellent, the environment was very friendly and open, and the work we did was challenging and interesting. Besides learning new technologies, we also learned how it is to work in a company, with all the good practices like doing documentation, testing and planning. Because all this, I would very much like to come back one day.”, Ana Margarida Silva
“My experience as an intern in PaddyPowerBetfair was much more than I could imagine. In 2 months, I learned new technologies and I applied a lot of things that I learned in the university. In a very friendly environment and in a team that was always available to help and to teach us. I loved the idea of working in a big company as PaddyPowerBetfair and I learned a lot about all the work methods and projects of the team I was in.”, Pedro Quintã