Data processing at the highest level

Over the past two years, Hidde van Heijst (Team Lead Data Engineering) and his colleagues have achieved impressive changes within their data architecture, which are unique in the Dutch data landscape. The result? A department of T-shaped talents processing huge amounts of data from various suppliers. Linking applications and creating security best practices: in this blog, Hidde talks about his job as a Data Engineer at Nederlandse Loterij.

Processing event-driven data

Since the online gaming market opened up two years ago, Hidde and his team have been facing serious data processing challenges. “We process event-driven data for TOTO. For example, when a player places a bet on a football match or plays a slot machine in a casino, it generates a message, also known as an event. When a match or bet is completed, it also results in an event. We regularly deal with peaks of tens of thousands of events per minute. To handle this efficiently, we have divided our platform into domains with unique characteristics, so these can scale up independently. When a Champions League match is played, Sportsbook will automatically scale up, without affecting other processes such as Casino or the Payments domain.”

This is exactly one of the biggest challenges: Nederlandse Loterij uses third-party systems for online casinos and sports betting. “When a player places a bet in the casino, he interacts with the system that manages their ‘wallet’. Money is withdrawn when a bet is placed. If you win, money is automatically added to the wallet. All this activity ends up on a vendor’s Kafka cluster. We consume the data from these Kafka clusters on behalf of Nederlandse Loterij. We utilize Event Sourcing to process the data into the organisation’s domain model.”

Event Sourcing

What does data from a vendor imply for Nederlandse Loterij? And how do data experts name a particular field, for example? A lot of thought goes into processes like this for Hidde and his team. “We store all the events we receive in an immutable database. Every new event that arrives, triggers a process. All events are replayed in order, so you always know the latest status. Think of a player’s account; all updates – such as a change of address – lead to the latest version of someone’s account.”

What is the main advantage of this approach? “It helps us to reconstruct sequencing, because we have no guarantee on ordering from the supplier. It is also useful for auditing. For example, time travel allows us to see what an account looked like a week ago.”

What happens next? “The NLO has other applications that use our API and Event Hubs. It allows them to act on anything suspicious in real-time. Usecases are anti-fraud and money laundering, but also to meet our duty of care in games of chance. Besides, by using Azure Data Explorer, analysts can come to us directly for queries and dashboards.

Security best practices

All the systems that Hidde and his colleagues use, need to communicate securely. “To ensure the security of a large amount of sensitive data, one of the things we use is identity-based access. And we strictly use private networks. We carry out penetration tests regularly, to ensure that our systems are airtight in terms of security. You have to be able to work according to security best practices to work well in our team.”

Working method of the team

All teams within NLO work according to Agile guidelines. “We start the day with a stand-up, where we go over current projects together. During these meetings, we check for any unforeseen issues and discuss the team’s progress. Problems can be anything, such as suppliers changing their data model without notice, resulting in events no longer being validated and ending up in a dead-letter queue. We are also trained to spot unusual patterns. If we see a big spike in traffic that we cannot explain, we figure out why.”

Although Hidde tries to keep his schedule clear to work on projects, there is also plenty of time to share knowledge within the Data Management team. “Every week, we have a refinement and every other week we have a team day. During this day, a demo is shared, and we have a retrospective to share knowledge and experiences between the various data teams.”

T-shaped profiles

The complex data issues within Nederlandse Loterij, require all-round experts, or, in other words: people with T-shaped profiles. Hidde himself is a good example. “As an Engineer, I need to have programming skills, but I also need expertise in security, Devops and Cloud Architecture. This is crucial, because you need to understand how to set up systems in a scalable way.”

According to Hidde, what makes Nederlandse Loterij unique is that “we are given the freedom to set up our own initiatives. An example of such an initiative is the creation of an operations team. Our team pointed out that certain security practices recur everywhere, and that we need standards in terms of cloud infrastructure. The operations team is now working on those standards, ensuring better security across the organisation. Teams can easily adopt these standards, which has many benefits in terms of security, management, and cost.”

Want to explore the data vision at Nederlandse Loterij? Read all about it here. Or would you rather discover whether we have a vacancy that suits you? Check out our vacancies here.