Cloud Platform Scalability & Optimisation

Challenge

Data Science and behaviour analysis

An intelligence-driven data science and behavior change company was experiencing scalability and performance bottlenecks in its existing data architecture. They needed to scale and optimize their platform to collect and analyze data from mobile devices.

The company had built an AWS-based cloud platform to process data from their mobile application SDK by collecting smartphone sensor data. This platform was designed according to a Lambda architecture with a speed layer for real-time processing and a batch layer for the daily processing of sensor data. While the platform leveraged cloud computing and modern technologies, there were still many performance and scalability bottlenecks, and this way, only a limited number of users were supported.

Contact usLook behind the scenes

Solution

In only three months, Klarrio

1

analyzed the existing cloud architecture,
2

conducted load tests to identify the performance bottlenecks,
3

this way resolving the most important bottlenecks in a short period of time,
4

improving the messaging layer from 4K to 200K supported persistent connections,
5

while the AWS cost remained the same despite the increase in scalability.

Results

200
From 4K to 200K persistent connections
50
more concurrent connections
0
difference in cloud infrastructure costs

Behind the scenes

Our customer built an AWS-based cloud platform to process data from their mobile application SDK by collecting smartphone sensor data.

It was designed according to a lambda architecture with both a speed layer for real-time processing and a batch layer for daily processing of sensor data.

After a cloud architecture analysis and load tests, we improved the messaging layer from 4k to 200k and supported persistent connections using Scala, Apache-Spark, Kafka and elasticsearch.

The Technology

  • Scala
  • Apache Spark
  • Kafka
  • Elasticsearch

The Expertise

  • Technical optimizations
  • Data science code adjustments
  • Messaging layer improvement

The Expert

At Klarrio we learned the hard way that the real scalability challenge goes beyond optimizing the CPU and memory of the current processing jobs. We found that the real issues lied in the resources that cannot be easily scaled up, such as network, single point of failures, recovering from failed components, read/write postgresql limitations, …
We really had to think outside the box here. Besides the technical optimizations, we also had to adjust the data science code written in Python, and solve queuing problems in the APIs between micro services as well as in Kafka caused by bursty traffic patterns.

Solving scalability issues without additional compute resources and costs is a tough, but fun challenge for any data engineer.

Bruno De BusCTO

Join us!

Want to work on similar projects?

Introverts and extroverts, geeks, nerds, and digital poets... Klarrio is the perfect place to learn and teach, experiment and brainstorm, exercise your brain, and feed your passion. Surrounded by people with amazing, world-changing talents.

We're hiring

Contact us!

We're your one-stop cloud-native partner

Big Data engineering. Data science. Data Analytics. Site Reliability Engineering. Consulting. And customized Open Source projects for companies of all sizes. What can Klarrio do for you today?

Contact us

Other Projects

Just a few projects examples.