Author at Infinum

Building a Tailored Recommendation System for Philips NutriU

Marin Krešo — Fri, 18 Oct 2019 10:05:00 +0000

Note: Since the publishing of this article, the Philips NutriU app has grown in scope and changed its name to HomeID. You can learn more about the project by visiting our work page.

Recommendation systems play a big part in our lives without us even realizing it. Famous web services like Youtube, Amazon, Netflix, and Spotify use them to make content suggestions tailored to a specific user, improving user experience by a significant margin.

These systems put together tailored playlists, suggest which movie to watch next, or reveal what products are often bought together.

Various services observe user actions like clicks and likes, and use the gathered data as input for the recommendation system to generate personalized recommendations. Using recommendation systems is a better and faster way of discovering relevant content for different people.

The award-winning mobile application NutriU is a companion app for Airfryers, blenders, and juicers. It allows users to control their appliances remotely but also helps them create and share recipes and find culinary inspiration.

When we set out to build a recommendation system for Philips’ NutriU, we realized that while there was plenty of literature online providing an overview of the algorithms and explaining how to train and evaluate such systems, there were scarce resources on how to build recommendation systems for production.

In this blog post, we take you behind the scenes of our project and show you how we’ve built a high-performance, fast and scalable recommendation system in a real-time setting.

If you’re considering integrating AI into your digital product, the first step is to evaluate your current state. Fill out our survey and get a free personalized assessment of your business’s readiness for AI implementation.

A recommendation system for recipes

Besides being a companion app for controlling Philips’ smart kitchen appliances, an integral part of the NutriU app is to provide a platform for users to add, browse, and share recipes.

With the previous solution, the user would open the home screen or the search screen, and the app would show them a certain number of recipes from the database.

What criteria were the recipes being shown based on? They were sorted according to simple, manually done criteria and then displayed top-to-bottom.

There were a couple of shortfalls to this approach:

All users would see exactly the same recipes (they are not personalized), therefore are not particularly interesting to the users
The food recipe app didn’t use the data about user interaction with the app and the recipes they were looking at in any way

They were deteriorating the user experience, hence there was a need for a recommendation system. It would be implemented as REST API in Python using the famous Flask library.

Philips had a couple of demands for the recommendation system:

Users should only get recommendations for recipes they had not seen before
Predictions of recommendations would have to be dynamic and include users’ recent actions, which means these recommendations could not be precalculated
Updates of the recommender model would have to be frequent and should not interrupt REST API in any way

Its majesty, data

For building a recommendation system, we needed its majesty, data. We were lucky that the NutriU already had an implemented collection of data about user interactions with the app-relevant recommendation system, such as recipe views, as well as recipe likes and comments etc.

We focused on making use of the data on recipe views. The amount of collected data for this interaction was more ample than for other interactions. Take into consideration that NutriU doesn’t have a “real” rating system, thus we only had implicit user feedback to work with.

NutriU used the Firebase platform for event tracking and analytics, so user interaction data was stored in BigQuery, the service which enables fast and interactive analysis of massive datasets. It also enables easy integration with Python’s most popular library for data analysis and manipulation–Pandas.

Acceptable execution time?

However, this is when we stumbled upon our first obstacle. Because we had to calculate recommendations dynamically, we had to fetch the data about user interactions every time anew. Although BigQuery runs blazing-fast SQL queries on gigabytes to petabytes of data, execution time for smaller queries is not that fast.

Let’s say you have 1 million records in your database. In that case, query execution time is around 2-3 seconds, which is amazing. However, if you have a lot less data, e.g. a hundred of records, query execution remains around 2-3 seconds. Not that fast anymore.

In fact, this pace was not acceptable for our recommender. Typically, using BigQuery as a main database in a web application is not suggested. So we had to build additional database which would regularly pull a relevant subset of data from BigQuery and communicate with REST API.

For teams dealing with this at scale, a lakehouse approach with Databricks offers a more integrated path by co-locating data, pipelines, and retrieval on a single, governed platform.

Lightning-fast recommendation calculus

There are several common approaches used for building a recommender system, the most popular being:

Collaborative filtering: This approach uses collaborative information, i.e. implicit or explicit user interaction with items (such as recipes viewed or liked). It doesn’t use information about the actual items or users.
Content-based approach: This approach works purely on user’s or item’s characteristics. They completely ignore interactions between users and items.
Hybrid approach: A combination of the previous two approaches.

There are also other, more advanced approaches, such as matrix factorization and sequential models created with deep learning. Taking each approach into consideration, the team finally settled on collaborative filtering for multiple reasons.

For starters, collaborative filtering is very interpretable, making it simple to explain why some recipe was recommended. It is based on the principle: “If users A and B liked recipe X, and user B likes recipe Y, there is a huge chance that user A will also like the recipe Y”. Furthermore, other approaches require a lot more work fine-tuning the models for predicting meaningful recommendations, while collaborative filtering works on the plug & play principle.

We implemented collaborative filtering with an implicit library. This amazing library has a simple interface, comes with support for implicit feedback datasets and all its operations and routines are vectorized and multi-threaded, so recommendation calculus is lightning fast.

However, the collaborative filtering method will not yield quality recommendations for cold-start users, i.e. the users who have had no previous interaction with recipes. To them, the app will recommend the recipes with the most views.

Putting theory into practice

The recommender system is implemented by microservice architecture. It is comprised of two components:

Scheduler
Rest API

Collaborative filtering algorithm requires calculation of similarity matrix which is then used to produce recommendations. Similarity matrix calculation is a time-consuming task, which is why we decided to create a scheduled task which will execute the whole process of similarity matrix calculation:

pull new data from BigQuery
store BigQuery data into PostgreSQL
use stored data to create similarity matrix

Rest API component responds to a user request for recommendations and returns the recommended recipes. We already mentioned that the model needs to be frequently updated without interruptions. Communication between the two components is achieved with Amazon’s S3 buckets.

The scheduler component generates the similarity matrix and saves it, while the Rest API component loads the generated matrix into the memory once it is generated for the faster serving of recommendations.

To avoid interruptions, Rest API runs a background job for loading new similarity matrix and switches it with the current one. We ensured that there could be only one job run in parallel.

We benchmarked recommendation request execution time for a cold-start user and for a non cold-start user. Specifically, we picked the user with the most interactions with the app. Benchmarks were created with 2,4 GHz Intel Core i7 processor:

Execution time for cold start user is 0.1 s
Execution time for non-cold start user is 0.05 s

Does the new NutriU work better?

To see if our recommender beats the previous solution, we conducted A/B testing experiments. Here are the results for Android.

The following are the results for the iOS platform.

Experiment results are crystal clear. Our proposed variant performed 168% better for the search screen, and 70% better for the home screen.

A simple choice for an efficient result

The solution described in this post uses one of the simplest algorithms for recommendations. Yet, it achieved impressive results in A/B experiments with additional gains in the interpretability of recommendations and extremely fast execution.

The intended takeaway from this blog post is that with careful and structured implementation, it is possible to significantly improve users engagement by using simple algorithms and create the foundation for future, more complex improvements.

If you’d like to learn how else we can utilize machine learning and artificial intelligence to help you achieve your business goals, check out our AI business solutions page.

The post Building a Tailored Recommendation System for Philips NutriU appeared first on Infinum.

Fast License Plate Concealment with Deep Learning

Marin Krešo — Fri, 01 Feb 2019 16:00:00 +0000

As technology advances, we have new and improved ways of dealing with various problems. One example would be the use of high-resolution cameras in traffic. They can be placed on congestion points for the purpose of traffic reports (so commuters would know which places to avoid on their way to work).

Also, we can monitor car flows for making a decision about the development of future roads, and use them for quick reactions in case of accidents or similar problems. These tasks are a lot easier if cameras have high resolution. But with that improvement, often we have a picture so clean that we can read license plates of the cars passing by.

Our task was to perform the detection and concealment of such readable license plates.

Why conceal license plates?

Besides running from police, one couldn’t think of a good reason for concealing license plate number. Well, that was maybe the case, but lately, there has been a lot of concern regarding the private data of individuals.

The main drive of this is the introduction of GDPR, a set of European Union regulations that govern how companies have to manage personal data. GDPR covers any data that can be used to identify a person, even indirectly. License plate numbers are personal information under GDPR because you can theoretically identify the owner of the car if you know their license plate number.

This could be exploited to track someone’s whereabouts, so we need an efficient way to conceal readable license plates. To conceal them, we must first be able to detect them. This is a complex task due to the continuous flow of cars passing by the cameras.

Because of its complexity and tediousness, this task is not suitable for humans, but it is perfect for a machine system. A system for this task needs to be fast, precise, and generalized to do the same thing on all cameras.

How do we build such a system? Well, the answer is a technology called Deep Learning. Because of simplified ways to collect huge amounts of data, machines that can perform fast computations, and advances in the field of AI, Deep Learning today is the defacto state-of-the-art approach for Computer Vision.

Overview of the system

The system is made up of three major parts:

Object detection
Landmark detection
REST API

system overview

Object detection

First, we want to detect all license plates in one frame (one frame could contain multiple cars). The output here is a bounding box of detected license plates for concealment. We have around 13.000 labeled images for this task.

Distribution of license plates on frames

We could build the model from scratch, but there are other already trained models on similar tasks which we could use. For this, we use the Tensorflow Object Detection API, because it makes it easy to construct, train and deploy object detection models. This API also has a number of different trained models; therefore we can choose the best suitable model for our needs. These models are not trained for detecting license plates, but we can use them and fine-tune them on our dataset for our specific task Transfer learning.

We mentioned that our system must be both fast and precise, so we have a trade-off between speed and accuracy. The most precise model for object detection is Faster R-CNN model, but the fastest one is SSD MobileNet (source). After experiments with both, we chose SSD MobileNet.

Although Faster R-CNN has magnificent accuracy, this model is very slow. Cameras send frames every few seconds, and we must process the images and detect all license plates in a short period of time so our system would be efficient. With advanced hardware, this is achievable but also very expensive. Also, it is not scalable – If we add more cameras, the cost will increase rapidly.

SSD MobileNet is a lot faster than Faster R-CNN but it has lower accuracy. Is there a way we could improve accuracy of our system with SSD MobileNet as an object detection model, but without losing too much speed? Well, this is the task of the next part of our system.

Landmark detection

The task of this model is to give us finer and better detections of license plates. The idea is simple: For each detection in a frame we send a bounding box of detected license plates to a landmark detection model to find corners of a specific license plate.

We built this model from scratch in KERAS with Tensorflow as backend. By using a Convolutional Neural Network (CNN) that takes an image with 100×100 size of car with license plate as input, and outputs x,y coordinates for all 4 corners of license plate.

The model was trained on 20.000 images of license plates and annotations of the license plate corners. By detecting corners, we have a much better approximation of license plate location, and thus, improved precision.

The model is not complex (for CNN) – we set up simple neural network architecture with two convolutional layers followed by two fully connected dense layers and an output layer (the model has 3 million trainable parameters), therefore its predictions are very fast (3 ms per license plate on 2,4 GHz Intel Core i7) and have a tiny effect on the speed of detections, so this gives us precisely what we want.

But with this model, we gain another huge advantage. Object detection models always output rectangular shapes of their detections. Because of the angle of the camera watching the traffic, license plates on the frames don’t have a proper rectangular form. With corner detection, we can get much finer detection and practically get some quadrangular shapes of the detected license plates.

Comparison of the object (in yellow) and landmark (in red) detection

It is important to notice that we expand the bounding box of detected license plates because we want a bigger context and more information for our landmark detection model to achieve better generalization.

Another reason is mentioned lower accuracy of SSD MobileNet – this model sometime returns a bounding box that doesn’t cover the whole license plate.

REST API

The purpose of the REST API is coordination between cameras and deep learning models for detection. The app takes requests (video streams frame by frame) from clients (traffic cameras) and delegates them to the previously mentioned modules. After detection and blurring part frame is sent back to the client. We got an additional significant boost in the computational speed, by building a Tensorflow package from the source code.

Summary & results

To summarize the system, this is what it all looks like:

A camera records the road and every few seconds sends a frame to a web interface of our system.

Each frame is preprocessed and then sent to an object detection model for recognizing all license plates in the frame.

An expanded bounding box of each detected license plate is sent to the landmark detection model, which returns predictions of each of the four corners of a given license plate.

For each detected license plate in the frame, we blur the part that is between corners.

Frame with blurred license plates gets returned to the client.

License plate detection pipeline

In this blog post we have shown that it is possible to build a license plate concealment system that is both fast and precise. Furthermore, the system is generalized to work equally well on all cameras. All this is possible with emerging deep learning techniques.

Need help with your Machine Learning projects? Get in touch, we’d love to help with your challenge.

The post Fast License Plate Concealment with Deep Learning appeared first on Infinum.