IoT HealthTech App: Building Heartbeat Detector Using Phone Camera

A few months ago I bought a smartband. The main idea being prosaic - I wanted to receive notifications from a smartphone as well as have a good pedometer. A few weeks after the purchase, the same company came out with a new version of the device that included one great feature… a heartbeat detector.

I would not be a true developer if I didn’t try to write an add-on for this one, right?

I went through the very first requirements. Dug deep into the app architecture. Actually it took me some long, long night hours (I should probably spend sleeping...). BUT, I finally came up with an idea tailored perfectly for a regular smartphone user.

For my concept I required a few things: a flashlight, a physical camera and a non-trivial algorithm.

The main profile of the person using our healthcare application is a normal smartphone user (not a power user, only an average person without any technical know-how). First, I thought about a cross-platform application, but after a few hours of reading about Ruboto integration with the camera API, I decided to create an IoT application in native Java.

How our health app works

The application uses the API to fetch the frame from the camera. Every valid image is processed (NV21 method) and the most interesting parameters are stored. Next, the algorithm compares the frame with latest cached frame, and (again) the results are stored.

The main issue at the beginning of this idea was to compare images. In my programming experience, I hear all the time about the expense of an operation that compares pictures, but I was not going to give up easily.

This problem encouraged me to improve and optimize this service. After 2 weeks of testing, I found a new solution.

Calculating the number of red pixels in an image is the most important thing! When I tried to detect a heartbeat, I needed only the number of red pixels from the image. If the number is higher than in the previous image (for example, 10 frames), then I can assume that the heart is now beating. Why only red pixels? Blood is, of course, red and so blood and a flashlight directed on a finger should give a very bright, red image. We don’t need anything else.

What problems we had?

Unfortunately, great news also brought bad news.

Finally, I thought about storing and managing data. After one pulse check, the phone stores approximately 2500 values related to the count of red pixels in every frame.

Some might ask - “Why don’t you only store the average value?” Ha! This leads me to discuss the most valuable function in my application. An atrial fibrillation detector. Wait… how can you check heart issues with just a phone, camera and flashlight?

From the beginning… after a few evenings spent analyzing tables with raw values from the images, I found one intriguing fact. When checking my own pulse, I saw a very similar time between each beat, but when I got results from a person who has an arrhythmia disease, the time between each beat had been quite different.

Okay, so back to the question - “Why don’t you only store the average value?”

When checking for heart disease, I want to calculate the standard deviation for time between each heartbeat. This function is very experimental, but we want do something better than a normal heartbeat detector.

For this (by the way - innovative) feature we need as much low level information as possible. It’s very important for checking if the user has an irregular heartbeat issue. When the user is resting and wants to check their heartbeat, it’s simple; when a person running or training checks their pulse, the application won’t have the necessary low level data and can be inaccurate. As a result, we need to know about every millisecond of delay in the data.

What is wrong with health apps we have now?

On mobile App Stores we can find a huge number of fitness/health applications that measure a similar number of body parameters, but the majority of them display dull health statistics without a wider context. They are also focused on social media sharing options for competing with a user's friends or gaining various achievements.

I admit, it’s important to encourage users with the gamification approach, but it’s also important to develop an awareness about the user’s health condition and be able to detect any deviation in status. IoT has already been noticed as a potential source of health improvements and this is an approach we should develop.

With this point came with my teammate from IoT development team the idea to design an application that will guide users through an educational approach, help them understand the readings as well as detect deviations in heart work - atrial fibrillation for example.

Lucky for me, we’ve done some internal IoT projects from the scratch (like our Breweree app), so I had a backup and support in my colleagues.

Elderly people and a large number of young users want to use applications designed this way. They want to extend their health awareness and be guided by a teacher who will tell them what's wrong with their readings. It’s also important to facilitate using the application, starting with the process of health inspection and easy access to historic data from past examinations. These features will provide a better user experience to the end user.

Design approach

The main design challenge I faced was creating an application from the human side. The majority of current health applications on the market are more or less built in the same way. A big start button, four or five charts and a table with measurements were not the best idea for an application that must tell sensitive readings about a user’s heart rate, pulse, arrhytmia disease or other heart disorders.

Interactions in application

The concept for how to solve this problem was a continuous application response based on each reading. For example, changing the background color to a vivid orange when the user’s pulse increases or a restful green when their pulse slows down. Or also a pulsatory red screen informing the user about encountered deviations from standard values. All of that and even more during a live EKG preview. The user can confidently verify their own health status by using the live EKG preview which has default reference values of a healthy person in the background.

A story told through short messages will lead users through a pulse verification process followed by push notifications which remind the user about regular examinations after atrial fibrillation detection. These application features don't leave the user with useless information or numbers, but actively support them in taking care of their own health.

What’s the status?

Our healthcare IoT app is not available to the market at the moment, but it’s being tested on a group of 20 people. To be honest, at the beginning I thought it would be exclusively a part of my engineering thesis and didn’t expect that our IoT development team would like to continue the project in any way.

I am considering releasing the app for public beta tests, but it’s not as easy as one could expect. It’s not just about development, but also proper market research. Interviewing doctors and potential target group is a challenge when you work full time... Trust me, it’s not one-man-job.

Summary

One of many things which we learned with our IoT R&D Team is fact that healthcare is a big undiscovered field of IoT industry. For next two, maybe three years, this kind of solutions will be of daily occurrence, cause it's very expensive part of business. Maybe the next quantum leap is syncing data with our doctor?

Networks full of sensors, devices connected to the internet and meaningful design interfaces will help to increase their popularity more and more and reduce costs of maintenance whole industry. That caused a question - why not try to make your own IoT healthcare application?

Maybe you have a better idea for checking many disease or performance of your app will be much better. Tell us! We wait for your comments and valuable experience!