Digital Health Tech Review: WHOOP Strap 3.0

Posted on by personalsensing
Source: Wearable Technologies

Whoop Strap 3.0

Next up on our digital health technology review is the Whoop Strap 3.0. We completed this review using the evaluation framework for human performance measurement technologies.

What is the Whoop Strap 3.0 and what does it do?

The Whoop Strap 3.0 is a wrist-worn wearable device that claims to provide the “most accurate and granular understanding of your body”. Whoop is designed to guide you through exercise, recovery and sleep. The Whoop takes measurements 24/7 of heartrate, heartrate variability, sleep, skin conductivity, and ambient temperature. Data is collected using a tri-axial accelerometer, body temperature sensor and photodiode LEDs optical HR sensor. Heartrate and heartrate variability are measured using photoplethysmography. It is accompanied with a dedicated app which can be downloaded for Android or iOS.

Who develops the Whoop?

Whoop was founded by Will Ahmed in 2011, Boston, USA. The company has an estimated yearly revenue of $3 million dollars. Up to now, they have raised $49.8 million through five rounds of funding, and investors include Twitter CEO, Jack Dorsey and the NFL Players Association.

How much does it cost and where can I buy it?

In 2018, Whoop transferred to a subscription service. Customers receive a free Whoop with subscription packages as follows: 6-month membership: $30 per month; 12-month membership: $24 per month; 18-month membership: $18 per month; Lifetime membership: $399 onetime payment (only available to founding members who had bought a Whoop prior to the establishment of the subscription service).

Whoop is available internationally through their website. The company pays import duties and taxes for EU and the USA only. Outside of this, consumers pay these taxes.  Shipping companies vary per country (EU=GLS; USA/rest of world=UPS).

What is the science behind the Whoop Strap 3.0?

Whoop make a massive claim their device offers customers the “most accurate and granular understanding of your body”. It is only right therefore, we examine the science behind their marketing.

Whoop’s unique selling points are to offer consumers feedback and data about their strain and recovery. Strain is calculated through maximum heartrate, which makes it quite individualized. It measures your daily accumulated strain score using a scale 0-21: Light: 0 – 9.9; Moderate: 10 – 13.9; Strenuous: 14 – 17.9; and All-out: 18 – 21. Strain is fed back to the consumer in two ways, Day strain and a strain score for a discrete workout or activity. Day strain is the total accumulation of strain you have completed per day. A workout strain score provides detail about cardiovascular effort for a set bout of time. 

Source: Slideshare – Power

Whoop also provides a recovery score, expressed as a percentage score calculated from multiple metrics such as heartrate, HRV and sleep. The feedback will then suggest the degree of strain to subject yourself to, as a way to control your exercise load or to decrease risk of injury.

Recovery is analysed through three key metrics: Resting heartrate, HRV, and sleep duration (hours). For example, an elevated HRV and a decreased heart rate may suggest you are recovering well when compared to your baseline. As you wear the Whoop, your baseline is calculated, and the algorithms calibrate the device accordingly after 4 recovery sessions. Recovery is determined when your sleep is complete. HRV is captured by the WHOOP during your last period of Slow Wave sleep each night, a high HRV as mentioned above, suggests you are well recovered and ready for a solid workout. This is one of the key metrics used to determine recovery score. Resting heartrate is also captured during the last period of Slow wave sleep, a lower resting heart rate also suggests you are adequately recovered. Personalized feedback is then provided to the consumer.

A core factor in Whoop’s calculation of strain and recovery is sleep. Measuring sleep accurately has been the problem child for many consumer wearable devices but is Whoop any different? A study conducted by Whoop aimed to test their device against the gold-standard for sleep measurement, polysomnography (PSG), at a sleep laboratory. Thirty healthy participants with varying sporting backgrounds were recruited. Participants spent one night in the lab and their sleep was measured simultaneously using actigraphy and PSG. According to their findings, Whoop was 93.1% accurate with a sensitivity of 95.6% a specificity of 80.3% and Cohen’s Kappa2 k value of 0.75 indicating a high level of statistical agreement compared to PSG. However, this work was not published in a peer-reviewed journal so it hard to determine the applicability of their methodologies, or the credibility of their findings. For example, a sleep validation study conducted by Castner et al., 2018 used a Fitbit Charge™ and compared it against an Actigraph wGT3X+. This study included 50 women with asthma and were monitored for a combined total of 978 nights, of which 738 were available for data analysis – the paper can be found here: https://www.tandfonline.com/doi/full/10.1080/02770903.2018.1490753. In comparison, the Whoop study involved 30 participants who spent only one-night in a controlled sleep laboratory. The setting and sample size of their study suggest the reliability and real-world accuracy of their findings should be viewed cautiously.

Source: AllAroundJoe

Interestingly, Whoop have a section titled “Validation” on their website. Depending on the lens you read the contents through, some may feel when you scratch the surface, there is very little in the shape of trustworthy validation findings. It’s much more a case of ‘Whoop, there it isn’t!’ in our opinion. Much of what is presented are short white papers aiming to make a case for the metrics Whoop incorporates and uses. The only peer-reviewed paper they cite on their website is a 2019 study by Sekiguchi et al., titled “Relationships between resting heart rate, heart rate variability and sleep characteristics among female collegiate cross‐country athletes”. The aim of this study was to examine the changes in resting heartrate, HRV and sleep characteristics across weeks and to conduct an investigation into the relationships of these variables during a female cross‐country competitive campaign (note nothing is mentioned about validating the Whoop in these aims). A Whoop device (which version is not stated) was used as a data-collection tool to capture resting heartrate, HRV and sleep data. The website however, states: “The study’s findings show the ability of Whoop to accurately measure these metrics, as well as how we can use them to promote recovery and improve athletic performance.” Even before reading this paper, the aim of the study should make you very skeptical of this claim. The study can be found here: https://onlinelibrary.wiley.com/doi/full/10.1111/jsr.12836. Upon reading this paper, the only thing validated about the Whoop will be your skepticism of their “accuracy” claim. The study in fact mentions nothing beyond using the Whoop for data collection and outlining some specifications, namely its’ sampling frequency. The reason they do not mention anything about the accuracy of the device, is because it was not a validation study. It is important to mention however, that Whoop are conducting a study to “investigate the effect of the Whoop Strap 2.0 device on sleep perception and perform a methodological study to validate the accuracy of the Whoop Strap 2.0 device when measuring HR accuracy and HRV accuracy, and sleep quality and quantity with respect to PSG in healthy volunteers with no self-reported sleep disorders or debilitating medical conditions.” (Clinicaltrials.gov).

What happens to the raw data and can I access it?

User data is wirelessly uploaded to smart phone using the accompanying app. The strap and app are connected to the user’s computer or cloud where it is available to them. It is unclear if this is raw data or not. The user has the right to access their data upon request. It is unclear whether the data transferred to the user’s computer/cloud is available as raw data. According to some user forums, accessing Whoops’ raw data appears to be difficult and requests to customer service have not resulted in access been granted, see comment section here: https://bengreenfieldfitness.com/article/biohacking-articles/review-of-whoop-wearable/.

What about human factors?

The Whoop has a lithium-ion polymer battery providing a reported 5 days of use according to manufacturers. Charging is reported to be only 1-2 hours, which is super considering the duration of battery-life. Other reviews online have confirmed these estimations, and it appears the battery life is spot on. This is great news considering the device can be worn 24/7, and very active people will appreciate not having to worry about the device if they outside for long periods of time. Unfortunately, there are no published papers (or even white papers) investigating the user-experience or usability of the Whoop. Online reviews again provide very favourable feedback on comfort and form factor, particularly the range of strap designs that can cover a range of user needs. Charging is simple, the device comes with a small battery pack for the user to slide on top of your strap to charge. In terms of usability, the fact that the Whoop requires little or no direct interaction would suggest it is easy to use. There is no display or buttons, and to check the battery life the user just needs to double tap the device and the LEDs on the side display the charge level.

Source: WHOOP | Accessories

The bottom-Line

Whoop is a seriously slick looking wearable that is trying to do something different. We like the idea of strain and recovery scores, but such overall scores have traditionally failed to offer any real impact in the long term for consumers, it remains to be seen if Whoop can crack this. We also feel the subscription model is cheeky, particularly when so much literature and anecdotal evidence exists that consumers find a neat place in their cupboard for their wearable within 6 months of purchase. It will be interesting to see if Whoops’ subscription service creates previously unseen levels of sustained engagement. Indeed, Whoop does lack an evidence-base, but this can be levelled against so many wearable tracker companies that it’s the exception not the rule to have little or no peer-reviewed studies to back up claims. We are looking forward to seeing the results of their clinical trial and they would do well to try to publish the findings as currently we are unconvinced by the accuracy they offer in relation to measuring sleep, especially as their device lacks rigorous validation testing against either electroencephalography or polysomnography technologies.

If you have any questions about this Digital Health Tech review, the research conducted by our group or the Digital Health Technology Evaluation Framework, please contact us via the blog or twitter!

Digital Health Tech Review- ŌURA Ring

Posted on by personalsensing

To kick off our series of sensor and digital health technology reviews we took a look at the ŌURA ring – a smart ring which claims to be ‘the most accurate, comfortable and beautiful wearable available’. The question is, in a wearables market that is full of big competitors and even bigger claims, can the ŌURA ring sustain being put through the ringer…..

As will be the case with all our reviews, we will be leveraging the evaluation framework for human performance measurement technologies published in Nature Digital Medicine. An overview of how these reviews will be conducted was published on our blog last week.

What is the ŌURA Ring and what does it do?

The ŌURA ring is a smart ring, designed to be worn on any finger (8 sizes available) and ‘help you get more restful sleep and perform better’. It strives to do this by collecting data about body temperature (negative thermistor sensor), heart rate [HR]/heart rate variability [HRV] (polysomnography) and activity/sleep tracking (tri-axial accelerometer and gyroscope). It is designed to be used in conjunction with a free android or iOS mobile app.

The ring provides you with metrics about sleep, activity and readiness.

The ŌURA Ring sleep, activity and readiness dashboard.

Who makes the ŌURA ring?

The ŌURA ring is a piece of wearable jewellery developed by ŌURA Health OyLtd, a health technology company founded by Kari Kivelä, Markku Koskela and Petteri Lahtela in Oulu, Finland in 2013. To date, the company has received upwards of $33 million in funding over four rounds and has an estimated annual revenue of approximately $2 million. Sports and TV fans might be interested to know that investors include the likes of Lance Armstrong, Shaquille O’Neal and Will Smith.

How much does it cost and where can I buy it?

Those interested in the ŌURA ring have a selection to choose from, ranging from the ‘Silver’ and ‘Black’ options (€314), to the ‘Stealth’ option (€419) to the diamond version (€1049). The ring is available internationally online, with no minimum or maximum order requirements. Importantly, the company will play import taxes for EU and USA customers only, while customers outside of this will have to fork out for the taxes themselves.

What is the science behind the ŌURA Ring?

Given the big claims by ŌURA that it is ‘most accurate, comfortable and beautiful wearable available’, does the science live up to the hype?

There are currently three papers available which have evaluated the validity (ability to measure what it’s supposed to measure) of the ŌURA ring. One internal white paper on the accuracy of the HR/HRV and two papers (one peer-reviewed paper and one internal white paper) on the accuracy of the sleep/ sleep quality estimations

Firstly, the internal white paper comparing nocturnal HR and HRV to the gold-standard Echocardiogram [ECG] measurement:

  • Ten healthy adults wore the ŌURA ring and a Faros 360 ECG device (Mega Electronics, Kuopio, Finland) over-night. Individuals wore the smart ring on each hand.
  • The ŌURA ring software labelled inter-beat intervals to estimate HR and HRV, while the Kubios software was used to obtain the comparison reference HR and HRV.
  • Pearson correlation was then used to investigate the relationship between the two measures.
  • The authors reported ‘ŌURA Ring provided reliability as compared to ECG of (r = 0.999) and (r = 0.984) for HR and HRV, respectively’.

While the authors conclude that ‘The nocturnal resting HR and HRV numbers measured by the ring represent similar values to those derived from the ECG’, we have a couple of major issues with the methods used:

  1. This paper only focused on nocturnal HR and HRV, arguably the easiest time to quantify this, not taking into account the accuracy during bouts of activity throughout the day – this would likely reduce the accuracy of the estimates.
  2. This analysis is only based on periods where 50% (HR) and 30% (HRV) of the intervals were labelled as ‘normal’. However, they don’t provide information on how often the ŌURA ring labelled data as abnormal. To me this is essentially cherry-picking nice clean data for comparison, ignoring the ‘abnormal’ data.
  3. This is a statistical point, but one we cannot ignore. The authors conclude that the ring can provide ‘reliability as compared to ECG’. This is not true. What the authors did do, was show a high correlation between the ring’s estimations, but did not (as they have stated) demonstrate ‘reliability’ or indeed ‘agreement’ – that requires a completely different type of analysis.

In short, this internal paper showed that when they exclude unwanted ‘abnormal’ data, the ŌURA rings estimations of HR and HRV have a high correlation with the gold standard ECG; however, I would be intrigued to see what the accuracy of the measurement would be if they didn’t simply exclude unwanted data.

Secondly, an external peer-reviewed paper on the accuracy of the sleep/ sleep quality estimations published in the journal of Behavioural Sleep Medicine. This study was conducted by an external research group based in California and provides a decent unbiased analysis of the sleep estimation.

  • Forty-one healthy adolescents/ young adults wore the ŌURA ring in a single laboratory overnight.
  • Sleep data were recorded using the ŌURA ring and the gold-standard polysomnography [PSG].
  • The ŌURA ring metrics were compared to the PSG equivalent using Blant-Altman and epoch-by-epoch analysis.
  • The authors concluded that the ŌURA ring tended to underestimate Deep Sleep (approx. 20 min) and underestimate rapid-eye-movement (REM) sleep (approx. 17 min). Total sleep time and wake after sleep onset lay within the ≤30 min a-priori-set clinically satisfactory ranges for 87.8% and 85.4% of the sample, respectively.
  • The ŌURA ring could detect sleep 96% of the time (excellent) but could only detect waking 52% of the time (poor). Additionally, agreement in detecting the stages of sleep (light, deep and REM sleep) was poor, ranging from 51-65%.

Interestingly, the authors conclusions about the ŌURA ring are relatively kind:

“Multisensor sleep trackers, such as the ŌURA ring have the potential for detecting outcomes beyond binary sleep-wake using sources of information in addition to motion. While these first results could be viewed as promising, future development and validation are needed.”

However, we’d like to draw your attention to a few key aspects related to the results of this study:

  1. While the ŌURA ring was excellent at detecting sleep, the sophisticated flipping a coin method would have similar accuracy in detecting wake, based on this study.
  2. While there was an underestimation for light sleep (approx. 4 min), underestimate deep sleep (approx. 20 min) and underestimate REM (approx. 17 min), the Bland-Altman analysis highlights just how much the ring differed from the gold-standard measure. For example, the ŌURA ring overestimated deep-sleep by as much as ≈100 min and underestimated it by as much as ≈65 min. Seeing as the average deep sleep time was only 97 min, this shows a marked (and non-systematic) difference between the ŌURA and the gold-standard alternative. Similar findings are seen for REM sleep estimation.

In other words, the ŌURA ring can accurately detect sleep, but is pretty poor at detecting wake and the various different stages of sleep. Based on this, and contrary to the authors conclusions, I would argue that the ring is not very good at detecting sleep outcomes ‘beyond binary sleep-wake’.  

What happens to the raw data and can I access it?

A common trait that we see within the consumer wearables space is a lack of information related to the nature of their algorithms and whether raw sensor data can be obtained from the device. The ŌURA ring is no different, with no clear information provided about the methods used in the data analysis and if the user can in-fact access the raw sensor data. While the user can access the summary metrics through the mobile application, for research purposes, this may not be enough.

As with many consumer wearables, ŌURA are attempting to try to make sense of the data capture by the ring through a composite ‘Readiness’ score; a metric derived from sleep, activity balance, body temperature and resting heart rate. This is a common theme across the health and wellness tech sector, however we’ve yet to see anyone develop any magic score that is actually related to injury, illness or any other factor for that matter.

What about human factors?

With the statement that the ŌURA is the ‘most accurate, comfortable and beautiful wearable available’, one would assume that the ŌURA has undergone some rigorous user testing. While they do provide quite a detailed overview of their design process on their website, there is no objective published data on this.

Aside from this, the ring requires little interaction apart from the obvious wearing of it, synching the data through the mobile application and wirelessly charging the device. In fact, one of the most impressive aspects of the smart ring is the ability of the company to have shrunk this set of sensors into such a small device that boasts an impressive week-long battery life. It will be interesting to see where this technology brings us in the next 10 years.

Is it being used in any clinical trials?

As one of the biggest interests in our research group is essentially understanding how technology can be used to improve health and performance, we are always interested to know if the technology in question is being used in any clinical trials.

Currently, there have been no published clinical trials which have leveraged the ŌURA ring; however, there is one registered trial of 1060 participants currently underway seeking to develop a set of typologies of users of mobile sleep tracking technologies, and optimal sleep health educational strategies among users of mobile wearable technologies that track sleep.

The bottom line

The ŌURA ring boasts itself as the ‘most accurate, comfortable and beautiful wearable available’. While you may agree that it is indeed ‘comfortable’ and/or ‘beautiful’, based on this evaluation of the current evidence available, it is hard to fully agree with the statement that it is ‘accurate’ – especially if you are looking to get detailed information about the stages of sleep. The ring is not cheap, coming in at €314 – €1049, a substantial price to pay considering you could arguably get a wearable with similar accuracy, for substantially cheaper. However, if a ring-based wearable is more desirable than a wrist-worn one, then the ŌURA may be suitable for your needs – but only if you are looking for high-level activity, sleep and HR data. It will be fascinating to see the outcome of the ongoing clinical trial using the ŌURA ring, and see where the technology goes over the coming years.

If you have any questions about this Digital Health Tech review, the research conducted by our group or the Digital Health Technology Evaluation Framework, please contact us via the blog or twitter!

Louise Brennan, Dr. Alison Keogh & William Johnston

Digital Health Technology Reviews

Posted on by personalsensing

With the rapid evolution of mobile sensing and computing technology, we have seen a surge in the amount of technology readily available at our fingertips. This has resulted in a surge in the number of companies producing both consumer and healthcare facing digitally enabled technologies.

These technologies span a wide range of use-cases; ranging from fitness and wellness, all the way through to healthcare.  Over the past decade, the world of human performance and behaviour measurement technology has rapidly moved forwards; starting with iconic wearables such as the Nike+, Fitbit and Google Glass. Since 2014, dubbed “The year of Wearable Technology” by some, we have seen a huge growth in the popularity of consumer wearable tech, with the introduction of the Apple Watch forcing numerous competitors to jump on the bandwagon.

With this, it is expected that the number of connected wearables will increase from 325 million in 2016 to 929 million by 20211, with the digital health consumer base expected to grow in parallel2.

While the increased availability of such devices has the potential for many positive outcomes in the spaces of health and fitness research and Digital Health, it also has the potential to result in huge confusion about what technologies are fit-for-purpose.

For example, the requirements of a piece of tech that simply need to detect the number of steps a recreational athlete takes during the day are massively different to the requirements needed if the user needs to detect if an individual has atrial fibrillation.

This led members of our research group and the Applied Research for Connected Health centre to develop a structured evaluation framework.

This framework allows for the:

“refining [of] the requirements for a specific application, and then evaluating the available devices against those requirements”

Over the coming year, members of our research group at the Insight Centre will leverage this framework to evaluate human performance measurement technologies, making these objective evaluations available online in the form of a blog post!

The structure of these blog posts will evolve over time, and will vary from evaluation-to-evaluation but will broadly focus on components of the evaluation framework.

A rough structure will be as follows:

  • What is the technology and what does it do?
  • Who makes the technology?
  • How much does it cost and where can I buy it?
  • What is the science behind the technology?
  • What happens to the data and can I access it?
  • What are the human factors associated with the technology?
  • Has it, or is it being used in any clinical trials?

1. Statista. Connected wearable devices worldwide 2016-2021. https://www.statista.com/statistics/487291/global-connected-wearable-devices/ (2018).

2. Berg Insights. mHealth and Home Monitoring – 8th Edition (Gothenburg, Sweden: Berg Insight, 2017).

3. Caulfield, B., Reginatto, B. and Slevin, P., 2019. Not all sensors are created equal: a framework for evaluating human performance measurement technologies. npj Digital Medicine, 2(1), p.7. 3.

Personal Sensing – What do we do?

Posted on by personalsensing

Recent years have provided us with unprecedented access to a wide range of inexpensive sensing, aggregation and communication technologies that can transform society through provision of access to data relating to human behaviour and performance in health and sport.  However, this ever-advancing sensor web needs to be carefully matched to relevant data analytics and domain expertise to unlock the inherent value in the complex datasets that it produces. 

This research strand brings together expertise from life and clinical sciences, material science, computer science, and biomedical engineering to enhance the application of the sensor web to challenges in connected health and sport.  Whether dealing with chronic disease or performance enhancement in elite sport, a connected ecosystem is dependent on an efficient and unobtrusive approach to sensing, sharing and analyzing data to facilitate timely delivery of accurate information to all stakeholders in the process. 

In this strand of the Insight Centre for Data Analytics, a research centre spanning Dublin City University, NUI Galway, University College Cork, University College Dublin and other partner institutions, we address fundamental research activities related to optimizing the sensing, measurement and understanding of human behaviour and performance, and the implementation of feedback strategies that are designed to enhance it.  A further aim is to contribute to the expansion of sensing capabilities of the sensor web through development and validation of novel sensing technologies.

O’Brien Centre for Science, University College Dublin, Ireland

As part of our research, the personal sensing team in University College Dublin also conduct regular structured evaluations of health and wellness technology which are currently available on the market, with the view to keeping up-to-date with recent advances in the field, as well as ensuring that the technology which we may implement in our research is fit-for-purpose.

Over the coming months, we will be making our objective and structured reviews, conducted in accordance with the evaluation framework published in Nature Digital Medicine (Caulfield et al., 2019), available online through this blog post.

You can also follow us on twitter: @personalsensing, or contact us through this blog.

Visit the Insight Centre webpage for more details on what we do

Caulfield, B., Reginatto, B. and Slevin, P., 2019. Not all sensors are created equal: a framework for evaluating human performance measurement technologies. npj Digital Medicine, 2(1), p.7.