What are the barriers for consumer sleep wearables being taken seriously as accurate, reliable sleep measurement tools by the research and clinical communities?
In the second part of this discussion we talk about the problems of proprietary algorithms, raw data access, and the challenges that need to be met by both science and commerce.
For Part 1 of the discussion see: https://sleepjunkies.com/the-great-sleep-tracker-debate-part-1
Prefer to read? Download the full episode transcript here
Skip to sections:
- 2:52 The difficulty in creating validation standards for consumer sleep trackers
- 4:17 Do consumer sleep trackers need to be regulated?
- 5:45 Getting ‘under the hood’, proprietary algorithms, raw data access
- 7:02 Understanding commercial incentives, intellectual property
- 8:25 Stability of data, algorithm updates, lack of transparency
- 9:12 People are basing health decisions on consumer sleep trackers
- 10:18 Consumer wearables have great potential but they are ‘black boxes’
- 11:00 Data privacy, cloud-based platforms, security
- 12:00 Data loss. How reliable are consumer wearables?
- 15:23 The danger of using a sleep wearable if you have a sleep disorder
- 19:10 Putting manufacturer’s claims into the spotlight
- 24:08 Wearables that have already been validated, Fitbit, Jawbone, Oura Ring
- 25:19 Choosing a demographic for the validation studies
- 26:28 Total sleep time (TST) estimations are pretty good in consumer wearables
- 27:21 Will consumer wearables ever be able to do accurate sleep staging?
- 29:33 Sleepscore – are commercial, proprietary validation systems useful?
- 31:54 What’s are some general tips for choosing a sleep tracker?
- 35:43 Why there’s no such thing as the ‘best sleep tracker’
- 36:38 Orthosomnia, when sleep tracking can cause anxiety
- 40:47 A summary of all the issues discussed
This episode’s guest:
Jesse Cook is a doctoral student within the Clinical Psychology program at the University of Wisconsin-Madison under the primary mentoring of David Plante, MD, PhD. Previously, he completed his undergraduate degree at the University of Arizona, whereby he assisted in projects directed by Dr Richard Bootzin.
Jesse’s research primarily focus on the assessment and treatment of persons with unexplained excessive daytime sleepiness. Additionally, he has published multiple papers evaluating the utility and ability of wearable consumer technologies as a sleep assessor, relative to PSG.
You can reach Jesse by email at jdcook4@wisc.edu
Resources:
Jesse Cook Researchgate: https://www.researchgate.net/profile/Jesse_Cook3
Jesse on Twitter: https://twitter.com/SleepAndSports
Review: Wearable Sleep Technology in Clinical and Research Settings: https://bit.ly/2K4e9MV
Fitbit/NIH research collaboration: https://bit.ly/2RUzsDa
Fitbit Alta HR validation study: https://bit.ly/2Ufbqor
Fitbit Flex validation study: https://bit.ly/2TVPLga
Jawbone UP3 validation study: https://bit.ly/2UeilhT
American Academy of Sleep Medicine: https://aasm.org/
More Episodes:
[powerpress_playlist limit=20]
Full Transcript
Jeff Mann: 02:12 I just want to talk about some of these dilemmas and just outline them. These are like the problems that need to be solved if we’re going to have a situation where consumers are able to buy a device to monitor their sleep and be confident what is telling them is actually what it’s doing. But also from the research side and medicine are able to rely on these devices as well and have some kind of structure and framework where they’re not guessing.
So some of the problems are, there’s no standards for validating, for designing studies. So briefly, you know, how do we start to tackle that situation?
Jesse Cook: 02:53 Yeah. That one’s challenging. You know, we’ve talked about this quite a bit as we’ve jumped around. One are the main problems that arises here that even as an isolated researcher doing prudent, effective research, I’m usually evaluating a device that’s been on the market for multiple years. So in the best sense, these things have been available without any sort of information backing them for multiple years.
Jesse Cook: 03:22 So staying on that horizon of technological evolution is seemingly impossible. And we’ve talked earlier about a relationship between manufacturer and researcher and the production of agnostic results in that regard. But as far as actually having systematic standardization, the review does outline a very thorough approach and how these devices need to be evaluated. And then it becomes, I think an impetus for maybe having like an approval label of some sort that says this device went through the appropriate testing under the parameters outlined by such and such. Similar to like a non GMO product or, you know, FDA approved in some aspects. But I think that would be useful in at least providing some comfort in the estimations that it’s producing.
Jeff Mann: 04:17 Who do you suggest comes up with that? Because big business and the world works in general, a lot of countries are anti-regulation and there’s a valid argument to say that regulation slows down innovation, all that kind of stuff. But if we do have something like non-GMO, you know, we need an external body to provide that rubber stamp. So would that be something like the AASM, the American Academy of Sleep Medicine, one of these bodies, or something on a high level above that, where would you think that would come from?
Jesse Cook: 04:53 Yeah, I think that’s a great starting point, at least jumping off point, at least getting having, first of all, the ASM adopt the standardization procedure because one does not exist. One is not outlined for actigraphy either. That’s a whole separate issue. But for them to adopt a validation approach and then having them have a stamp of approval or another governing body that’s on the same tier as that that’s related to sleep in that regard.
Jeff Mann: 05:23 Yeah. So we need some kind of framework some kind of infrastructure for regulation and validation. And as you said, Jessie, the review, it does flesh a sort of a step by step process to how we might start achieving that. And we’ll put all the links on the website. Maybe the biggest problem in terms of a technical nature is this idea of these consumer devices as black boxes. You know, a bit like the Google algorithm or the Facebook algorithm. Nobody knows how it works. And again, the review talks a lot about this in a few different aspects. You don’t have access to the raw data. So with an actigraph you can basically look at the raw data that’s coming off of the sensor and then there are algorithms which been validated scientifically. But with these consumer devices, you’re seeing what the consumer sees. Maybe you’ll get a spreadsheet if you’re lucky, but I think you told me at one point you were literally having to graphically pull data from a graph or a screenshot and somehow put that into your study. I mean that, that sounds like a nightmare.
Jesse Cook: 06:40 It was an extremely cumbersome process. Thankfully at the tail end of it I did adopt and train some, some very talented undergraduate research assistants to move that ball for me. But yeah, that’s a huge challenge right now and a barrier for implementation. And you know, we’ve, we’ve talked about this already, I understand as a researcher their desire to keep their proprietary information secret as it is a consumer market.
Jesse Cook: 07:12 It is in their best interest to produce a product that cannot be mirrored by other corporations and manufacturers. And as such, revealing what that product actually does would inhibit their growth as a company. But at the same time, if we have no idea what’s going on, how can we trust and how can we truly interpret the dynamics of the output. At the same time, I’ve actually had it happen to me where data that I acquired a year ago was in accessible through their online platforms. So now you start raising concerns about stability of data. If they make an algorithmic modification that they don’t alert people on and it changes the data. How does that fit into research and clinical domains? That’s a whole other point of complication, but it ties into this whole lack of understanding of what’s going on.
Jeff Mann: 08:14 It’s not helpful. Yeah. It’s in fact, it’s completely obstructive to the whole point of what you’re doing as a scientist and trying to present objective data. As you say, you may go back to a device and they’ve done a firmware update or whatever, and the results you’d get will be completely different. You would have no way of knowing.
Jesse Cook: 08:37 Yeah. And did my sleep changed that night? No. So what actually happened? Where in the algorithm schema did they modify it. Did it improve or not improve their congruent with PSG? These are important questions and at this time we just have an inability to assess that.
Jeff Mann: 08:56 Yup. And I mean, if it was a piece of technology, like a calendar or scheduling app or something that you use in your daily life, but it’s just, it’s just an app that’s fine. Update the algorithms, that’s fine. But if people are using something like this to monitor their sleep and trying to make life decisions based on it, this is people’s health.
Jesse Cook: 09:22 It’s their well-being. So, say Jeff, I’m tracking my sleep and I’m starting to notice that my device keeps telling me that I’m waking up 20 times during the night for a total of two and a half hours across the entire night. So I’m eventually prompted. I decide to go see my primary care physician and I present them to these data. And then as I go see my primary care physician, the data actually changes. And I’m now only waking up five to seven times a night for an hour.
Jeff Mann: 09:53 Because they’ve improved their algorithm.
Jesse Cook: 09:56 Yeah, or they’ve adjusted it in some fashion. So that makes the complexity of interpretation just astronomical and unresolvable.
Jeff Mann: 10:07 Yeah. So maybe the biggest sort of nut to crack this, this idea that in business you have competitors, you have to have a competitive edge. You don’t want to give away trade secrets. So all of these big companies in sleep tech have proprietary algorithms and they’re all black boxes. No really easy answer to that is there?
Jesse Cook: 10:32 No, absolutely. And you know, one of the main benefits of these devices is their ability to provide real time feedback. Somebody wears the device, it syncs to their phone or computer and they can see real time information, which is super cool. Whereas if I gave them an actigraph, they would have to bring it back to the laboratory, we would download it, we would process it and then we could give them some feedback. So much more cumbersome process in that regard. The problem that comes into play here is when we start utilizing these cloud based platforms, we start now bringing confidentiality issues as well. How secure are these databases? How protected are they? And then they can say over and over again that they meet certain standards and so on. But that’s certainly something that at least leaves an uneasy feeling if it’s truly going to be utilized for clinical purposes or other aspects as well?
Jeff Mann: 11:30 I haven’t really considered that this in conversation, but yeah, the whole idea, if there is going to be some kind of cross over between consumer sleep technology and medicine and research, who owns the data? How does the data get anonymized, where does the data reside. So that’s another big one.
Jesse Cook: 11:51 And that’s not just unique to sleep obviously. That’s on the forefront of all this health technology revolution that’s going on right now.
Jeff Mann: 12:00 Yeah, that’s massive, massive. Reliability as well. Another thing they talk about in the report, was it one of your studies where you lost…
Jesse Cook: 12:09 Yeah, significant portions of data. In just about every investigation I’ve ran, whether it be 10 to 20% of my participants data was inaccessible. Nothing changed in our process and our end as far as how we apply the device, how we synchronize the device, how we tried to extract the data. But for some reason or another, the device malfunctioned And that was similarly problematic for, I believe, Elisa Meltzer’s team experienced something similar in that regard and a Max’s team. The de Zambotti SRI International Group has also experienced data loss in that regard.
Jeff Mann: 12:50 That must be so frustrating.
Jesse Cook: 12:52 Yeah. When you’re on a strict budget provided by grant funding, that also is an issue as far as I don’t want to recruit more participants.
Jeff Mann: 13:02 Throw in the bin. Sorry, that’s a bit dismissive there.. Right, so we’ve got all these obstacles, but the fact is, at this point in time, and certainly five years and 10 years in the future, there’s no doubt that this sleep tech is going to be so much better and it’s going to be super, super useful to consumers. But also to researchers as well. So we need to find a way for industry and science to put their heads together and find some way of coming together,
Jesse Cook: 13:40 Absolutely. And I’ve been very careful in using any sort of manufacturer name explicitly for a myriad of reasons. But I will say that I’ve had direct contact with many of these manufacturers at past sleep conferences, over the digital technology that’s available. And most at least endorse similar sentiments that we’ve expressed today. They want congruency, they want alignment with the scientific and medical field. And they want transparency. But it just hasn’t unfolded yet. So, as you said, this is a field, this is an area that in 10 years it’s not going to look anything like it is right now. And that’s just the way our society from a technological standpoint is evolving. And you know, soon with Elon Musk we’re going to have a chip in the back of our brain anyways, that’ll probably track our sleep. So it’ll completely change in that dynamic. So I think at this time it’s a really challenged field, but I think I have the utmost confidence that in the coming years it will be resolved in a very appropriate manner.
Jeff Mann: 14:57 Yeah, I think so as well. We’re just a bit of a junction in the road at the moment. People have to decide which way to go. But bottom line is, it’s going to be good for business isn’t it? You get a rubber stamp of approval.
Jesse Cook: 15:13 Absolutely.
Jeff Mann: 15:13 So in that sense it’s in the interest of these companies to work this out with the scientific and the medical community. One other thing about these dilemmas that I was describing, sleep disorders as well. That’s a whole different kettle of fish as well, isn’t it? Because if people are buying these devices to find out if they do have a physical problem with their sleep rather than an anxiety related problem, then where does the responsibility lie with the manufacturer, you always see a disclaimer somewhere saying this isn’t a medically approved device…
Jesse Cook: 15:53 Well, the interesting thing is that it’s not always explicitly clear from these companies what their true purpose and utility is. For instance, one company had two different settings for their product. A normal and a sensitive mode. And on their actual help page it was explicitly laid out that it said people with – I’m paraphrasing now I guess in some aspects – people would sleep disorders should utilize the sensitive setting. And first of all what sleep disorder? Sleep disorders are vastly different across the board. And two, when I evaluated the sensitive setting, it underestimated total sleep time by two hours. And so it was kind of just this blind consideration or endorsement for their product that has vast negative implications for the individual. And so we in the research and medical community really just want them to characterize their product as a sleep estimation tool, a sleep tracker and not a measurement device.
Jesse Cook: 17:05 And it may seem trivial in the nomenclature, but just clarifying that to the consumer is really important that it’s not a surrogate for in-lab testing. And that if you have any concerns over your sleep that you should go to your primary care physician and maybe triage to a sleep professional in that regard. And most of the companies are getting on board with that, but it needs to be explicitly laid out in that fashion. There’s a variety of sleep disorders.
Jesse Cook: 17:31 For instance, I tend to study people who fall under this categorization of central disorders of hypersomnolence. Hypersomnolence being excessive daytime sleepiness. So individuals, persons with narcolepsy fall into that category. And within narcolepsy, you have those who have cataplexy and those who don’t have cataplexy, meaning that their muscles give out on them involuntarily and things of that nature. But you have your general insomnia and some people believe that there’s primary insomnia, which is more physiological and neurophysiological versus environmental insomnia, which is more based on how you manage your stress, how your sleep environment is set up. Are you appropriately handling sleep hygiene? You have just a myriad of different disorders, sleep apnea, various levels of sleep apnea. These devices currently provide a very, very, very limited lens of sleep apnea. You can look at your number of awakenings during the night and try and extrapolate that to, well I wake up 60 times during the night. There’s probably something going on there, but they’re not assessing breathing or oxygen saturation in any fashion.
Jeff Mann: 18:46 Yeah, I mean you said interesting space that one I’ve seen in the last year or so quite a few different wearables come into the market and measuring oxygen saturation and obviously going for that sleep apnea detection even though they’re not medical devices. But that’s a whole separate conversation. Not for today. I want to move on and I’m going to read out some manufacturer’s claims. I’m not going to mention the manufacturer and then what I’d like Jesse is for you to hear some of these claims and then mention some of the studies and again you don’t have to mention the actual devices themselves because we’re not trying to make any specific judgments and say this device is great, this device is bad. And just see how the claims match up to the peer reviewed science that you’ve done. So first I’m going to just read, these out really quickly.
Jeff Mann: 19:44 I’ve got one, two, three, I’ve got six things and I found all of these, these aren’t buried in blog posts or whatever. These are actually on the homepages of these websites. So one of them says “The most accurate sleep tracking next to clinical sleep measures. “One of them says it’s “the most accurate sleep and activity tracker.” One of them says “a degree of precision comparable to that of laboratory.” Another one says “access to information previously only accessible in a sleep lab.” Another one says “built for hospital grade accuracy.” Another one says “monitor up to two sleepers at once with pinpoint accuracy” Okay. Now in your experience, you’ve studied a lot of these devices. Do any of these claims sound vaguely like any of the results that have come out of your tests?
Jesse Cook: 20:41 In some aspects, yes, Jeff, to be honest. The word comparable, is a useful one when considering consumer sleep trackers and actigraphy when it comes to their estimations of total sleep time. So it’s really important to think about what the accuracy pertains to. Because accuracy could relate to its ability to detect the amount of sleep somebody getting versus the type of sleep that somebody’s getting. And as we kind of mentioned earlier, these devices seem really poor currently or limited in their ability to actually categorize sleep across the different stages. But for my research, these devices have improved to a point where their estimations of sleep duration, you know that 500 minutes or 420 minutes, whatever is output on the phone or the desktop that the individual sees, is comparable to that of clinical actigraphs from the data that I produced it and seen across other literature.
Jeff Mann: 21:53 Just to pause you there very briefly. They’ve thought about these descriptions here, but for me, when I see something like “a degree of position comparable to that of a laboratory” I’m thinking a sleep lab, somebody wired up to PSG. Or “the most accurate sleep tracking next to clinical sleep measures” I’m thinking what’s the most accurate? But they’re probably not hinting at that, they’re probably hinting at next to actigraphy. So to me I find these claims problematic,
Jesse Cook: 22:28 They’re very much hyperbolic, right? And that’s their intention is to attract the consumer. These devices still will overestimate total sleep time and will bias relative to PSG. But that doesn’t mean that they’re bad or useless as we’ve discussed. But again, you’re absolutely right that there’s a degree of fabrication in their very poetic description of their device trying to entice the consumer. And it’s amazing when you read through these and as you’ve showed each company says that they’re the most accurate product.
Jeff Mann: 23:08 Yeah, that’s quite interesting, isn’t it?
Jesse Cook: 23:09 How is that possible? But for my understanding, unless I’m completely amiss, the devices are different. So they can’t all be the most accurate.
Jeff Mann: 23:23 This is the argument for regulation and people like the FDA and having Kite Marks here and CE Marks in Europe. But again some people would argue that’s anti competitive..
Jesse Cook: 23:35 ..and potentially limiting, right? Has the boom that we’ve seen in technology due to the fact that it’s not being regulated at this time. Whereas if a governing body came out and said, we fully endorse this product for the sleep estimations, then do the other companies fall by the wayside and we don’t get any sort of competitive growth in that sense, which I think is what you were trying to hint at.
Jeff Mann: 23:56 Yeah, there’s no easy solution here because we can’t have a situation where everyone’s claiming they’ve got the best sleep tracker. But at the same time, we can’t have overburdensome regulation just slowing the innovation down. So, actually I am going to read out a few of these models because I’m going to put links to all these papers on the website as well. So some of these devices that you’ve measured, they’ve actually been through clinical validation. The Fitbit’s a couple of them, the Fitbit flex, the Fitbit, Alta HR, the Jawbone UP3, the Oura ring, another Fitbit, the Fitbit Charge 2, all of these have been studied in the lab properly. Can you just encapsulate the broad spectrum that they’ve fallen in with regards to accuracy and measuring up to the gold standard and actigraphy as well.
Jesse Cook: 24:45 Yeah, absolutely. So a couple of those devices you mentioned are the older models. And so for utilizing those you’re going to want to be skeptical of any of the outputs that they’re producing. But some of the newer models are getting really good at estimating sleep duration. So if you’re buying a device that has been recently produced and manufactured from one of these corporations, I think you can put a reasonable amount of confidence in the total sleep time estimations it’s producing. One thing that’s important to clarify though, Jeff, is that even in our kind of rigorous validation structure, we’re validating within a specific sample of individuals. So a lot of times that may be an adolescent group versus an adult group. That may be quote unquote healthy sleepers versus disordered population. And so it’s truly unclear whether findings from a study that I performed that’s in a disordered population would translate one to one to a healthy population. So although a device has been quote unquote evaluated, it was evaluated in a very specific circumstance.
Jeff Mann: 25:58 For instance you potentially might get different results between using for adults and using for kids.
Jesse Cook: 26:06 Absolutely. Kids are more likely to move around during the night. And we mentioned earlier that these devices have a lot of issues. Their estimations become much more inaccurate when there’s more movement. And so that’s a major challenge when considering the results that we produce in science.
Jeff Mann: 26:27 So you’re saying that the newer models, with regards to total sleep time, they’re getting pretty good.
Jesse Cook: 26:35 For total sleep time they’re getting pretty good. When it comes to the other components. Whether it be the REM sleep classification or your deep sleep classification, those outputs I would take with a grain of salt. Again, some of the research that I’ve produced has highlighted a range of 30% to 60% ability of these devices to correctly identify REM sleep. So if that’s something you’re really interested in, I would monitor it through these devices, but I wouldn’t think of it as like a liturgical output in that sense.
Jeff Mann: 27:21 Are we ever going to get there with these movement and heart rate based devices in your opinion with regards to sleep staging?
Jesse Cook: 27:30 That’s a great question and one that I think about a lot. If you looked at my a PubMed search history, you’d see Bluetooth, EEG, single sensor, typed in a lot into my search history. I think the technology’s there. I think the ability for somebody to manufacture an electrode that a consumer can put on their forehead in a specific location and it could detect wavelengths of brain activity during sleep, which then gets Bluetooth synced to your device, I think it’s there. And it’s probably going to be the intent of a manufacturer that pushes us past this, this necessity and it makes it actually implementable. I don’t see a research team doing it. But I do foresee that in the future the staging limitations will not be completely removed or reduced or washed, if you will. But I think they’ll be markedly improved for sure.
Jeff Mann: 28:39 I love technology as well, so I hope at some point there’s going to be something like that. I’m not for technology for technology’s sake, but just technology that going to have lots of practical uses, whether it’s in health, or research, some people using this stuff for biofeedback, meditation.
Jesse Cook: 28:56 Yeah. There’s a lot of companies out there nowadays that are utilizing EEG headbands of some sort and purporting that these devices are detecting alpha rhythms or assisting in augmenting alpha rhythms or whatever it may be. So it seems the technology seems available. It’s just translating it for the purposes of sleep tracking at this time.
Jeff Mann: 29:17 Okay. So you’ve seen in your studies and the body of work is quite limited, but there are a few of these devices that have been validated. You’ve seen them improve over the years, so there’s no doubt they’re getting better and better. Something I want to touch on briefly as well, a company called SleepScore and but they’re very much doing a commercial version of a validation system and rubber stamping sleep products based on their validation techniques. How do you think these kind of things fit in?
Jesse Cook: 29:56 Well I think first and foremost, I like to stress that we really encourage individuals taking an interest in their sleep in any form, whether it’s just jotting down thoughts about sleep or actually creating a diary where they monitor their sleep in that fashion to these apps. Most individuals tend to have better sleep because they’re now taking an interest in it and holding themselves accountable. So I think all that’s great when it comes to these like systems that tell you how quality your sleep was, how rested you are, how likely you are to perform optimally today, I get a bit squeamish because I want the individual to be able to assess their existence on their own and not be dependent upon a data point, telling them how they’re supposed to feel.
Jesse Cook: 30:55 So I can see a scenario where somebody looks at their output from one of these applications and it says that they’re at 80 out of a hundred on their overall score, whatever that means. And in truth, they actually may have gotten an appropriate amount of sleep and they feel really good. And if they hadn’t actually looked at that school where they would have felt that they were at a hundred. And they may be at a hundred or they may be at 80, that’s a complication. But I don’t want people to base their subjective experience, which is what really matters in a lot of ways, solely off of these unvalidated techniques and scores. And I think that’s happening more and more these days that people’s wellbeing’s are being derived. Their state of existence is being derived by a technological score that is largely unsubstantiated
Jeff Mann: 31:51 Just to try and round this off. I just want to try and give people a couple of takeaways. You know, people who are listening to this are probably interested in sleep tracking, and they hear all these negative or complicated point, and think, Oh God, what do I do now? But there are lots of good products out there and there’s stuff that’s getting better all the time. So what would you say, just broad advice for somebody who’s considering getting something to start measuring their sleep and I like your use of the, the term sleep estimation device as opposed to something that measures objectively your sleep.
Jesse Cook: 32:33 Yeah, I’m so overwhelmed by the sheer volume of models available. You know, it’s seemingly an unavigatable terrain when trying to figure out which sleep tracker to buy. When you type in Google ‘best sleep tracker in 2019’ and even within that, there’s still 20 recommendations. And those may or may not have merit to them but that kind of gets away from the point that as humans we have this challenge when we have a ton of options, it’s called the paradox of choice. The more options we get, the worst we get at making a decision. And this falls right into this issue with the sleep trackers, I think it relates really well. So I think the first thing you have to do if you’re thinking about purchasing one of these products is, think about its purpose in your life and who you are as an individual. If you’re an individual that just wants something that they can wear on their wrist, that measure steps, that can give you some heart rate feedback and has the ability to produce some estimations of sleep time and maybe even classify your sleep across stages and you could be looking at getting one of the low end devices that are $30 to $50 and feel pretty comfortable in what it’s producing relative to the more expensive products.
Jesse Cook: 33:59 It doesn’t have to be one of the more name brand products. You may get better technological support, IT support if you do go with one of the more name brand products. So that may be advantageous in that regard. Starting there and then as you get more thorough in your use. If you’re a training triathlete, if you do have some medical concerns and you want better, more reliable, more frequent estimations of heart rate monitoring while you’re exercising or while you’re going through your day to day, then doing some homework and typing into the Google machine, the specific characteristics you’re looking for, whether it be the ability to swim with the device or the ability to not have to charge the device for 20 plus days. Many of these devices only have a battery life of five to seven days. If you’re someone who doesn’t want to have to continually charge your device, maybe you should opt for one that has a longer battery life but doesn’t have all the bells and whistles that the five to seven days device has.
Jesse Cook: 35:05 If you’re a triathlete or are looking for fitness than maybe you want to prioritize the device that has been evaluated for its movement tracking abilities and heart rate sensing and GPS capabilities that are additional components to the sleep tracking. Because in a lot of ways these devices aren’t that unique in their abilities to track sleep. They are in my understanding, utilizing relatively similar technologies as their underpinnings. And again, as we’ve pointed out, I have no idea what their algorithms are doing so I can’t make any comments there. So it really comes down to the individual. There’s no panacea of the devices, meaning that there is no device that performs the best across every single domain of activity tracking, heart rate tracking, sleep tracking. And so figuring out what is the most important characteristic for you, how much you’re willing to spend, what having a name brand linked to your device means for your comfort level. If it makes you quote unquote sleep better at night, knowing that you have one of these main manufacturer products, then do that. But if you’re perfectly comfortable using one of the more knock off, low expensive, not necessarily lower grade products and it doesn’t disrupt your sleep in that fashion, I say do that as well. So it really just comes down to being comprehensive with what you want to utilize the device for
Jeff Mann: 36:39 And I know you wanted to mention as well, this whole idea of sleep tracker induced anxiety, this term orthosomnia.
Jesse Cook: 36:49 The last thing that we want to do as sleep researchers and individuals in the medical field of sleep is cause worst sleep, right? Our goal is to have everyone sleep optimally and it definitely occurs in some individuals where having more information, complicates things and actually causes problems. The orthosomnia terminology has become a topic and it actually was a seminar at the sleep conference last year and it definitely pertains to a certain subset of individuals, specifically those who have a propensity for elevated clinically significant levels of anxiety. And so if you’re one of those individuals who may be susceptible to having these data negatively maladaptively influenced your sleep quality, then that’s something you need to be honest about with yourself as well. And then maybe just don’t do it, maybe not have one of the sleep trackers. Or the other option is to address that component of your life and potentially seek help in developing novel cognitive techniques to help ameliorate those issues. But that’s an entirely different topic and one that I’m not an expert in. But that’s a big thing. But as I mentioned earlier, and it’s really important to point out, we tend to see that for most individuals who monitor their sleep in some fashion, their sleep improves. So that’s a stamp of approval there.
Jeff Mann: 38:20 Yeah. Somebody who actually takes the act of expressing an active interest in their sleep, that in itself is hopefully going to have some positive impacts.
Jesse Cook: 38:32 Yeah. For the most part it should.
Jeff Mann: 38:34 Yeah. So maybe unless you’re one of these people who thinks they might be worrying more, if you’re one of those people, then maybe don’t buy a sleep tracker.
Jesse Cook: 38:45 If you’re going to define yourself based on an output of a device that may or may not be truly reliable and accurate, then the sleep tracker may not be the best thing. I guess if you’re going to critically evaluate yourself, then maybe a sleep tracker is not the best decision for you to utilize. But again, most individuals don’t fall into that camp and they tend to be pretty useful at least in maintaining or improving sleep schedule consistency. That’s a big one we see. People going to bed at similar times and rising at similar times rather than having spontaneity in their sleep patterns and then recognizing what characteristics associate with better quality sleep for you.
Jesse Cook: 39:36 You know, as you mentioned earlier, Jeff, a couple beers, doesn’t improve our sleep quality. Maybe the fact that I wasn’t on my computer for the hours leading into bed. Well, oh man, my sleep score was better. Maybe I shouldn’t do that. And just being your own scientists in a way, and comprehensively evaluating your life in the context of not just the output of this device, but how you actually feel as well.
Jeff Mann: 40:00 Yeah. Okay. Awesome Jesse. Do you mind if I put you on the spot slightly?
Jesse Cook: 40:11 With trepidation? I say yes or no, I’m comfortable, I guess.
Jeff Mann: 40:16 We’ve covered an awful lot of ground in this conversation and I just wondered because I really think this is important, both for consumers and the business community, but also for science and for medicine. Would you be in any way be able to encapsulate the main overriding message from this review, which we’re basing a lot of these conversations on today and also from, from your experience?
Jesse Cook: 40:49 Sure I will help do my best Jeff in that regard. So the ability to objectively measure somebody’s sleep in their natural environment is really powerful and useful. Bringing someone into the lab is just not practical. It costs a ton of money. It costs a lot of time. It can be an artificial setting. We’ve touched upon those types of things. Actigraphy, actigraphs have existed as the medium utilized to facilitate this need for many decades.
Jesse Cook: 41:28 Recently, wearable consumer technology has provided another means, another medium, another lens that’s much more affordable and is extremely prominent in our public to facilitate the objective, quote unquote measurement or estimation of sleep and natural environment. At this time though, these devices, although improving from earlier generations when they advanced from single sensor to multisensory, are still limited in their abilities to truly and comprehensively estimate sleep.
Jesse Cook: 42:07 They have demonstrated the ability to produce estimations of sleep duration, so your total sleep time at at night that are comparable to clinical actigraphs. However, these estimations are still not congruent with our gold standard polysomnography. Meaning that they’re still biased in some fashion, they’re typically overestimating total sleep time. Additionally, their ability to classify sleep, so the light sleep, deep sleep, the REM sleep that’s output by the device seemingly is much less accurate than their total sleep time estimations.
Jesse Cook: 42:54 So although it’s very useful and very powerful to see that information, there needs to be at least a pause or hesitation in digesting that information at this time. Currently there are some major barriers, specifically around the proprietary nature of the devices, their algorithms, the raw data that truly complicate the integration of these devices and the progression of these devices for clinical and research purposes.
Jesse Cook: 43:28 However, myself, researchers, our governing bodies in the sleep field, they are all encouraged by the prospects of these devices going forward. Their ability to acquire large datasets is extremely unique. Again, they’re very advantageous with their realtime feedback, their affordability, the sleekness and design, their ever evolving technology and growth. These are all very advantageous characteristics and we just ultimately as researchers want the best products to be available and described in the most appropriate and transparent ways. Ultimately we see this happening in the coming years and we’re very encouraged by the horizon.
Jeff Mann: 44:13 Awesome. Well, thanks for that. Like you, I’m optimistic. People being able to take control of the lives, expressing an interest in their sleep and their health. If there’s companies out there who want to get into this space and want to push the boundaries, business and commerce is always going to move faster than science. So the two just have to come to a point where they meet. Hopefully we’ve contributed to this conversation, snowballing into something, you know, a better outcome in the future.
Jesse Cook: 44:52 Yeah, we put the sheets on the bed or something of that nature, so hopefully produce the best quality sleep for society.
Jeff Mann: 45:00 Fantastic. Thanks so much, Jesse. I’ll let you get on with the rest of your day and maybe when we’ve recovered from this conversation we can revisit some of these topics a later date.
Jesse Cook: 45:15 Absolutely Jeff, I really appreciate you having me on. This has been a true joy and it’s a very important area to dissect at this time. It’s very challenging and then I appreciate you giving me the time and the opportunity to discuss.
