High-Frequency Data
Spoiler, it's a LOT of data
The following section will discuss our High-Frequency Data model and help you understand the limitations, processes, and methods for accessing High-Frequency Data.
What is High-Frequency Data
In the scope of this conversation, we're referring to events captured at a rate of at least one reading per minute or higher. Currently, data is limited to activity, blood glucose, and heart rate data.
Intraday vs High-Frequency
As you learn more about High-Frequency, you may also hear people describe this data differently. Some data sources refer to metrics calculated at this frequency as intraday, and others call it epoch data. Since Validic receives so much data over the day (i.e., Intraday), we've made the distinction of calling a single event, which records multiple moments in time, "High-Frequency Data." Yet, because engineering moves faster than branding, you'll notice our API endpoint is named Intraday. It's confusing, we know. Tomayto, Tomahto?
Supported Data Sources
At this time, there are limited data sources that expose High-Frequency Data. Below, we've outlined the requirements to access, the frequency it's delivered, and the consistency of the data.
Enabling your Validic account to access High-Frequency Data
Before your account can request High-Frequency Data, you must contact your Validic representative or contact our client services at [email protected].
| Data Source | Supported Metrics | Freqency | Limitations |
|---|---|---|---|
| Fitbit | steps heart_rate elevation distance | Per Minute (IE count of steps accumulated over one minute) | 3rd-party developers who want access to retrieve other Fitbit users’ Intraday data through the “Client” or “Server” application type is granted on a case-by-case basis. Applications must demonstrate necessity to create a great user experience. Fitbit is very supportive of non-profit research and personal projects. Commercial applications require thorough review and are subject to additional requirements. Only select applications are granted access and Fitbit reserves the right to limit this access. https://dev.fitbit.com/build/reference/web-api/intraday-requests/ |
| Apple Health | steps | Per Minute (IE count of steps accumulated over one minute) | Requires customers use the Validic Mobile Library to access data. Steps are calculated from the Apple Watch only. Steps are an average over the reported period. |
| Dexcom | blood_glucose | 5 Minutes, reported Hourly (IE One event, every five minutes) | Dexcom requires pre-approval and must review your application before you're granted full access to their data. You can learn more here. When you're ready to get started, contact [email protected] and we'll help you navigate the application process. This is currently only supported in the US. "Data from the Dexcom API is available with a one-hour delay. This delay is enforced on the data upload time, not on the timestamps of individual data points. The G5® and G6® Mobile applications upload once per hour, so the data will (on average) be 1.5 hours delayed. On the other hand, data uploaded directly from a receiver over USB (via the Dexcom CLARITY® uploader) is available immediately because it is viewed as an active, rather than passive, upload." |
| Garmin | steps distance | 15 minutes (IE: count of steps accumulated over 15 minutes) | No restrictions. |
Data Accessibility Notes
In all circumstances, the frequency and consistency of data delivery are determined by the end-user and how often they submit data to the cloud. In most cases, high-frequency data appears on predictable intervals of 5, 10, 15 Minutes, up to 1 hour. However, when working with this data, consider a user may only sync every few days. We suggest you replicate these conditions when testing to ensure your systems can scale appropriately.
Code Samples
Below are samples of High-Frequency Data. Spoiler, it's a LOT of data. Below you'll find examples to help set your expectations. While the identities have been changed to protect the innocent, these represent actual data collected from Validic employees.
event:data
data:{
"checksum":"9a442c33c5415f3757421b1c107e7a0d",
"created_at":"2019-10-13T14:59:50.094Z",
"deleted_at":null,
"end_time":"2019-10-13T14:45:00Z",
"granularity":{
"interval":15,
"unit":"min"
},
"id":"f814a36d2SAMPLE33740e5481555c",
"log_id":"30b5984SAMPLEd6ff4cf75f17e",
"metrics":[
{
"data_points":[
{
"time":"2019-10-13T04:30:00.000Z",
"value":28
},
{
"time":"2019-10-13T05:15:00.000Z",
"value":52
},
{
"time":"2019-10-13T08:15:00.000Z",
"value":17
},
{
"time":"2019-10-13T13:45:00.000Z",
"value":7
},
{
"time":"2019-10-13T14:45:00.000Z",
"value":41
}
],
"origin":"device",
"type":"steps",
"unit":"count"
},
{
"data_points":[
{
"time":"2019-10-13T04:00:00.000Z",
"value":0
},
{
"time":"2019-10-13T04:15:00.000Z",
"value":0
},
{
"time":"2019-10-13T04:30:00.000Z",
"value":22.68
},
{
"time":"2019-10-13T04:45:00.000Z",
"value":0
},
{
"time":"2019-10-13T05:00:00.000Z",
"value":0
},
{
"time":"2019-10-13T05:15:00.000Z",
"value":42.11
},
{
"time":"2019-10-13T05:30:00.000Z",
"value":0
},
{
"time":"2019-10-13T05:45:00.000Z",
"value":0
},
{
"time":"2019-10-13T06:00:00.000Z",
"value":0
},
{
"time":"2019-10-13T06:15:00.000Z",
"value":0
},
{
"time":"2019-10-13T06:30:00.000Z",
"value":0
},
{
"time":"2019-10-13T06:45:00.000Z",
"value":0
},
{
"time":"2019-10-13T07:00:00.000Z",
"value":0
},
{
"time":"2019-10-13T07:15:00.000Z",
"value":0
},
{
"time":"2019-10-13T07:30:00.000Z",
"value":0
},
{
"time":"2019-10-13T07:45:00.000Z",
"value":0
},
{
"time":"2019-10-13T08:00:00.000Z",
"value":0
},
{
"time":"2019-10-13T08:15:00.000Z",
"value":13.77
},
{
"time":"2019-10-13T08:30:00.000Z",
"value":0
},
{
"time":"2019-10-13T08:45:00.000Z",
"value":0
},
{
"time":"2019-10-13T09:00:00.000Z",
"value":0
},
{
"time":"2019-10-13T09:15:00.000Z",
"value":0
},
{
"time":"2019-10-13T09:30:00.000Z",
"value":0
},
{
"time":"2019-10-13T09:45:00.000Z",
"value":0
},
{
"time":"2019-10-13T10:00:00.000Z",
"value":0
},
{
"time":"2019-10-13T10:15:00.000Z",
"value":0
},
{
"time":"2019-10-13T10:30:00.000Z",
"value":0
},
{
"time":"2019-10-13T10:45:00.000Z",
"value":0
},
{
"time":"2019-10-13T11:00:00.000Z",
"value":0
},
{
"time":"2019-10-13T11:15:00.000Z",
"value":0
},
{
"time":"2019-10-13T11:30:00.000Z",
"value":0
},
{
"time":"2019-10-13T11:45:00.000Z",
"value":0
},
{
"time":"2019-10-13T12:00:00.000Z",
"value":0
},
{
"time":"2019-10-13T12:15:00.000Z",
"value":0
},
{
"time":"2019-10-13T12:30:00.000Z",
"value":0
},
{
"time":"2019-10-13T12:45:00.000Z",
"value":0
},
{
"time":"2019-10-13T13:00:00.000Z",
"value":0
},
{
"time":"2019-10-13T13:15:00.000Z",
"value":0
},
{
"time":"2019-10-13T13:30:00.000Z",
"value":0
},
{
"time":"2019-10-13T13:45:00.000Z",
"value":5.67
},
{
"time":"2019-10-13T14:00:00.000Z",
"value":0
},
{
"time":"2019-10-13T14:15:00.000Z",
"value":0
},
{
"time":"2019-10-13T14:30:00.000Z",
"value":0
},
{
"time":"2019-10-13T14:45:00.000Z",
"value":33.22
}
],
"origin":"device",
"type":"distance",
"unit":"m"
}
],
"offset_origin":"source",
"source":{
"device":null,
"type":"garmin"
},
"start_time":"2019-10-13T04:00:00Z",
"tags":[
],
"type":"intraday",
"user":{
"organization_id":"59ba7cSAMPL001c29216",
"uid":"garmin-intraday",
"user_id":"5c65a7bSAMPLE700019df1a5"
},
"utc_offset":-14400,
"version":"1.0"
}
event:data
data:{
"checksum":"0b59aa330b4b7fc632fb018d7c1f11a1",
"created_at":"2019-09-24T16:49:01.475Z",
"deleted_at":null,
"end_time":"2019-09-24T03:59:00Z",
"granularity":{
"interval":1,
"unit":"min"
},
"id":"bcd0f59daSAMPLE08e2e685ff",
"log_id":"afbbfSAMPLE84b6bc9f3295",
"metrics":[
{
"data_points":[
{
"time":"2019-09-23T20:14:00Z",
"value":4
},
{
"time":"2019-09-23T20:15:00Z",
"value":7
},
{
"time":"2019-09-23T20:16:00Z",
"value":30
},
{
"time":"2019-09-23T20:17:00Z",
"value":80
},
{
"time":"2019-09-23T20:18:00Z",
"value":102
},
{
"time":"2019-09-23T20:19:00Z",
"value":110
},
{
"time":"2019-09-23T20:20:00Z",
"value":110
},
{
"time":"2019-09-23T20:21:00Z",
"value":112
},
{
"time":"2019-09-23T20:22:00Z",
"value":108
},
{
"time":"2019-09-23T20:23:00Z",
"value":110
},
{
"time":"2019-09-23T20:24:00Z",
"value":110
},
{
"time":"2019-09-23T20:25:00Z",
"value":27
},
{
"time":"2019-09-23T20:40:00Z",
"value":7
},
{
"time":"2019-09-23T20:41:00Z",
"value":16
},
{
"time":"2019-09-23T20:42:00Z",
"value":34
},
{
"time":"2019-09-23T20:44:00Z",
"value":31
},
{
"time":"2019-09-23T20:45:00Z",
"value":26
},
{
"time":"2019-09-23T20:46:00Z",
"value":104
},
{
"time":"2019-09-23T20:47:00Z",
"value":42
},
{
"time":"2019-09-23T20:48:00Z",
"value":9
},
{
"time":"2019-09-23T20:49:00Z",
"value":18
},
{
"time":"2019-09-23T20:57:00Z",
"value":14
},
{
"time":"2019-09-23T20:58:00Z",
"value":18
}
],
"origin":"device",
"type":"steps",
"unit":"count"
}
],
"offset_origin":"profile",
"source":{
"device":null,
"type":"fitbit"
},
"start_time":"2019-09-23T13:54:00Z",
"tags":[
{
"name":"range_start",
"value":"2019-09-24T09:54:00+00:00"
},
{
"name":"range_end",
"value":"2019-09-23T23:59:59+00:00"
}
],
"type":"intraday",
"user":{
"organization_id":"59ba7cdafSAMPLEc29216",
"uid":"fitbit-intraday",
"user_id":"5c1003aSAMPLE012a6fc1"
},
"utc_offset":-14400,
"version":"1.0"
}
event:data
data:{
"checksum":"0c1e07c59b1e2b03bf0cffdf633e8e5e",
"created_at":"2019-10-14T12:29:55.743Z",
"deleted_at":null,
"end_time":"2019-10-14T12:28:00Z",
"granularity":{
"interval":1,
"unit":"min"
},
"id":"9ae640f4219SAMPLEc7d912df027c",
"log_id":"25bb6e041SAMPLE9ee53000a14a6",
"metrics":[
{
"data_points":[
{
"time":"2019-10-14T11:28:00Z",
"value":86
},
{
"time":"2019-10-14T11:29:00Z",
"value":82
},
{
"time":"2019-10-14T11:30:00Z",
"value":74
},
{
"time":"2019-10-14T11:31:00Z",
"value":72
},
{
"time":"2019-10-14T11:32:00Z",
"value":72
},
{
"time":"2019-10-14T11:33:00Z",
"value":70
},
{
"time":"2019-10-14T11:34:00Z",
"value":70
},
{
"time":"2019-10-14T11:35:00Z",
"value":71
},
{
"time":"2019-10-14T11:36:00Z",
"value":71
},
{
"time":"2019-10-14T11:37:00Z",
"value":71
},
{
"time":"2019-10-14T11:38:00Z",
"value":71
},
{
"time":"2019-10-14T11:39:00Z",
"value":70
},
{
"time":"2019-10-14T11:40:00Z",
"value":69
},
{
"time":"2019-10-14T11:41:00Z",
"value":70
},
{
"time":"2019-10-14T11:42:00Z",
"value":72
},
{
"time":"2019-10-14T11:43:00Z",
"value":75
},
{
"time":"2019-10-14T11:44:00Z",
"value":69
},
{
"time":"2019-10-14T11:45:00Z",
"value":70
},
{
"time":"2019-10-14T11:46:00Z",
"value":69
},
{
"time":"2019-10-14T11:47:00Z",
"value":68
},
{
"time":"2019-10-14T11:48:00Z",
"value":69
},
{
"time":"2019-10-14T11:49:00Z",
"value":70
},
{
"time":"2019-10-14T11:50:00Z",
"value":68
},
{
"time":"2019-10-14T11:51:00Z",
"value":70
},
{
"time":"2019-10-14T11:52:00Z",
"value":70
},
{
"time":"2019-10-14T11:53:00Z",
"value":69
},
{
"time":"2019-10-14T11:54:00Z",
"value":67
},
{
"time":"2019-10-14T11:55:00Z",
"value":67
},
{
"time":"2019-10-14T11:56:00Z",
"value":67
},
{
"time":"2019-10-14T11:57:00Z",
"value":67
},
{
"time":"2019-10-14T11:58:00Z",
"value":67
},
{
"time":"2019-10-14T11:59:00Z",
"value":67
},
{
"time":"2019-10-14T12:00:00Z",
"value":65
},
{
"time":"2019-10-14T12:01:00Z",
"value":66
},
{
"time":"2019-10-14T12:02:00Z",
"value":65
},
{
"time":"2019-10-14T12:03:00Z",
"value":64
},
{
"time":"2019-10-14T12:04:00Z",
"value":66
},
{
"time":"2019-10-14T12:05:00Z",
"value":65
},
{
"time":"2019-10-14T12:06:00Z",
"value":66
},
{
"time":"2019-10-14T12:07:00Z",
"value":66
},
{
"time":"2019-10-14T12:08:00Z",
"value":66
},
{
"time":"2019-10-14T12:09:00Z",
"value":66
},
{
"time":"2019-10-14T12:10:00Z",
"value":66
},
{
"time":"2019-10-14T12:11:00Z",
"value":65
},
{
"time":"2019-10-14T12:12:00Z",
"value":64
},
{
"time":"2019-10-14T12:13:00Z",
"value":65
},
{
"time":"2019-10-14T12:14:00Z",
"value":64
},
{
"time":"2019-10-14T12:15:00Z",
"value":65
},
{
"time":"2019-10-14T12:16:00Z",
"value":66
},
{
"time":"2019-10-14T12:17:00Z",
"value":66
},
{
"time":"2019-10-14T12:18:00Z",
"value":64
},
{
"time":"2019-10-14T12:19:00Z",
"value":65
},
{
"time":"2019-10-14T12:20:00Z",
"value":67
},
{
"time":"2019-10-14T12:21:00Z",
"value":67
},
{
"time":"2019-10-14T12:22:00Z",
"value":64
},
{
"time":"2019-10-14T12:23:00Z",
"value":63
},
{
"time":"2019-10-14T12:24:00Z",
"value":63
},
{
"time":"2019-10-14T12:25:00Z",
"value":64
},
{
"time":"2019-10-14T12:26:00Z",
"value":66
},
{
"time":"2019-10-14T12:27:00Z",
"value":61
},
{
"time":"2019-10-14T12:28:00Z",
"value":62
}
],
"origin":"device",
"type":"avg_heart_rate",
"unit":"bpm"
}
],
"offset_origin":"profile",
"source":{
"device":null,
"type":"fitbit"
},
"start_time":"2019-10-14T11:28:00Z",
"tags":[
{
"name":"range_start",
"value":"2019-10-14T00:00:00Z"
},
{
"name":"range_end",
"value":"2019-10-14T23:59:59Z"
}
],
"type":"intraday",
"user":{
"organization_id":"59ba7cdaSAMPLEc29216",
"uid":"fitbit-intraday",
"user_id":"5c1003SAMPLE00012a6fc1"
},
"utc_offset":-14400,
"version":"1.0"
}
event:data
data: {
"checksum": "SAMPLE98dbf4f3663c31ce49c5987ce0",
"created_at": "2023-11-09T15:35:00.638Z",
"deleted_at": null,
"end_time": "2023-11-09T11:59:20Z",
"granularity": {
"interval": 5,
"unit": "min"
},
"id": "8c6088d419ce5f98ee4a6e2c17ebff4c",
"log_id": "2023309bSAMPLEc6b31527fc8b8e2023",
"metrics": [
{
"data_points": [
{
"time": "2023-11-09T11:59:20Z",
"value": 137
},
{
"time": "2023-11-09T11:54:19Z",
"value": 134
},
{
"time": "2023-11-09T11:49:20Z",
"value": 136
},
{
"time": "2023-11-09T11:44:20Z",
"value": 135
},
{
"time": "2023-11-09T11:39:20Z",
"value": 136
},
{
"time": "2023-11-09T11:34:20Z",
"value": 137
},
{
"time": "2023-11-09T11:29:20Z",
"value": 134
},
{
"time": "2023-11-09T11:24:20Z",
"value": 142
},
{
"time": "2023-11-09T11:19:20Z",
"value": 146
},
{
"time": "2023-11-09T11:14:19Z",
"value": 148
},
{
"time": "2023-11-09T11:09:19Z",
"value": 148
},
{
"time": "2023-11-09T11:04:19Z",
"value": 147
}
],
"origin": "device",
"type": "blood_glucose",
"unit": "mg/dL"
}
],
"offset_origin": "user_defined",
"source": {
"device": null,
"type": "dexcom"
},
"start_time": "2023-11-09T11:04:19Z",
"tags": [],
"type": "intraday",
"user": {
"organization_id": "568bSAMPLE65630008000050",
"uid": "sample_user",
"user_id": "64b68SAMPLE4d9000e569174"
},
"utc_offset": -21600,
"version": "1.0"
}
Updated about 1 month ago