Question on FIT file parsing: Misaligned Splits / "Summary First" vs "Summary Last" handling

[davidxcz]

Hey everyone,

It’s not clear that this is even a bug. But there is no better category

I’ve been digging into some FIT files generated by OpenRowingMonitor (ORM) because I noticed the active laps and rest intervals weren’t aligning quite right when creating intervals from laps. ORM’s files seem to be very standards compliant and using the “from laps” feature gets it damn near perfect, off-by-one, or rarely two. Which made me think it’s a programming/timing thing

While poking around in FitFileViewer and doing some research, I noticed that all the lap messages in the ORM files have the exact same timestamp (the start of the workout), while start_time marks the actual start of the lap.

This was ununtitive to me at first, but according to the Garmin FIT SDK documentation and forums:

"For files using the Summary First method, the timestamp approximates the file creation time. In either case, the timestamp should not be used to determine the start or end time of the summary message. To properly decode FIT Activity Files… the End Time for the summary messages should be calculated as: end_time = start_time + total_elapsed_time "

Garmin notes that historically their devices used “Summary Last”, leading many platforms to assume that timestamp represents the end of the lap, and therefore failing to properly handle “Summary First” files.

My direct question is: How does Intervals.icu handle lap message parsing under the hood? Does the parser rely on the timestamp to derive the lap span/anchor records, or does it calculate start_time + total_elapsed_time as Garmin recommends for Summary First files?

It is totally plausible that ICU is doing this correctly, and this is actually intended behaviour (top level “intervals” vs “laps” being parsed, etc). It would align with how Strava is doing things too (though I do not have faith they would actually answer the question if I asked lol).

Just thought I’d ask!

Thanks!

[R2Tom]

Can you show an example what’s the bug report about? Because this is more a question about the implantation.

@davidxcz

My direct question is: How does Intervals.icu handle lap message parsing under the hood?

[davidxcz]

It is a question based on implementation, yep! And based on the answer, could be a bug report?

I’ll put together some screenshots, though.

[Jaap_van_Ekris]

My direct question is: How does Intervals.icu handle lap message parsing under the hood? Does the parser rely on the timestamp to derive the lap span/anchor records, or does it calculate start_time + total_elapsed_time as Garmin recommends for Summary First files?

Just tested it, it seems to ignore the timestamp. ORM will use the file creation time in the future.

[davidxcz]

So, the maybe bug:

Here are the intervals I get when I use the “Use Laps” Action:

and here are the laps according to fitfileviewer

It seems like it picks up the right number of laps, but the other data seems pretty far off?

[Jaap_van_Ekris]

@davidxcz

It seems like it picks up the right number of laps, but the other data seems pretty far off?

Did you upgrade the install, because this still is start-time based where current 0.9.7 uses write-time as timestamp?

[davidxcz]

The workout predates that commit by a few days

But my read of this:

@Jaap_van_Ekris

it seems to ignore the timestamp

Was that it didn’t matter — am I misunderstanding?

Also… it sounds like you’re switching the format to summary-last:

For files using the Summary Last method, the timestamp approximates the end time of the summary message.

Despite having the Summary “first”?

Which makes pragmatic sense and I understand (and appreciate) the rationale. But. If there’s a spec out there that’s not being followed, and that bug is on the consumer side, and the consumer is willing to adopt the spec behavior, wouldn’t that be preferable?

I guess there’s always the possibility of switching back later, when/if the spec is adopted… by everyone

[Jaap_van_Ekris]

Lets get things straight here. When I look at the official Garmin FIT documentation:

The sequence of messages in an Activity file typically follows one of two patterns: summary first or summary last.

With the summary first message pattern, all of the summary messages are grouped together at the beginning of the file, followed by the moment-by-moment messages like Records and Events.

Whereas with the summary last sequence pattern, the summary messages are written to the file as they occur during the recording of the activity. With this pattern, the summary messages are written to the file at the end of the time interval that they represent.

So 'Summary First has a lot more to do with file structure (something that fitFileViewer.com will not show, but RunAnalyze’s viewer does) than just timestamps.

Looking at the timestamp requirements, Garmin tells the following in their announcement:

For files using the Summary Last method, the timestamp approximates the end time of the summary message. For files using the Summary First method, the timestamp approximates the file creation time. In either case, the timestamp should not be used to determine the start or end time of the summary message. ​

As we create the file at the end, our interpretation is valid for me. I’d agree that Garmin uses the activity start time as timestamp for the summary messages, which was the reason I copied their approach. However, it creates weird semantic situations: a file starts with a ‘session’ record, timestamped on the session start, but representing the end state (including the total elapsed time of the session and average HR etc.). It gets even more weird for the lap summary, as it is timestamped on the start time of the session, but the second lap starttime is in the future.

So for me, session endtime is more semantically correct as it is internally consistent. And I don’t have the impression Garmin had an issue with it.

[davidxcz]

@Jaap_van_Ekris

As we create the file at the end, our interpretation is valid for me.

Huh, alright — if you say “Summary Last” is what you’re doing anyway, far be it for me to argue! Based on how on top of the fit file stuff you’ve been, I’ve got all the faith in your approach/interpretations

@Jaap_van_Ekris

With the summary first message pattern, all of the summary messages are grouped together at the beginning of the file, followed by the moment-by-moment messages like Records and Events.

My naive read of this was that the file order mattered; in both the fitfieleviewer and runalyze tools, it looks to me like the Session is near the top/“start” of the file. But. I think I’m now understanding that “session” ≠ “summary”

So I will mark as resolved!

[Jaap_van_Ekris]

@davidxcz

if you say “Summary Last” is what you’re doing anyway, far be it for me to argue

OpenRowingMonitor 0.9.7 uses “Summary First”, and ORM 0.9.6 uses “Summary last”. We changed as the recording structure for 0.9.6 is more complex (it requires quite a hierarchical structure) and writing in one go was always our approach anyways.

So we write in one go, starting with a session summary, Laps summary, etc. (this is a copy of Garmin’s structure as I want it to be digested by Garmin as well).

[davidxcz]

… alright. I’m going to attempt to summarize for posterity. Please correct me if I’m wrong:

@Jaap_van_Ekris

ORM 0.9.6 uses “Summary last”

@Jaap_van_Ekris

with the summary last sequence pattern, the summary messages are written to the file as they occur during the recording of the activity. With this pattern, the summary messages are written to the file at the end of the time interval that they represent.

@Jaap_van_Ekris

For files using the Summary Last method, the timestamp approximates the end time of the summary message.

Where a summary message includes Session and Lap; so this would be the end of each individual Session(multisport)/Lap.

and

@Jaap_van_Ekris

OpenRowingMonitor 0.9.7 uses “Summary First”

@Jaap_van_Ekris

With the summary first message pattern, all of the summary messages are grouped together at the beginning of the file

@Jaap_van_Ekris

For files using the Summary First method, the timestamp approximates the file creation time.

So every Session(multisport)/Lap would have the same timestamp: file creation time.

---

The decision made here is

@Jaap_van_Ekris

ORM will use the file creation time in the future

where

@Jaap_van_Ekris

we create the file at the end

So the same timestamp for every session/lap, and that timestamp is the end of the workout:

timestamp: fitWriter.time(workout.endTime)

---

This is a change from what 0.9.7 was doing prior to this discussion. It used:

@Jaap_van_Ekris

write-time as timestamp

Interpretted NOT to mean file write time (this is where a lot of my confusion stemmed from), but “flywheel spin” write time, or workout start time:

timestamp: fitWriter.time(workout.startTime)

---

Thoughts/feelings about using start time include:

@Jaap_van_Ekris

Garmin uses the activity start time as timestamp for the summary messages

@Jaap_van_Ekris

it creates weird semantic situations… [with] session record… [and] lap summary

Feelings about using end time/file creation time:

@Jaap_van_Ekris

it is more semantically correct as it is internally consistent.

@Jaap_van_Ekris

I don’t have the impression Garmin had an issue with it.

---

Fit file standard and ORM implementation aside, the answer to the intial question:

@davidxcz

How does Intervals.icu handle lap message parsing under the hood? Does the parser rely on the timestamp to derive the lap span/anchor records[?]

Jaap suggests that the answer here is “no, it does not rely on timestamp”:

@Jaap_van_Ekris

Just tested it, it seems to ignore the timestamp.

But there is hope that the “Use Laps” behaviour will be correct after this latest change. Which means that other changes to the the latest version of 0.9.7 does things to correct one of the other 2 variables:

@davidxcz

calculate start_time + total_elapsed_time as Garmin recommends for Summary First files

I will test to confirm this in about a week, when I will have access to my ORM machine again

…

ALL THIS TO SAY THAT INTERVALS IS CALCULATING THINGS PROPERLY, GREAT JOB @david

[Jaap_van_Ekris]

@davidxcz

It seems like it picks up the right number of laps, but the other data seems pretty far off?

Aren’t these laps automatically created based on intensity (and thus ignoring any structure ORM provides)? I read the files in several apps, including FitFileViewer, Garmin Connect, Strava and they all work flawlessly with the splits provided.

[davidxcz]

There is a “use laps” function under “Actions” at the bottom of the page that should also auto-create interval!

[Jaap_van_Ekris]

Found this option. When looking at the resulting splits table, I am puzzled as well.

I tested with a 10K steady state with 500 meter splits, and my conclusion is that it seems to ignore the lap-table data completely?

The reason I say this, is because OpenRowingMonitor reports the interval at the exact crossing and interpolates if needed (for a Concept2, we can measure distance only in 3 to 4 cm accuracy). So my split table in fitfileviewer.com will allways report exact 500.00 meter splits. Same goes for intervals: a 5000 meter interval will be exact 5000.00 meters.

When I open my fit-file in intervals.icu, that data seems to be ignored and splits are 498 meters, and sometimes 504 meters?

I strongly suspect the distances are based on some interpretation of something, but it certainly isn’t our splits table. The weird thing is, we actually artificially inject the splits and interval boundaries as records (these are ‘strokes’ that repeat the same stroke number, but hit the boundary exactly) to anchor these boundaries there as well.

[Jaap_van_Ekris]

I have recorded a session with my Garmin Epix Gen2 via Ant+, with an automatic lap every 500 meters. No weird stuff, just a 10k steady state from start to finish. This reports per second, so stroke info is lost here.

Garmin connect also shows these laps as is: a lap every 500 meters.

Looking at the laps on intervals.icu, they show weirdly inconsistant distances:

lap	distance
1	502 meters
2	506 meters
3	370 meters
4	496 meters
5	503 meters
6	498 meters
…	… meters

So my conclusion is that intervals.icu is messing up lap data, regardless of source…

[pepe]

Your analysis is missing very important points. Intervals does fit file Elapsed Time Reconstruction and Lap Boundary Reconciliation validating against record data streams. This reduces raw FIT file quantification errors and ensures consistent second-level timing accuracy.

[app4g]

there’s no way to actually get clean laps down to the accuracy of rounded numbers like 500m. It’s actually approximated of sorts (I asked once in the Garmin Fit forum) based on speed and such.

Garmin writes these laps as rounded numbers (which I like) but in actual fact, they are not and that’s how Intervals.icu is doing it.

like what @pepe said - “elapsed time reconstruction” using the actual data written to the FIT file and just doing the calc of end time - start time (of lap) and distance.

[Jaap_van_Ekris]

@pepe

Your analysis is missing very important points. Intervals does fit file Elapsed Time Reconstruction and Lap Boundary Reconciliation validates against record data streams. This reduces raw FIT file quantification errors and ensures consistent second-level timing accuracy.

Thanks for this clarifying answer. But my summary of this behaviour: this post processing completely messes up perfectly valid data.

As I can’t speak for Garmin’s implementation, I can speak for OpenRowingMonitors.

OpenRowingMonitor works with nanosecond accuracy and reports with milisecond precission. For distance, we work on milimeter precission. While the timestamps for records can’t handle that, the periods in laps, splits and session can. To help post-processing analysis, we actually inject position reports in the records that are on the exact boundary of the lap, split or session. So why is this messing about with our data? I simply don’t see a trigger to move a lap indicator over a 130 meters (!).

[Jaap_van_Ekris]

@app4g

there’s no way to actually get clean laps down to the accuracy of rounded numbers like 500m. It’s actually approximated of sorts (I asked once in the Garmin Fit forum) based on speed and such.

Garmin writes these laps as rounded numbers (which I like) but in actual fact, they are not and that’s how Intervals.icu is doing it.

This actually depends on the origin. A watch only gets a position report every 400ms, so some interpolation/extrapolation is needed indeed.

However, on a rowing machine we get an internal position report every 2 to 5ms, and we actually interpolate between these datapoints to get to the true crossing. So when we say that it took 120.000 seconds to travel 500.00 meters, we truly mean it.

@app4g

like what @pepe said - “elapsed time reconstruction” using the actual data written to the FIT file and just doing the calc of end time - start time (of lap) and distance.

This leads to splits being off by 130 meters? While the underlying data is sound? Shouldn’t happen, especially as the underlying data is accurate to the nanosecond and consistent across streams. There is no “end time” in the lap stream. There is in the split stream, but that isn’t that interesting I think. I think there is a lot of guesswork going on here that simply goes horribly wrong here…

[R2Tom]

And what does your speed stream look like? Because if your average speed in this interval would be 200m/min - then what would be correct? Distance or speed?

As intervals was a cycling platform at first, I think it consider speed as more trustable (because on a bike a speed sensor is more reliable than GPS distance).

Of course this shouldn’t be the case for indoor rowing. But anyway, if speed /time is different than distance, which one would you trust more? And can you confirm, that your speed stream is accurate?