Two weeks ago I was in NYC for a slew of events, product pickups, and meetings. Most of which you’ve heard about by now. Be it the DJI Mavic 2 launch, the Garmin Vivosmart 4 announcement, or the Fitbit Charge 3 announcement. And that’s only the half of it. But in that 36 hours I had on the ground in NY, I snuck in a run around Central Park…starting from Times Square.
I’ve done this countless times before. I usually make a point of getting in at least one run in Central Park anytime I’m in town. And while I was exhausted from an early morning transatlantic flight and a long day of meetings/shooting, I went out for an evening run to ‘freshen up’. Or something like that.
For whatever reason, I decided to shoot a YouTube video about it. Seemed like a fun vlog style thing to do. Then I decided to write a post about it. After which I decided to go down a rabbit hole of looking at data nearly a decade old…so the post got a lot longer. But, if you don’t have time for all that – here’s the video:
If you still have time and want to get a bit geeky (as opposed to the more fun video), continue on!
Oh, and I totally get that thousands of people run in NYC every day with GPS devices. But very few people run with 3 GPS devices and then chart/plot the results. So…here I am in an old-school DCR style totally geeky deep dive.
A NYC Test Run:
In case you didn’t watch the video (why?!?) the premise here is simple. I took three watches with me out for the run (fear not, I had an entire suitcase full of other watches). Each watch was configured with the best GPS settings offered on each unit that would, in theory, give me the most accurate data:
I know you might ask why not the Suunto 9 instead? Quite frankly I get better results from the Suunto Trainer Wrist HR. My Suunto 9 review is done, so I’m back to using the Suunto Trainer Wrist HR as a reference device. Love that little thing. Why not the Polar V800? Cause I’m still trying to find it…from last December. I’m starting to wonder if it was a casualty in the move. Maybe when I unpack later this month when construction in the new DCR Studio is finished I’ll find it.
The COROS Pace was in the mix because I’m prepping an in-depth review on that. I wore the Garmin and COROS Pace units each on one wrist, and then simply hand-held the Suunto, keeping the satellite bump pointed at the sky at all times. I thought about adding my phone to the mix, but honestly my phone battery had a long day already and was about to die. So no point in dragging that along only to have it die mid-run.
I started the run from Times Square, as it made for a solidly tough place to begin a run from a GPS acquisition standpoint. I’ve started, give or take a few blocks, from here over the years, and it used to take half a dozen minutes to get GPS signal. In this case – a mere 20 seconds for the Garmin Fenix 5 Plus. A couple minutes for the Suunto, and over 5 minutes for the COROS Pace.
With that, I headed out towards Central Park via Broadway. So basically a straight but not terribly wide road (in that section) snaking between the towering buildings of Manhattan:
Here’s the results of that first section (you can click on that link to dive into the DCR Analyzer files for the entire run):
You’ll probably notice two things immediately. First, it’s a mess. But second, the COROS Pace decides to wait 5 blocks to start plotting its GPS track. While all devices did start with full bars and satellite lock, apparently that was a fake-out play on its part. Once it decided to start plotting points, it wasn’t any better than the others.
I’d argue for this specific section the Fenix 5 Plus was closest to the road, but that’s like giving an award to someone who gets a ‘D’ in class, merely because the only two other people in class got an ‘F’. Roughly. Though, these were hardly the worst GPS tracks I’ve ever seen in NYC. So that’s positive I suppose.
In any case, crossing the street into Central Park the COROS Pace was the closest to where I went. After which, the Suunto took over the lead on accuracy for the exact path I was on (or closest to it). But once I hit the main road in Central Park, things equalized between the units.
Below left, showing my entry into Central Park. And at right, on the inner side of the pathway.
One could quibble about which side of the road each GPS track said I was on (I was physically always on the innermost side), which meant that for the outbound portion the Garmin was generally most correct while the other two were on the other side of the road. But I suspect most wouldn’t care about this tiny difference.
Plus, on the return coming back down the other side of the park the Garmin got an odd offset of perhaps 10-meters into the woods for almost the entire length of the park. The other two stayed on the road (right or wrong side be damned).
Offsets like this aren’t super uncommon actually, and do hit all models from time to time on various workouts. Today was just the day for such an event for my watch. Win some, lose some.
Oh – along the way a DCR Reader spotted me and took a selfie as we ran. You can find this whole interaction at about the 6:08 marker of the video above. It’s pretty funny and worth the quick watch!
Close to the end of the park though, all three units joined back up again and ended happily together:
From a distance totals standpoint, here’s where they sat:
Ignore the zero-value elevation gain on the Suunto. Unfortunately Suunto’s Movescount doesn’t properly write the .FIT file header data, so in the DCR Analyzer we don’t have the ‘known’ data to pull from (we don’t want to recalculate it, as that’s not what appeared on your watch and is making assumptions).
I’m not sure which watch is ultimately the ‘winner’ here, if there is one. I suppose it’s not the COROS Pace merely because it fumbled the play at the beginning. So that leaves the Suunto and Garmin. The Garmin has a (barely) better track in the tougher buildings. And on the outbound portion the Garmin was on the correct side of the road. But on the return the Garmin was off in the woods geocaching or something with the weird offset. That wouldn’t have impacted distance meaningfully, but certainly could have impacted Strava Segments or just the vanity of a perfect GPS track (which, is important to me). Your call.
(Note: All of the charts in these accuracy portions were created using the DCR Analyzer tool. It allows you to compare power meters/trainers, heart rate, cadence, speed/pace, GPS tracks and plenty more. You can use it as well for your own gadget comparisons, more details here.)
Was it better years ago?
Still, I was curious – has much changed here in terms of GPS accuracy?
People always seem to say that previous devices were better than current devices. I often hear that a device from 2008 or 2010 made better GPS tracks than a device from 2018. I suspect the vast majority of that is just the normal human brain’s ability to slowly over time view everything through rose-tinted glasses of times (except for bad airline flights). Still, off I went searching.
While I’ve run probably dozens of times in NYC over the years, my ability to find all these runs and the corresponding watches is a bit of a mess. In theory it should be easy, but in practice not so much. Part of the complexity is that the further back you go in time, the less likely you used cloud services. For example, my older Garmin FR305 certainly wasn’t uploading to Garmin Connect in its heyday.
So I dug into each platform to see what older runs I had. I started with Suunto, because they have by and away the best way to search old activities from a heatmap standpoint. I simply zoom into NYC on the map and select to just show my runs. And done.
Only problem was that I’ve only got one other NYC run on a Suunto device (remember, I don’t live in NYC). Ironically enough, almost precisely one year prior on the exact same Suunto Trainer Wrist HR:
Looking into that track, the vast majority of Central Park was perfect, but it did bobble at the start for some odd reason:
Next, I tried to do the same within Polar, and it too would have been super easy. Except this:
Next, Garmin Connect. I can’t seem to find any way to zoom around my own personal map there anymore – so I had to filter by activity names. I did searches on both New York and NYC and some other variants. They turned up a handful of items over the past 4-5 years. Still, it was also definitely missing some items within that timeframe too – probably because I renamed them something other than New York for some reason.
In any event, going back to some of the older ones, there was one from 2015 that almost mirrored the same route – starting a few blocks from Times Square and then doing a loop around Central Park. I did this on that day with both a FR920XT and a Forerunner 630. Clicking on either of the images below takes you to those Garmin Connect files to look at.
On that same street as before, it was pretty much equally as sucky. Here’s the FR920XT track for that starting point:
And here’s the FR630 – it was worse:
The rest of Central Park was fine though – nothing notable there.
So let’s turn back the clock again a year prior. Again, a FR920XT (here’s that file), but also a Fenix 3 (original edition, no HR yet – here’s that file). In this case, I started a few blocks away, did a full Central Park loop and then went back into the buildings down to Times Square. As you can see, that was a crap-show too, though slightly better on the return. Barely. However, it still had a rough time getting into Central Park and getting out of Central Park as you can see at the edge of the green.
The Fenix 3 meanwhile was marginally better in that area/stretch. By the way, in case you’re wondering why these screenshots are so small, it’s because you can’t make the map any bigger on Garmin Connect (or at least, I’ve never found a way to do so…why Garmin, why???).
Neither watch displayed much oddity once in Central Park – all was good there.
Prior to that point though, I can’t seem to find much older on Garmin Connect. So I turned to Training Peaks. This would have a record of all my runs/rides/swims since around 2007 or 2008. So digging into that I went.
Like Garmin, they don’t offer a way to filter on a map. But even worse is that back in the day most workouts weren’t named by location either. So I had to simply hope that I had put something about NY in the description to my coach that I could find in a text search. Only four results came back, one of which was a passing reference to NYC from a run in Paris. One was a bike, and the final two a run. Of which, one of those runs was my NYC Triathlon Race, the run portion:
If we look closely at the portion that goes from the river to the park, you’ll see it does roughly follow the road. But that road’s a heck of a lot wider and with shorter buildings:
Not only that, but it still meandered through buildings. According to my race report from the NYC Triathlon that year, I was wearing a FR310XT. I was also wearing something on my other wrist, but I’m honestly not sure what it was. I can’t quite tell. However, doing some digging I was wearing/testing the older Forerunner 110 in an older post a few weeks prior – and it looks pretty similar. So maybe that’s it.
Then we’ve got a run back from 2010. This was starting a handful of blocks away from Central Park, a loop or so around Central Park and then back. While I don’t know the precise path I was taking back then, I think we can all agree it wasn’t diagonally through these buildings.
The sections in Central Park were pretty much the norm, all good there:
Unfortunately, Training Peaks doesn’t display the older device types here. So I had to download the original file, then crack it open and look at the file contents to see which device it was – the good ol’ FR310XT:
Phew.
And that’s where I run out of searchable/findable data. I’ve no doubt got older runs probably on Garmin Connect and Training Peaks, but they certainly aren’t filterable easy enough to find them. And then there’s likely runs somewhere in an offline file of the desktop version of SportTracks from before that. But I haven’t had SportTracks desktop installed in years. Plus, this rabbit hole is deep enough already…for now.
Still, it’s clear that at least in the random snippets here, things weren’t really any better back then. I’d say they trended either roughly the same or perhaps slightly worse in the buildings. Plus, what I do remember (and some of my older posts point this out) is that I’d often wait 5-8 minutes for GPS satellite acquisition. As you saw in the video – it was under 20-25 seconds for the Garmin, with the Suunto not terribly far behind. And that’s considered slow – purely because of the buildings.
Ultimately though – for now, big tall buildings are still big tall buildings, and by and large, they’re going to block GPS signals. Even my iPhone – the latest and greatest available, still would struggle as I walked around town to place the blue dot on the correct street. Whether you’re in NYC or Dubai, I wouldn’t expect massive improvements in GPS accuracy anytime soon. Instead, I think you’ll continue to see minor improvements like you can see with the data here. Or ancillary type improvements with GPS lock time that are clearly far faster than before. Not to mention today’s devices have far more features and functionality packed into the same form factor (or smaller).
In any case – hope you enjoyed this random jaunt down the rabbit hole (or memory lane).
With that – thanks for reading!
FOUND THIS POST USEFUL? SUPPORT THE SITE!
Hopefully, you found this post useful. The website is really a labor of love, so please consider becoming a DC RAINMAKER Supporter. This gets you an ad-free experience, and access to our (mostly) bi-monthly behind-the-scenes video series of “Shed Talkin’”.
Otherwise, perhaps consider using the below link if shopping on Amazon. As an Amazon Associate, I earn from qualifying purchases. It doesn’t cost you anything extra, but your purchases help support this website a lot. It could simply be buying toilet paper, or this pizza oven we use and love.
Interesting would be to do a run with only a GPS watch and a GPS + Glonass enabled watch and see if the GPS + Glonass is getting beter position than the GPS only watch…
There’s a lot of other factors involved to eg position in sky of satellites giving a dilution of precision. you’d have to factor that in for each constellation combo used.
I borrowed a forerunner 305 last yeat (IIRC). I always remembered it was ‘great’ at some point ‘back in the day’. unfortunately when i got rid of my 305 I bought some rose tinted spectacles – aka, as dcr says, it really wasn’t any better back then despite what we all remember
Thanks a lot for digging! This was insightful and even more so with all majors releasing new flagships this year. We already heard from Garmin and Suunto, and now Polar is about to follow up on veritable V800.
On a separate note, next time you in NYC, you should probably do DC Rainmaker Club meetup. Central Park or not. Not everyone will be running, but everyone can be drinking ?
I have read all your reviews. But this one is not GPS accuracy testing. The reviews are more close to what you call GPS accuracy testing. Not this post man. Not this post.
Ray’s ‘accuracy’ testing has always really been just eye-balling it. No binning, no RMSE, no bias. It’s… interesting, but doesn’t really show or prove anything.
But that’s not why people come here. Ray has a very readable and entertaining writing style, and lots of good photography.
Depending on this you can at least predict or “know” at what conditions the watches performed.
What I mean:
On a good day and time that a lot of satellites are available it must be easy for most, like perhaps your historical runs and comparisons that you mention.
On a bad day it shows which device is superior when less satellites are visible.
Check it out.
I am here to help perhaps “upgrade” your testing procedures.
Polar v800 still the most accurate. Great reply Jimmy BTW. Funny how Ray couldn’t get any polar data. Garmin accuracy not even in the same league as the v800 have had both to
Compare. Currently using Fenix 5 (other features outweigh the accuracy for me also use stryd).
Cool tool you have, I’ve seen it before. But that’s actually not covering the main functionality of the DCR Analyzer. The main focus of the DCR Analyzer was initially around sensor metrics (power, HR, cadence, etc…). These days it covers GPS, elevation, distance accrual, etc… However, a foundational aspect of the tool is to never ‘make-up’ data. So we don’t backport in incorrectly filled data, such as most of Suunto’s file exports.
RE: Fellrunner
I think Fellrunner’s stuff is interesting, but super misleading. As you noted, you can get different results on different days (and even different arms). Thus, it’s strange to use that as an example when he’s running a single segment potentially months apart and implying they were all gathered at once. Also, that loses the ultimate reference, because no person can run precisely the same plot hundreds of times and then try and compare that data.
I think the data he has may have a place, but it’s often presented well out of context with what it actually is. Subdividing a quarter mile segment into 50 chunks doesn’t make you have more data. It just gives you smaller pieces.
RE: GPS Accuracy Testing
Actually, it is. It’s merely a sliver of a single day, as I noted in the opener. Of course, then I also went back over a decade of data. Again, I’m curious where your data plots are?
RE: GPS Accuracy Testing Methods
Finally, and this is something that I often see folks get wrong. And I don’t think Jimmy you are directly, but I think Dan is: GPS Accuracy testing is the validation of where I actually ran versus where the device said I ran.
Period.
Seriously, double period.
No amount of scientific analysis can magically make-up and figure out where I ran. I have to tell you where I ran. I have to tell you which side of the road I ran on, and whether I crossed the street where I did, or whether I started on one side of Times Square or another. If a watch fails to show exactly where I ran, then it failed to record where I was. End of story.
RE: Ways to improve accuracy testing
Definitely, there are ways to improve everything in the world, and I’m always looking into those. The GPS satellite link you noted is a cool one that I hadn’t seen before. But I also think that’s ‘cheating’ in a sense. Regular consumers don’t go out and check that database (nor should they). They just got for a run. Just like I did. I got to the end of my day after a long travel day and decided to go for my run and document it for the fun of it here.
RE: One last point
Too many smart people try and ‘scientize’ GPS accuracy testing, to the point it because non-realistic and non-representative of what people are actually doing. People don’t run the same 1/4th of a mile section a thousand times over in real-life. Nor does every person wear their watch on the same wrist. That’s precisely why I test in all sorts of places and aim to match where I actually ran versus where the device says I ran (or rode or swam or whatever). It’s why I don’t use automation in testing (such as power meter tests), because they’re too easily tricked.
So, to Dan’s point – yes, I’m “proving” where the device went compared to where I went. Which frankly is the point.
Regarding the satellite tool I mentioned this as not to predict the best time but to get an insight post run of a potential factor.
Regarding fellnr methodology I think you are wrong there. Yes a device has to be consistent even if it is a small segment and that guys also has a trail performance table that unfortunately most devices suck at it.
Let me ask, as a runner is it the point to know which side of the road you run or having a good pace and distance ?
If its the first as you mentioned in your review its not justifying the claim that the fenix offseted but measured the distance correctly.
There are many factors to consider on GPS testing.
Anyways, perhaps we will never stop debating on this but please consider:
I would love to see a GPS accuracy roundup, where you run multiple paths , multiple times with different watches at the same time. Same arm etc.
Remember re arm position one arm is inferior and one superior for GPS testing.
Let me explain. If you wear a watch at the right arm and run at the right side of the road or trail then that watch is on inferior position since on your right your view is more blocked by buildings or trees. That is not the rule but a most probable scenario.
Anyways perhaps last night I got a bad sentiment because I happily saw an accuracy post by you, but was not so much close to what I expected.
PS. Re quantified self , I don’t stab any data and most fields are supported. I think you can also filter ibi data etc. Has some way cool stuff there and does not need a backbend since it only runs on the browser. If you take a look at the graphs you can easily overlay most of the data, compare etc. Anyways enough of the self promo.
Have a great day pal and for me I’ll leave this behind and expect your next review on perhaps the new polar.
Enjoy.
BTW I just noticed. Saying that fellnr methodology is wrong is just ridiculous. Did you really read that.
I think you are comparing potatoes to cucumbers here if your methodology , (not that you claimed that) is better. Its not and that is the whole point about it. You just run 1-2 times and compare tracks.
In my 5 years of GPS testing I had different results from different even seasons of the year for the same device etc.
You asked about my data and testing. I ll be pleased to give you as many files you want for watches such as polar m400 , v800 (not that many anymore), Suunto and Garmin. Feel free to contact me via email as I don’t share any private data.
I think by showing you my online tool of anaylisis you ll get what kind of data geek I am , please do not try to turn this around.
You can find me in several online services as well, such as Garmin connect, flow and Strava and Movescount but I would prefer to keep this private. An email is fine.
Overall Ray, we are not in 2008 that we compare how pretty tracks from one runner to another.
Listen to this very very carefully and you might understand why some people can become haters or lovers:
You hold a blog with great power that can damage or praise sales of GPS device companies. A wrong test, a bug or even luck can damage the prestige of a product , and if you don’t try to follow a methodology or give a methodical testing people will be mislead. I don’t think you would like that.
I don’t think you would like people to buy an x device because it had 3 days in a row of good data while in most of the cases it sucks.
But it feels like that’s what you are doing in the end.
If you don’t want to spend time in GPS testing maybe an idea is to leverage community power for that, other reviewers or link to fellnr tables.
PS. Fellnr has nothing wrong on those tables. They align with reality and its great resource for you as well.
You know what to do pal. You know better. And to be honest you can just ditch my words and go on the same way you are used to. Its fine.
Have (again) a great day and feel free not to reply.
I would appriciate if we don’t get thus debate far away my rant is over.
Fellrnr makes (and continues to make) a massive mistake that’s been pointed out numerous times to him: He uses smart recording on Garmin devices.
As such, he only records Garmin data points every 4-7 seconds on average (but can easily range up to 20 seconds between points). You can’t do statistical analysis on that kind of data, especially while trail running. He says it doesn’t matter. Obviously, it does. Anyone who analyzes data knows it does. And one wonders why all his Garmin tracks look less ideal when he uses 1-second recording on Suunto and Polar devices.
RE: “I don’t think you would like people to buy an x device because it had 3 days in a row of good data while in most of the cases it sucks.”
I think you’re missing the point of this post. This wasn’t a product review. There’s no word ‘review’ at the top. The point of this post was looking into a single run, and then a pile of historical runs over the course of nearly a decade. This was never intended to be a crazy review. In fact, I say that in the intro. It started off as a simple fun YouTube video that I decided to expand.
In fact, all of my review posts include data over usually a month or longer (many cases, many months). I’m not sure why you’re saying I only run 1-2 times and compare tracks. In the reviews I usually have half a dozen or more tracks I’ve analyzed, but often links to piles of extra tracks for folks to dive into if they want to. I try and pick ones to dive more deeply to in the post that are most representative of what I saw at large.
RE the smart recording. Fellrnr might have a point here, if he presents that as the out-of-the-box experience because it’s the default option.
It took me few months of owning my first Fenix to find out such an option even exists.
It’s also nothing rare to see a question about cutting corners on garmin forums.
I’ve long argued to Garmin it’s stupid to leave that as the default setting, nobody else does. As you noted, the forums are full of people cutting corners only to find out smart recording was on. The storage cost these days is laughably negligible (it doesn’t impact battery any, as Garmin themselves have noted).
Fellrnr doesn’t present that as the reasoning though, and certainly he’s more than technical enough to change it.
I know its not a review.
As I said I’ve read literally all your review.
I do know you do a lot of tests, you are higher than the experienced consumer and you do look at data. I don’t doubt your data testing and its quite OK.
I also noticed , mentioned here that you had them on the same arm which is how it should be. Mentioning this because I do understand your methodology and its evolving which is great man.
Re market. You have a point here. Perhaps it can’t affect market. Perhaps this is were I stuck and got a bad sentiment about this post. My mistake.
Regarding fellnr I did not know that. Does he do that also with the f5? I’ll have to ask him perhaps.
That is totally unfair!!!
Thanks for the add at the list, much appreciated.
I wish you a great day further and perhaps talk again at another post.
Regarding the GPS accuracy: While you can pre-calculate similar satellite constellations to ensure the same or similar GDOP, you cannot do this with the atmospheric effects. The ionosphere and especially the troposphere can greatly disturb the signal from the satellites and thus create worse positional accuracy (you can experience this a bit, when you compare the amount of time it takes for your watch to obtain a position fix in nice weather and in cloudy weather (preferably the next day)).
As of such, the only ‘scientific’ way to compare the accuracy of different watches would be to take them all with you on the same day and mount them on some form of trolley.
At the end of the day, and I’m saying this as a surveying engineer, I don’t really see much need for extended accuracy tests. Unless the watch has a serious flaw in the antenna design or the chipset is very weak in poor signal-to-noise ratio scenarios, the accuracies should be pretty similar. And no matter how good the chipset/antenna – you will still experience poor distances i.e. on a track, where your body hides the watch from the satellites at some point and thus usually returns more than 400m for a lap.
“Unless the watch has a serious flaw in the antenna design or the chipset is very weak in poor signal-to-noise ratio scenarios”
And you see no point in testing this? Also you seem to assume the GPS data is shown raw. There’s a lot of software involved to compensate for bad resdings and the quality of this software varies of course.
I would never buy a watch without reading some tests first.
He said he tried a few tests on one watch, in which case he said it didn’t matter. He didn’t show any data from that.
There’s a decade worth of overwhelming evidence that Smart Recording cuts corners and of course misses track points – that’s fundamentally what it’s designed to do (in the trade off of saving space for devices a decade ago when it mattered). He’s the only person that believes it doesn’t.
Others have tried to reason with him on this point, to no avail.
On what route? A straight road? The usual test? For how long? And where’s the actual tracks?
Again, he even admits in the reviews the impact of this:
“I also use smart recording rather than 1-second as this only changes recording frequency, not the frequency of GPS polling.”
And then goes onto use said plotted data that’s missing 75-80% of what other devices use. For someone so smart, I really don’t understand why he’s so set on Smart Recording when it carries zero benefits in 2018 (or anytime this decade).
The numbers match exactly the usual test he has covered extensively. So obviously that is where the test took place.
Anyway, I trust his tests by far the most until I come across something better.
The smart recording won’t matter because the DATA POINTS fellrnr uses are the LAP DISTANCE recorded by the watch, which means they ARE 1S DATA calculated using the full 1-second sampling — regardless of the “smart recording” setting, which only determines how frequently x,y coordinates get written to the data file.
The recorded lap distance is compared to the known, actual lap distance and the difference between the two is the measure of the watch’s GPS accuracy. (It’s a fixed route that he has carefully measured and that he can re-run very closely, and he manually triggers a lap every time he reaches this calibrated lap point — like you might lap when you see a mile sign in a marathon race, except here all the markers are much more accurate, and they aren’t always in a race :). Ray did something very similar ~5 years ago when he tested some devices in a parking lot and in the park.) This addresses what Ray says above, something like “to make the comparison, I have to tell you exactly where I know I was, you can’t pull that information out of the fit file” (well said).
When he repeats this test by re-running the route many times, he finds that the measurements behave statistically, so you have to test the watch statistically, too: i.e. there’s a spread around a mean, and for each watch you will find, after many runs, how close the mean is to the true distance, and how wide the spread around the mean is. That’s how you arrive at saying one watch is good and another is bad.
Smart recording only matters a/ for plotting your route on a map — a “smart” route will look “stupid” ;) and cut through buildings going around a corner even though the distance was calculated using data points properly going around the corner ; b/ if you want to recalculate the distance or other quantities yourself from the x,y coordinates in the data file. But fellrnr doesn’t do that, and almost nobody would do that manually, but 3rd party analysis software that you might want to use may do it. So, I personally turn on smart recording, and I don’t see a good reason why fellrnr doesn’t, but it has no impact on the validity of his work.
The second criticism I read here sometimes “fellrnr always runs the same route! Who does that?”. Well, most of us do have a handful of routes that we run over and over :). But, sure, maybe the fellrnr route is particularly difficult somehow (like Ray’s route between tall NYC buildings above), and maybe watches would rank differently for someone running in the city or in the desert or in a place with better or worse satellite coverage? We would get the most comprehensive study of a watch if we could get statistical sampling (lots of repeated runs) for a representative ensemble of routes. In the ideal case we would recruit, say, 20 fellrnr’s across different routes all over the world (different in terrain or location relative to GPS systems). Maybe it can be crowdsourced one day. In our dreams watch manufacturers are already doing it. ;)
Finally, what does this say about Ray’s work? Ray slices through the ensemble in a different way: use a watch once across various routes; as opposed to using a watch many times on one route. He uses the GPS track, plotted on a map, as metric, as opposed to lap distance. Strictly for GPS accuracy, I expect Ray’s approach to capture some of the statistical variation and ‘the truth’, but somewhat less rigorously (I have a feeling it introduces more variables, has fewer data points, and is less quantitative). BUT imo THE REAL DIFFERENCE is that the two authors ultimately don’t care about the same things as much. Ray thrives on, and excels at, looking at the total usefulness of a watch, and clearly he relishes providing watch makers with user perspective, being a user advocate, and thinking about all the things that will ultimately make people want to pick up a watch and go run. When he labels a watch “good” or “bad”, it reflects a broader judgment of which accuracy is only a component. A valid approach. (I like my Stryd but it hasn’t changed the way I run ;).) Obviously this paragraph is total personal opinion, and I’ll fully defer to Ray on how he defines the great volume of work he’s done in this space.
In the end, I see that watch manufacturers have worked with both authors in the past to troubleshoot and improve their devices. A cue that both have merit. I wish that the long-going, and in my view incorrect, fellrnr bashing on this website would be laid to rest (and find it unbecoming a person with a technical background).
— Disclaimer: I’ve read DCR and fellrnr and 5krunner for many years. I like fellrnr for the detail on trail running, DCR for the wider view across multiple sports and the detail of the reviews, and 5krunner for the opinionated and spirited pieces.
My takeaway from this exchange is it’s a waste of time to engage trolls who call you pal. And Jimmy, for someone who’s read all the reviews on this site for the past 5 years, you sure are a dick.
However, I think you’re misunderstanding why smart recording is so problematic for Fellrnr. Specifically, this part:
“The recorded lap distance is compared to the known, actual lap distance and the difference between the two is the measure of the watch’s GPS accuracy.”
He only uses lap to establish a starting gate, after which the core graphed metrics shown on every one of his pages are using the GPS coordinates. This is explained on his explainer page, and in particular he notes that the lap marker GPS positions themselves are only used as approximates to help match chunks (“These estimations are only used for determining the nearest lap marker, so accuracy is not critical.”)
After which point, everything is based on plotted GPS points. He goes through multiple passes for different graphs, charts, and trueness values based on these GPS points.
I actually don’t have a problem with any of this bits. In fact I think it’d kinda interesting.
But, all of this gets shattered when you use smart recording. It makes everything except that entry gate lap marker totally useless, especially so on trail running (which is the vast majority of his predefined route). You could almost get away with smart recording on straight farm roads (though, it could inflate accuracy). But not when he’s including charts like the attached.
As you allude to midway through, while I can appreciate Fellrnr’s approach for trail runners, my concern is that most people read without the understanding that virtually all of that is on trainers. It’s also all skewed by usage of footpods in the ranking tables. Given that 99.x% of people don’t trail run (nor use a footpod), I think a much more valuable metric to folks is on a variety of circumstances and labeled as such. Here’s how it does on road, here’s how it does on trails, here’s how it does in cities, etc…
That’s what I try and do (as you noted) in each review as best as my travel/geography permits. That let’s people figure out based on their own requirements whether a watch is applicable to them.
If anyone is interested, I have painstakingly (using USATF guidelines) measured a forest trail course, a wide open bike path course, and (publishing next week) a 10K course with a wide variety of GPS challenges. On each course I’ve run a bunch of laps over several days measuring the mean error and standard deviation for each watch footpod. I only have a few devices so far, but will add more over time. Of course, it’s impossible to do tests holding everything constant, but I feel like this is an interesting approach for those who are curious about performance in specific environments.
what does your own experience tell you?
i have just checked my runs on my fenix 3 – I run af few different courses again and again (5-20k) – they are all within 10-30 meters the same – (about 200 runs the last year).. I call that accurate – or rather who cares if one ran 10k or 10.02k…
I generally don’t use any desktop software anymore for analysis. I just don’t see the reason to. Web platforms work well enough.
I use Training Peaks for core data analysis, and Strava for more social stuff. And I use the DCR Analyzer to compare tracks and metrics related to power/cadence/developer fields.
The one exception to this is somewhat new, which is I’m using Golden Cheetah to do some Aero analysis bits due to some sensors I’m testing.
“I generally don’t use any desktop software anymore for analysis. I just don’t see the reason to”
Simple to me… I don’t want to give anybody else my data and typically I find desktop app to give more detailed analysis that online apps
“Ignore the zero-value elevation gain on the Suunto. Unfortunately Suunto’s Movescount doesn’t properly write the .FIT file header data, so in the DCR Analyzer we don’t have the ‘known’ data to pull from (we don’t want to recalculate it, as that’s not what appeared on your watch and is making assumptions).”
And yet Strava feels the need to do so all the time even though they’ve received endless feedback that users want them to match.
LOL, You asked why so as I am obviously the “outlier” in the world I’ll say why I didn’t watch the video. They are too slow! I just do not get it, I can read much faster than any video can give me information and yet EVERYWHERE that is all people want to do anymore is make videos. I just want to read. I can read, re-read if i need to and digest at a pace that preserves the amount of time I have. I can also easily search right to a specific world or phrase if I want to come back to reference something. Maybe its just that I grew up with books I’m not sure but I really do not see the need for so much video content in the world.
Although on a plus note youtube did lead me down a rabbit hole the other day to some Tori who is some sort of Vegan that really like her bicycling kit and posts videos about it ….. so there is that…..
You can read text without disturbing anyone else. My wife would go nuts if I sat next to her on the couch watching Rays videos. Long live text and the long in depth reviews!
Yeah, I don’t understand the global fascination with videos. But we are indeed outliers — just look at how many views Youtube videos get.
I wonder if (good) video production makes a better return on time invested than (good) writing and photography. Given the monetisation of videos v text, I’m thinking it probably does. Plus why not make two revenue streams from each piece of content inspiration? New watch? Trip overseas? A video *and* a written post gets you twice as much income from each.
YouTube monitization is horrible. Really, it’s horrible.
On this video, with say 10,000-20,000 views (if it gets that much, it likely won’t, but yeah never know), I’ll probably make between $10 and $20. Obviously, other content sometimes works out for me – but rarely is there a meaningful ROI on YouTube for me.
But, it’s ultimately a different audience. There’s plenty of people that watch me on YouTube that have never been to the blog (seriously, I run into them all the time, go figure).
Same for me. I read all articles, am a supporter and don’t block ads on this site (its whitelisted since the beginning), but I barely watch any videos.
I disagree with that statement. The new capabilities in Galileo and next-gen GPS include some specifically to improve accuracy in urban environments.
Phones however are likely to improve much faster than wearables — there are now standards for time-of-flight ranging and thus triangulation of wireless access points, which will enable metre-precise navigation inside large buildings and in dense urban areas with lots of wifi.
Part of the problem is multi path interference. Broadcom is apparently releasing a chip soon that will also utilize another code on the GPS signal to do better in urban canyons.
Maybe in urban environments with tall building could help the use of multiple constellation simultaneously.
We saw in the test of DCRainmaker that the Fenix5plus, that was the only watch that used two constellations of satellites (GPS + Galileo), was the best of the three in the section of Manhattan with tall building, even if the quality overall was bad and if the difference in quality between the three watches was not so the big, in fact Fenix5plus got a D and the other two a F (Ray writes: “I’d argue for this specific section the Fenix 5 Plus was closest to the road, but that’s like giving an award to someone who gets a ‘D’ in class, merely because the only two other people in class got an ‘F’.”).
Maybe a further improvement in accuracy could be obtained using simultaneously the 4 constellation available today (GPS, Galileo, Glonass, Beidou.) We could imagine that in the section of Manhattan with tall building, the satellites in view could increase from a number of, let’s try to guess, maybe 5 in the configuration of two constellation (GPS+Galileo) to 9 in the configuration of 4 constellation (GPS+Galileo+Glonass+Beidou). Maybe with this increase of the number of the satellites in view the accuracy of the position would be better, even if the position of the satellites would be not ideal because they would be high in the sky without some satellites in a low position above the horizon (this means high DOP -dilution of precision – and so less accuracy).
It would be nice if the watches could report second by second the number of satellites in view and the DOP value, and if these values could be plotted after the run in a graph, for example in Garmin Connect. I think that a watch knows exactly second by second the number of satellites in view and when a watch finds a position fix, it knows exactly the DOP value for the satellites that it uses. So I think that the watches could write in their memory second by second number of satellites in view and value of DOP. It would be interesting to ask the brands (Garmin, Suunto, Polar, Coros) if this is possible and why they don’t give this information.
The problem is that the use of 4 constellation simultaneously has a hit on the battery, so I don’t know if they will ever produce this type of chip for wearables and sport watches. This type of chip that can use simultaneously 4 constellations already exists for cars, but I don’t think for watches and wearables that till now can have maximum just two constellation simultaneously.
By the way I found the test nice and well down. I liked in particular that it analyzed in the same test and under the same condition a section in urban environment with tall building and a section in open sky (central park).
For example the new GNSS chipset of Broadcom for smartphones BCM47755, installed in the Xiaomi MI8, can receive simultaneously all the 4 constellations (GPS, Galileo, Glonass, Beidou) and also SBAS satellites (WAAS in America, Egnos in Europe and QZSS near Japan). The chipset of the same family for wearables, the BCM47758, is capable to receive maximum two constellations simultaneously, GPS+Glonass or GPS+Galileo (I don’t think that it can receive SBAS satellite, but it’s not clear from the technical data that Broadcom has released in the pdf product brief that can be found here link to broadcom.com ).
Ray, what firmware were you using on the Fenix 5 Plus? The recent betas are giving me massive improvements in terms of gps tracking compared to the latest official firmware..
2010 isn’t quite far enough back for when Garmin had to switch from SiRF, though, is it? Most of the GPS complaints I see about Garmin say that the Mediatek chips they use today give worse tracks than the SiRF’s they used to use. I own a couple of Garmin’s with SiRF (60CSx, and I think my 705) but I’ve never gone into the weeds comparing tracks (in any case, there are other reasons a 60CSx, like the HONKING HUGE ANTENNA, might give better results than an Edge 1000).
I have; my FR310XT was consistently better than the tracks my friends got from the likes of 920s and 235s. But my FR645 (which replaced the 310) is just as good. Manufacturers switched to smaller designs, sacrificing antenna size for style, a little before tech could mitigate the change.
I think the loss of accuracy in current GPS-watches is due to the fact that engineers want to save energy for longer batterie life. So they have to choose low-power chipsets which, as a negative effect, are less accurate than older chips.
I’ve had a FR935 — really nice watch with a stunning long batterie life. But GPS was very poor compared to my old V800. So I dropped the FR935 (an other reason was the horrible Garmin Connect site and the feeling that my watch collected a bunch of data, but nothing really important came out of it. With Polar Flow I can always read a lot more out of the data. But that’s another topic …)
Generally speaking it’s actually more the antenna.
Most older watches have huge antenna bumps on them. For example, the Suunto Trainer (and Ambit before it) has that big bump at the bottom, that’s where the antenna goes. The old FR305? Yup, same place.
Whereas these days companies try and integrate the antenna into the bezel. Which isn’t to say companies didn’t do that before. Many other Garmin Forerunner watches do that just fine. But, there’s plenty of examples where Garmin and others have hosed that up too.
That you are wearing is a Perth based Western Force rugby union jersey, formerly from Super Rugby now in some kind of Twiggy named billionaire supported limbo!
Sorry but can’t find the V800 is akin to dog ate my homework excuse?
I’d like to see Polar V800 vs Suunto Spartan Ultra vs Garmin Fenix 5 Plus, or the Polar Vantage once announced?
Seriously, I wish I knew where it was. I’m going to be in a pickle soon if I can’t find it. I’m probably going to spend 2-3 hours today looking for it actually. Boxes upon boxes. Sigh.
I had actually set it aside back in December prior to the move, because I wanted to take it to Australia for the three months I was there. I did that because I suspected Polar would be launching their new watch during that timeframe and wanted to compare it side by side. This meant I took it out of the device bin and put it somewhere, and then it got put somewhere else in the shuffle. Likely an errant box.
Very interesting. As you may have surmised, GPS tracks do not necessarily give much insight into a watch’s accuracy. I’ve created the definitive test loop for watches and will be publishing the results soon.
Yeah, I do think that’s an area where Suunto is doing some really cool stuff – and I’m excited to see what that looks like in say 9-12 months from now once they’ve had more time to finesse it.
1. Use Chrome, I used it on macOS (both current versions)
2. Right click map, choose Inspect Element
3. Edit the highlighted ‘height’ parameter shown in the screenshot.
4. Move the map around a bit to trigger it loading the missing tiles.
5. Voila: link to images.paulwakeford.info
You can make it more user-friendly with a ‘bookmarklet’. Create a new bookmark in the bookmark bar (a favorites, whatever your browser calls it), make a nice label like ‘Big map’, and add the following as URL: javascript:(function(){document.getElementById(‘activity-map-canvas’).style.height=’800px’})();
If this shows up as a clickable link here, you can also drag it to the bookmarks bar.
Usually GPS tracks are examined and judged by using streets and buildings on Google maps as a reference.
I wonder how precise these maps are? If Google used GPS measurements to align images on its model of the world the map could be shifted as well. Lenses and even fluctuations in the atmosphere will introduce distortion when the images are taken. The vector maps are almost certainly based on GPS measurements.
Do you know how precise these maps actually are? Does it vary globally? By how much?
I generally try and use satellite imagery in GPS track comparisons when it gets to the detailed points, because the non-satellite maps (vector maps) can often be inaccurate at that level of detail (especially near water).
The challenge with satellite maps though is that when you zoom in on Google map-view it’ll go into the angled view (usually taken from an airplane) when you get too close to the ground, which then distorts where the lines are. I suspect there’s a Google Maps API option to tweak that I could look into.
Wasn’t there some noise several years back of the Chinese deliberately offsetting maps of China (as if reconnaissance satellites didn’t exist) and causing some trouble for tourists using GPS?
“I suspect there’s a Google Maps API option to tweak that I could look into.” It is the ’tilt’ option, on initialisation of the map use `tilt: 0` to avoid the angled view.
Great writeup Ray. These are the kinds of posts that got me checking DCR the time – reading about you travels and generally getting lost in the geekery of the data. Let’s have more of them!
Hope you’re feeling better….
Test wasn’t fair for the Suunto. You should know the watch is meant to be on your wrist and the gps module directed up to sky! That is the whole idea to have separate module in the wrist band.
And with a Mediatek 3339 (GPS only). Antenna is very important. All standard GPS chipsets have now high sensitivities. Is not for that. What is the common domineering between Ambit … and Suunto Spartan Trainer? The design at bottom. Generally i see points of you, Jimmy, and Jonathan (Fellnr). And you all have valid points. We have to read carefull. And i do understand that your post have other intention. But i have to agree with Jimmy. DC, is important to test watches, in same conditions (GNSS system equal, recording frequency, in some wrist, etc) watches with same range, and several runs in same route path. It will not 100% equal, but it will show tendencies of accuracy. I have one suggestion, if you would have patience. Pick a route, not many miles/km, half city, half open. Put watches in same conditions and repeat more than 5 times in this route. Use what software you like most. Plot for we see as you do in your reviews. Then, we can see tendencies in different days (with different weather, with different available satellites). Should we take any conclusion or not? I think it will bring many people with attention. Can i suggest first one? S9, Polar Vantage V and Fenix 5 Plus. Good work and thanks for reading!
Generally speaking I only wear a single watch per wrist, because that impacts optical HR sensors (hugely if more than one device per wrist).
Hence why i’ll usually hand-carry a secondary watch (3rd watch I suppose) in an orientation that doesn’t block any GPS aspects. That hand-carried unit is typically connected to a chest HR strap (to use as reference against the optical HR sensors). If a fourth unit is carried in the other hand, then I usually connect it to a TICKR-FIT or Scosche sensor.
Just in case you are interested, I’m conducting a series of tests for GPS watches and foot pods, reporting two numbers: average error from the true distance and the standard deviation of the sample.
A couple of question, hoping you can find a minute to answer.
1) Not too long ago I read about Broadcom developing (or having developed) a new GPS chip that uses a new/better signal. If I’m not mistaken, accuracy would be in the <0.5m area. Did you hear about that and if so do you know if there is any R&D with that chip in wearables?
2) I'm currently using Movescount and TrainingPeaks. While they are both ok, they lack a ton of analyzing capabilities. For example – I have found no way to compare three bike rides. I'd just like to see where I was faster/slower compared to other rides.
Oh, and my favorite – there is no way to find out what distance you did on your bike in e.g. 2017 in Movescount. Just simply not possible.
Unfortunately you don't talk/write much about the software side of things. So my question is, if it wouldn't make sense to write a review or comment about software every now and then?
Thanks up front for answering my questions and keep up the good work!
That‘s why I love Strava. You can create your segments (maybe make them private if you‘re only interested in your own progress) and compare your rides. There is even a nice diagram where you can choose which rides to compare and it shows you exactly when you were ahead or slower (note: the diagram is not included in the free version of Strava). Also you get a perfect summary of miles for every bike. If you want more, you can connect to VeloViewer, which is awesome and gives you dozents of further statistics.
Hi Dirk,
thanks for the tipps. I’ll look into VeloViewer this weekend.
I did try Strava with their segments, but couldn’t get it to work properly. Only slight deviations from the route and it won’t compare the two rides. Maybe I should try it again. Who knows, maybe it was just my incompetence …. :-)
If you create segments by yourself you should have no problems then riding them again with the same device. (Check GPS accurancy on DCR site ;) )
What I also like is that you can also log parts of your bike, eg you can easyly see how many kilometers your chain is already on your bike, or your brake pads and so on …. (especially useful if you‘ve a lot of bikes to think of).
Yes it it. Click the year in calendar.
Now you see totals of that year. If you want to see all the statistics only from cycling, you van additionally filter the data to show only cycling.
“Oh, and my favorite – there is no way to find out what distance you did on your bike in e.g. 2017 in Movescount. Just simply not possible.”
This is actually not true, Flo. You can do this rather easil by opening “me => yearly summary”. This will give you the totals for the current year (i.e., now, 2018), From there, you just need to switch from the calendare view to the table view (second little circle on the top left of the calendar) and click on the “2017” year on top. This will automatically select all mvoes from that calendar year, with totals below. This workd for every calendar year, or calendar month (if you stick to the calendar view instead of using the table view).
Anyway, the Fenix 5 and 5 plus both have new beta builds available with the purpose of improving GPS. My initial testing is promising. I’m not sure if it’s actual improvements to accuracy, or just smoothing.
“GPS FW (2.24) should improve pace accuracy for running/walking activities and track performance for open water swimming activities.
Our public beta users are encouraged to load the new GPS firmware and use it for their next activities. If there are any activities where the GPS performance did not improve, we would like to see them. This also includes if comparing to previous activities….”
Good old FR310XT… I started reading DCRainmaker, because I found the manual and the product review more detailed and useful on your page then on a hard copy attached…
Bought in 2009 for 310EUR and sold in 2012 for 210EUR when swapped for FR910XT…
Anyone can say whatever he wants, but when I used 310XT and 910XT at the same time, the former seemed to be more accurate in terms of GPS accuracy, although the difference was not so big.
Due the barometer inside I kept only 910XT.
Now when I use 910XT and Fenix 3 HR together, 910XT is a clear winner. Almost always, I mean at least for 90% of the activities. Full stop.
Fenix family may never be as accurate as the rectangle Garmin watches.
So you mentioned a couple of times, frustratingly, that there is no good way to look at a map to find your old routes/tracks. A personal heat map, if you will. WHY OH WHY NOT?? This frustrates me to no end! I’ve been logging rides all over the place for years, and I would LOVE to look at a map to pull out some old rides, to either analyze progress, or just to see if that familiar looking place is somewhere I have actually been before. Why is this not a thing? I would use it all the time. Just like doing a photo search using a map. I may not be able to remember when I took it, or what I may have called it, but I can often find it by location.
“and then simply hand-held the Suunto, keeping the satellite bump pointed at the sky at all times”
in the table the Suunto device shows an avg. HR –> did you wear an HR-strap? But if so, why is there a 3bpm difference between the Suunto Trainer and the Fenix?
Yes, it was connected to a TICKR. Though, it seemed to have dropped the connection at one point. Not sure if that was the result of the TICKR or something else. I suspect that battery was dying, as I changed it the following weekend.
Read the comments section after the reference to it in the newsletter. Wow. Intense.
Ray – you’re the man. Huge fan of your blog and appreciate your expertise and neutrality. Thanks for all you do.
Having almost the same distance the Suunto (11.32 Km) and the Garmin (11.28 Km),
how is it possible that the avg_speed is more than .5kmh difference?? (12.86 kmh vs 12.14kmh )
This is why, living in London, I have a Stryd footpod, and as long as I calibrate it periodically, it works a treat everywhere. It’s especially great in races where there are tunnel sections like London Marathon or Big Half.
I’ve read that the Fenix 5 watches with maps (5X, and all 5 plus) have an option “keep on track”.
This is supposed to keep your location within the streets. It would be very interesting to see if that would make a difference in your test. Also it would be interesting how the modes “GPS only”, “GPS+Galileo” and “GPS+Glonass” would give different results.
It is not surprising GPS+Galileo does not give a significant improvement. Until october 2018, US laws prohibited use of Galileo and the sattelites were not yet launched. As of the last months of 2018, 2 sattelites are added making the total 22. But only when the last 2 sattelites are launched in 2020, US users will have the full benefits. Galileo supports precision up to 50cm for consumers, compared to the 3 to 5 meters for GPS and 4 to 10 meters for Glonass.
I live in Europe and noticed a significant improvement.
I’m pretty sure it was enabled for Garmin devices prior to that.
Still, of course, the constellation continues to grow and will only get better. That said, I haven’t noticed a major difference for my devices living in Amsterdam in terms of accuracy.
Maybe your watch was receiving Galileo satellites because you had a european version. It’s possible that american version of the Fenix had Galileo satellites filtered (like american version of phones seemed to have Galileo satellite filtered in firmware).
Well, Garmin doesn’t let us know any more with wich satellites the watch is connected, so we will never know.
I know that the article is outdated, however, the Garmin 45 or Coros Pace better? (Because the price are same.) Could you advise which of them would be the better option for running? Thanks!
I swim, bike and run. Then, I come here and write about my adventures. It’s as simple as that. Most of the time. If you’re new around these parts, here’s the long version of my story.
You'll support the site, and get ad-free DCR! Plus, you'll be more awesome. Click above for all the details. Oh, and you can sign-up for the newsletter here!
Here’s how to save!
Wanna save some cash and support the site? These companies help support the site! With Backcountry.com or Competitive Cyclist with either the coupon code DCRAINMAKER for first time users saving 15% on applicable products.
You can also pick-up tons of gear at REI via these links, which is a long-time supporter as well:
Alternatively, for everything else on the planet, simply buy your goods from Amazon via the link below and I get a tiny bit back as an Amazon Associate. No cost to you, easy as pie!
You can use the above link for any Amazon country and it (should) automatically redirect to your local Amazon site.
Want to compare the features of each product, down to the nitty-gritty? No problem, the product comparison data is constantly updated with new products and new features added to old products!
Wanna create comparison chart graphs just like I do for GPS, heart rate, power meters and more? No problem, here's the platform I use - you can too!
Think my written reviews are deep? You should check out my videos. I take things to a whole new level of interactive depth!
Smart Trainers Buyers Guide: Looking at a smart trainer this winter? I cover all the units to buy (and avoid) for indoor training. The good, the bad, and the ugly.
Check out
my weekly podcast - with DesFit, which is packed with both gadget and non-gadget goodness!
Get all your awesome DC Rainmaker gear here!
FAQ’s
I have built an extensive list of my most frequently asked questions. Below are the most popular.
You probably stumbled upon here looking for a review of a sports gadget. If you’re trying to decide which unit to buy – check out my in-depth reviews section. Some reviews are over 60 pages long when printed out, with hundreds of photos! I aim to leave no stone unturned.
I travel a fair bit, both for work and for fun. Here’s a bunch of random trip reports and daily trip-logs that I’ve put together and posted. I’ve sorted it all by world geography, in an attempt to make it easy to figure out where I’ve been.
The most common question I receive outside of the “what’s the best GPS watch for me” variant, are photography-esq based. So in efforts to combat the amount of emails I need to sort through on a daily basis, I’ve complied this “My Photography Gear” post for your curious minds (including drones & action cams!)! It’s a nice break from the day-to-day sports-tech talk, and I hope you get something out of it!
Many readers stumble into my website in search of information on the latest and greatest sports tech products. But at the end of the day, you might just be wondering “What does Ray use when not testing new products?”. So here is the most up to date list of products I like and fit the bill for me and my training needs best! DC Rainmaker 2023 swim, bike, run, and general gear list. But wait, are you a female and feel like these things might not apply to you? If that’s the case (but certainly not saying my choices aren’t good for women), and you just want to see a different gear junkies “picks”, check out The Girl’s Gear Guide too.
Interesting would be to do a run with only a GPS watch and a GPS + Glonass enabled watch and see if the GPS + Glonass is getting beter position than the GPS only watch…
There’s a lot of other factors involved to eg position in sky of satellites giving a dilution of precision. you’d have to factor that in for each constellation combo used.
I borrowed a forerunner 305 last yeat (IIRC). I always remembered it was ‘great’ at some point ‘back in the day’. unfortunately when i got rid of my 305 I bought some rose tinted spectacles – aka, as dcr says, it really wasn’t any better back then despite what we all remember
Thanks a lot for digging! This was insightful and even more so with all majors releasing new flagships this year. We already heard from Garmin and Suunto, and now Polar is about to follow up on veritable V800.
On a separate note, next time you in NYC, you should probably do DC Rainmaker Club meetup. Central Park or not. Not everyone will be running, but everyone can be drinking ?
I saw the selfie picture and my first reaction was “What’s the antenna sticking out of Ray’s head? An unannounced product?”
Which firmware of the Suunto 9 did you use ?
Sorry, I meant the Spartan. The latest I assume ?
Correct. Latest as of about 15 minutes before the run. I had synced with desktop and charged directly before.
I am sorry to say so, but GPS accuracy testing is not called accuracy testing with one run only.
You lost me pal. There are more accurate and methodical ways to do so.
Cool, pal, do you have a link to your GPS accuracy testing?
Also, have you read my reviews?
Of course but let me raise the competition
link to fellrnr.com
I even made a tool that is free and does what the analyser does. Compares tracks and shows differences between stats even (declination)
https://quantified-self.io
Just a tip: Even if you run with 3 of the same units you would have different results most of the times.
For that you need to repeat the tests and prove that a devices accuracy is consistent and not just a days celebration.
I am sorry if I offended you by my comment but I first expected something more concrete from you. Been a fan since 2015.
BTW the tool is free and open source
I have read all your reviews. But this one is not GPS accuracy testing. The reviews are more close to what you call GPS accuracy testing. Not this post man. Not this post.
Ray’s ‘accuracy’ testing has always really been just eye-balling it. No binning, no RMSE, no bias. It’s… interesting, but doesn’t really show or prove anything.
But that’s not why people come here. Ray has a very readable and entertaining writing style, and lots of good photography.
BTW here is a nice tool to show how many SATs were or will be available in your future tests.
link to satpredictor2.deere.com
Depending on this you can at least predict or “know” at what conditions the watches performed.
What I mean:
On a good day and time that a lot of satellites are available it must be easy for most, like perhaps your historical runs and comparisons that you mention.
On a bad day it shows which device is superior when less satellites are visible.
Check it out.
I am here to help perhaps “upgrade” your testing procedures.
Polar v800 still the most accurate. Great reply Jimmy BTW. Funny how Ray couldn’t get any polar data. Garmin accuracy not even in the same league as the v800 have had both to
Compare. Currently using Fenix 5 (other features outweigh the accuracy for me also use stryd).
Good job grinding that axe. Should be nice and sharp now.
Hi Jimmy-
To cover a few points you raised.
RE: Analyzer tool vs Quantified Self Tool
Cool tool you have, I’ve seen it before. But that’s actually not covering the main functionality of the DCR Analyzer. The main focus of the DCR Analyzer was initially around sensor metrics (power, HR, cadence, etc…). These days it covers GPS, elevation, distance accrual, etc… However, a foundational aspect of the tool is to never ‘make-up’ data. So we don’t backport in incorrectly filled data, such as most of Suunto’s file exports.
RE: Fellrunner
I think Fellrunner’s stuff is interesting, but super misleading. As you noted, you can get different results on different days (and even different arms). Thus, it’s strange to use that as an example when he’s running a single segment potentially months apart and implying they were all gathered at once. Also, that loses the ultimate reference, because no person can run precisely the same plot hundreds of times and then try and compare that data.
I think the data he has may have a place, but it’s often presented well out of context with what it actually is. Subdividing a quarter mile segment into 50 chunks doesn’t make you have more data. It just gives you smaller pieces.
RE: GPS Accuracy Testing
Actually, it is. It’s merely a sliver of a single day, as I noted in the opener. Of course, then I also went back over a decade of data. Again, I’m curious where your data plots are?
RE: GPS Accuracy Testing Methods
Finally, and this is something that I often see folks get wrong. And I don’t think Jimmy you are directly, but I think Dan is: GPS Accuracy testing is the validation of where I actually ran versus where the device said I ran.
Period.
Seriously, double period.
No amount of scientific analysis can magically make-up and figure out where I ran. I have to tell you where I ran. I have to tell you which side of the road I ran on, and whether I crossed the street where I did, or whether I started on one side of Times Square or another. If a watch fails to show exactly where I ran, then it failed to record where I was. End of story.
RE: Ways to improve accuracy testing
Definitely, there are ways to improve everything in the world, and I’m always looking into those. The GPS satellite link you noted is a cool one that I hadn’t seen before. But I also think that’s ‘cheating’ in a sense. Regular consumers don’t go out and check that database (nor should they). They just got for a run. Just like I did. I got to the end of my day after a long travel day and decided to go for my run and document it for the fun of it here.
RE: One last point
Too many smart people try and ‘scientize’ GPS accuracy testing, to the point it because non-realistic and non-representative of what people are actually doing. People don’t run the same 1/4th of a mile section a thousand times over in real-life. Nor does every person wear their watch on the same wrist. That’s precisely why I test in all sorts of places and aim to match where I actually ran versus where the device says I ran (or rode or swam or whatever). It’s why I don’t use automation in testing (such as power meter tests), because they’re too easily tricked.
So, to Dan’s point – yes, I’m “proving” where the device went compared to where I went. Which frankly is the point.
Fair points.
Regarding the satellite tool I mentioned this as not to predict the best time but to get an insight post run of a potential factor.
Regarding fellnr methodology I think you are wrong there. Yes a device has to be consistent even if it is a small segment and that guys also has a trail performance table that unfortunately most devices suck at it.
Let me ask, as a runner is it the point to know which side of the road you run or having a good pace and distance ?
If its the first as you mentioned in your review its not justifying the claim that the fenix offseted but measured the distance correctly.
There are many factors to consider on GPS testing.
Anyways, perhaps we will never stop debating on this but please consider:
I would love to see a GPS accuracy roundup, where you run multiple paths , multiple times with different watches at the same time. Same arm etc.
Remember re arm position one arm is inferior and one superior for GPS testing.
Let me explain. If you wear a watch at the right arm and run at the right side of the road or trail then that watch is on inferior position since on your right your view is more blocked by buildings or trees. That is not the rule but a most probable scenario.
Anyways perhaps last night I got a bad sentiment because I happily saw an accuracy post by you, but was not so much close to what I expected.
PS. Re quantified self , I don’t stab any data and most fields are supported. I think you can also filter ibi data etc. Has some way cool stuff there and does not need a backbend since it only runs on the browser. If you take a look at the graphs you can easily overlay most of the data, compare etc. Anyways enough of the self promo.
Have a great day pal and for me I’ll leave this behind and expect your next review on perhaps the new polar.
Enjoy.
BTW I just noticed. Saying that fellnr methodology is wrong is just ridiculous. Did you really read that.
I think you are comparing potatoes to cucumbers here if your methodology , (not that you claimed that) is better. Its not and that is the whole point about it. You just run 1-2 times and compare tracks.
In my 5 years of GPS testing I had different results from different even seasons of the year for the same device etc.
You asked about my data and testing. I ll be pleased to give you as many files you want for watches such as polar m400 , v800 (not that many anymore), Suunto and Garmin. Feel free to contact me via email as I don’t share any private data.
I think by showing you my online tool of anaylisis you ll get what kind of data geek I am , please do not try to turn this around.
You can find me in several online services as well, such as Garmin connect, flow and Strava and Movescount but I would prefer to keep this private. An email is fine.
Overall Ray, we are not in 2008 that we compare how pretty tracks from one runner to another.
Listen to this very very carefully and you might understand why some people can become haters or lovers:
You hold a blog with great power that can damage or praise sales of GPS device companies. A wrong test, a bug or even luck can damage the prestige of a product , and if you don’t try to follow a methodology or give a methodical testing people will be mislead. I don’t think you would like that.
I don’t think you would like people to buy an x device because it had 3 days in a row of good data while in most of the cases it sucks.
But it feels like that’s what you are doing in the end.
If you don’t want to spend time in GPS testing maybe an idea is to leverage community power for that, other reviewers or link to fellnr tables.
PS. Fellnr has nothing wrong on those tables. They align with reality and its great resource for you as well.
You know what to do pal. You know better. And to be honest you can just ditch my words and go on the same way you are used to. Its fine.
Have (again) a great day and feel free not to reply.
I would appriciate if we don’t get thus debate far away my rant is over.
Fellrnr makes (and continues to make) a massive mistake that’s been pointed out numerous times to him: He uses smart recording on Garmin devices.
As such, he only records Garmin data points every 4-7 seconds on average (but can easily range up to 20 seconds between points). You can’t do statistical analysis on that kind of data, especially while trail running. He says it doesn’t matter. Obviously, it does. Anyone who analyzes data knows it does. And one wonders why all his Garmin tracks look less ideal when he uses 1-second recording on Suunto and Polar devices.
RE: “I don’t think you would like people to buy an x device because it had 3 days in a row of good data while in most of the cases it sucks.”
I think you’re missing the point of this post. This wasn’t a product review. There’s no word ‘review’ at the top. The point of this post was looking into a single run, and then a pile of historical runs over the course of nearly a decade. This was never intended to be a crazy review. In fact, I say that in the intro. It started off as a simple fun YouTube video that I decided to expand.
In fact, all of my review posts include data over usually a month or longer (many cases, many months). I’m not sure why you’re saying I only run 1-2 times and compare tracks. In the reviews I usually have half a dozen or more tracks I’ve analyzed, but often links to piles of extra tracks for folks to dive into if they want to. I try and pick ones to dive more deeply to in the post that are most representative of what I saw at large.
Oh, and just to be clear, I really do think your tool is cool. I’ve added it to the larger tools collection here: link to dcrainmaker.com
RE the smart recording. Fellrnr might have a point here, if he presents that as the out-of-the-box experience because it’s the default option.
It took me few months of owning my first Fenix to find out such an option even exists.
It’s also nothing rare to see a question about cutting corners on garmin forums.
I’ve long argued to Garmin it’s stupid to leave that as the default setting, nobody else does. As you noted, the forums are full of people cutting corners only to find out smart recording was on. The storage cost these days is laughably negligible (it doesn’t impact battery any, as Garmin themselves have noted).
Fellrnr doesn’t present that as the reasoning though, and certainly he’s more than technical enough to change it.
Fellrnr has compared Fenix 5X with and without smart recording. There was virtually no difference. It just performs really bad. Period.
I know its not a review.
As I said I’ve read literally all your review.
I do know you do a lot of tests, you are higher than the experienced consumer and you do look at data. I don’t doubt your data testing and its quite OK.
I also noticed , mentioned here that you had them on the same arm which is how it should be. Mentioning this because I do understand your methodology and its evolving which is great man.
Re market. You have a point here. Perhaps it can’t affect market. Perhaps this is were I stuck and got a bad sentiment about this post. My mistake.
Regarding fellnr I did not know that. Does he do that also with the f5? I’ll have to ask him perhaps.
That is totally unfair!!!
Thanks for the add at the list, much appreciated.
I wish you a great day further and perhaps talk again at another post.
Best wishes.
Regarding the GPS accuracy: While you can pre-calculate similar satellite constellations to ensure the same or similar GDOP, you cannot do this with the atmospheric effects. The ionosphere and especially the troposphere can greatly disturb the signal from the satellites and thus create worse positional accuracy (you can experience this a bit, when you compare the amount of time it takes for your watch to obtain a position fix in nice weather and in cloudy weather (preferably the next day)).
As of such, the only ‘scientific’ way to compare the accuracy of different watches would be to take them all with you on the same day and mount them on some form of trolley.
At the end of the day, and I’m saying this as a surveying engineer, I don’t really see much need for extended accuracy tests. Unless the watch has a serious flaw in the antenna design or the chipset is very weak in poor signal-to-noise ratio scenarios, the accuracies should be pretty similar. And no matter how good the chipset/antenna – you will still experience poor distances i.e. on a track, where your body hides the watch from the satellites at some point and thus usually returns more than 400m for a lap.
“Unless the watch has a serious flaw in the antenna design or the chipset is very weak in poor signal-to-noise ratio scenarios”
And you see no point in testing this? Also you seem to assume the GPS data is shown raw. There’s a lot of software involved to compensate for bad resdings and the quality of this software varies of course.
I would never buy a watch without reading some tests first.
Exactly what tom said
Re: Smart recording
He said he tried a few tests on one watch, in which case he said it didn’t matter. He didn’t show any data from that.
There’s a decade worth of overwhelming evidence that Smart Recording cuts corners and of course misses track points – that’s fundamentally what it’s designed to do (in the trade off of saving space for devices a decade ago when it mattered). He’s the only person that believes it doesn’t.
Others have tried to reason with him on this point, to no avail.
He did show data.
That’s not really showing data.
On what route? A straight road? The usual test? For how long? And where’s the actual tracks?
Again, he even admits in the reviews the impact of this:
“I also use smart recording rather than 1-second as this only changes recording frequency, not the frequency of GPS polling.”
And then goes onto use said plotted data that’s missing 75-80% of what other devices use. For someone so smart, I really don’t understand why he’s so set on Smart Recording when it carries zero benefits in 2018 (or anytime this decade).
The numbers match exactly the usual test he has covered extensively. So obviously that is where the test took place.
Anyway, I trust his tests by far the most until I come across something better.
The smart recording won’t matter because the DATA POINTS fellrnr uses are the LAP DISTANCE recorded by the watch, which means they ARE 1S DATA calculated using the full 1-second sampling — regardless of the “smart recording” setting, which only determines how frequently x,y coordinates get written to the data file.
The recorded lap distance is compared to the known, actual lap distance and the difference between the two is the measure of the watch’s GPS accuracy. (It’s a fixed route that he has carefully measured and that he can re-run very closely, and he manually triggers a lap every time he reaches this calibrated lap point — like you might lap when you see a mile sign in a marathon race, except here all the markers are much more accurate, and they aren’t always in a race :). Ray did something very similar ~5 years ago when he tested some devices in a parking lot and in the park.) This addresses what Ray says above, something like “to make the comparison, I have to tell you exactly where I know I was, you can’t pull that information out of the fit file” (well said).
When he repeats this test by re-running the route many times, he finds that the measurements behave statistically, so you have to test the watch statistically, too: i.e. there’s a spread around a mean, and for each watch you will find, after many runs, how close the mean is to the true distance, and how wide the spread around the mean is. That’s how you arrive at saying one watch is good and another is bad.
Smart recording only matters a/ for plotting your route on a map — a “smart” route will look “stupid” ;) and cut through buildings going around a corner even though the distance was calculated using data points properly going around the corner ; b/ if you want to recalculate the distance or other quantities yourself from the x,y coordinates in the data file. But fellrnr doesn’t do that, and almost nobody would do that manually, but 3rd party analysis software that you might want to use may do it. So, I personally turn on smart recording, and I don’t see a good reason why fellrnr doesn’t, but it has no impact on the validity of his work.
The second criticism I read here sometimes “fellrnr always runs the same route! Who does that?”. Well, most of us do have a handful of routes that we run over and over :). But, sure, maybe the fellrnr route is particularly difficult somehow (like Ray’s route between tall NYC buildings above), and maybe watches would rank differently for someone running in the city or in the desert or in a place with better or worse satellite coverage? We would get the most comprehensive study of a watch if we could get statistical sampling (lots of repeated runs) for a representative ensemble of routes. In the ideal case we would recruit, say, 20 fellrnr’s across different routes all over the world (different in terrain or location relative to GPS systems). Maybe it can be crowdsourced one day. In our dreams watch manufacturers are already doing it. ;)
Finally, what does this say about Ray’s work? Ray slices through the ensemble in a different way: use a watch once across various routes; as opposed to using a watch many times on one route. He uses the GPS track, plotted on a map, as metric, as opposed to lap distance. Strictly for GPS accuracy, I expect Ray’s approach to capture some of the statistical variation and ‘the truth’, but somewhat less rigorously (I have a feeling it introduces more variables, has fewer data points, and is less quantitative). BUT imo THE REAL DIFFERENCE is that the two authors ultimately don’t care about the same things as much. Ray thrives on, and excels at, looking at the total usefulness of a watch, and clearly he relishes providing watch makers with user perspective, being a user advocate, and thinking about all the things that will ultimately make people want to pick up a watch and go run. When he labels a watch “good” or “bad”, it reflects a broader judgment of which accuracy is only a component. A valid approach. (I like my Stryd but it hasn’t changed the way I run ;).) Obviously this paragraph is total personal opinion, and I’ll fully defer to Ray on how he defines the great volume of work he’s done in this space.
In the end, I see that watch manufacturers have worked with both authors in the past to troubleshoot and improve their devices. A cue that both have merit. I wish that the long-going, and in my view incorrect, fellrnr bashing on this website would be laid to rest (and find it unbecoming a person with a technical background).
— Disclaimer: I’ve read DCR and fellrnr and 5krunner for many years. I like fellrnr for the detail on trail running, DCR for the wider view across multiple sports and the detail of the reviews, and 5krunner for the opinionated and spirited pieces.
My takeaway from this exchange is it’s a waste of time to engage trolls who call you pal. And Jimmy, for someone who’s read all the reviews on this site for the past 5 years, you sure are a dick.
Thanks Kevin for the thoughts.
However, I think you’re misunderstanding why smart recording is so problematic for Fellrnr. Specifically, this part:
“The recorded lap distance is compared to the known, actual lap distance and the difference between the two is the measure of the watch’s GPS accuracy.”
He only uses lap to establish a starting gate, after which the core graphed metrics shown on every one of his pages are using the GPS coordinates. This is explained on his explainer page, and in particular he notes that the lap marker GPS positions themselves are only used as approximates to help match chunks (“These estimations are only used for determining the nearest lap marker, so accuracy is not critical.”)
After which point, everything is based on plotted GPS points. He goes through multiple passes for different graphs, charts, and trueness values based on these GPS points.
I actually don’t have a problem with any of this bits. In fact I think it’d kinda interesting.
But, all of this gets shattered when you use smart recording. It makes everything except that entry gate lap marker totally useless, especially so on trail running (which is the vast majority of his predefined route). You could almost get away with smart recording on straight farm roads (though, it could inflate accuracy). But not when he’s including charts like the attached.
As you allude to midway through, while I can appreciate Fellrnr’s approach for trail runners, my concern is that most people read without the understanding that virtually all of that is on trainers. It’s also all skewed by usage of footpods in the ranking tables. Given that 99.x% of people don’t trail run (nor use a footpod), I think a much more valuable metric to folks is on a variety of circumstances and labeled as such. Here’s how it does on road, here’s how it does on trails, here’s how it does in cities, etc…
That’s what I try and do (as you noted) in each review as best as my travel/geography permits. That let’s people figure out based on their own requirements whether a watch is applicable to them.
Cheers.
If anyone is interested, I have painstakingly (using USATF guidelines) measured a forest trail course, a wide open bike path course, and (publishing next week) a 10K course with a wide variety of GPS challenges. On each course I’ve run a bunch of laps over several days measuring the mean error and standard deviation for each watch footpod. I only have a few devices so far, but will add more over time. Of course, it’s impossible to do tests holding everything constant, but I feel like this is an interesting approach for those who are curious about performance in specific environments.
what does your own experience tell you?
i have just checked my runs on my fenix 3 – I run af few different courses again and again (5-20k) – they are all within 10-30 meters the same – (about 200 runs the last year).. I call that accurate – or rather who cares if one ran 10k or 10.02k…
Reading through this entertaining thread taught me to turn off smart recording on my Fenix3!
+1
“ut I haven’t had Sport Tracks desktop installed in years. ”
What do you use for your go-to software analysis : Goldencheetah ? MyTourbook ? TP WKO+ ? Strava ?
I generally don’t use any desktop software anymore for analysis. I just don’t see the reason to. Web platforms work well enough.
I use Training Peaks for core data analysis, and Strava for more social stuff. And I use the DCR Analyzer to compare tracks and metrics related to power/cadence/developer fields.
The one exception to this is somewhat new, which is I’m using Golden Cheetah to do some Aero analysis bits due to some sensors I’m testing.
“I generally don’t use any desktop software anymore for analysis. I just don’t see the reason to”
Simple to me… I don’t want to give anybody else my data and typically I find desktop app to give more detailed analysis that online apps
“Ignore the zero-value elevation gain on the Suunto. Unfortunately Suunto’s Movescount doesn’t properly write the .FIT file header data, so in the DCR Analyzer we don’t have the ‘known’ data to pull from (we don’t want to recalculate it, as that’s not what appeared on your watch and is making assumptions).”
And yet Strava feels the need to do so all the time even though they’ve received endless feedback that users want them to match.
LOL, You asked why so as I am obviously the “outlier” in the world I’ll say why I didn’t watch the video. They are too slow! I just do not get it, I can read much faster than any video can give me information and yet EVERYWHERE that is all people want to do anymore is make videos. I just want to read. I can read, re-read if i need to and digest at a pace that preserves the amount of time I have. I can also easily search right to a specific world or phrase if I want to come back to reference something. Maybe its just that I grew up with books I’m not sure but I really do not see the need for so much video content in the world.
Although on a plus note youtube did lead me down a rabbit hole the other day to some Tori who is some sort of Vegan that really like her bicycling kit and posts videos about it ….. so there is that…..
You can read text without disturbing anyone else. My wife would go nuts if I sat next to her on the couch watching Rays videos. Long live text and the long in depth reviews!
Yeah, I don’t understand the global fascination with videos. But we are indeed outliers — just look at how many views Youtube videos get.
I wonder if (good) video production makes a better return on time invested than (good) writing and photography. Given the monetisation of videos v text, I’m thinking it probably does. Plus why not make two revenue streams from each piece of content inspiration? New watch? Trip overseas? A video *and* a written post gets you twice as much income from each.
YouTube monitization is horrible. Really, it’s horrible.
On this video, with say 10,000-20,000 views (if it gets that much, it likely won’t, but yeah never know), I’ll probably make between $10 and $20. Obviously, other content sometimes works out for me – but rarely is there a meaningful ROI on YouTube for me.
But, it’s ultimately a different audience. There’s plenty of people that watch me on YouTube that have never been to the blog (seriously, I run into them all the time, go figure).
+1
+1
Same for me. I read all articles, am a supporter and don’t block ads on this site (its whitelisted since the beginning), but I barely watch any videos.
“Whether you’re in NYC or Dubai, I wouldn’t expect massive improvements in GPS accuracy anytime soon.”
:-(
I disagree with that statement. The new capabilities in Galileo and next-gen GPS include some specifically to improve accuracy in urban environments.
Phones however are likely to improve much faster than wearables — there are now standards for time-of-flight ranging and thus triangulation of wireless access points, which will enable metre-precise navigation inside large buildings and in dense urban areas with lots of wifi.
Until the gps signal can go through steel and concrete buildings it won’t matter if your on gps, Galileo or the Soviet one. Urban accuracy will stink.
Part of the problem is multi path interference. Broadcom is apparently releasing a chip soon that will also utilize another code on the GPS signal to do better in urban canyons.
I’ll let Ray try it first…
link to spectrum.ieee.org
Maybe in urban environments with tall building could help the use of multiple constellation simultaneously.
We saw in the test of DCRainmaker that the Fenix5plus, that was the only watch that used two constellations of satellites (GPS + Galileo), was the best of the three in the section of Manhattan with tall building, even if the quality overall was bad and if the difference in quality between the three watches was not so the big, in fact Fenix5plus got a D and the other two a F (Ray writes: “I’d argue for this specific section the Fenix 5 Plus was closest to the road, but that’s like giving an award to someone who gets a ‘D’ in class, merely because the only two other people in class got an ‘F’.”).
Maybe a further improvement in accuracy could be obtained using simultaneously the 4 constellation available today (GPS, Galileo, Glonass, Beidou.) We could imagine that in the section of Manhattan with tall building, the satellites in view could increase from a number of, let’s try to guess, maybe 5 in the configuration of two constellation (GPS+Galileo) to 9 in the configuration of 4 constellation (GPS+Galileo+Glonass+Beidou). Maybe with this increase of the number of the satellites in view the accuracy of the position would be better, even if the position of the satellites would be not ideal because they would be high in the sky without some satellites in a low position above the horizon (this means high DOP -dilution of precision – and so less accuracy).
It would be nice if the watches could report second by second the number of satellites in view and the DOP value, and if these values could be plotted after the run in a graph, for example in Garmin Connect. I think that a watch knows exactly second by second the number of satellites in view and when a watch finds a position fix, it knows exactly the DOP value for the satellites that it uses. So I think that the watches could write in their memory second by second number of satellites in view and value of DOP. It would be interesting to ask the brands (Garmin, Suunto, Polar, Coros) if this is possible and why they don’t give this information.
The problem is that the use of 4 constellation simultaneously has a hit on the battery, so I don’t know if they will ever produce this type of chip for wearables and sport watches. This type of chip that can use simultaneously 4 constellations already exists for cars, but I don’t think for watches and wearables that till now can have maximum just two constellation simultaneously.
By the way I found the test nice and well down. I liked in particular that it analyzed in the same test and under the same condition a section in urban environment with tall building and a section in open sky (central park).
For example the new GNSS chipset of Broadcom for smartphones BCM47755, installed in the Xiaomi MI8, can receive simultaneously all the 4 constellations (GPS, Galileo, Glonass, Beidou) and also SBAS satellites (WAAS in America, Egnos in Europe and QZSS near Japan). The chipset of the same family for wearables, the BCM47758, is capable to receive maximum two constellations simultaneously, GPS+Glonass or GPS+Galileo (I don’t think that it can receive SBAS satellite, but it’s not clear from the technical data that Broadcom has released in the pdf product brief that can be found here link to broadcom.com ).
I wonder why one of these brands can’t make a snap to roads feature for their watch software.
Ray, what firmware were you using on the Fenix 5 Plus? The recent betas are giving me massive improvements in terms of gps tracking compared to the latest official firmware..
2010 isn’t quite far enough back for when Garmin had to switch from SiRF, though, is it? Most of the GPS complaints I see about Garmin say that the Mediatek chips they use today give worse tracks than the SiRF’s they used to use. I own a couple of Garmin’s with SiRF (60CSx, and I think my 705) but I’ve never gone into the weeds comparing tracks (in any case, there are other reasons a 60CSx, like the HONKING HUGE ANTENNA, might give better results than an Edge 1000).
I have; my FR310XT was consistently better than the tracks my friends got from the likes of 920s and 235s. But my FR645 (which replaced the 310) is just as good. Manufacturers switched to smaller designs, sacrificing antenna size for style, a little before tech could mitigate the change.
I think the loss of accuracy in current GPS-watches is due to the fact that engineers want to save energy for longer batterie life. So they have to choose low-power chipsets which, as a negative effect, are less accurate than older chips.
I’ve had a FR935 — really nice watch with a stunning long batterie life. But GPS was very poor compared to my old V800. So I dropped the FR935 (an other reason was the horrible Garmin Connect site and the feeling that my watch collected a bunch of data, but nothing really important came out of it. With Polar Flow I can always read a lot more out of the data. But that’s another topic …)
Generally speaking it’s actually more the antenna.
Most older watches have huge antenna bumps on them. For example, the Suunto Trainer (and Ambit before it) has that big bump at the bottom, that’s where the antenna goes. The old FR305? Yup, same place.
Whereas these days companies try and integrate the antenna into the bezel. Which isn’t to say companies didn’t do that before. Many other Garmin Forerunner watches do that just fine. But, there’s plenty of examples where Garmin and others have hosed that up too.
That you are wearing is a Perth based Western Force rugby union jersey, formerly from Super Rugby now in some kind of Twiggy named billionaire supported limbo!
Yeah, a DCR reader gave it to me down in Perth when I was there. We went for a run together. Cool dude!
Sorry but can’t find the V800 is akin to dog ate my homework excuse?
I’d like to see Polar V800 vs Suunto Spartan Ultra vs Garmin Fenix 5 Plus, or the Polar Vantage once announced?
Seriously, I wish I knew where it was. I’m going to be in a pickle soon if I can’t find it. I’m probably going to spend 2-3 hours today looking for it actually. Boxes upon boxes. Sigh.
I had actually set it aside back in December prior to the move, because I wanted to take it to Australia for the three months I was there. I did that because I suspected Polar would be launching their new watch during that timeframe and wanted to compare it side by side. This meant I took it out of the device bin and put it somewhere, and then it got put somewhere else in the shuffle. Likely an errant box.
“The dog ate my homework” is a perfectly good excuse when the dog actually did eat your homework!
Very interesting. As you may have surmised, GPS tracks do not necessarily give much insight into a watch’s accuracy. I’ve created the definitive test loop for watches and will be publishing the results soon.
What about the upcoming Polar Vantage watches?
Have you seen them?
What can you tell us?
Would be nice to see if the Suunto 9 would have had a correct GPS track without GPS.
Yeah, I do think that’s an area where Suunto is doing some really cool stuff – and I’m excited to see what that looks like in say 9-12 months from now once they’ve had more time to finesse it.
Bigger Garmin maps!
1. Use Chrome, I used it on macOS (both current versions)
2. Right click map, choose Inspect Element
3. Edit the highlighted ‘height’ parameter shown in the screenshot.
4. Move the map around a bit to trigger it loading the missing tiles.
5. Voila: link to images.paulwakeford.info
Not exactly user friendly but it works.
Huh, nice trick! Handy for next time!
Great tip, thank you!
It works in Opera browser too, just tested it
You can make it more user-friendly with a ‘bookmarklet’. Create a new bookmark in the bookmark bar (a favorites, whatever your browser calls it), make a nice label like ‘Big map’, and add the following as URL: javascript:(function(){document.getElementById(‘activity-map-canvas’).style.height=’800px’})();
If this shows up as a clickable link here, you can also drag it to the bookmarks bar.
Let’s make it even easier, go to link to gcmap.s3-website-eu-west-1.amazonaws.com and follow the instructions.
Apparently, Garmin is reading this as well: there is a button now to maximize the map on Garmin Connect :-)
Usually GPS tracks are examined and judged by using streets and buildings on Google maps as a reference.
I wonder how precise these maps are? If Google used GPS measurements to align images on its model of the world the map could be shifted as well. Lenses and even fluctuations in the atmosphere will introduce distortion when the images are taken. The vector maps are almost certainly based on GPS measurements.
Do you know how precise these maps actually are? Does it vary globally? By how much?
I generally try and use satellite imagery in GPS track comparisons when it gets to the detailed points, because the non-satellite maps (vector maps) can often be inaccurate at that level of detail (especially near water).
The challenge with satellite maps though is that when you zoom in on Google map-view it’ll go into the angled view (usually taken from an airplane) when you get too close to the ground, which then distorts where the lines are. I suspect there’s a Google Maps API option to tweak that I could look into.
Wasn’t there some noise several years back of the Chinese deliberately offsetting maps of China (as if reconnaissance satellites didn’t exist) and causing some trouble for tourists using GPS?
“I suspect there’s a Google Maps API option to tweak that I could look into.” It is the ’tilt’ option, on initialisation of the map use `tilt: 0` to avoid the angled view.
Great writeup Ray. These are the kinds of posts that got me checking DCR the time – reading about you travels and generally getting lost in the geekery of the data. Let’s have more of them!
Hope you’re feeling better….
Thanks Neil!
Hey,
Test wasn’t fair for the Suunto. You should know the watch is meant to be on your wrist and the gps module directed up to sky! That is the whole idea to have separate module in the wrist band.
Which is why I specifically noted that when I ran I held the watch with the pod module pointed at the sky.
Still, I’m confused at folks saying the Suunto was somehow disadvantaged…given it did better in Central Park than the Fenix.
Ok. Thanks for explaining!
I’m using now S9 and trail running. Isn’t yet as precise as A3P was. I think the placing of the gps chip is the main reason
And with a Mediatek 3339 (GPS only). Antenna is very important. All standard GPS chipsets have now high sensitivities. Is not for that. What is the common domineering between Ambit … and Suunto Spartan Trainer? The design at bottom. Generally i see points of you, Jimmy, and Jonathan (Fellnr). And you all have valid points. We have to read carefull. And i do understand that your post have other intention. But i have to agree with Jimmy. DC, is important to test watches, in same conditions (GNSS system equal, recording frequency, in some wrist, etc) watches with same range, and several runs in same route path. It will not 100% equal, but it will show tendencies of accuracy. I have one suggestion, if you would have patience. Pick a route, not many miles/km, half city, half open. Put watches in same conditions and repeat more than 5 times in this route. Use what software you like most. Plot for we see as you do in your reviews. Then, we can see tendencies in different days (with different weather, with different available satellites). Should we take any conclusion or not? I think it will bring many people with attention. Can i suggest first one? S9, Polar Vantage V and Fenix 5 Plus. Good work and thanks for reading!
Generally speaking I only wear a single watch per wrist, because that impacts optical HR sensors (hugely if more than one device per wrist).
Hence why i’ll usually hand-carry a secondary watch (3rd watch I suppose) in an orientation that doesn’t block any GPS aspects. That hand-carried unit is typically connected to a chest HR strap (to use as reference against the optical HR sensors). If a fourth unit is carried in the other hand, then I usually connect it to a TICKR-FIT or Scosche sensor.
How about one single number representing rms deviation from actual. That would be worth writing about.
But how would you plot actual? Manually?
Just in case you are interested, I’m conducting a series of tests for GPS watches and foot pods, reporting two numbers: average error from the true distance and the standard deviation of the sample.
A couple of question, hoping you can find a minute to answer.
1) Not too long ago I read about Broadcom developing (or having developed) a new GPS chip that uses a new/better signal. If I’m not mistaken, accuracy would be in the <0.5m area. Did you hear about that and if so do you know if there is any R&D with that chip in wearables?
2) I'm currently using Movescount and TrainingPeaks. While they are both ok, they lack a ton of analyzing capabilities. For example – I have found no way to compare three bike rides. I'd just like to see where I was faster/slower compared to other rides.
Oh, and my favorite – there is no way to find out what distance you did on your bike in e.g. 2017 in Movescount. Just simply not possible.
Unfortunately you don't talk/write much about the software side of things. So my question is, if it wouldn't make sense to write a review or comment about software every now and then?
Thanks up front for answering my questions and keep up the good work!
That‘s why I love Strava. You can create your segments (maybe make them private if you‘re only interested in your own progress) and compare your rides. There is even a nice diagram where you can choose which rides to compare and it shows you exactly when you were ahead or slower (note: the diagram is not included in the free version of Strava). Also you get a perfect summary of miles for every bike. If you want more, you can connect to VeloViewer, which is awesome and gives you dozents of further statistics.
Hi Dirk,
thanks for the tipps. I’ll look into VeloViewer this weekend.
I did try Strava with their segments, but couldn’t get it to work properly. Only slight deviations from the route and it won’t compare the two rides. Maybe I should try it again. Who knows, maybe it was just my incompetence …. :-)
If you create segments by yourself you should have no problems then riding them again with the same device. (Check GPS accurancy on DCR site ;) )
What I also like is that you can also log parts of your bike, eg you can easyly see how many kilometers your chain is already on your bike, or your brake pads and so on …. (especially useful if you‘ve a lot of bikes to think of).
Yes it it. Click the year in calendar.
Now you see totals of that year. If you want to see all the statistics only from cycling, you van additionally filter the data to show only cycling.
Thanks Dirk, I’ll have to try again.
@Janne: That works only for the current year, not e.g. for 2016 ?
“Oh, and my favorite – there is no way to find out what distance you did on your bike in e.g. 2017 in Movescount. Just simply not possible.”
This is actually not true, Flo. You can do this rather easil by opening “me => yearly summary”. This will give you the totals for the current year (i.e., now, 2018), From there, you just need to switch from the calendare view to the table view (second little circle on the top left of the calendar) and click on the “2017” year on top. This will automatically select all mvoes from that calendar year, with totals below. This workd for every calendar year, or calendar month (if you stick to the calendar view instead of using the table view).
Hope it helps.
Wow, people lose their minds over GPS testing.
Anyway, the Fenix 5 and 5 plus both have new beta builds available with the purpose of improving GPS. My initial testing is promising. I’m not sure if it’s actual improvements to accuracy, or just smoothing.
“GPS FW (2.24) should improve pace accuracy for running/walking activities and track performance for open water swimming activities.
Our public beta users are encouraged to load the new GPS firmware and use it for their next activities. If there are any activities where the GPS performance did not improve, we would like to see them. This also includes if comparing to previous activities….”
Galileo (in current sat fleet) covers also NYC??
Galileo is a global network like Navstar and Glonass. Of course it cover!
Oh, Boy!
Good old FR310XT… I started reading DCRainmaker, because I found the manual and the product review more detailed and useful on your page then on a hard copy attached…
Bought in 2009 for 310EUR and sold in 2012 for 210EUR when swapped for FR910XT…
Anyone can say whatever he wants, but when I used 310XT and 910XT at the same time, the former seemed to be more accurate in terms of GPS accuracy, although the difference was not so big.
Due the barometer inside I kept only 910XT.
Now when I use 910XT and Fenix 3 HR together, 910XT is a clear winner. Almost always, I mean at least for 90% of the activities. Full stop.
Fenix family may never be as accurate as the rectangle Garmin watches.
So you mentioned a couple of times, frustratingly, that there is no good way to look at a map to find your old routes/tracks. A personal heat map, if you will. WHY OH WHY NOT?? This frustrates me to no end! I’ve been logging rides all over the place for years, and I would LOVE to look at a map to pull out some old rides, to either analyze progress, or just to see if that familiar looking place is somewhere I have actually been before. Why is this not a thing? I would use it all the time. Just like doing a photo search using a map. I may not be able to remember when I took it, or what I may have called it, but I can often find it by location.
You mean like this? link to jonathanokeeffe.com (assuming all your data is on strava)
Awesome post & comments!!!
“and then simply hand-held the Suunto, keeping the satellite bump pointed at the sky at all times”
in the table the Suunto device shows an avg. HR –> did you wear an HR-strap? But if so, why is there a 3bpm difference between the Suunto Trainer and the Fenix?
Yes, it was connected to a TICKR. Though, it seemed to have dropped the connection at one point. Not sure if that was the result of the TICKR or something else. I suspect that battery was dying, as I changed it the following weekend.
Read the comments section after the reference to it in the newsletter. Wow. Intense.
Ray – you’re the man. Huge fan of your blog and appreciate your expertise and neutrality. Thanks for all you do.
Having almost the same distance the Suunto (11.32 Km) and the Garmin (11.28 Km),
how is it possible that the avg_speed is more than .5kmh difference?? (12.86 kmh vs 12.14kmh )
Even the simplest metrics you cannot believe!
This is why, living in London, I have a Stryd footpod, and as long as I calibrate it periodically, it works a treat everywhere. It’s especially great in races where there are tunnel sections like London Marathon or Big Half.
Great post DC… grief seems to be a bit random given the nature of the post… keep sloggin away
Hi,
I’ve read that the Fenix 5 watches with maps (5X, and all 5 plus) have an option “keep on track”.
This is supposed to keep your location within the streets. It would be very interesting to see if that would make a difference in your test. Also it would be interesting how the modes “GPS only”, “GPS+Galileo” and “GPS+Glonass” would give different results.
Cheers
It is not surprising GPS+Galileo does not give a significant improvement. Until october 2018, US laws prohibited use of Galileo and the sattelites were not yet launched. As of the last months of 2018, 2 sattelites are added making the total 22. But only when the last 2 sattelites are launched in 2020, US users will have the full benefits. Galileo supports precision up to 50cm for consumers, compared to the 3 to 5 meters for GPS and 4 to 10 meters for Glonass.
I live in Europe and noticed a significant improvement.
I’m pretty sure it was enabled for Garmin devices prior to that.
Still, of course, the constellation continues to grow and will only get better. That said, I haven’t noticed a major difference for my devices living in Amsterdam in terms of accuracy.
Maybe your watch was receiving Galileo satellites because you had a european version. It’s possible that american version of the Fenix had Galileo satellites filtered (like american version of phones seemed to have Galileo satellite filtered in firmware).
Well, Garmin doesn’t let us know any more with wich satellites the watch is connected, so we will never know.
Is there a way to change the Coros Pace’s background to white with black numbering? Thank you.
I know that the article is outdated, however, the Garmin 45 or Coros Pace better? (Because the price are same.) Could you advise which of them would be the better option for running? Thanks!
Hm