(Also note: A few weeks after the test was published Garmin released a new firmware update that fixes the FR610 issues, and I redid some of the tests here).
Last fall to the request of many I put together a bit of a Sport Device GPS Accuracy Showcase Showdown. That test had just four devices – the Garmin FR310XT, Edge 500, Edge 800, and the Timex Global Trainer. The goal of it at the time was to look into how accurate sports GPS devices really were against a measured course, and that goal remains unchanged this year.
This time though, I doubled-down and went to nine devices:
– Garmin FR310XT
– Garmin Edge 800
– Garmin FR610
– Nike+ GPS Sportwatch
– Garmin FR210
– Garmin FR305
– Polar RCX5
– iPhone 4 (with RunKeeper)
– Sony Ericsson X8 ANT+ Android phone (with RunKeeper)
I wanted to take out the most popular new devices (and a few vintage/popular devices) and see how they faired against each other at the same time on differing courses. In doing so I scouted out five different locales that covered a variety of scenarios, and then did each course three different ways – walking, running and cycling.
Finally, with so many of the devices now using the exact same underlying GPS chip – how important is the software that runs the device when it comes to distance accuracy? And would devices made by the same manufacturer with the same chips yield the same results?
Let’s get into how it went down.
Methodology:
Over the course of a number of weekends I did five separate and distinct tests. The tests involved five different courses with varying levels of ‘difficulty’, these included:
1) The Straight and Narrow: A perfectly straight out and back course with low tree coverage – exactly 1.00 miles.
2) The Rambling Loop: A half a mile loop with plenty of turns, a bit of trees and some ups and downs – basically your average run.
3) Circles around the track: Two laps around a 400m track, for a total of 800m, all on the inside lane.
4) The underpass and bridge test: Many of us run/ride under bridges and overpasses – how would a short out and back session under the four lane bridge fair? Would the GPS handle drops in satellite availability and then regain the signal?
5) The Deep Tree Adventure: Into the woods I went, on a fairly twistyroute on dirt trails.
Of course, you may be wondering how I was measuring these. Well, you might remember back to some of my previous course measuring posts, where I picked up a rolling measuring tape – similar to those used to mark out distances for cross country meets.
I took that little roller and then strapped all of the GPS devices onto it. By doing so I could guarantee they were all going the same distance at the same time. Plus, it looked way less sketchy than strapping them on my wrist. For more on my methodology see the end of the post.
Test 1: The Straight and Narrow:
This is my defacto first test, which is about as simple as it gets for a GPS device. All it’s gotta do is just go half a mile out, and half a mile back on a almost perfectly straight course with little tree cover, no telephone poles or anything else blatantly blocking the view.
Here’s what the trail looks like from the ground:
Last year when I did this, all of the devices were within 1.2% of the baseline distance, with the majority even closer than that.
For the methodology I staked out a start line with some pink tape (easier to see from a distance than chalk), and then walked outwards half a mile until the measuring wheel read 2,640 feet (half a mile). At which point I planted another pink flagged into the ground, did an about face, and came back. I gave the units about 8-10 seconds to stop and register the turn – the same process as last year. I did this once walking (with the measuring wheel), and then running, and then finally cycling. I worked to stay within one foot from the path’s edge in both directions.
This year I saw a tiny bit more variance in the numbers, but still all contenders did very well on this result (as they should) – all but one of which performing within the usual 2.5% that GPS is quoted as being accurate to, here’s the results:
I thought it was interesting to see that the units seemed to do best at speed, which is a trend I’ve noticed a bit over the past two years of this. It’s almost as if the walking allows the GPS to wander a bit and decrease accuracy.
The only outlier was the walk test with the Android phone. It’s a bit unclear why it went over so much, but it seemed to get back on the mark with the run and the bike portions.
With the straight and simple tests complete, it was time to kick it up a notch.
Test 2: Circles around the track:
I added this test this year for two reasons. Firstly, because you asked me to…and secondly…it’s really easy to do. Since I don’t have to measure the track (known 400m per loop), and I’m not allowed to ride my bike on it, it makes it a rather quick test to knock out and at the same time answers many questions about how well the units perform on a track.
Now, I should point out that generally speaking when I’m running on track during a workout I’m using the track as the distance ruler – not my GPS (I use that instead as a general reminder of roughly where I am in a workout). But I do understand that some prefer the GPS measurements from a tracking standpoint.
This test was two part. First, I ran 800m (two loops) on the inside lane. Following which I walked 800m, again on the same inside lane. I started and stopped at the same line both times.
Here’s the results of the first test (running this time). Keep in mind that since we’re only talking half a mile – that just .01mi gets a unit knocked down 2% to 98%. Which means it’s plausible that the watch was all of a few more feet from being over/under. In other words, in shorter distance tests don’t overthink too much only .01mi off.
As you can see, most of the devices did quite well. Track corners do tend to throw GPS for a bit of a loop in general (no pun intended), but these are pretty close to spot on. And as I kinda said before, if you’re on a track, use the known distances there if getting down-to-the-second precision is necessary. The RCX5 did perform a bit lower than the rest, a data point that seemed to continue on the track in the walk test. The Edge 800 also had some issues on the run on the track, but did better in the walk test.
Now let’s look add in that walk. For most of the devices we see pretty much the same, except there’s one big outlier:
That’d be the one at 43% marked in red, by the way. So what happened? How does it go from what should be .50 miles to 1.15 miles? Well, it didn’t actually get lost…it just got distracted. You can see that below in the map, where it starts off good for the first 30 or so meters, and then poof – it goes and visits some houses, before visiting another set of houses:
The catch is, that it’s not terribly uncommon for GPS data for iPhone users, or some cell phone users in general. You’ll find plenty of complaints about incorrect data points, often caused when phones attempt to pickup location data using WiFi hotspots instead. In fact, if you look carefully you’ll notice it’s interesting that it seems to go straight to some houses on the right side, and then again back to houses on the left side – exactly as if it picked up WiFi as I got close to each set of houses (with nothing bit thin air in between the track and the homes). In my tests I didn’t disable WiFi, and while I could have – to me that’s just an annoyance that many users wouldn’t bother with. Additionally, this is the only time I saw this occur throughout the tests. And it only occurred with the iPhone.
All in all though, most units did just fine with the track test, so let’s move onto a more common scenario.
Test 3: The Rambling Loop
The rambling loop is simply what most people likely run on a day to day basis. It’s half on a running path/sidewalk next to the street before cutting across a parking lot, ducking under a small grove of trees onto gravel road near a small pond before looping back around. All in the loop is .52 miles long. Measured this day as 2,727 feet (last year it was 2,733 feet, so only five feet difference – impressive year or year measurement). Here’s an overview:
And here’s few pictures from the ground. First, looking across the parking lot, I run towards the camera, and then the trail behind the waterpark, where I run away from the camera:
For this test, I did three variations: Walk, Run, Bike. Each variation got progressively faster with the change from the walk to the bike – which is the idea, to see how it tracks at varying speeds.
With that, here’s the results for the rambling test:
As you can see – almost all the units were within 2% on all the tests. The only outlier this time was the Garmin FR610, which was steadily set in thinking it was .50 miles each time. It should be noted that the FR610, FR305 and RCX5 all were set to 1-second recording rates, while the other units were using either Smart Recording or proprietary recording algorithms (could not change). And finally, again you notice a very subtle trend where higher speed tests actually do better than lower speed tests.
Mini-Wrap Up
So with Part 1 complete you’re seeing that the majority of the time in ‘normal’ conditions the units do fairly well. Sure, there are some outliers, but we’re not seeing huge 5-10% issues like in the past. It’s also really important to stress that the colors are there merely to allow the brain to quickly see the differences, but in some cases – we’re talking only .01-miles differential between a ‘green’ and a ‘yellow’.
Tomorrow though, we’ll dive into the trees and tunnels and start to really separate the boys from the men…or the girls from the women, and then we’ll wrap up with all of the details consolidated so you don’t have to scroll up and down a bunch.
For the rest of the tests, as well as the summary results, head on over to Part II…
Q&A Post Show For Part I:
When you start doing testing of any sort there are invariably just as many questions as answers. So, I wanted to outline some of the reasons I did what I did, in an effort to answer as much as possible. Do keep in mind that while my goal is to be as accurate as possible, it’s also to be as realistic as possible to a normal runner. Additionally, I wanted to cover as many devices as possible in as many scenarios as possible…while still trying to make it possible to spend some time with my fiancée. Sometimes you have to eventually draw the line on data collection methodology (especially since this isn’t my day job).
Q: Why didn’t you include XYZ watch?
A: It depends, but in most cases it’s because I didn’t have it or already tested it, or it’s duplicate with one I tested. For example, I didn’t do the FR410 because I never re-purchased it after the review since it’s too similar to the FR405/FR405CX (same chipset) and I’m not personally a huge fan of the bezel. I didn’t add the FR110 because it’s just the exact same watch as the FR210, except without footpod support and instant-pace data field. I didn’t add the Timex Global Trainer because I did that last fall, and saw no reason to beat that dead horse again. And finally, I didn’t add any Suunto stuff…simply because I don’t have any Suunto stuff.
Q: You should have used XYZ phone app, it’s better.
A: True, there are no doubt better apps. And any time you choose apps there are difficult decisions – you know, like Angry Bird Classic or Angry Birds Rio. But I specifically used RunKeeper for a few reasons. First, it’s offered on three platforms (iPhone, Android, Windows 7 Phone), so it makes it super easy for me to consolidate the data afterwards since it ends up in one account. Second, they have the largest install base as far as running apps go – thus, the most people are probably using it. Third, they’ve done incredible work to try and fix phone GPS data coming into it. Of course, others have done this work as well, but then see point #1 again. :) Fourth, since they offer it on multiple platforms you can attempt to compare hardware a bit more than comparing developer coding skills (though, not 100% of course). Fear not though, I’ll be doing at some point an app comparison, though I realize I’ve been saying that for some time. Oh, and as a side note, the iPhone was running version 4.3.2 and the Android phone kernel 2.6.29.
Q: Why don’t you change the position/orientation/sunlight levels/etc of the watches/phones?
A: I tried to remain as close as possible to how a person would normally wear their watch or phone. For the phones, they went in pouches designed for them and onto my arm – exactly as intended. For the watches, they went on the pipe portion of the measuring wheel than I ran in front of my chest with – positioned about the same distance to my body as my wrist would be. While it’s plausible that one watch got a slightly better ‘deal’ placement wise than another – that’s life. If a watch can’t get good results because I moved it three inches to the left, then quite frankly…the watch sucks.
Q: What about XYZ watch that comes out in XYZ timeframe?
A: Eventually I had to draw a line on waiting. If/when Garmin and/or Timex and/or someone else comes out with a new watch, I’ll circle back and do accuracy tests with those. At this point I’d suspect I’ll wait until all of the summer related releases occur (including things leading up in EuroBike/Interbike in late August).
Q: Why didn’t you do any really long courses – like 30 miles?
A: I have no desire to either run or walk 30 miles, it breaks a key running rule I have: Thou shalt not utilize thy feet to go more than 26.2 miles without wheels. (Exceptions to the rule are made if people are running after me with weaponry, or a giant Smurf is chasing me.) Even if I had ignored the walking/running piece and simply done bike tests – they would only be comparing against themselves. While you could use a speed sensor, once you look into that a bit more – they’re not quite as accurate as a Jones Counter bicycle, and that involves a lot more complexity around PSI and everything else.
Q: Why did you reduce from exact meters down to one-hundredths of a mile?
A: I did this for two reasons. None of these units display meters to the end user, but rather all display in .01 increments, thus, it seemed a bit odd to measure more precisely if at the end of the day rounding would show simply one only show the non-rounded value to the user. After all – it’s what you see that matters, not what the unit internally calculates. Secondly, I did this to make it more manageable for me. With nine devices, five umbrella tests, and upwards of three subtests per major test, I was looking at upwards of 135 files to consolidate, analyze and record – across many different platforms, file types and services.
Q: Why don’t you pickup a Jones counter bicycle to measure the routes?
A: Because my fiancée would kill me if I added yet another bicycle to the stable. But more practically, some of the routes I’ve chosen only permit running/walking – such as the one in the park in Part II and the track, where bicycles are not allowed on the trails/tracks. Further, my little roller does just fine for my purposes. :)
For the rest of the tests, head on over to Part II…
FOUND THIS POST USEFUL? SUPPORT THE SITE!
Hopefully, you found this post useful. The website is really a labor of love, so please consider becoming a DC RAINMAKER Supporter. This gets you an ad-free experience, and access to our (mostly) bi-monthly behind-the-scenes video series of “Shed Talkin’”.
Support DCRainMaker - Shop on Amazon
Otherwise, perhaps consider using the below link if shopping on Amazon. As an Amazon Associate, I earn from qualifying purchases. It doesn’t cost you anything extra, but your purchases help support this website a lot. It could simply be buying toilet paper, or this pizza oven we use and love.
The 610 doesn’t look like its doing too well for the gps based watches
Love the test, I only ever test my 205 on the road against the km marking of the local Marathon. I have, in the past, before I had the 205 borrowed the clubs wheel, but the wheel is old and not so accurate. I checked it on the track.
I also have a Casio GPS watch, but don’t ever use it as on GPS mode it dosn’t even last an hour and I guess it’s out by at least 10%…
I have also found that what I run/walk up the side of a mountain Mr. Garmin often reads funny, like I can watch him count back as I’m going forward!!! That said when I down-loaded the data it looked a whole lot better. (eg the watch said I had run 1.4km up while once down-loaded the map said 1.8km).
Ever had that sort of problem?
Thanx
Got one question regarding a problem I always have with all the Garmins I had till now (110, 405 and now the 310XT):
The accuracy at the beginning of my workout is horrible! It locks the satellite after a few minutes, the accuracy says something around 5 to 7 meters, then when I look at my workout the first 1km of the workout is always around 100m off (with the 405 it was always to the south, with the 310 it is always to the north).
Resetting the satellites (power on while holding down) doesn’t help.
So I need to wait at least 5 more minutes after it locked the satellites before I start the workout, ever had this problem?
Thanks for doing such a thorough research – great work and very revealing! Looking forward to Part II.
I Call this type of work “Public Service” :)
Looking forward part II (no pressure intended).
BTW, Sunnto people, come on, a T6d, please, multi sports Pack, ok? Thank you…
Do you use the 610 in smart recording or in 1s recording ?
Interesting to see that the 305 does so well albeit such an old device.
Looking forward to part 2 of the test!
I am impressed with the 210, getting almost a 100% in all situations. Thanks for your huge test, very nice to see all your tests.
I really enjoyed reading your GPS watch test, even though I’m not that interested in the actual results. More than anything, I enjoy hearing about how you set up the test, the lengths you went through to ensure valid and reliable results, and the way you express yourself. Keep on blogging.
This is fascinating stuff! Thanks for taking the time to develop the methodology and then execute it. I am personally very interested in the race between the 310xt and 610!
Thanks Ray. Interesting to see the RCX5 performing fairly well. I’d love to see a comparison of the new Polar GPS brick to the last one, to see if there’s been an improvement. But, since I’m running with the 310XT mostly these days, I’m glad to see a somewhat aged device performing so well against newer ones.
Also, as an aside, where is that track? I’ve been going up to T.C. Williams, but that doesn’t look like their track. Is there one closer to you?
Thanks for the post.
You are correct that some devices will “go for a wander” at slow speeds or when stopped. GPS data is very sensitive too all sorts of things in the atmosphere, and it is down to the software to filter the data and workout what is actually happening. Kalman filters can be used to constrain your movements, and make a better estimate of your position. When you are moving in a certain direction at a decent speed it can be assumed that you are not suddenly going to be 1 meter in the other direction or have suddenly stopped. However when your still or moving slow, the filter has a hard time determining what is movement and what is just timing inconsistencies. i.e. the device doesn’t really know what your doing, it’s only making an educated guess.
@coach dion
I would guess it’s not picking up your elevation correctly and throwing it off. Garmin connect will use elevation info from mapping data instead of the gps data, which can change your distance if it’s off.
What a surprise ;) that the 610 comes up short on a regular, turning rambling route. I wonder where I’ve seen that before….
Did you clear any of the windows that pops up when you start the watch, etc.? That seems to help somewhat.
Excellent detailed comparisons among all the GPS based watches. Just wondering if you did take note of the GPS accuracy at that point of time for some of the watches like the Forerunner series.
Outstanding work here Rainmaker…thanks for the effort.
I have no idea how you’d conduct this test, but would it be possible to test elevation accuracy as well? I’ve got the 210, and love it, but am not sure I trust its elevation reporting. I’ve biked the same route multiple times and come up with different elevation gains each time. Also when I run on bridges (over water), the altitude goes negative.
Any thoughts you might have on this would be greatly appreciated! And thank you for all the tests and reports you’ve already done.
This comment has been removed by the author.
Great stuff…. it would be interesting to see one longer test, as a 10 meter error is 1.25% of 800 meters but only 0.1% of 10 km. A 1.25% error on a 10 km time would be roughly equivalent to a 30 second error in running time, by 10 meters would be only 2.4 seconds. So it’s not easy to extrapolate results from short tests.
Still, fantastic work – if I’d done your pre-test “taper” workout I’d have been napping, not conducting quantitative analysis….
Run!
link to bradaptation.com
Excellent and thorough testing. I’m actually quite amazed with all the devices as they all only seem to deviate from 0 to 32 m, where a normal gps fix on my phone has an accuracy of around 8 m. And you can’t wear that brick on your wrist.
I agree with djconnel about the longer testing, probably the absolute accuracy will remain about the same, but in percentage, it should go down. But that’s the theory, it’d be interesting to see if that really happens.
Great work so far!
Thx for the test.
As some poster said before it’s really interesting how well the FR305 does considering how old (and cheap) it is in comparison to many of the other devices.
This is awesome!! And totally fascinating. I’m thinking I should have gotten a 210. My 305 is holding its own, but I want perfection =)
How wide is the stick you used on the track? If it’s .5 meters, the inside watch traveled .5x2x3.14×2=6.28 meters less than the outside watch. Over 800 meters, that’s .79%, or almost 1% difference.
As always, your stuff is thorough and a pleasure to read.
I was surprised at the numbers of Garmin based units in your tests and no Timex Global Trainer as an alternative. But in remembering the results of last year’s tests, perhaps it was a bit reluctant to show up for the rematch.
Thanks¡¡¡¡¡ Good job¡¡¡
I’m not as worried about the accuracy of the GPS devices as I am about the accuracy of your rolling measuring tape. There’s a reason they don’t use something like that to measure certified race courses.
My biggest problem with accuracy is like Andre mentions, at the beginning of my run. It has especially been a problem with my FR410. At times the same .5 mile beginning section of my run has ranged from .35 to .76. While most of the time the difference is more like +- .05, that is still 10% off in the first half mile. So how long did you wait before you started the tests? Did you check to see if the on a map whether any simply started in the wrong place. I am thinking about getting rid of the 410 because of this problem and from what I’m seeing maybe I should go to the 210 instead of the 610.
Did you use the footpod for any of the garmin watch devices by any chance?
Have you done any accuracy tests in hilly areas? Appreciate your tests. Bought my wife a Timex Run Trainer GPS based on your review.
Interesting review but you have only collected a single sample for each course. Even as few as 3-5 replicates would be useful in evaluating the variation within each model. I realize this would take additional time to repeat the courses multiple times but honestly, running .5-1 miles sevral tiems isn’t a huge time suck.
From Italy and sorry for my bad English!
For my test, I run on inline lane and on outline lane of a track with my Garmin 210.
The first lane measures 400 meters; the 8^lane measures 450 meters.
I run for several times 2 loops on the external lane (900 meters) and on the internal lane (800 meters).
Then, on the external lane I got 900 meters by my 210 (OK!!!); but on the internal lane I got the same 900 meters!!! 100 meters more!!!
I repeat the test in another day end the results were the same.
I don’t understand as your test with 210 on the track was very good and the mine no!
Ettore
Great Job there!
And really interesting reading! It’s a huge help for both curious beginners and experienced runners as myself!
I’ve been searching for a long time for these GPS watch comparisons, But they have been really hard to find. I have the Nike/Tomtom watch, It’s an amazing watch,based on design and functionality, but its have a lack of accuracy in heavy woods, and for every time you stop it, fx when you’re doing intervals, and then continues you’re run, the watch “loose” 30-50 meters, because it needs time to locate satelitets after youre break – That’s bad!. So during a long run with intervals, such as 5x1000m, its a lot of meters you’re losing,and by that important data to your speed. Thats really anoying!
But your test have been really usefull and by the results, my choice of the Garmin forerunner 210 has just been increased! :-)
Viktor
Over-reporting the distance when you go slow makes total sense. The distance is measured by polling the position regularly, say every second, then add up the movements.
But GPS ain’t totally accurate, thus each position may be off by a meter or two. Being off by a meter to the side if the real movement was 10 meter straight ahead doesn’t matter much. if you do the pythagoras, you find it’ll result in reporting a movement of 10.05 meters when 10 is correct.
However, when movement is slow, the same error is devastating: if you really moved 3 meters ahead, but the GPS plots you 1 meter left of where you really went, then it’ll report you moved 3.16 meters when 3 was correct – percentually the error is much bigger, even though the position is still only one meter inaccurate, like before.
I’m not sure if someone commented on this earlier, but a 400 meter track is not 4 laps/mile, i.e. 800m does not equal 0.5 mi.
HTH.
Great GREAT comparison and thanks for doing this! I am trying to figure out if running with RunKeeper on my iPhone 4s or Garmin 305 is a better GPS.
Based on your analysis it seems – it’s not that big of a deal and turning wifi off on the iPhone is not that tedious a task prior to the run.
There are some BLACK SPOTS that my iPhone seems to hit and there are Satellite issues at times with the Garmin.
I guess it all depends on where you are and what time of signal you’ll get from time to time.
That being said – that was a great read and analysis for ‘us’ run and gadget geeks!
Hi – thought I’d share my gps accuracy experience with a Garmin 310XT and my old Garmin 305. The 310XT didn’t appear to be able to get an accurate GPS lock until it had been switched on for 5-15 minutes (it varied a lot). This initial inaccuracy varied between 30-150m. This was so bad that I contacted Garmin and over a period of 4-5 months I had 3 replacements. All 4 (original and 3 replacements) had exactly the same behaviour. In the end I sent it back to Amazon for a refund. My old Garmin 305 (which was broken but since repaired) always works correctly. My experience over thousands of runs is that you can switch the 305 on, wait until it locks and then press Start and at the end of a run, the run distance and track on the map will be very accurate and consistent between runs. In my experience (and from reading a lot of forum comments) it seems to me that no Garmin device since the 305 perform can perform that task with any reliability.
Very interesting! Have you done track accuracy tests on any other models? I own the Forerunner 10 and it is between 89-91% accurate at my 400m track. I know the GPS is going to be inaccurate, but just curious if all the Forerunner 10 (and other budget GPS watches) have such a lower accuracy than the ones you tested here.
Hi, recently I have tried another iPhone app for GPS-based distance measurement – ‘Measuring Tape’, and it did quite well, but it only measures the air distance (from A to B), so this can only be used for informative measurement of a straight track, real estate, etc.
I forgot to mention, that the Measuring Tape app. was available here for testing: link to itunes.apple.com
As this test is already some years old is it possible to do an update on this?
They’re some new devices nowadays..
thanks
DC,
What firmware did you use for all the watches? I think that would be an important point. I know that the Garmin 610 released FW 2.8 that took care of some of the shortage issues. Have you re-run the tests periodically after pertinent firmware releases?
You’ll notice the very first line of both Part 1 and Part 2 talks to that… ;)
Hi Ray,
It would really be great to see an update with many of the newer watches. I can imagine this is a complicated, time consuming test to “run”, but the info would be very useful for many of your readers. If you have already done an update and my search has been unsuccessful, can you help me find it on your site.
Thanks!
Hi Ray,
digging out a very old post, as I am a bit stumped: my trusty old 910xt (5 years old) always shows too much distance on track workouts: on a 12 lap workout (continuous), which is 4.8k it usually shows 5.2. The 10% over are pretty consistent. I always thought it couldn’t be close on the track due to persistently wrong error correction on turns, until I saw the files of later watches (a friend has a 620).
Are you aware of any changes over time that happened in terms of firmware that would explain this, or is it just the chipset that is worse than that of newer watches?
Thanks
Nicolas