2011 Sport Device GPS Accuracy In Depth: Part II

(Note: If you’re just joining me in Part II, be sure to read Part I, which covers the first half of the tests. Also of note: A few weeks after the test was published Garmin released a new firmware update that fixes the FR610 issues, and I redid some of the tests here)

As we left off in Part I, the tests thus far had been largely easy.  By that I mean that the first test was a straight out and back, the second was a 800m track test, and the third was a loop with some simple terrain and trees. None of these though really tested GPS accuracy in a satellite challenged environment or situation.

In those tests we found that the majority of the devices did quite well, with most of them being within one-half of one percent compared to the rolling tape-measure accuracy.  As you remember, all the courses are measured using a rolling tape measure, and then each course is tested with three different methods: Walking, Running and Cycling.

But that’s where the easy stops, and the tough get going.  In Part II we’ll explore forests which can degrade signal, as well as well a bit of tunnel that shows how well the devices can track despite potential drops in signal.

Test 4: The Deep Tree Adventure

Perhaps the most talked about scenario for GPS sport devices is how they handle in the trees.  Trees make very natural obstructions to GPS signal, and depending on the type of forestry being discussed – can be rather tall and dense.  While I’ve yet to find a forest or jungle anywhere in the world where I couldn’t get a GPS signal successfully, I have – like many people – seen it have an impact on tracking.  The secondary reason for inconsistencies is that most forest trails aren’t straight and narrow affairs.  Instead, they tend to wander.  With trail switchbacks it’s rare to just be going on a single compass heading for an extended period of time.

Switchbacks in particular can also cause issues if the recording interval is set such that it doesn’t catch the turns being made.  For example, if a data point is only recorded every 5 seconds, and within that 5 seconds you complete an out and back switchback on a bike – it may appear you’ve only gone 10 feet, when in reality you’ve gone 50 feet.

There’s a pretty substantial wooded park not too far from my house, complete with some running trails that wander aimlessly through the forest.  So I headed on over and setup the test.

Below is an overview of the route, which was one-way along a single-track trail that did a lot of twisting and turning.  I then went back to the start and did it over again.

Unfortunately, park rules dictate that you’re not allowed to bike on these trails, so the tests are only walking and running…but, I ran pretty fast to make up for it. :)  Though, I believe once you see the test results, you’ll see that cycling probably wouldn’t have made things dramatically different…except perhaps made it worse.

Like always, I started with walking the rolling tape around the course, before running it.

The course ended up being 2,157 feet long, or .41 miles:

Also of note is that during my time in the forest I got a near equal number of mosquito bites as there are meters in each lap.  I was getting eaten alive!

Last year, we saw significant differences in data length – and that continued to this year as well. Check out the results:

You’ll notice how every single unit except the FR310XT and X8 while walking came up short (these two unit went long while walking).  This tends to be one of the key issues with using GPS for short and quick switchbacks, like those present on my route.  The units tend to miss those switchback data points and in turn present a shorter and more direct track/route than you actually ran.

Despite common misconceptions, it’s not typically GPS reception that’s the issue here in forests – but rather tracking of rapid changes in direction that are short lived and non-constant.  Meaning, had you I just made a 90* turn and kept the course – everything would have been fine.  But because I turned 90*, then back 90* a few meters later, it misses that section where I jetted out a bit – and thus gave me a short distance.

This is one reason why many trail runners still use foot pods in lieu of GPS for speed, or in the case of Garmin and Polar units – choose to select footpod within the unit’s settings for speed but keep the GPS for the map plotting and elevation.

Test 5: The Bridge Underpass Test

This last test I wanted to perform was a bridge test.  Many of us have seen that bridges and tunnels can cause our GPS reception to drop out, must frustrating during races.  But the loss of reception in and of itself is completely normal – after all, one can’t fully except the devices to track well under many feet of thick concrete and steel.  It’s the recovery that’s primarily of concern.  So I hit up a newly completed running underpass just down the street, which passes under a four lane road.

Here’s a view of what it looks like from above.  Since it was just completed a few weeks ago, the satellite image is a bit older – hence all the construction going on.

(This little path just ‘opened’ two weekends ago, and by ‘opened’ I mean someone – not me – took down the banner closing it off.  Nonetheless it was flooded with runners/cyclists while I was there testing.)

The goal here was really to try and cause one of the GPS’s to lose satellite lock and then falsely pick it up with lower accuracy results, to see how it handles such a situation.  See, when a GPS unit loses satellite reception it goes back through an acquisition phase.  It’s during this acquisition phase that the level of accuracy is less, in other words – instead of being +/- 10ft, it’s now +/- 200ft.  Over the course of a few seconds (or longer), it then normally work it’s way back down to acceptable levels.  But sometimes, when it’s still acquiring satellites it’ll drop incorrect data points – perhaps hundreds of feet or even meters away.  In fact, for those of you that have tried swimming with your Forerunner watch (and not using the FR310XT openwater swim mode), what occurs here is the rapid loss and gain of satellites as your arm goes below the water and associated inaccurate accuracy levels upon reception during the recovery phase of the stroke, thus plotting incorrect data points.  You can actually look at your satellite accuracy level on your device, and I walk through how to do this at the end of this post.

In the below image, you can see a very tame version of this, where as I run in both directions – satellite reception drops, and then picks up closer to the other side, just not quite in the right place.  A few dozen feet later it gets back on track.

The above data map was from the Edge 800, while running.   But it gets more interesting if you look at the three different tracks – walk/run/bike – you can clearly see how speed affects map accuracy.

But you want to know the real kicker?  Despite showing vastly different tracks – the actual distances weren’t terribly different.  So while the unit showed .21 for both the cycling and running tests, it showed .20 for the walking test.  Yup, you read that right – the one test where it showed the least direct track was actually the shortest.

So I dug a bit deeper – what about the exact track lengths – in meters?

As you can see, the difference between the unit showing .20 and .21 in this case was only 5 meters.  Yup, that’s it.  And the reason why it was actually shorter is because while it looks wonkier, it actually cuts the corner a bit.

Now, let’s look at how the units actually performed…cause that’s what really mattered.  The wheel itself measured 1,111.4 ft round-trip – or 338.8 meters. I’ll note above that in the above running test, the Edge 800 managed to get within 6”.

Here’s the full test results:

All in, there were no vast differences between reality and the data the units recorded – except for the FR610.  In fact, if you look at the results, you’ll see that all of them except the FR610 were within .01.

For fun, I went and dug into those data files and found that for the FR610 on these tests they were 305m, 307m, and 313m meters respectively – or about 25-30 meters off from the stated 338m.

The key takeaway here isn’t so much that GPS can recover from dropouts – but rather that speed absolutely affects how long the dropout will be, as well as track position.

Summary of Results:

All in, there were over a hundred tests files created, covering the most common scenarios.  But with so many different scenarios and data chunks, I figured it’d be more useful to consolidate them down into a single chart, showing both average and median values, as well as how each unit performed in summary.  Remember from high school math that median values work by simply eliminating the top and bottom values to find the middle ones – which is why you can have a sub-100% average and still end up with a 100% median.  I showed both metrics as a way to remove cases where a device had a single bad test (i.e. RunKeeper’s 1.15mi in the 800m test).

(Click the chart to expand)

As you can see, the majority of watches were within 3% on median – the only outlier being the FR610 (and if using the averages instead, the x8 and Nike+ would also fall outside).  Consistently the FR610 came in about 2% short – which is definitely interesting.  While one could note that the shorter distance results skew the percentages (and that would be true), the 1-mile test and the half-mile loop still hold being off 2%.  What’s interesting though is that it’s consistently 2%, meaning that no matter the distance its short by that same 2%, as opposed to being 2% at half a mile, and 4% at a mile.  The test that brought down the FR610 considerably in the averages was the tree test – which, to be honest, brought down everyone – hence why I also showed median values.

That said, I wouldn’t consider that a reason alone to avoid the FR610, though it’ll be interesting to see if some of the rumored firmware updates for satellite acquisition issues fix that.  I’m waiting on an outstanding answer from Garmin on that one.

Looking at the remaining of the results though – you can see that like last year, in general the watches were within the typically advertised 2.5% accuracy rates – thus continuing to cement the validity of using GPS for most training and racing scenarios in sports today.

GPS Signal Tips, Tricks and Notes:

(Last year I added these and they helped quite a few, so I copied them over again as a good way to close out the post)

There are ways you can either directly or indirectly improve your devices GPS accuracy, and most of them are quite simple and easy.

1) Wait for the signal to get fully acquired

When you first power on your GPS device, it’ll typically go through a satellite location process.  This is simply the device finding its best satellite buds and getting to know them.  On older devices (pre-Forerunner 305), this used to take a heck of along time.  On the FR305, this was improved to usually less than a minute.  On the newest class of devices with the Hotfix technology, this is typically on the order of 10-20 seconds assuming you haven’t dramatically changed locations.

With the newer devices (FR310XT/110/210, Edge 500/800, etc…), the devices essentially remember which satellites you last talked to, and aim to find those same satellites again.  By remembering where they were in the sky, they can quickly find them and then get you running in short order.

And if it’s too cold (or warm) outdoors to wait around, remember the window trick to satellite accuracy.

If however, you don’t wait for the satellites to get fully acquired, then your accuracy is less than stellar…which I’ll talk about in the next item.

2) Look at the current accuracy

Did you know that on your Garmin you can see the current satellite accuracy?  While the unit says “Acquiring Satellites” simply hit the up button to see the satellite page.  The page will look like the below:

This page tells you a few things.  First off, it lets you know which satellites its connected to, and the strength is of each of those satellites.  Those are black bars you see.

Next, it tells you the current accuracy of your unit.  Meaning – exactly how far off the mark could your location be?  Typically the Garmin units need to be within 40-50ft before it’ll consider it ‘locked’.  If it shows 300+ft for example, that means that it could identify you as being almost a football field away.  This would have a direct impact on the distance of your activity, as well as speed and other items.

You may remember my In Depth Post on the Openwater Swim Mode, where I talk a bit more to how accuracy acquisition affects results.

3) Not getting correct mileage?  Try a soft reset.

I get a few e-mails a month from folks who see wildly inaccuracy distances – especially compared to friends running with GPS units.  For example, I’ll hear of someone running a 10-mile run with a group, and their unit will read 8.5 miles (that’s way off!).  In virtually all these cases a soft reset will resolve the issue.  A soft reset is kinda like rebooting your computer.  It doesn’t delete any profile or history data (a hard reset would), it simply resets a few things and most importantly – resets the satellite history.  Once you reset this satellite memory, the problem almost always resolves itself.

4) Update the firmware

You may remember my older post on the importance of updating firmware.  While many firmware fixes are addressing bugs or adding new features, there are certainly numerous instances where satellite acquisition and accuracy is being updated via firmware updates.  For example, there was a firmware update a while back that fixed a few issues with satellite accuracy on the Forerunner 310XT in certain situations.

In the case of the Timex Global Trainer, the September 2010 firmware resolved virtually all of the major satellite acquisition issues that the first firmware had – so that’s a biggie for TGT users.  However, it’s generally true that the longer distance accuracy issues are still outstanding for the TGT (being 2% short) – despite the Nov 2011 firmware update.

Useful Related Posts

If you found this post interesting, then you’ll probably find some of these related posts right up your alley – all part of the larger How-To section:

– Racing the Line: Understanding how courses are measured
– Understanding GPS Sport Device Elevation
– Garmin Openwater Swim Mode (Talks to satellite acquisition & accuracy in water)
– Importance of updating GPS firmware
– Tip of the day- Avoid Doing The GPS Dance
– Sport Device GPS Accuracy In Depth- Part I (2010 Edition)
– Sport Device GPS Accuracy In Depth- Part II (2010 Edition)

FOUND THIS POST USEFUL? SUPPORT THE SITE!

Hopefully, you found this post useful. The website is really a labor of love, so please consider becoming a DC RAINMAKER Supporter. This gets you an ad-free experience, and access to our (mostly) bi-monthly behind-the-scenes video series of “Shed Talkin’”.

Support DCRainMaker - Shop on Amazon

Otherwise, perhaps consider using the below link if shopping on Amazon. As an Amazon Associate, I earn from qualifying purchases. It doesn’t cost you anything extra, but your purchases help support this website a lot. It could simply be buying toilet paper, or this pizza oven we use and love.