It’ll likely come as no surprise that top racers are once again* cheating on Zwift. After all, cheating and cycling are as intertwined as peanut butter and jelly. And while historically cycling has seen cheating take the form of doping (or, motors), more recently that’s shifting into the digital realm as cyclists move indoors and compete there. Especially this year where outdoor racing is few and far between.
(*And by ‘once again’, I mean ‘still always’)
The last time there was a public kerfuffle with Zwift and cheating it was at their first indoor championship – broadcast live to millions of households around the UK (and beyond). A big production set and the whole bit. Only to find out later that the winner, Cam Jeffers, obtained the faster winning bike by using a bot to accumulate fake mileage. While some defended the rider’s actions due to the semi-muddy circumstances around it, that’s a much harder stance to take this time around.
But this time around, as first reported by Caley Fretz on CyclingTips, Zwift has banned two racers from future races for a period of six months each. Though, the incidents happened a month apart.
More notable though in this case is that Zwift clearly sought to throw down the virtual gauntlet with data that overwhelmingly shows the two women in question were falsifying files. But in my mind – that almost escapes the bigger issue as you’ll see: Why on earth does Zwift even allow this sort of pathway anyway?
And why exactly are these decisions being announced today specifically, versus when they actually happened? No worries, I’ll (try) and explain both the technical and non-technical bits. Hold onto your saddle.
It’s About the Coverup, not the Crime:
There’s two events in question here – both of them were 12 years ago by 2020 standards. The first was on August 17th, where Shanni Berger was competing in the ‘Off the MAAP’ series, specifically Race #2. You can watch that full race here. She had provisionally finished 2nd in that race. According to Zwift’s Performance Verification Board, this was case #80 (yes, there were 80 cases by this point, more on that later in this post).
The second event in question is on September 17th, 2020. This was an event where Lizi Duncombe was racing the Zwift Racing League – Women’s Qualifier #2. She provisionally finished 4th. According to Zwift’s Performance Verification Board, this was case #109.
One of the things that’s important to understand is that for certain levels of Zwift racing, there’s a slew of requirements. These include allowances on what types of gear/trainers they’re allowed to use, and how they must verify the results. Some of the requirements are merely window dressing. For example, Zwift allows certain types of trainers that are known to be inaccurate while disallowing others. Zwift themselves don’t do any actual true certification of trainers’ accuracy (else, they’d ban more trainers). In any case, at a high level, the following core things are required (the actual list is much longer, in the full ruleset listed):
A) During the race, riders must also record a secondary power source to another device (a Garmin/Wahoo/etc…), in addition to the trainer. Meaning, they need a power meter and must record another file. [2.7.1]
B) Riders are to submit a pre-race weigh-in video per a set instructions [Appendix A: Pre-Race information]
C) Riders are to submit a video recording of riding a specific Zwift test course [Appendix A: Pre-Race information]
D) Riders need to submit the exact model, firmware, serial number, and calibration information of their trainer and power meter prior to the ride, as well as photos of that. [Appendix A: Pre-Race information]
E) Riders need to submit an outdoor ride file on a gradient of 5% or more from the the last 12 months, with 5s/1m/5m/20min efforts. [Appendix A: Pre-Race information]
F) Riders need to submit calibration and spin-down video evidence for both the smart trainer and power meter [Appendix A: Pre-Race information]
G) Riders need to do the hokey pokey, recorded on video. [Appendix Z: Squirrel Section]
However, it’s this first one we’re gonna focus on today – 2.7.1 – the dual-recording of the race files. Riders are required to submit that secondary recording file to Zwift shortly after the event. In the case of Lizi Duncombe, she had a file from her warm-up, but didn’t have anything beyond that. Zwift assumed this was a mistake and reached out to the rider:
“The Board judged that this may have just been an inadvertent mistake by the rider, but nonetheless without an independent recording of the rider’s efforts it was not possible to verify their performance in the race and their result was therefore annulled. 2. Shortly after the rider submitted their first dual-recording file to ZADA, they uploaded a different dual recording FIT file to ZwiftPower. This second new file showed a dual-recording of the race labelled as coming from the rider’s Garmin and of their Assioma pedal based power meter.”
However, Zwift’s performance board then went on to crucify the file she sent in, noting (quoted sections are their words):
1) The rider had claimed the file was from an Edge 820, however clearly wasn’t, according to Zwift: “contained a “Version ID” label of “562”. This number is given to FIT files produced by Zwift. FIT files produced by a Garmin Edge 820 use a value of “1250”, and examination of multiple other dual-recording files from the rider from different races show they all contain values of “1250”.
2) Next, it noted that the file’s device timestamp (different than the date of the workout) was inconsistent: “It contained a “Device Timestamp” label of “6th Jan 2015 14:08”. This value is given to FIT files produced by Zwift. FIT files produced by a Garmin Edge 820 use a value of the time of the recording, and examination of multiple other dual-recording files from the rider from different races show they all contain values of the current time of the recording.”
3) The file had GPS coordinates in it. It shouldn’t have: “It contained GPS co-ordinates for the ride. This would be expected for a Zwift generated FIT file, but a Garmin recording a pedal based power meter would have no knowledge of the location in Zwift of the rider’s virtual avatar and would therefore be impossible for it to produce.”
4) The file was too perfect: “The power values contained in the file were all different from the Zwift recorded FIT file by the same percentage. For example, at every point in time when the Zwift recording recorded a value of 297 Watts, the second FIT file sent by the rider showed exactly 294 Watts. The correlation between the power recorded by Zwift from the trainer, and the power recorded by the Garmin from the pedals was mathematically perfect (i.e. Correlation Co-efficient: R=1.00). In any real-life scenario, for the trainer and power meter to be perfectly correlated across every single data point from rest to maximal sprint power, is impossible.”
So in a nutshell, she or someone on her behalf, tried to edit the original Zwift .FIT file, tweak the values by 1% across the board, and then call it a different power meter. Except, they didn’t understand enough about the nuances of this to make this even remotely passable. Thus, she was voted off the island Watopia for 6 months.
Next, we’ve got Shanni Berger’s violation. Her downfall was also falsification of files, but not actually in the same way. In her case, the event she participated in required that the trainer be a power source, not a power meter. While she had a Saris controllable trainer paired, she instead used a Stages power meter as the power source. This was likely just an innocent mistake. I’ve done the same thing too, not realizing it.
You can see in the below screenshot from Zwift that the trainer was paired as a controllable trainer, then the power source listed as the Stages power meter for both power and cadence source. This is from the files sent quietly behind the scenes to Zwift.
Having two different sources is something that likely thousands of Zwifters do every day without even thinking about it. And frankly, depending on the type of trainer and exact Stages model involved here – her decision would probably have been more accurate. For example, if she had a wheel-on Stages M2 trainer, then I’d defer to the accuracy of the Stages power meter (inversely, if she was on a Saris H3, I’d trust that over a Stages left-only unit). We don’t know the exact model of the trainer or power meters, it wasn’t specified.
However, that’s also not what matters here. The rules for that race stated a trainer had to be used as the power source (don’t worry, more on this later).
So, Zwift DQ’d her after the fact, based simply on that. This happens daily. It’s not a big deal, it’s not even the equivalent of getting a parking ticket. However, what happened after that though is the big deal.
According to Zwift and the full write-up about the incident, the company:
“received multiple emails from the rider’s Team Manager and team owner, protesting the annulment of the result. Their stated grounds for this were:
a) That the rider had been “adamant” that they had their trainer paired as the primary power source.
b) That the rider had provided the team with the Zwift log.txt file recorded on their own computer from the event which included the following section identifying the pairing used by the game (emphasis added):
Here’s the data snippet they included in those logs they sent, note the highlighted orange part. Also – I appreciate that Zwift chose to highlight the files in company orange, rather than any of the millions of colors available. Never in my IT career have I seen someone use orange as a text highlight choice. Kudos on keeping the brand on-point:
At the juncture above, they (her, her team, whomever) – had already screwed up. See, a power source can’t be paired in Zwift at the same time on both ANT and BLE. It’s physically not possible. And that’s where Zwift starts to throw down their technical gauntlet again:
“Expert evidence received from the Zwift software development team shows that the Zwift game does not produce log files in this format. Paired devices are either listed as being connected via “[ANT]” or “[BLE]” but never both.”
Then they go onto iterate one by one through a litany of issues with the modified log file (albeit, seemingly unable to know that power meter is two words, not one):
“The sections of the riders log.txt file that correspond to the game detecting the riders powermeter are all missing from the file. Whilst this might be possible if the rider had no powermeter, the rider freely admitted that they were using a power meter as a cadence sensor (and the riders log.txt file also confirms the use of the powermeter as a cadence sensor during the event). If the rider, and the riders file, are therefore to be believed as accurate, then the game would have to be using the powermeter as a cadence sensor, without ever having detected its existence.”
And again, where line items in the log file reference a difference between the two sources was left in:
“In total there are 27 lines in the riders log.txt file that differ from the original version – all of them relate to the powermeter; either lines that are completely missing in the riders version, or differ by referring to the primary power source as being the trainer rather than the power meter. However, not all of the lines referencing what the primary power source is are different. In the riders log.txt file there are still some lines that state that the primary power source is the riders powermeter. If the rider, and the riders file, are therefore to be believed as accurate, the game would have to be rapidly and repeatedly swapping between using the trainer and powermeter as the primary power source (and, as point (4b) above, without having ever detected the existence of the powermeter).”
Wait, there’s more. Here where they explain that while the rider/team changed all the references to the word ‘Stages’, in reality, Zwift actually uses ID’s behind the scenes, and these weren’t correctly swapped out.
“As noted in point 4c above, the lines in the riders log.txt file that differ all relate to the riders powermeter. Explicitly though, they all refer to the name of the power meter (i.e. lines that use the word “Stages”). However, the log.txt file also logs out internal game-specific numeric codes for all the equipment being used as well as human readable words. Notably, these game-specific numeric codes for the equipment being used that are in the two files are identical between the riders version of the log.txt file and the original version. This clearly indicates that the game is using the riders powermeter as the primary power source, and if the rider is to be believed, the game must have, for some unknown reason, chosen to change just the human readable names it gives to the equipment.”
Then Zwift tried to see if the rider was being aided (or tricked) by their team, and asked for the original files directly from the rider, rather than the team that had sent them:
“As noted above, the Log.txt file containing the discrepancies outlined above was passed to by Zwift by the riders team, and it was unclear whether these changes were present on the file stored on the riders computer, or had been made afterwards by their team. To help establish this, the rider was requested to send the files stored on their computer directly to Zwift. The files they sent were identical to those sent to Zwift by the team (i.e. contained the different primary power source, and incorrectly formatted data).”
So in other words, the rider didn’t throw the team under the bus, and just passed along the fake data file again (from the team), as opposed to her actual original data file.
Eventually, the rider admitted something:
“After numerous exchanges, Zwift received an email from the rider expressing their “apologies for any misunderstanding” and that it’s “very possible that I [the rider] made mistakes with the software due to human error”.
It’s not clear from that whether she’s admitting to simply selecting the wrong power source (a relative non-issue and frequent thing), or admitting to faking the data with her team (very much an issue).
Ultimately though, in both cases – had the riders simply accepted the DQ’s due to pressing the wrong buttons in the software, or forgetting to start their Garmin’s, this never would have been an issue. Neither Zwift nor anyone else is saying these riders cheated during the race. But rather, that they willfully “fabricated” and modified the data after the fact to avoid a DQ. As is often the case in mistakes, the coverup is far bigger than the crime (or, not even a crime at all originally).
Wait, why now?
So my first question about reading this was more simplistic: Why was this being just announced now?
After all, these events occurred exactly three months and two months ago, with the final Verification Board Decision occurring within 3-4 weeks – way back in September in the first case.
Well, the definition of ‘now’ is probably important here. After all, nobody knew about this in public until Caley’s article. According to Zwift though, Caley was in turn tipped off to the details by someone in the Zwift community (probably another racer). But also according to Zwift, the details of these have apparently been published for a while.
I can’t seem to find the exact date yet they appeared, but it was sometime this November. Zwift says that they were there sitting on an esports rules page that’s virtually impossible to find unless you know the secret handshake. But that almost immediately brought me to the next question: As of September there were 109 cases, what happened in the other 107 cases (plus how many more between now and then)? Why were these two riders being made a clear scapegoat?
Zwift said that basically those Performance Verification Board decisions cover a broad range of things, including cases where trainers or power meters are simply inaccurate enough to invalidate a rider’s race. Plus, in some cases the person was found totally innocent of any issue at all (such as being able to prove their data was correct). So ok…that removes a pile of them. But then what about people truly guilty? After all, I have a hard time believing that after 109 cases only two people are guilty.
Well, the company says that they’ve been internally trying to find the right line there for what to publish or not publish. At one extreme you’d publish everything, guilty or innocent of purposeful cheating. That’d be for everyone – random dad in their garage doing a community race at 5AM on a Tuesday, or WorldTour Team pro cyclist doing the Virtual Tour de France ride. All the same in that scenario.
While at the other end you’ve got only publishing guilty-type verdicts for pro racers on pro events of ‘value’, where a prize is being handed out, and where the riders are already held to a higher standard.
And that’s ultimately what Zwift has settled on. At present they aren’t publishing details of regular users doing bad things. That fart just goes away quietly into the night. It’s only the pro riders during races “of value” that will get nailed publicly.
(Fun screenshot tidbit for those eagle-eyed Zwifters and readers, despite the actual races where these riders were banned taking place a month apart in different race series, the two riders actually competed head to head and even in the same frame in the initial Aug 17th Off the MAAP race where Shanni Berger was DQ’d. You can see this in the video here, or just the screenshot above. It wouldn’t be for another month until Lizi Duncombe ran afoul of the rules and got her DQ.)
Zwift Needs to Improve:
It’s easy to blame these two riders. And yes – they deserve 100% of the blame they get here. But it’s important to point out that both of these bans were more for the coverup than the actual crime. In both cases, the riders would have simply been DQ’d due to the technicality of not having a secondary recording file for that event. There’s a million reasons that can occur – none of them nefarious (what was nefarious here was poorly creating a fake file to cover up the fact, attempting to avoid a DQ).
To put it in perspective, as one whose job is to create secondary recordings of power meter data – just last night I went for a ride, and two out of the four units I had recording failed in some way. One was a watch that wouldn’t start recording due to a bug (not my fault), and the other was a head unit I accidentally pressed pause on mid-ride during a swipe and didn’t realize it till later – basically invalidating that recording (totally my fault). Obviously, I wasn’t racing, but my point is that what these two riders did during the race wasn’t really a big deal in the grand scheme of things, it was effectively just missing paperwork. It’s what they did afterwards that got them in trouble (faking the paperwork and lying about it).
However, this ultimately brings up the most obvious of questions: Why on earth do athletes need to go through this trouble to have a secondary power source recording?
Seriously, why? So, I asked.
Zwift says that “it’s something that is challenging to implement but could be done and has been discussed. However, as mentioned, it boils down to prioritization at this time, and while dual power recording in-game would be valuable to one cohort, it would stop us developing another area which we currently feel would be of value to a greater number of Zwifters.”
Which basically sounds like every other response Zwift gives on why bugs or long outstanding issues aren’t resolved yet.
Zwift already pairs to multiple sensors as part of the pairing menu – why on earth doesn’t it just record those dual sensors and then upload them to Zwift? You can see the pairing data below, using an example of having a trainer connected and a different power source (notice the controllable trainer is listed as the Tacx NEO 2T, while the power source is listed as the Vector 3 pedals – V3 BLE):
While I try not to trivialize development, given the amount of logging Zwift already does here as evidenced by their own details in the above cases, it would be a trivial development effort to simply write/transmit this extra power data behind the scenes for verification. This is not hard stuff – so much so that every trainer and power meter company in the space has tools that do this sort of dual-recording for their own internal development purposes. But in this case – Zwift even has the darn menus already built for it. And they do about 90% of the work in the log files today already.
Zwift could add a bit more logic to validate there are two distinct sources based on manuf ID’s/GUID’s, plus ANT+/BLE ID’s/GUID’s. And they could require the racer have both connected and live before they pass under the starting banner. Without it, the racer simply doesn’t get to race.
Plus, it’s not as if we’re putting all the eggs in one basket, since Zwift already states that if Zwift’s network connection disconnects, per the rules the athlete gets DQ’d anyway (after 1 minute of lack of data connectivity), at all. In fact, this de-baskets the eggs. It means there’s two power sources that could be used as a safety net for the rider/race for any short or transient trainer/power meter connectivity issues.
Now like most questions around Zwift’s aging legacy infrastructure, a fix is always around the corner. In this case they additionally noted that with Zwift’s recent investment round raising $450M, that some of that money was going to revamping older infrastructure issues. And in particular, making these sorts of scenarios cleaner/easier for end users. But ultimately, there’s no clear answer on why this is as complicated as it is.
And even more than that, this would have tremendous benefits to the larger Zwift community of racers, allowing Zwift (and to an extension ZwiftPower) to validate results more accurately and instantly without the added uplift of dealing with secondary files. While also potentially minimizing transient connectivity issues. Seems like a no-brainer.
Summary:
Ultimately though, while these two women got themselves in trouble for the coverup more than the crime, I think it also uncovers the complexities that Zwift racing faces. And while there are many ways people can cheat on Zwift, the core of it roughly comes down to three simple things:
A) Is the racer’s weight accurate?
B) Is the rider who they say they are?
C) Is the racer’s power accurate?
The first one has largely been solved via video weigh-ins for pro riders (though is a dumpster fire for non-pro riders). The second one is solved in pro races via webcams and Zoom already. Whereas the third one could be easily mostly mitigated with real-time dual-power source verification. There’s no reason a rider should even be allowed to pass the start line unless the trainer and power meter are both paired, active, and transmitting different as expected. And there’s a massive number of things Zwift can do around technically validating that (as they’ve shown they’re already doing after the fact).
Of course – it’d also be easier to simply have accurate trainers or power meters. Today Zwift largely forces their pros to use the trainer power as the source for power data in a race instead of a power meter, a somewhat bizarre stance since overwhelmingly power meters made in the last 3-5 years are more accurate than trainers made in the last 3-5 years.
Still, that aside, Zwift says they’re working with the UCI to define ways to have an accurate trainer. Though, based on my conversations with various parties that have been part of that committee – the various parties are a long, long, long way away from agreeing on an even approach, let alone how to implement it. After all, the reality of that decision will ultimately make official what myself and others do on a weekly basis: Show how inaccurate many of the trainers being used are. Which in turn, will make brands unhappy.
And if there’s one thing more intertwined in cycling than cheating…it’s sponsorship.
With that – thanks for reading!
FOUND THIS POST USEFUL? SUPPORT THE SITE!
Hopefully, you found this post useful. The website is really a labor of love, so please consider becoming a DC RAINMAKER Supporter. This gets you an ad-free experience, and access to our (mostly) bi-monthly behind-the-scenes video series of “Shed Talkin’”.
Support DCRainMaker - Shop on Amazon
Otherwise, perhaps consider using the below link if shopping on Amazon. As an Amazon Associate, I earn from qualifying purchases. It doesn’t cost you anything extra, but your purchases help support this website a lot. It could simply be buying toilet paper, or this pizza oven we use and love.
Now it becomes obvious why Zwift is all in developing their own hardware which will likely at some point be the ONLY certified device for pro racing with any monetary rewards.
My guess is their bike will record weight for the rider/event.
Next they’ll add finger print recognition which must be paired to a passport image on file. :)
Zwift wants to build its own hardware because selling tens of thousands of units to recreational riders is profitable. They aren’t specifically targeting the subset of competitive riders and the even smaller subset where data integrity is of any real consequence.
Exactly. They don’t need an extremely accurate one – they just need all competitive riders to use exactly the same hardware. Then even if it’s wrong, it’s wrong in the same way for everyone.
Certainly not, cheap hardware would have no reason to be wrong the same way across the whole range. Engineering measuring devices is much more complicated that “if it was out of the same production line it must be the same”. In fact making ocnsistent product is probably the single hardest thing to get right in production.
In IRL racing if I use deep profile aero wheels and another rider uses medium profile aero wheels, then it is not equal either. Frankly, no bike of different brands rides 100% the same. Some are stiffer than others, some lighter. Having identical trainers in a virtual race would be even more accurate. It will never be perfect though.
So basically you are required to keep your power meter on the bike you put on your trainer? Even though the trainer already measures your power output.
Instead of using the trainers measurements for Zwift and your power meter on your winter bike for outdoor training.
Yeah but only if you’re a Pro. It doesn’t seem too unreasonable to ask them to swap the pedals over (or invest in a set for the trainer).
I like your proposal about built in dual recording and I’ve had some similar thoughts. There would be an additional potential benefit. Anyone who races on Zwift knows the agony of experiencing a wireless drop out, having your avatar coast at zero watts and watching the race speed off leaving you stranded. In theory dual connection would permit detection of this event and support automatically reading from the secondary source until the primary power source reconnects.
But is this due to connectivity issues between your power source and computer or rather insufficient internet connection (which would lose both power sources at once)?
I’d guess that the vast majority of Zwift connectivity issues are local to the household, things like WiFi blocking ANT+/BLE sensors/etc…
I solved most of my dropout problems by using a Tacx Ant+ antennae under my bb, and doing a frequency scan on my WAP to look for conflicts. Since I did that, I’ve not had a problem. Bluetooth is another issue altogether. I hate BT.
Yup, Almost all of my Zwift dropout issues went away after I bought a $5 15-foot USB 2.0 extension cable to position the ANT+ dongle next to the trainer.
There’s really no reason why smart trainers can’t come with USB and/or Ethernet connectors in 2020.
Our trainer had hi speed USB connection for many years. Data transmitted to/from computer at the rate of 250 samples/s
If only Zwift would work with hardware partners to utilize those wired ports. Kinetic has had them for years as well, and now Wahoo launches without any real confirmation from Zwift.
There’s an additional way of cheating that is an extension of “C) Is the racers power accurate?” and is manipulating the power signal between the source and the Zwift App. With cheap hardware and a bit of knowledge of the involved protocols is quite easy to do. If you are not “greedy” and add a bit of randomness to the manipulation is very hard to detect. At least when you are not in public or tightly controlled enviroments where the device could be harder to hide.
Correct, there are a handful of solutions out there – but I’d say that the required hardware + know-how significantly elevates it well beyond what’s accessible here. Plus, as those individuals that created those solutions would tell you, it’d be relatively trivial for Zwift in conjunction with the trainer industry to put in place a few things to make that immensely more difficult to execute.
In fact, we saw a few of those minor things in the Sterzo Smart, which, while defeated by two super-smart dudes in the power meter/trainer industry, wasn’t really designed to be a blocker. It was designed to be a half-hearted measure to keep paperwork happy.
But the way I see it – there’s far easier fruit to pick right now than trying to protect against a MITM attack with a sophisticated geek.
“Plus, as those individuals that created those solutions would tell you, it’d be relatively trivial for Zwift in conjunction with the trainer industry to put in place a few things to make that immensely more difficult to execute.”
TBH, its not that trivial to protect against cheating. Cryptographically protecting the BLE traffic is only going to make it more evident that the real way to cheat is attack Zwift’s executable in memory (via direct memory editing, network level attacks, etc). Since Zwift runs on windows, all bets are off since this system does not offer protection for processes – one can run self-signed kernel level code with minimal effort. Its better on IOS and MacOS but also not completely safe.
Also, preventing BLE MITM is not really that easy either, since Zwift would need to establish trust with the BLE device by verifying it’s signatures against some known to be trusted public key. Stored somewhere. Sent over the network to the process that does the verification. And that’s assuming that companies like garmin and tacx, long known for being extremely good with software engineering, don’t screw the implementation up in some entertaining way.
I did a lot of reverse engineering and anti-cheating consultancy a few years ago, for popular fps titles with millions of players. The conclusion back then was, and I think it still stands, is that the only unbeatable method for detecting cheaters is through statistics. I think private hacks are impossible to eradicate as long as personal computers stay personal computers and allow arbitrary code execution.
Sorry for the offtopic.
I said: “place a few things to make that immensely more difficult to execute.””
You said: “Cryptographically protecting the BLE traffic is only going to make it more evident that the real way to cheat is attack Zwift’s executable in memory (via direct memory editing, network level attacks, etc). ”
That’s exactly what I mean. The shift from editing a text file to developing legit code and spending real dev hours on something is a dramatic leap.
I agree there’s plenty of other ways to address this problem, including in the analytics side (as Zwift is already doing). But that doesn’t stop someone from standing on the (virtual) podium in most cases.
Zwift would do well to think about it now. That dramatic leap you are mentioning is not a very good way to describe it — it’s an impossible one for a layman, but a trivial one for a pro. Once posted online – it would be available for all to use.
Sadly the very best and most common strategy software vendors employ is denial. There are no american tanks in Baghdad. Cheating is not widespread, look, we just banned a few. And ultimately, for the vast majority its ok. Most people don’t care much about racing, and the weight distribution in any Zwift race is proof enough of low moral standards of the racing population. Best way to use it is to race against oneself.
True, video weigh ins can be manipulated as well. Someone with a bit of engineering skill could also attach a small electric motor to their setup, effectively turning it into an eTrainer. The only way around this is to place all racers in the same room and let them audit each other.
Exactly. Virtual racing is a joke. And that’s fine, untill you start pretending it’s not. Please just look at gaming. Good aimbots are a hundred times more difficult to make than manipulating one or two data points (like power). And yet you can buy those easily for games with a smaller player base than Zwift. The only way to really combat cheating is also shown in e-sports: all players physically present at one venue with standardised and monitored hardware and software. Just bring your own mouse / saddle. At least saddles don’t have programmable macros.
Or you could do what you wrote and will probably happen; pretend and deny.
I totally agree with you on that one, making a cheat for Zwift is so easy …
Why did they cheat? Was there prize money or was it just ego?
Cheating is fun in itself, for some people. The feeling of beating the system and it’s defenses is, for them, as rewarding as the one you get for beating the other players.
What stops a cheater from using an ebike? An modest power boost could be all it takes to win.
A modest…
The secondary power source would highlight the use of a motor.
How can a secondary power source differentiate between power generated by the rider and power generated by a motor?
It can’t.
Thank god you corrected that. :)
If your trainer is registering 300w and PM pedals or crank mounted PM are registering 200w would that not suggest an ebike is a possibility?
Ignore that, re-read the conversation.
Pedal and crank based power meters would not consider the power from a motor.
Or a tandem with two riders on a direct drive trainer but only a single rider’s weight, perhaps with each rider having the L or R pedal power meter.
As a software engineer what I find interesting is how “amateur” these attempts were. Taking the first example, the first three items would be trivial to correct. The fourth is slightly more interesting but applying some moderate jitter to the adjustment would have hidden that.
This all reminds me a little of the cycling equivalent of the “deep fake” problem we see in video and audio. The issues above highlight obvious oversights by the cheaters, but they’re also easy to fix. Others have no doubt already cheated and not made those mistakes, and even more can look at those examples and figure out pretty quickly how to cheat more robustly.
The right way to manage this kind of cheating (modifying log files) would be for devices to cryptographically sign their output so that any tampering can be immediately identified. If this sport is to be taken seriously, I’d think that Zwift could work with the device manufacturers to implement that. Of course more sophisticated cheaters could still modify the power source or, heck, stick a motor in their bottom bracket.
So if I read rule F and perhaps E(if you need the same powermeter) it is not possible timide a StagesBike for Zwift racing?
They’re using the technology wrong. Period
If Zwift thinks someone is tampering with their logs they should sign them (hello blockchain), but if they want to use ANT+ or BLE as a “certified power source” they just don’t understand how this protocols work.
Zwift has built their e-sport platform on the foundations that power-meters (new spelling ray ;-) signals can’t be faked and that’s simply not true. They need to tackle this issue asap before they lose the “community” trust, because as long as there’s some gain involved (money or ego) cheaters gonna cheat.
Either Zwift holds the races in a controlled location with Zwift provided trainers or they develop their own hardware were they can guarantee the power data “chain of custody”.
Part of the problem is that to date Zwift has resisted efforts at every turn to work with the industry. They’ve never attended an ANT+ Symposium. They ignored previous efforts by trainer company groups to try and start tackling data security type issues. And to the best of my knowledge, they’re not an active member of the BLE SIG’s either.
The funny thing is – this sort of power data connectivity tampering/issues isn’t new. We went through this entire circle a number of years ago with the pro riders in the tour, and MITM type attack concerns.
And what everyone came back to then was ‘Oh, we should introduce encryption between the power meter and the head unit’, followed by ‘Oh…that’ll break stuff’.
So ANT+ went away, worked on encryption, and came back. And sure enough, if the power meter ANT+ TWG wanted to, they could implement it (just as it would for the FE-C groups). And sure, it would break things depending on how it was implemented, but honestly…it wouldn’t have to.
Realistically speaking, all these races are occurring with pro riders almost exclusively riding on four companies trainers: Tacx, Wahoo, Elite, Saris. Sure, there’s some cats and dogs leftover, but basically those four makeup 99.9% of it.
And I can guarantee you that if Zwift went to those companies and said ‘Look, we want to introduce encryption on trainer firmware for the last 2-3 years worth of mid to high end units’…it wouldn’t actually be a big deal. We’re talking 1-3 models per company at most, and most of them share much of the same code anyway. Yes, work has to be done. And yes, that requires effort and engineering. But it also requires ‘teamwork’. Zwift has to actually work with these companies, instead of working against them on this stuff. It can’t be solved in a vacuum.
Unless I am missing something obvious, for this to work the power sources/trainers would need to have their private keys embedded in secure enclaves, sign their traffic with those securely held private keys, and local zwift runtime would only pass the signed data to zwift hq where it would be safely verified against known-to-be-trusted public keys associated with those devices.
I guess that could work reliably, and local zwift runtime could not be effectively hacked, but the investment on part of the hardware manufactures would be non-trivial. After having observed first hand how cynical and carefree are game developers when it comes to preventing cheating in popular games — i doubt it would happen here, with significantly smaller market.
I think it’s easier than that. Well “easier” for some definition of easy.
Firmware contains the public key. Sign with the public key. Zwift validates the signature with the powermeter company. I mean heck, the easiest way is the powermeter .com just throws up a web page. Copy/paste the entire log file into the webpage, it validates the signature using its private key, and returns green or red. Obviously you would make this more sophisticated for production use.
Of course these companies will be hacked and someone will steal the private key. Then you just generate a new pair, issue a new firmware with the public key, and Zwift can require that all racers have the latest firmware. Since if they don’t, their results will be thrown out.
Signing with a public key is pointless, as a public key is.. public: a man in the middle can resign altered packets with the same key, so it’s no longer possible to verify the data hasn’t been tampered with.
>”So ANT+ went away, worked on encryption, and came back. And sure enough, if the power meter ANT+ TWG wanted to, they could implement it (just as it would for the FE-C groups). And sure, it would break things depending on how it was implemented, but honestly…it wouldn’t have to.”
And then someone opens power meter and adjusts couple of resistors to change the slope of strain gauge to electrical signal conversion
“And then someone opens power meter and adjusts couple of resistors to change the slope of strain gauge to electrical signal conversion”
Nobody’s ever going to solve that angle. Well, Zwift of course eventually will. They’ll do it by only allowing specific trainers (non-secondary power source) that are pre-calibrated (akin to Tacx NEO series).
Ultimately, the 10 Immutable Laws of Security* defines that if someone can physically dork with something, it’s eventually always going to be end-game.
But, I’m already seeing a super-common issue in IT security happening above: Perfection is the enemy of progress. There’s no such thing as perfection in IT security (which is what this is). There’s increasing the bar higher and higher so that the risk/reward/payout equation isn’t worth it anymore. Plus, eventually, once we get past this pandemic, the goal of these high profile events (where the reward might be worth it), was to have them in-person, not distributed. But again, don’t let perfection get in the way of progress.
*https://www.marshall.edu/it/departments/information-security/10-immutable-laws-of-security/
why not just cheat the old fashioned way by roiding up and doing some blood doping? Much harder to detect in esports.
Yes, how will the prevent this from happening? Make everyone take a video of themselves sending a blood sample?
Not enough prize money in Zwift racing to offset the cost of the doping programme (*yet*).
We’re reaching the point where it starts to matter though.
If there are Zwift-only pro’s (meaning that they don’t have a real world racing license and therefore need to keep their whereabouts current and be subject to random doping tests conducted by their local anti-doping agency) then Zwift and the xADA’s need to start thinking about an approach for that for sure.
Agree that it would be best to have a single accurate power source. Especially for those who do not ride outdoors- seems silly to require them to have a separate power meter AND recording device.
That said, are there any power meters or trainers that record activity internally (like some of the HR straps)?
Wow… I had no idea of the complexities of the system related to log files. As you said, the cover-up was worse than the crime, which speaks to the morals of the rider more than anything. My worries are more real world, such as when my HR monitor drops when I enter the race pen, which will DQ me in B cat races once I soon rate up. Not much I can do about that other than get a new HR monitor which is why I visited your site today, ha! Like many, I am frustrated by cheaters, especially weight cheaters and would love for Zwift and ZP to put as much effort into solving such issues for us amateurs as they do for the pros. But as many say… I really just racing for myself and to improve my fitness. That’s my mark of success.
lol hokey pokey
> Which basically sounds like every other response Zwift gives on why bugs or long outstanding issues aren’t resolved yet.
This! As a software developer I know the BS here, i have the feeling functional and unit test was unprioritized and rely to much on manual testing, so scared to introduce any new features….
This is just supposition but the burn rate of fixing/delivering features is so low.
I am working for a small company proving software servcie running at large scale and as a dev I would be really frustrated to work for a company delivering or fixing so little.
Why is Zwift storing everything in text files anyways? They could store it in some binary format, or better yet binary with some sort of encryption scheme so that it’s nearly impossible for riders to go in mess around with data.
Then again, why is stationary racing even a thing.
Not sure you should have picture of an unrelated triathlete (QS) who has nothing to do with the article?
Yesterday I did a one hour long group ride om Swift with a Kickr bike.
Zwift saying the ride was 31 km long, but my Fenix 6X (also connected to the Kickr bike) says the ride was only 29.7 km. How comes?
Elevation?
Well, it has a quite flat course, so I do not think elevation is the (full) aswer here.
link to zwiftinsider.com.
Thanks!
Off the subject but related – I’ve followed you Ray since you first started DC Rainmaker. You’ve developed it into a superb wealth of information. You have a unique talent at analyzing products and succinctly expressing your thoughts about them in an objective way.
The facts surrounding this event are a little over my head but you’ve explained them very clearly.
Kudos to you and your lovely wife for all you’ve done.
I wish you health, happiness and a long life……….
So now they basically told the World How to cheat in zwift?
Kind of makes no odds at amateur level. Because if you want to cheat you just reduce your weight. No one checks.
Re: the orange highlighted text:
Always be branding!
“Never in my IT career have I seen someone use orange as a text highlight choice.” Except, of course, DC Rainmaker ;)
This! As a software developer I know the BS here, i have the feeling functional and unit test was unprioritized and rely to much on manual testing, so scared to introduce any new features….
What’s missing here is how the known actions might have been used to cheat / coverup cheating. In the Duncombe case, an incorrect power meter FIT file was initially submitted. Then, a replacement power meter file was submitted that was red flagged for two reasons: 1) the device ID and the time stamp indicated that the source was a Zwift FIT and not generated by a Garmin Edge 820 (as claimed) and 2) the ‘power meter’ FIT was exactly 1% different at every measure as the Zwift FIT file. The most likely explanation, IMO, is that the original Zwift FIT file was simply scaled by 1% and submitted as the power meter file. Let’s assume that. What are the possible reasons? Perhaps there was no power meter file – forgot to record, Garmin battery died, etc. So, to avoid a DQ, a ‘power meter’ file was created from the Zwift FIT. In this case, it is plausible that the power generated was the power reported to Zwift (I.e. no cheating from a power manipulation POV). Another possibility would be that the power measured by the trainer was manipulated on the way to Zwift, so that the actual power produced was less than the power sent to Zwift. If that MITM (presumably) editing occurred on the Zwift channel, perhaps it was not replicated on the Garmin channel (assuming here that a Garmin Edge 820 recording actually exists). This might happen if the trainer is connected by BT and the Garmin by ANT+ (with the MITM only on BT), for example. Thus, the Garmin recording would reflect the actual, lower power produced – submitting that power meter file would ensure the DQ. If the athlete had the capability to edit the Zwift FIT file by 1%, then they would, presumably, have the capability to edit the Garmin file by some percentage that would make it similar enough to the Zwift FIT file inflated by the MITM modification.
I’m sure that there are more possible explanations, but my bet is on case #1 – no power meter recording, about to face a DQ, take the only FIT file available from that ride (the Zwift FIT), make a 1% edit and try to pass it off as the Garmin file. The athlete (or file editor, which might not be the athlete) was not aware that the FIT file has a unique device ID and got tripped up by that (and the FIT file date and the perfect correlation of the two FIT files, offset by exactly 1%).
Now, maybe there are other file characteristics that are unique to Zwift FIT files that are undisclosed in this case. But in the absence of that, it seems clear that a Zwift FIT could be passed off as a power meter FIT by carrying out editing of the device ID and date consistent with the situation and then editing to power data by, say 1% with 1% random jitter (or similar). Even better, just do any old Zwift ride with the Garmin, take the two FIT files and calculate a vector of power differences as a function of time. Now do your Zwift race, grab the Zwift FIT, process the trainer power with those power offset values and correct the device ID and date. Now you have a FIT file derived from a Zwift FIT that ‘looks like’ it could have come from a Garmin. That’s a lot of work, and to what end? In that situation the power generated is the power sent to Zwift (I.e. no power ‘cheating’). So, to make this approach deliver higher power values to Zwift than measured at the trainer, you need a MITM that has a known value (let’s say inflating the trainer power by 5%) and then reduce the Zwift FIT by that same amount before applying the ‘Zwift-to-Garmin’ offset plus jitter (or similar).
This would all require a semi-automated approach (feasible) if Zwift requires the power meter file shortly after the event. If Zwift allows power meter file submission a day or two later, then there is sufficient time for a knowledgeable person to carry out all sorts of mischief.
I don’t have any relationship with the athlete in question, or Zwift, or the racing series involved. I’m not justifying the actions of the athlete or the response from Zwift. I’m just trying to figure out the likelihood of power manipulation consistent with the situation reported by Zwift and what could be learned from this case to enable undetectable power cheating by future Zwift athletes.
“E) Riders need to submit an outdoor ride file on a gradient of 5% or more from the the last 12 months, with 5s/1m/5m/20min efforts. [Appendix A: Pre-Race information]”
Seriously?? What if you live in a location where such a climb does not exist?
There is not a single climb within two hours of me that would take 20 minutes. I am not sure even two hours would get me to one.
Duncombe issues statement:
link to velonews.com
I’ve read the statement a few times and the way I read it she’s just making it worse. I don’t believe for a moment she didn’t know what the friend was doing given they’d produced full race data when it was likely only partial data was available but even if she didn’t, clearly the blame is still with the ‘friend’ for doing the tampering. However she repeatedly blames Zwift for their attitude and how they’ve handled it when she was the one that admits she submitted a tampered file
I’m sure all her other races are genuine but the moment you try to cheat the system and are caught it shows you can’t be trusted.
I would agree.
Her battery on her Garmin died?? When was the last time that happened? The same day as when the dog ate my homework.
I assume in this case, she’s talking about the battery dying (running out of battery). Pretty sure that happens to all of us here and there.
It’s what she did after that point that got her in trouble.
Great piece Ray. Very interesting and informative.
Sounds like a lot of technical challenges to complete in a virtual sport. I don’t see why a dual recoding is required for units that are verified to be accurate. I can forsee the pros slowly dropping out of participating if they keep getting dq’d due to technical issues.
If I was a pro I wouldn’t bother because one single drop and boom, your race is done.
The purpose of dual-recording is to eliminate doubt in odd-ball scenarios. It’s also a acknowledgement that generally speaking every trainer on the market except the Tacx NEO series can be faked/tweaked with. The new KICKR V5/2020 is a bit better in how they overwrite the calibration values, but that has its own current accuracy issues.
The KICKR Bike though is also an exception, but we’ll leave that off to the side for now.
I have a NeoBike, factory calibrated, and there is no way in hell I would pay another £400 to validate the power of a £2500 piece of hardware.
Zwift should agree a certification standard for trainer accuracy. Manufacturers would have to follow a prescribed testing criteria to certify the unit, and depending on accuracy it could get a grade. For example, the NeoBike and 2T could be grade A.
Zwift can then mandate that top races need to be on A standard trainers, or event organizers could say B standard or above (for example).
Dual recording would never highlight eBike usage, because no-one using an eBike would dual record unless they were also manipulating that data. Dual recording is a waste of time and money.
The ‘Wait, Why Now?’ Image:
There are now Zwift/Trainer Racing specific jerseys/skinsuits?!
“as one whose job is to create secondary recordings of power meter data”… That sounds a little sad.
However this was a hilarious piece and it had me laughing out loud a couple of times. I love the way you throw some dead-pan comedy into your supreme technical insight.
Advantages of using the trainer power: it increases consistency (as opposed to mixing data from one-legged Stages with Garmin pedals with SRM with Quarq with whatever). It reduces data lag (versus cranks which can only measure power each pedal stroke, which can be < 1/second). And it guaranteed drive train losses are counted for everyone (unless riders are on a WattBike or similar, which might not use a drivetrain).
All true. The only challenge is that the disadvantages are big: Almost never as accurate, often susceptible to sprint overages due to flywheel carry-through.
Very interesting articles
Real lies repeataedly. Must show conviction when doing so.
link to strava.com