I’ve never been good at writing titles. They’re all too factual, and usually too boring. The titles I considered writing here were:
1) UCI Esports World Championship Qualifier Banned after Spicy Real-Time Manipulation of Data Stream
2) Zwift Race Cheating Gets Even More Covertly Brazen During UCI World Championship Qualifier
3) Previously Outlined Zwift Cheating Hack Actually Implemented in UCI eSports World Championships
All would work, and there are many more potential good ones! However, I don’t think any of those titles really capture just how ballsy this particular attack is, and more importantly, how big of a deal it is going forward to UCI’s Esports World Championship series. As for the titles, feel free to add your own in the comments section. The winner gets nothing.
So why am I writing about this? Well, I’ve long found these Zwift cheating bans interesting. To be clear, Zwift is only doing cheating bans on essentially pro-level races. Races where you’ve agreed to a set of terms, agreed to certain verification standards, etc… These aren’t being done on your run-of-the-mill DIRT events.
However, what fascinates me about them is just how increasingly technically brazen riders cheaters are getting with these. Mind you, these are only the ones where people have been caught and publicly flogged. In this case, arguably it’s the very brazenness of not just the tech, but the race finish usage of it, that outed him.
An Unbelievable Mountaintop Finish:
Back on November 13th, 2022, Zwift had a continental qualifier event for the upcoming UCI Cycling Esports World Championships. This particular race was for Europe & Africa, though of course, the World Championships cover, ya know, the world. That doesn’t occur till mid-February, when everyone is solidly sick of trainers. That race even had a livestream of it, whereby 5,978 people watched the event unfold. This qualifier included 124 starters, with 50 of those advancing to the next race. Additionally, the winner got an automatic ticket to the world championships. Most notably, the livestream even had webcams from some riders – including the individual in question.
The 27.2km race of Roule Ma Poule had been progressing normally, but there was one final ~100m ascent for the last couple kilometers, with a hill-top finish. Two riders had broken away from the pack, while the main pack, including our man in question Eddy Hoole, were back quite a ways.
It’s at that moment, right before the offending action occurred, one of the two announcers, Dave Towle, says (at 34:38), and I quote:
“I don’t know all 7 of the deadly sins, but I know two of them are greed and sloth. And certainly sloth is not the issue out here, but greed might be.”
Mere moments later at 37:28, rider Eddy Hoole starts a breakaway from the pack at the base of the final climb of the race – a daunting multi-minute climb to the finish. In doing so he’s holding 8 w/kg up this hill for some 4 minutes in duration. At times breaking over 10 w/kg, in the middle of this extended climb. You can listen to the announcers astounded as Eddy inexplicably closes the gap from the peloton to the two leaders, passing them as Nathan Guerra says he “comes flying back like they’re standing still”, ultimately taking the win and qualification spot:
Just after crossing the line (41:23), announcer Nathan Guerra says:
“He took on with an amazing effort, something we’ve almost never seen before”…“that is one of the best efforts, I literally have ever seen for a catch on Zwift, to go flying right on by, Eddy Hoole just did what I thought was absolutely impossible”.
Which, would turn out to be true. It wasn’t possible, and is largely beyond known human performance levels.
And again, just three minutes later when they cut to a side-by-side of the announcers, announcer Nathan re-iterated the implausible nature of it, and even seemed to be contemplating it, and seemed less excited and a bit more like the gears were already turning in his head. Follow that another 2 minutes later and announcer Dave Towle actually mentions the verification process ensuring that this data is valid from the trainers and power meters.
The Datastream Attack:
Yesterday, December 7th, 2022, Zwift published their so-called “Performance Verification Decision” document to their site. This document is basically the final list of charges and included ban for the rider. This doesn’t get published immediately after the event, but rather, this is the culmination of that process. The process includes bilateral communication with the rider, teams, etc… It’s aimed at figuring out whether cheating was purposeful, or perhaps accidental (such as a misconfiguration/miscalibration of a trainer/power meter). And in fact, it goes through those very steps here.
I think in the past, some of Zwift’s initial performance verification decisions were on shaky grounds (while others were very solid), however, they’ve gotten better over the last two years, and it seems the only cases making it to public view (and thus bans) are the most damning cases. Obviously, Zwift wants to limit its legal liability here.
Here’s a chart of the entire race for the rider in question, Eddy Hoole:
The data sets are as follows:
• Terrain/Altitude – Grey
• Power – Green
• Estimated Energy Reserves – Yellow
• Heart Rate – Red
(In case you’re wondering, Estimated Energy Reserves is W’bal, which is a way of estimating potential wattage over time – roughly like MPA from Xert. Here’s a bit more detail on W’bal.)
The focus area here is on that final climb, basically where all that blue text is at the top, and where the power jumps up. It’s also the section I outlined earlier. The rider in question, according to Zwift, nailed the following wattages:
- Total effort = 4min 16sec @ 526 Watts average
- Best 4min average power = 526 Watts
Zwift says that given the rider’s weight, this equates to a sustained output of 8.5W/kg, which in turn would require a VO2Max of 90. Zwift goes on to note that the highest-known Tour de France or Olympic Pursuiters have a VO2Max of about 85. As usual, Zwift then gives the rider the opportunity to have an independent lab conduct a VO2Max test, which, the rider accepted. Here’s that excerpt:
As you can see, at this point you’re thinking ‘Oh, just a miscalibrated power meter’. Except, remember that Zwift requires dual-recording for these races from both a power meter and a certified trainer. Now, Zwift (still, seriously, 3-4 years later), doesn’t dual-process this data in real-time as has been begged for, for years. Instead, it’s post-processed when required for stuff like this. So, what’d that data look like? Well, in short Zwift says the two were basically the same:
Even more, Zwift threw a bit of a knife-to-the-heart in there by noting that the trainer the rider was using is ‘self-calibrating’, which is basically a recent Wahoo KICKR or TACX NEO series device. I can’t quite tell from the webcam angle what he’s using.
However, here’s where the spicy part finally comes in. Zwift noticed that after the rider joined the pen, there was a brief disconnection that occurred to Zwift’s servers. Interestingly, no other riders had this happen. Yet inversely, this rider had this happen in every race, but never any regular Zwift training rides. This particular data channel included analytics information about the sensor:
Note about Zwift says:
“Zwift considers the absence of this analytics information to be equivalent to the presence of a masking-agent in anti-doping – for example, it would allow the rider to change their paired device from their trainer to a computer-controlled device that gave falsified power information, without such a change being recorded by Zwift’s servers.”
In translation: The rider is inserting a device/software into the middle of the (basically open) data stream to dynamically change it, providing an offset (increased power), that gives the rider a boost.
When asked about this, the rider had no answer, but instead deleted 150 publicly visible dual-recordings from ZwiftPower (a website used for displaying these recordings post-race to prove your data). The rider has since deleted or made private all his social media accounts, including his Instagram account which listed him as a “Web Software Developer”.
Based on that, Zwift says they’re satisfied that the rider knowingly cheated, saying:
“The Performance Verification Board is comfortably satisfied that the power recorded by the trainer and used in-game did not match the actual power produced by the rider and/or was not the actual power measured by the trainer, and therefore that the rider’s performance in the event cannot be verified.
Further, the Board is comfortably satisfied that this was a result of deliberate manipulation of data, masked by the deliberate disconnection of the Zwift analytics datastream channel, rather than accidental miscalibration of two independent pieces of equipment by the same amount coupled with a coincidental accidental loss of analytics data.”
As a result, the rider received a 6-month ban, given it fell under a Tier 3 section – specifically, “Bringing the Sport into disrepute”. In case you’re wondering what else is in a Tier 3 ban, I asked Zwift:
Tier 3: Bringing the sport into disrepute
● Examples include, but are not limited to, the following:
– Fabrication or modification of any data
– Equipment modification or other external trainer control
– Use of bots / simulated riders
– Identity fraud
– Abuse of race officials
● Sanctions include, but are not limited to, the following:
– First violation: Six month ban from Zwift Cycling Esports events.
– Second violation: One year ban from Zwift Cycling Esports events.
– Third violation: Lifetime ban from Zwift Cycling Esports events.
Finally, Zwift ended the performance decision with the usual taunt, saying, ‘if you can prove it, we’ll drop it’, which is basically a CYA in case the rider claims their first test was on a bad day.
“If, within 1 month of the issuing of this decision, the rider can perform an independent laboratory test & antidoping test to the satisfaction of Zwift that shows that they are naturally physiologically capable of producing the results they have recorded in this event (including, but not limited to, an average power output of 8.5 W/kg for 4 mins), the Board will happily reverse its decision, reinstate the rider’s results, and additionally reimburse the rider for the cost of the tests.”
Concurrently, the team he was riding for has terminated their relationship with him, as noted in a statement they published.
“On Saturday 3rd December 2022 Esports Team Toyota CRYO RDT terminated their relationship with rider Eddy Hoole. The team was requested not to make any public statement while a Zwift Accuracy and Data Analysis Group (ZADA) investigation was ongoing. The results of this investigation were released today Wednesday 7th December 2022….
…As can be seen from ZADA’s determination the nature of this case is such that the team would not have the means to suspect / identify / investigate circumstances such as these as they require access to Zwift Server log files and an in-depth knowledge of how to interpret these. However, it was clear to the Management Team that as a result of initial information received from ZADA without any plausible explanation from the rider there was only one decision open to us.
Esports requires a basis of trust on the part of all involved to ensure that the sport is fair, and we have and will continue to work with ZWIFT / ZADA in an effort to achieve this. We are saddened by this situation and will now study in detail ZADA’s report to establish any lessons which we can learn from it.”
However, while the story for the rider ends here, it doesn’t end here on the tech side.
Previously Demonstrated at Hacking Conference:
Now, the kicker about this whole thing is that this exact attack vector was previously shown at a security conference back in August 2019 by Brad Dixon and Mike Zusman, from a security consulting firm. However, prior to that, on March 5th, 2019, Uncle Keith Wakeham demonstrated a very similar variant of this too (I say Uncle, because in the sports tech industry, everyone knows Keith and his Titan Labs, and almost all roads involving quirky data things and fun data projects, tend to lead back to Keith). When in doubt, Uncle Keith probably knows the answer, or why it exists. I actually wrote about both back in 2019.
Both of these attacks essentially did the same thing – they were so-called ‘Man in the Middle’ attacks. Essentially, they took the data stream from your trainer or power meter (or heart rate, as also shown), and then tweaked the values before sending it onward. This is relatively easy to do over ANT+, but slightly more complicated over Bluetooth Smart (still, not impossible, just a bit messier).
In the case of the attack that Brad/Mike showed, it was a bit more simplistic in terms of how it added spice to your ride. It offered either a set multiplier for your power, or it could just ride for you, and generate a fake HR number. However, the fake HR number wasn’t super believable, given it was a bit more of a static value. It didn’t have the human nuance that would show like a human suffering with slight ups and downs. Same goes for power. However, the multiplier mode (called EPO mode in their presentation) would be believable, since it was using your real power as a baseline.
But it did show how they could execute it, on different channels and even control it via a game controller:
Meanwhile, Keith’s hack wasn’t re-transmitting things, but simply acting as the sole source of data. In other words, he could just sit there and do nothing and control a rider profile to whatever power/HR he wanted. In his case, he did add variability to the power/cadence/HR numbers so that it was believable (or at least, more believable).
Given in this case we saw the rider actually riding his bike via webcam, it’s unlikely that he was just using an Xbox controller as in Keith’s hack. However, that doesn’t mean aspects weren’t leveraged.
The implementation shown by the rider in question seems to be closest to what was demonstrated at the DefCon event. However, what’s interesting is the channel drop portion. Zwift isn’t clear in their document precisely what channel was dropped. However, given that Zwift notes *both* datasets transmitted/recorded for the race showed similar values, that tells you that this cheat was being applied to not just the trainer or power meter, but actually both of them.
That’s because if it applied to just one of them, it would have demonstrated a difference between the two required data sources. And notably, in this cheat, the rider was seemingly able to turn it on or off at will. Or, perhaps he just rode easy for the majority of the race with the multiplier always on.
Going Forward:
The challenge with this cheat is that it’s relatively hard to detect when used properly. In this case, the rider effed up by using it on a mountain-top finish with a crazy breakaway win. Had he stayed with the leaders and then just edged them out at the line, he’d probably never been flagged. Further complicating things is that he’d apparently been using the cheat for *all races*, effectively establishing a very good baseline that large data set algorithms might have ignored. Though, he screwed up by also not using it for training – which would have given him some plausibility excuse of a weird technical issue.
The mitigation for this type of attack is the same as it was in 2019 when I posted about: Encryption or digital signing at the trainer level, which ensures the data stream isn’t tampered with. In fact, Keith goes into detail on this in the second half of his video (there are YouTube chapters in it).
Of course, the challenge there is that getting the trainer industry to agree upon even the most basic of standards has been impossible in recent years. They can’t agree on how to implement steering; how are they going to agree to fundamentally change the direction of smart trainer protocols? All at a time when one trainer company is suing other trainer companies, and the remainder are closing their openness doors.
The problem is: The cat’s out of the bag. Sure, the idea was published years ago, but there wasn’t much proof anyone was using it. Now, not only are people using it, but someone that had won an automatic ticket to the UCI World Championships used it. And he would have used it successfully had he not been, to quote Dave Towle, “greedy”. And there’s no doubt he would have eventually used it in the actual UCI World Championships, likely to outright win.
It’s at this point that I remind you that the UCI has a person (division in fact), dedicated to esports racing and trying to establish believability in the sport. It’s a heck of a lot different when the UCI knocks on Zwift/Wahoo/Elite/Tacx/Saris/etc’s door and says they need to implement something, than if someone like me says it. But at this point, the presence of this cheat in the wild demonstrates that if the UCI wants this title to have any meaning at all, then they need to start demanding some changes.
With that – thanks for reading!
FOUND THIS POST USEFUL? SUPPORT THE SITE!
Hopefully, you found this post useful. The website is really a labor of love, so please consider becoming a DC RAINMAKER Supporter. This gets you an ad-free experience, and access to our (mostly) bi-monthly behind-the-scenes video series of “Shed Talkin’”.
Support DCRainMaker - Shop on Amazon
Otherwise, perhaps consider using the below link if shopping on Amazon. As an Amazon Associate, I earn from qualifying purchases. It doesn’t cost you anything extra, but your purchases help support this website a lot. It could simply be buying toilet paper, or this pizza oven we use and love.
Good article, one typo I saw:
“Even more, Zwift THROUGH a bit of a knife to the heart in there”
Thanks!
Hy”s ‘n poes!
Here’s a small typo
Olympic Persuiters
I’m not sure how he even got on the bike with stones big enough to try to pull this off.
I guess go big or go home. He selected both.
I see what you did there… LOL
One of the many reasons not to ride indoors on Zwift.
I couldn’t disagree with you more. Zwift is an excellent indoor training and social tool (albeit with some bugs), allowing cyclists to both improve their fitness, meet new friends, and challenge themselves further, all in the safety of their own home. What this will hopefully motivate is continued insights into detecting and banning cheaters from ruining a great platform for others, as was done here. To me, this is pretty much the equivalent of warning, you should never go online because you could get a virus.
Because someone, somewhere might be cheating? If that’s a good reason not to do something we should stop most of the activities in our lives.
Yeah, why would this stop you from using Zwift? Would you stop riding outdoors if you heard that someone was doping in outdoor races?
I’m not sure there’s a better example of “Throwing the baby out with the bath water.”, Frank.
As covered well by JP, MicahlW and Paul S., that is a very bad reason to discount Zwift in it’s entirety. There are MANY ways to use Zwift that don’t touch on racing. And for the many that do race, bad power data (intentional or otherwise) happens.
The key to using Z racing is understanding that those examples exist, recognizing your interest in racing and choosing to ignore the stuff that is out of your control. I and others happily race and take what we need from it, along with the many other aspects Z offers.
As mentioned, if that’s your litmus test for life (any cheater invalidates the whole entity), you must not take part in many other things in life, since cheating and/or rule breaking are parts of just about everything I can envision :P
Well Said Chad. s
Fully agree. 90% of the time I am on Zwift for structured training and workouts. I value this a lot however I have alternatives as good as Zwift for that (RGT or even MyWhoosh). Even free ones. What creates a real difference in Zwift is that notion of community for racing on top of the rest. Therefore valuing Zwift does not prevent from pointing at the incoherences of Zwift towards fair racing. They are the ones throwing ads on Eurosport claiming that Zwift competition is very similar or the same than IRL. IMO this is plainly deceiving. Plus there is a huge gap between claimed intentions and actual facts and investments in order to solve the issues that are well known for ages. I believe there are different ways to look at things, one of them has also to do with making the platform better.
“Guy that openly and clearly cheated on his own youtube stream and called out by hundreds finally banned”
Ohh…that’s a good one!
(For context for others, the rider had previously posted videos of himself in various sticky watts type situations, though, perhaps those were just this in testing…)
Took a lot longer than I thought for this to show up in the wild. I personally enjoy Zwift racing, but it has a very very long way to go before it can be trusted.
Is there any reason this can’t be done without the tell-tale channel drops?
It’s a bit fuzzy what ‘channel’ precisely Zwift is referring to, but, this sounds like something that Uncle Keith would know…
There are others out there clearly using the same method to cheat in races and stream. Happy to point it out privately if you are able to get someone like Keith Wakeham to check. Reaching out to zwift and organisers of the larger race series, seems like either they don’t care or some folks can already do this without these ‘channel drops’ to give it away.
“It wasn’t possible, and is largely beyond known human performance evels”.
BTW, the Open House last weekend was great…thanks again for this splendid event with the chance to meet you, The Girl and Des and to enjoy free beer/food and great company!!!
Thanks Hoot, and glad you enjoyed the event. Super fun to catch-up with everyone!
I do have some code out there which fakes ant power, if you have two ant usb key it would fetch the value from one source (the real power meter) and replay it with an increase on the other side…
I never actually used it (I don’t really care about racing, just do workout) but will be happy to share it if you want to write a followup article..
(it uses python-libusb running on a rpi)
(not planning to share to anyone else than ray, before someone ask ;))
that is sooooooo bad…. where is the bug bounty program Eric Min promised to the community 9 months ago?
DC, the mitigation for this is a very easy outdoors verification protocol. On top of any technical solution, there is nothing like IRL for this. This one was highly highly predictable doing basic comparisons with Strava IRL rides.
Specially at the level we are talking here.
Outdoors protocol and ebiopassport.
Such protocol has been uses by ZADA and race organizers for years.
the ZADA protocol is a joke, Eddy Hoole completed it…
Protocol is good. Someone did a bad job analyzing his IRL activities. Of course, even those can be potentially faked. The only solution for this is IRL verification during live event or done by certified bike shop/gym/lab/individual.
Well, I insist that the protocol is flawed and that there are KPIs you can look at IRL that would definitely solve the issue. I will certainly publish something about it with some other people in the next days.
Live final events need to make a comeback to discourage ‘virtual’ riders.
This. Stopping the slight insanity of attempting hosting prestigious events with people participating from their basements, should effectively discourage those who see an unethical opportunity to make a “career” in the discipline. And probably make the situation in rest of the “unprestigious” events better too with those gone.
Would put I think an increased healthy business demand for more indoor-cycling studios facilities that could get “certified” to allow riders to participate from, when the event require it. So riders don’t have to do ironically immense travelling across the country to participate in something run over the internet anyway.
So the cynic’s version of the lesson here is to have your multiplier electronics in line and ready to go before you connect to Zwift so that the connection never drops.
I’m not sure encryption will solve this problem. If the box in the middle can pretend to be Zwift to the trainer and the trainer to Zwift, wouldn’t it have all the keys it needs?
It would. But, I don’t think that’s an issue. There’s only a handful of trainer companies, and if they wanted to, they could get their act together pretty easily and coordinate this.
It would provide a minor stumbling block for startups, but nothing of meaningful value since it could perhaps only be enforced at these pro level events, so by that point most startups would have had time to get their docs in order.
Keith actually goes into this a bit in his video, in terms of things like no second chances, etc…
at some point the keys will need to be exchanged on the device where the client run and it would be not too difficult to figure out how to intercept or extract them….
except if they go full on cloud gaming, i don’t think there is many ways to secure that…
> So the cynic’s version of the lesson here is to have your multiplier electronics in line and ready to go before you connect to Zwift so that the connection never drops.
Is that what the nature of the attack was or since Zwift found that the calibrations were out of whack after the race, was the defendant cutting the stream while changing the calibration factor?
I don’t understand how the MIM attack would result in power meter calibration factors being off. Doesn’t this look more like the calibration factor itself was changed so that the reported powers would be higher?
No, you could simply have handshake between the trainer (at firmware level) and Zwift and an encrypted stream. Any alteration to the firmware would show.
Not exactly. There are a number of encryption protocols which do not rely on sharing keys. Banking (and most web traffic) now use HTTPS which is, at least at this level, not crackable.
One way to do it is with a set of keys in the stored in silicon (inside a physical chip) in the trainer. At some point someone may “decap” a chip, break the security and get the keys but that would take nation state level tools and experience.
If, for example, the trainer implements a server and Ethernet/WiFi connection, it can then make its own connection to a 3rd party audit server in the cloud.
Zwift locally accesses the data via HTTPS Websocket (or SSH) from the trainer.
After the event, the data saved by the Zwift application can be compared to the data saved in the audit server. That could be part of the post-race results processing.
When you look at indoor rowing, where they had indoor races and world records for decades, they solved it by encryption. The PM5 monitor of the Concept2 (the main manufacturer of these machines and the defacto main racing platform) simply creates a hash of the timestamp, distance and achieved time, and signs it. This “verification code” can only be obtained by this PM5, which is a black box for everyone except the manufacturer.
For example, as developer of OpenRowingMonitor, we can mimic a PM5’s bluetooth protocol to the bit (even fooling Concept2’s own smartphone app), except we can’t sign the result as we don’t have the right key. So from that point, you can remove this way to cheat. But such signature then needs to become part of the protocol when completing a session.
HOWEVER, you basically move the cheating further down the line. For rowers, it is possible to replace the sensor with a simple signal generator and generate or replay previously recorded signals from the sensors (and even speed them up a bit). A bit more complex attack, but doable with a Raspberry Pi. Unless companies start investing in securing the sensors and the communication to the monitor as well, that is a doable approach.
And especially for bikes, adding a small electromotor could do a lot, especially when there is no physical inspection of the equipment. For indoor rowing they solved this issue (as well as the replay attack) by requiring certain world records to be conducted at racing vanues where stock machines are provided to the athlets, which probably isn’t accepted by most cyclists.
The estimated power reserves look like W’bal, based on Dr Skiba’s model. Pretty sure it’s a screen gram from Golden Cheetah.
Indeed, Zwift just confirmed as well. More info for those that want it: link to em-sportscience.com
“Hoole tool ruled fool on Roule Ma Poule”
❤️❤️❤️
Bravo!
Hands down!
We have a winner!
That was unbelievably good.
You rule!
“And there’s no doubt he would have eventually used it in the actual UCI World Championships, likely to outright win” Don’t they do the worldchamps in person with supplied kit (bike aside…), how would this cheat work in that situation?? If it wouldn’t, what’s the point of cheating knowing that you’ll be “unmasked” then….? Crazy….
That would require said cheater to think logically about the future and the implications of their actions.
Its amazing to see how much enrgy peaple use to cheat, and what do they get with is? its frustating to see if all this energy would be used for the good … like improuving Zwift or other software we would be much better
intresting that this risk was shoen in 2019 and Zwift did not react, i thin this is the second tinme hapening….
someting where zwift should take more leadership…
Why do people cheat on Strava challenges or segments? There’s nothing to gain there. Here, at least, there’s probably money available, either prizes or potential salaried contracts with actual teams.
If one day you find an explanation to human being stupidity you will be eligible to Nobel prizes in several disciplines
Why cheat on Strava segments? Bragging rights in your riding circle and the inflated ego that goes along with being #1.
When you’re really #2…
I could never figure that out. But most of the popular segments near me have impossible times.
Before even reading the conclusion I came to the same one, a fix for this is simple but very hard to implement because it would mean cooperation of all powermeter companies.
I think the simplest way to do this is to have new firmware that would allow to register your power meter outside of the normal transmission of data and when doing so, a secret key would be stored on the Zwift server and powermeter device. Then, there should be a new signing channel using private key encryption to send a combination of the secret power meter key and the power value, encrypted using a Zwift public key so only it can decrypt it.
Just need to yoink the key out of a PM and then you can do the same MITMing. Fighting this would be the same kinds of arms race that has effectively allowed ripping of Blu-Ray discs: a losing battle. (Blu-Ray has a similar problem… Encryption keys on players that keep getting grabbed and used for ripping. These keys can be invalidated, but so long as there are crappy manufacturers that don’t sufficiently protect the keys in the hardware the window, it’s a losing battle.
Maybe but trying to extract the key from the hardware is much harder to do than a man-in-the-middle attack. The firmware shouldn’t even have an option to read the key. At that point you would have to physically tamper with both devices to read that value from its internal memory. That is a pretty high bar to go over in order to cheat that other forms of cheating may seem more accessible
All it takes is one hardware hacker to do it, though, and then write a tool and the cat’s out of the bag. After that point that key would need to be invalidated, making all those trainers not work unless their private key was updated. That’d be horrible for users and a black mark on the brand.
Why do you need a secret key on the Zwift server? The power meter itself could simply sign its data using PKE, and that would prevent a MIM device from modifying the readings. The only way for this to be defeated by a bad actor is to physically recover the power meter’s private key, which if implemented well would require physically exposing the silicon.
The problem is power consumption.
BluRay had a problem where the “locked” item (the disc) was unable to perform computations so it needed to be pre-shared keys. That problem was compounded by the fact that the BluRay group wanted everyone to be able to sell decryption hardware and so it was inevitable that someone would fail to protect the keys.
In this case, we want the power meter to be the locked item, and it can not only perform computations (which means public key encryption can be used) but there is also no expectation that every fly-by-night company is able to use the keys.
You could do a different key for every session that gets negotiated at the beginning so you don’t need to store it on the PWM but it should definitely be stored at Zwift for audit purpose otherwise you wouldn’t be able to validate the signed data.
yeah it’s a loose battle….
we just need to have matchign irl results!
@Laurent
If a power meter signed its data packets with its private key, anyone with the public key could verify the unmolested authenticity of said data packet. No need for a different key every session. No need for key negotiation. No need for Zwift to store a secret. No ability to MIM the data packets unless the private key gets physically recovered from the silicon of the power meter. If implemented correctly it would take the physical removal of the power meter’s encryption chip packaging and looking at the chip under a microscope to have a chance at finding the key. You’re hung up on private key encryption which isn’t the way to do this.
Also remember, we’re not trying to hide any data from anyone. We’re simply trying to prove authorship of data – trying to prove the data came from X power meter and not Y man-in-the-middle device. There’s no need to make this a bi-directional conversation (no “negotiating keys”) because we don’t care who can see the communications.
Also remember, unlike the broken BluRay analogy, we’re not trying to both share data (a movie) to “trusted” users (authorized BR players) while hiding the exact same data from untrusted users (pirates). BluRay HAD to share the private key to make that system work. What we need here is for the hardware to sign its data, thus the key never needs to leave the hardware. A very different problem.
All you need is to get the key out of one device and turn it into a cheat device. After that point the key used for cheating needs to be revoked and all devices using that key updated to have a new one. There are timing issues around when to revoke and when to reissue, and any device that doesn’t get updated won’t work after the revocation happens.
I just don’t see it happening because either every participant in the ecosystem (say, Zwift) needs to cooperate, or no one does. It’s a prisoner’s dilemma. There’s also no central group as there was for Blu-Ray, and personal fitness devices don’t exactly have the huge money backing that the movie industry does.
Or, a third option for those who take racing really seriously, would be someone (say, Zwift themselves) making a certified trainer that signs things. Then you can have most folks with normal power devices, select people on Certified Power Devices, and those devices required for certain races. It’d be kinda like the virtual power and power meter tiering there is now, but one more.
Yeah, that’s why the key can’t be tied to the device because it can be compromised. A session-based handshake would mitigate this problem.
As far as how to do this as a business problem, if only one company was on board, they could market their PWM as “secure” or “cheat-proof” and individual team could even market themselves as using this PWM. It wouldn’t even need to me required for a while but just used as a transparency good faith token from teams like hey, we are committed to racing clean
@ Steve
> All you need is to get the key out of one device and turn it into a cheat device. After that point the key used for cheating needs to be revoked and all devices using that key updated to have a new one.
Not if you require people to register on Zwift with their specific power meter(s). This isn’t encryption with 1 end goal, it’s signing with a unique key. Every device would have a unique key.
> I just don’t see it happening because either every participant in the ecosystem (say, Zwift) needs to cooperate, or no one does. It’s a prisoner’s dilemma. There’s also no central group as there was for Blu-Ray, and personal fitness devices don’t exactly have the huge money backing that the movie industry does.
There is no threshold problem with a solution like this. Zwift need not care what JoneRacing.com does.
A central body isn’t needed. The attack (physical inspection of the chip under a microscope) is incredibly hard and can’t scale. There’s no reason, at this time, to suspect there would be widespread cheating.
> Or, a third option for those who take racing really seriously, would be someone (say, Zwift themselves) making a certified trainer that signs things. Then you can have most folks with normal power devices, select people on Certified Power Devices, and those devices required for certain races. It’d be kinda like the virtual power and power meter tiering there is now, but one more.
Or Zwift can just work with 1 power meter company to make a signing power meter and then open the standard and tell people that starting next year only signed meters will be allowed. No need for them to make hardware or to make it exclusive.
@Laurent
> Yeah, that’s why the key can’t be tied to the device because it can be compromised.
Handshakes are subject to more trivial attacks than pulling a private key off of silicon. Handshake attacks scale better as well.
@Dr Jones!
Aaaw I see what you are saying! It should probably be a short-lived dynamic key though because the amount of signed data generated by the PWM could increase the odds of brute forcing the private key. So maybe the public key is only valid for a month for general purpose and a new one needs to be generated before major events.
@Laurent
It’s easier to make the private key longer than to rotate keys. With sufficient length nobody is brute forcing the key before either the heat death of the universe or quantum computing. PKE is routinely used on far more valuable data streams with far larger corpuses.
I wonder how much the race format exposed the cheat. The UCI and ZGP races both place a lot of emphasis on repeated big finishes.
I wonder if this would be even easier to do with the network-connected trainers because then you can skip the RF layers.
Kinda wish I had one; it’d be a fun winter project.
(Just need to introduce some noise and variability with the offset and it’d be really, really hard to detect.)
Maybe Zwift, one day, will set it up where Zwift can connect to multiple sources, of the same data, for comparison purposes. Even if it only for this type of event. Personally, I don’t have a need for such an option, but it sure does like Zwift needs it. And for things like channel disconnects, that is up to the rider to ensure is working prior to the event. Maybe another screen/prompt to let them know it is missing. If they can’t meet the requirements, then can’t race. Simple as that. It is no different if the team didn’t bring enough tires to a race or didn’t bring the tools to change a tube.
Yup, I’d agree. In fact, I think it’s actually super useful for helping both prove cheats, and help accidental calibration issues.
Every trainer/power meter has relatively unique ID’s that are broadcast to Zwift. Sure, ANT+ ID’s are re-used to a degree, but when combined with other model/FW details transmitted from the sensor + your actual account, it developed a pretty unique fingerprint of that unit.
Having dual-recorded built-into Zwift so that every time you ride you’re building up a ‘database’ of that data would be super helpful. Heck, it’d even be helpful in helping people know when something is amiss – a system that automatically points out super high (or low) values due to miscalibration or whatever.
And this is something that people have been asking for, for years, and it’s all that complex.
Don’t you think Dual Recording has contributed to many of these issues? It gives a cloud of authenticity even though it is completely meaningless, as the secondary power meter in almost all cases can be easily configured to match the power of the primary source, whether that primary source is accurate or not.
A rebroadcast signal can easily rebroadcast the fingerprint from some other device or the same device if you use proper fairday cage with receiver for computer and broadcaster from man in the middle in day a microwave.
There is no way to solve this without broadcasting more data. The trainer could cryptographically sign the data it’s broadcasting using a private key on the trainer but how do you keep an end user from hacking their trainer to extract the key?
Aka “why Zwift is a game and not a sport unless it’s done in person live on competition-supplied trainers, etc.”
What other eSport has this easy of ability for participants to change outcomes? Surely not Madden, or COD, etc. right? I mean at the less than PRO level it’s as easy as dropping your weight or height. Even and the PRO level we now see this type of thing occurring.
The names Hoole, Ass Hoole…
Good article Ray!
One question I do have on this though. The “independent test” was this actually a lab test or a case of the rider got asked to ride outside and provide a 4min effort with data to validate against? Was GPS/VAM used from outdoor to show the PWM was over reading or was this a separate lab test where an independent person/company tested him on their equipment with his own PWM to establish was over reading?
That then leads onto a slight issue with the theory in that the PWM was found to be over reading independent of the at-home MITM device setup. Your theory is that the device was boosting the watt data output from both the trainer (to game) and PWM (to head unit)….. but if the PWM was over reading by the near 25% ZADA claim when it was not present in the MITM device situation then I can’t understand how said device was boosting both sources like your theory claims. More like the primary source was being boosted by the device to closer match the already sloped/offset/boosted PWM which is reading high all the time regardless of the MITM device.
Love to hear your thoughts or discuss. Feel free to email me :)_
I had the same question. How was the independent test actually performed? If it was a 3rd party which used their own devices to record power, I suspect Hoole knew he was about to be caught?
Six month ban is a joke. This was a lengthy, deliberately planned cheat over many races leading up to and including this event. A lifetime ban is the only acceptable penalty IMO.
100% agree. The guy has been confirmed to have been cheating in almost every race he’s done in last what, 2 years? At very least, ban him for length he’s been cheating.
well there’s big bucks available on the mywhoosh races apparently – he would just go there i guess
It’s wild that a reputable team in Zwift, at the highest level of e-sport, has obviously done ZERO due diligence into the riders on their team. Hoole has no outdoor results to speak of yet the team blindly accepted that Hoole could compete at the highest level? Shame on them too. As usual, Zwift not doing near enough to address cheating.
Yeah it’s easy to release a statement after the fact, but to go so long (taking money in to the club directly from Zwift too) with the attitude ‘it’s not a problem until it’s a problem’ is a major issue.
A 6 month ban seems light for intentional and provable cheating. I understand that type of ban for outdoor riding if someone took ped’s since it is a bit harder to prove. Imo this is similar to catching someone red handed trying to mechanical dope outdoors. Obviously for this rider, there were ramification outside of the ban like getting dropped from his team but for situations such as this, in arguably one of the bigger Zwift events, the punishment should have been harsher.
Especially given that this intentional cheating went on for at least 150 races. Don’t know how often the rider races, but I’d guess that means at least a year (thought maybe I’m just a wimp and people at that level are frequently racing multiple races a day).
Great article! Thank you.
It could be just me … but doesn’t the penalty seem a bit light? Why not 2-3 years and the something lighter for the team? I state that acknowledging that getting these things right can be tricky. On one hand, you have the decision to strip Cameron Jeffers of his title, which I found to be a rather strange. [Why does it matter if the guy obtained a bike, available to everyone through nefarious means? So he didn’t spend enough time in Zwift prior to the race to get the superbike. So what? Why wouldn’t you open up all the bikes to all the competitors equally prior to the race anyway?]
Not just you. Others have already mentioned the shortness of the penalty in comments above (literally one reply ahead of you in one case ;) ). I agree that 6mo is laughable considering all that is known in this case.
Totally agree with Fritz here, the ban when put alongside the punishment of other ‘e-dopers’ seems very light. I’m not even that aware of many others outside of Cam’s but the two side by side are arguably totally different and should be totally different in scale. Yet they’re not. Both 6 month bans, a light fine and stripping of titles/wins.
One was knowingly manipulating power data over several years to unfairly gain an advantage across the board in ‘every’ competitive race. The other working ‘around the rules’ to gain access to a bike frame others where riding in the same event – to effectively level his playing field with them. Yes they made an example of Cam for being ‘the first’, but a larger example should have been made here.
Just ask many of the community racers/admins. Hoole is well known for being a cheat and totally unbelievable performances. He just hadn’t been caught until now.
“Zwift then gives the rider the opportunity to have an independent lab conduct a VO2Max test, which, the rider accepted.” Is that confirmed a lab conducted the test? From Zwifts somewhat vague wording it sounds like the actual power the cheater put out (~400W) was maybe calculated based on a ride they did for verification?
It is getting harder and harder to win Zwift races. First they make it impossible to change weight during the race. Can’t sandbag anymore due to autocat. And now it is forbidden to alter our signal?
What’s next?
Does anybody know if this still possible on whoosh? (Asking for a friend)
PerfPro CT SmartBridge is a commercial product that demonstrates this type of bridge.
It bridges a Computrainer to ANT+ in precisely the same manner, allows for control of the CT and power data etc is sent back via ANT+. This provides a more robust way to use your CT with Zwift.
The only (slightly) tricky part of this hack is ensuring you modify both sets of power data to make them look similar enough to pass muster but still be dissimilar sufficient to look unmodified.
This is also not limited to ANT+, BLE data streams would be as easy to manipulate. And if fact BLE data could be modified and streamed out of ANT+ or vice versa.
The data could be protected cryptographically. Trainers are not power-limited (like power meters). Just having the equivalent of a Raspberry Pi with Ethernet/WiFi embedded in the Trainer would allow an SSH or TLS connection to collect the data and control the connection.
I’ve got a hack that turns me from a 8w/kg to 3W/kg type person, happy to share, note that the training does involve copious amounts of beer.
Colin.
It’s obvious that ZADA, ebiopassport, etc… are all easy to fake and get approved while still being a cheat. The one constant I’ve seen with riders who are good racers on Zwift is that they have good AND transparent results IRL. Publicly posted IRL results, whether on Strava or even athlinks.com or similar is a must. Any team worth it’s salt should be confirming their riders are backing up indoor results with outdoor results.
Good writeup. So I looked at the Youtube livestream before reading the article and I noticed if you pause on his Zoom session camera Hoole’s stuff is sitting on a trainer box that I believe says “Direto”. So if I had to take a guess, he was using an Elite Direto of some vintage.
Looking at the Elite website and support site, not only is it not a self-calibrating trainer, but the calibration itself is ripe for exploitation. Again, I’m just going off of a grainy screencap of a box his laptop and bottles are sitting on.
“The procedure must be conducted only after a 10-minutes long warmup session (with power ranging from 100 to 150w).
Please follow these steps to conduct the procedure:
– Load the myE-Training app.
– Power up the home trainer (on the Kura model, pedal faster than 5 km/h for a few seconds until the red LED light is always on and the other two blink)
– Warm-up for 10 minutes (for instance with a Level training)
– On the main screen select Settings – Home Trainer calibration
– A message will pop up reminding you to conduct a 10 minutes warm-up session. Tap on “OK” if you’ve already done it”
link to elitesrl.zendesk.com
Could be an Elite Justo. Those are self-calibrating and the only one made by Elite able to meet the 1% accuracy required for this particular race.
Ray,
You may need to chalk this up to the stupid question bin. But, in the photos provided, not only can you not tell what trainer he is using but what bike he is using. Wouldn’t it be much easier to cheat with an e-bike or manipulation of the flywheel weight than this electronic manipulation of data? I don’t race on Zwift and don’t intend to but there is no way, in my mind, you can ever eliminate cheating on this when you can’t personally view everything the rider is doing. Esports will always be subject to cheating unless it’s done in a controlled environment.
His watts speak for themselves.
The Zwift cheating / chess cheating crossover joke we needed
Except that, unlike this case, there’s no evidence that Niemann cheated in the game against Carlsen. Or anywhere else, for that matter, except for the self-confessed cheating online years ago. Carlsen played a poor game. It happens, even for the soon to be ex-World Champion.
I agree that the punishment seems too light for the infraction. BUT Z seems to be following its own rules about the max penalty for a 1st infraction. Maybe modifying one power stream is infraction #1 and modifying power stream #2 (the ‘dual power’) is infraction #2 and modifying the HR data is infraction #3. There, fixed it for you – lifetime ban (according to Z rules).
27,000 joules / 250 seconds… = ~108 joules per second… 526-108 = 418 CP
CP w/kg 6.85
Hooooooooorse shit
when cheating… cheat by 1%… not 20%
Avid Zwifter … I would love to average 8 WKG for 4 minutes
Isn’t this why most esports are on controlled devices in a stadium, when it gets to the “main event” plus that adds to ability for media and “gaming” which yes, integrity, but gambling needs….
Can we pressure Eddy to release the list of all the peeps that bought the program?
Good point. Was the the coder selling a product, (one of) the seller’s main advertising point(s), or just one (of many) consumer or a hack?
Man in the middle attack of this was obvious from the beginning. I mean the testing told that were available could mostly do this.
Transmitting the data securely isn’t that simple of a challenge, it’s more like DRM and console machines with “secure” bootloaders.
Reminds me when Lance armstrong was destroying the tour and quite a few medias were doubting his performance and yet he kept repeating it was “all hard work”.
I wonder if Eddie Hole has spoken publicly ?
Pro Racer Cheats His Way To World Qualifier Slot But Data Gets Him Suspended
What a poepol… I hope he gets banned for life. From Call of Duty too.
This proved a point that Zwift as a social tool for cycling indoors for general fitness and social reasons remained ideal for most people.
Virtual racing can be taken out of context entirely with such possibilities to enhance performances and affect result of races. Which is why I decided a few years ago to stop racing seriously on Zwift and focus largely on community effort which is the real gratification.
Anyway, hope this saga will discourage anyone in future to try cheating for results. I am sure this will not be the last but the next may just take a much longer time to be discovered.
I wonder if tech like the layer2 crypto currentcy could be used as a method of validation.
This article describes quite an elaborate cheat, but my reading of the Zwift reports is a lot less elaborate cheat. I read it as…
Has dodgy Power meter (say pedals) that reads 20% higher.
Has a legit, accurate trainer (say Wahoo Kickr)
He connects the dodgy power meter to a bike computer as Ant+ (say Garmin Edge)
He connects Zwift to the legit Kickr and joins the pen (this gives his primary source fit file the Kickr stamp)
He disconnects the Kickr in the pen and selects his dodgy power meter as his primary as Bluetooth.
He is then using his dodgy power meter as the same source for Primary and Secondary. But critically because he joined the pen in the Kickr, Zwift thinks he did the whole rider with the Kickr as his primary.
Comparing both fit files from the same source would looks slightly different due to the fractional differences when Zwift / Garmin Edge would take the data points. Making it all look legit.
Thoughts?
What’s funnier is he doesn’t even have to do that. Wahoo or Elite trainers have a ANT+ power meter pass-through feature.
1) Buy Favero Assioma pedals.
2) Install on 165mm cranks.
3) Set crank length to 210mm in the Assioma app or on a Garmin.
4) Permanently link to a Wahoo KICKR via the “control w/ANT+ power meter” option.
5) Use a spider-based power meter which supports scale factors as a secondary power source for dual recording,
6) Set a scale factor of 1.27 on the spider.
7) Done. No disconnects or UDP port blocking necessary.
Setting aside all the other ways he could cheat as Eugene noted, the key item in Zwift’s document that alludes to this not just being hardware swappage is that it was “missing a data analytics channel”. If it was just a source swap, or even a trainer using power match, that data would still be there (and state as such in the data stream).
They’ve previously busted others on hardware swappage, and have stated that as the exact cause in the finding.
Eddy’s LinkedIn profile gives some clues as to his technical ability:
link to linkedin.com
Cheating is endemic in Zwift at every level. They don’t really want to stop it because they only care about subscriber numbers.
I was amazed that it wasn’t the guy whose name reads as Pirate.
Looks like he might be using an Elite Direto trainer? I can see what appears to be the word “DIRETO” on the box under his bidons / bottles in the second picture.
I think both Zwift and Strava aren’t incentivized to go after cheaters, so they don’t unless it’s an event like this in the public spotlight. Man-in-the-middle attacks such as this have long been known to be theoretically possible, and are virtually undetectable. If this guy had paired with the malignant device originally, instead of enabling it in the pens, it would have been much more difficult to prove.
Zwift’s old school method of doing an independent lab test to confirm a race result, and to only moderate important races, such as UCI sanctioned events, is also evidence of laziness in moderation. Notice how if we keep using that ideal we’ll not be likely to catch cheaters unless they’re being very obvious (as with this brazen individual).
The unfortunate reality is cheaters and weight dopers tend to be Zwift users that are very active at renewing their subscription; they’re racers, not casual Zwifters. Zwift could be afraid of losing users if they crack down on cheating in any meaningful way, and we (the public) have no real idea as to how wide spread cheating and weight doping is across the platform.
We must remember that at the end of the day Zwift is still a game.
Helpful for fitness, training and a great community but still a game.
People will cheat, even more when there’s money, fame, recognition and sponsors in the mix.
Others games like COD, Apex, CSGO, etc. After 5-20 years of experience still battle to catch cheaters.
Way too much work.
Just hook your cordless drill up, like I do.
This type of power output is not out of the question. It’s not even extraordinary. I remember freeriding in Zwift on the backside of the Epic KOM climb. Some guy passed me like he was on a virtual motorcycle. There was barely anyone else on the climb so I could see his stats for minutes as he was holding high-singles/low-teens w/kg. Must mean plenty of people can do it.
…./s
Just fyi, the racers you referred to as Persuiters: it should be pursuiter, in reference to the track events Individual Pursuit and Team Pursuit.
Thanks Mike!
I don’t get why, it’s just a game. I guess we can be sure there are others, because they can’t all be dumb enough to put out numbers rivaling Pogacar and Vingegaard. I’m also curious now how he got his power meter in the lab to read the same without being detected for tampering.
We are taking indoor “riding” seriously now? Zwift… who cares?
My cousin does all her riding and running indoors, bragging about her hard instructors and intense workouts.
We went for a brisk walk and she couldn’t keep up with me on the moderate hills, 15-minute miles, lol.
Indoors is a last resort for when you can’t get outside. It’s an artificial substitute.
Here in Belgium we ride for reals!
Who cares? Apparantly you do because you took time out of your day to write a comment here when you could have been outside. Also I checked the forecast for Belgium, looks like you have a high of 11 C coming up this week. That sounds horrible, you’d probably have to put on a long sleeve jersey! You are tough to ride through that!!! I worship you sir.
This article gets mentioned in this piece on road.cc. Zwift say it was a different hack to what is mentioned here but don’t actually say what hack was used.
link to road.cc
Interesting. Not sure why they’re saying more than they did back then. Either way, still not saying very much, and still implying some sort of data stream intercept along the way, given they say this:
“There are multiple ways to hack servers, intercept signals, and manipulate them before they hit Zwift servers or block one signal and send another.”
At the end of the day, all but the first one is considered a man-in-the-middle type scenario. Unless they’re saying Zwift’s servers got hacked…. (the first one).
There will always be cheaters as long as there will be stakes and competitions. That ego has to be fed to grow and then fed more…
Funny, I remember Zwift saying cheating dosen’t really happen to us after our article got published (Richardson, Smith, Berger, 2022) then this happens hahaha
Got another 3 coming out on cheating/doping on zwift and ecycling
Hope you are keeping well and great article! well written.
Better Title = ‘US Postal? Hold my beer.’