[Update – Feb 26th, 9:30AM US Eastern: See ‘Updated’ section below – Zwift has backtracked and rescinded the ban.]
On the doorstep of Zwift’s biggest event of the year – the UCI sanctioned Esports World Championship, which is later today – Zwift has managed to get themselves into another cheating and rider ban debacle. This time, for the banning of an individual that published a post of a previously known bug that allowed competitors to change their weight values mid-race without being detected, potentially significantly altering the results of said race. The published post included numerous requests to Zwift to address the issue.
To be super clear: Zwift confirms they did not ban the individual for actually using said cheat, but rather, for publishing it. And like any good drama – the coverup is often far worse than the actual crime. The question is, who was doing the cover-up here? Let’s dive into it.
What Happened:
Earlier this past week, Luciano Pollastri published a post titled “The Ultimate Undetectable Weight Cheat on Zwift”, on a burner WordPress (a blog hosting platform), with the publishing designed to draw attention to the bug. The article was then posted to a handful of Zwift Facebook groups.
The article essentially outlined that you could actually change your weight mid-race (such as just after the start), which would immediately take effect (such as before a climb, making you lighter and thus faster in the game). However, the key ingredient was that you could change it again just before the end of the race, and essentially go undetected. The since-removed article outlined, in excruciating detail, numerous tests of this (in an individual time trial where it didn’t impact other competitors), that the issue was indeed reproducible and real. And also undetectable.
However, it should be noted that the instantiation of a burner WordPress site wasn’t actually the initially planned venue for this post. Instead, it was ZwiftInsider.com (an independent site, but one that receives support from Zwift). As outlined by founder Eric Schlange in this post, notes that they didn’t think the bug would actually work. Turns out, it did, and as Eric from ZwiftInsider rightfully pointed out, it would be logical to hold up a moment and ensure Zwift had been notified first, with a chance to respond. The image below from Zwift Insider’s article (text from Eric to Luciano):
However, during that timeframe, after discussing it on a private Discord with a small number of other Zwifters, Luciano became aware that this was previously disclosed on Zwift’s own ZwiftPower forums some two years prior, ultimately without any subsequent fix.
At this juncture, rather than waiting for Zwift Insider to validate with Zwift, Luciano decided to publish the details of the issue publicly. And, while he was at it, gave the post the aforementioned cheating-forward title. The post was shared to a number of very large Zwift Facebook groups including Zwift Racers, Zwift Forum, and Reddit. Some of these groups immediately removed it, since it discussed or promoted cheating. That’s fair, given that such a restriction was a well-known caveat of some of those groups.
Shortly thereafter, Luciano received a generic notice from Zwift’s Customer Service that he’d been banned, without any context for why.
A subsequent follow-up included this slightly more detailed but arguably pretty unprofessional e-mail with further details:
The distinction between ban and shadow ban is basically that the user can continue to use Zwift, but that their results aren’t recognized in races.
In my follow-up conversations with Zwift, the company’s Chris Snook confirmed that Luciano violated their terms of service:
“First, I just want to clarify the ‘ban’. Luciano will have restrictions placed on his account for a period of 30 days. These restrictions will prevent Luciano from showing in group rides, races and will also not show on results. The ban will also restrict him from chatting with other Zwifters during that time. It does not prevent him from using the platform.
He went on to specify exactly what was wronged:
“The reason the ban has been enforced is because his actions have breached Zwift’s terms of service namely, users are forbidden to “Use our Platform other than for its intended purpose and in any manner that could interfere with, disrupt, negatively affect or inhibit other users from fully enjoying our Platform or that could damage, disable, overburden or impair the functioning of our Platform in any manner;”
This is referring to section 5 part VII:
Certainly, it’s within Zwift’s rights to temporarily ban, shadowban, or outright cancel any account for basically any reason. Except, not even the most liberal reading of that terms of service would cover publishing an article on a 3rd party platform outlining an unfixed bug with a plea to fix it, as a violation of that line item.
When I pushed back on this to Zwift, it was noted that it was less about publishing the bug, and specifically more about two core things: Publishing it with a clickbaity title, and then sharing it on social media. With Zwift saying:
“Promoting information on how to exploit the platform constitutes a violation of these terms as it can negatively impact the enjoyment of other Zwifters. Luciano has not been banned for highlighting an issue, it is because he chose to host a WordPress site titled ‘The Ultimate Undetectable Weight Cheat on Zwift’ promoting this exploit and shared this on forums and Zwift community groups (some of which also forbid members from sharing information on how to cheat).”
At this point, this starts to feel less like concrete reasoning, and more whataboutism.
But, now’s a good time to back things up momentarily. Assuming that Luciano’s intent was for good (and, I have every reason to believe it was – and I think even Zwift would agree here too), that doesn’t mean the execution was good. Luciano’s choice of titles was at best designed to attract cheaters to cheat, and at worst, designed to raise the profile of such an exploit just days before the biggest event of the year.
For as much #FreeLuciano as one might be, let’s be clear – this title was 100% about cheating – not about fixing cheating. No part of the title, subtitle, or intro suggested Zwift fix it. However, to his credit, if one read past the title area, the third and fourth paragraphs did both ask Zwift to fix it, and suggest how to fix it, saying:
“We believe it is already widely exploited in competition and affects race
results as some indirect conversations occur among riders. In the interest of
fairness of competition, we believe such a simple and definitive way to cheat,
such a substantial hack should be addressed immediately. As most races are
decided on very small variations and in short time periods up to 5 minutes,
this is the simplest and most effective cheat we know so far.
Fix seems simple: disable weight change feature through companion app.
Though ZADA seems to have made Zwift aware of the hack, nothing has been
done so far to solve the issue.”
And the article also ends with a plea to fix the cheat:
“Zwift: do something please!!! At least sticky-watters needed to train a little bit
to cheat! This one feels like you left the door of the safe opened!!!”
That does still though ignore Luciano’s rush to publish without waiting for Zwift’s official stance. After all, if this had been in the public for two years, why was there an immediate need to publish this post this very minute – versus waiting a day or two? I don’t know. Certainly, I can understand the publishing desire to get something out and ‘beat the crowd’. But even if I did, I certainly wouldn’t have given it that title. Still, the way the data was presented is super clear that he did his homework on this cheat and the implications it has for Zwift. And ultimately, he repeated multiple times in the article he wanted Zwift to fix it.
Sliding back into the technical question for a moment, in a since-deleted response from WTRL in their Facebook group, was this message (captured by ZwiftInsider):
As you can see, it implies that WTRL (Zwift’s official race organization partner organization) was aware of this for some two years. A fact that is directly challenged by Zwift themselves. Zwift’s PR lead, Chris Snook, stated in an email that:
“Regarding WTRL’s post, this was issued without consultation with us, so I am not able to provide a comment on this at this time. I am aware of a two-year claim on the cheat. This claim is something that is currently being investigated however, the only known ticket relating to this bug at this time is the one raised a few days ago. The product team is working on a fix now and I’d like us to provide an update on that fix when we are able.”
Of course, in this choose-your-own-adventure plot, you can decide which of the following you want to be true:
A) Zwift knew about it two years ago but never filed the bug or it got closed, or the person responsible moved on
B) WTRL knew about it two years ago but didn’t tell Zwift
C) Zwift never knew about it until this week
Or, some blend of that. There are infinite combinations of the above. In the same way, there are infinite ways to cheat at Zwift. You’re never going to solve them all, though, this does seem like a big and obvious gap. And if WTRL knew about it, why wasn’t it addressed with Zwift (and raised as a priority)? And further, I question WTRL’s claims that they acted upon instances of this being utilized. I’m skeptical that the logging is actually in place for them to do that today.
Finally, the classification of this ‘issue’ from a technical standpoint is debate-worthy. Some have called it a “security bug”, others just a “bug”, and others an “issue” (meaning, it can be a bug but not a bug depending on your use case – such as realizing your weight was incorrect). And some further, merely a policy issue. I suppose that’d depend on your perspective. From the UCI standpoint, I could see how this is effectively a security bug – with the security being the awarding of World Championship rainbow jerseys. Inversely, it’s not security in the sense of a potential breach of your confidential information.
However, Zwift lacks any sort of official security/bug bounty type program, or tracking system. Nor any clearly fast-tracked way to submit such a security bug. Perhaps that would have prevented much of the following from occurring. Though, perhaps not. After all, in most responsible security disclosures, the bug reporting person has a set timeline after notifying the company before the disclosure (e.g. 30 days). Certainly, not 0 days (or even negative days), as was the case here.
Update – Zwift Rescinds Ban:
This section was added on Saturday, Feb 26th, 2022 – at 9:30AM US East Coast Time, about 6 hours after the initial post was published.
Zwift has just announced they’ve rescinded the ban against Luciano, as well as apologized for the situation. To summarize, Chris Snook of Zwift says:
“The decision has been made to rescind the ban on Luciano. Zwift is working on a priority fix for this particular exploit and plans to introduce a new bug bounty scheme to incentivise people to highlight potential performance exploits directly with Zwift.”
However, further than that, the CEO of Zwift, Eric Min has also apologized and outlined in more depth here, also copied below.
I would like to personally issue an update on a situation that has escalated over the last 48 hours, concerning a ban imposed on a Zwift community member.
Having been brought up to speed, it is clear to me that this situation could have been better handled by both parties. The performance increasing exploit was until now, relatively unknown both within Zwift and outside, but this is no excuse to not have addressed it. The exploit is detectable, and we have the ability to look back and identify those to have used it. That said, our priority is not to look back, but to look forward, and fix this as a matter of priority in one of the upcoming game releases.
For this reason, we have taken the decision to lift the 30-Day shadow ban issued to Luciano. For clarity, a shadow ban does not prevent a Zwifter from using Zwift, they simply do not show to others.
Neither party had ill intent and I can only apologise to all involved, but in particular to Luciano himself. We have an obligation to the community to address exploits on the platform and will fix this particular exploit as a matter of priority.
It is important for us to uphold our terms of service as they exist to protect the enjoyment of the majority of Zwifters. Rather than share information on how to exploit a performance bug, we would always encourage members of the community to come forward to Zwift with performance exploits they find. The process on how to bring such issues to the attention of Zwift hasn’t always been clear, so in order to improve this, we plan to introduce a bug bounty program that will not only make it easier for Zwifters to highlight issues but will also reward them for doing so. We will need time to develop this program but will share information in due course.
Thanks,
Eric Min
Co-founder & CEO
This is a good step, and I look forward to seeing the details on the bug bounty program – and ideally in a reasonable timeframe (e.g. weeks, not months or years). All too often we’ve seen Zwift promise things down the road, and not deliver on them. I’d say having a clear bug bounty program in place and public by the end of March would be reasonable.
[Note: The remainder of this post remains as of the original publishing]
Update 2 – Cheat Fixed:
This update was added March 4th, 2022 at 3:34AM US East Coast Time. Zwift says they’ve now put in place a stop-gap fix that prevents height and weight changes from occurring during a race. At present, these fixes are largely shims, until a proper fix can be put in place. In the case of the website, it’ll give an error if you try and change your weight/height mid-race. Whereas in the case of the companion app, it’ll pretend to accept the change, but won’t actually change behind the scenes.
Zwift has outlined both of these in a post on their forums here:
“Today we are beginning a series of security changes to address an exploit in game where a Zwifter could change their weight while in an activity in an attempt to gain an unfair advantage in competition. This exploit could be detected on Zwift servers, but would be hidden from public view, therefore impacting community racing. The first fix, which is live today, addresses competitive integrity and ensures greater fairness, specifically in events.
What does today’s fix entail?
Starting today, weight and height will remain locked when you are in an event.
If you are in an event and you try to make a change to your weight or height via your zwift.com web profile, you may be presented with a generic error message. If you try on Zwift Companion, changes will not save, and therefore performance in game will not be impacted.
When can I change my height and weight?
You’ll still be able to change your height and weight when you are logged out of the game, or when you are logged in, but not active in an event.”
The likely reason for the unpolished-looking fix, is that Zwift operates on a monthly release cycle to platforms like the Apple & Google app stores. Thus, doing an out-of-cycle update makes their development/engineering life moderately miserable. So in this case, they’re handling it behind the scenes for now. I suspect in the next companion app update (later this month), we’ll probably see a more polished fix to this issue.
[Note: The remainder of this post remains as of the original publishing]
Going Forward:
It’s easy to pick on Zwift, in the same way, it’s easy to pick on Peloton. Both are large companies that experienced significant growth in a short period, with often a heavier internal focus on sustaining that growth rather than addressing gaps. Both have communities of devoted fans, and yet both have continued to manage to stumble into self-inflicted PR wounds for often unnecessary reasons.
In talking to a bunch of people on both sides of the issue, I get the impression that this situation escalated faster than Zwift realized, and that adults might not have been present ‘in the room’ when the initial ban decision was made. By any logical PR or technical-security standards, there’s no reason this should have ever made the public’s radar. From a corporate communications standpoint, this should have been handled quietly behind the scenes. Certainly, the adults in the room understood the implications of banning a key ZwiftInsider.com contributor, especially over something ultimately as trivial as pointing out a bug? Zwift has both a very competent external PR agency/team (in my direct experience) that’s well regarded as one of the best in the industry, and they have (also, in my direct experience) a very competent internal PR team. I don’t get the impression either had been engaged this time until it was far too late. Now the situation has escalated to waves of people posting screenshots of them canceling their accounts on Facebook, Reddit, and elsewhere – in support of Luciano.
And from a technical standpoint, certainly, the right public response from any competent engineer would have been “Wow, thanks for pointing this out, we’re gonna escalate this quickly with a temporary fix, and then a longer-term fix”. No matter how frustrating it might have been for said engineers to see the clickbait title that Luciano wrote that triggered this avalanche, that doesn’t remove the technical issue that was the true foundation for the avalanche to occur.
Either of those two groups should have prevented this from occurring in the lead-up to Zwift’s biggest event in the last few years. And ultimately, as it stands now, the longer Zwift waits for Mea Culpa, the more media attention this is going to get. And certainly, some of those media are eventually going to ask the next most logical question: “Will you ban my account the next time you don’t like our article title”?
On the bright side, Zwift’s Chris Snook did confirm a fix is on the way and that Zwift themselves is able to detect this specific cheat for this weekend’s UCI World Championships. Further, a fix seems more imminent than previous statements from Zwift that were saying “long term”, with him noting that it’s actively being worked on now, going on to say they’ll provide an update as soon as it’s implemented.
Of course, the problem is – it shouldn’t have taken this giant kerfuffle for that to get a fix for this. It should have simply been just a normal day in a software company. And the fact that it wasn’t is more of an issue than the title of a post.
With that, thanks for reading.
FOUND THIS POST USEFUL? SUPPORT THE SITE!
Hopefully, you found this post useful. The website is really a labor of love, so please consider becoming a DC RAINMAKER Supporter. This gets you an ad-free experience, and access to our (mostly) bi-monthly behind-the-scenes video series of “Shed Talkin’”.
Support DCRainMaker - Shop on Amazon
Otherwise, perhaps consider using the below link if shopping on Amazon. As an Amazon Associate, I earn from qualifying purchases. It doesn’t cost you anything extra, but your purchases help support this website a lot. It could simply be buying toilet paper, or this pizza oven we use and love.
Having reported a few bugs in my time some companies are brilliant and will feed back about what they intend to do, others are so belligerent it makes you want to write click bait articles so somebody will listen.
Exactly. Zwift should have swallowed their pettiness and oversized ego and just said “thank you for bringing this issue to our attention. Our software team is working on a fix to address this issue” it’s like amateur hour with their PR team. This isn’t hard. Trying to scrub the internet of references to the original post predictably backfired and is hilarious.
I promise there was nothing belligerent in our process. We all love (loved?) Zwift and thought we were doing something to preserve the fairness of the competition. We did not look at it from an “exploit” perspective as we did not even know exactly what the term meant.
And with that…onto the next Zwift ban dilemma:
link to rad-net.de
“BDR bezieht Stellung: Sperre von Schallau nicht gerechtfertigt” = Official German cycling foundation says zwift ban is not justified
(after validating power themselves)
And 1400 posts of mostly gossip starting here:
And link to forum.tour-magazin.de
Yeah, that one is a giant pile of mess.
The official ban here: link to content-cdn.zwift.com
Would love to read your take on this.
Her outdoor speeds do not match her power numbers. Either she has the CdA of a house on flats and rides a 30kg bike up hills, or her power sources are off… And what are the chances that a smart trainer (KICKR) and Assiomas both being off by the same general percentage without malicious intent/tampering?
Quite frankly, she got caught because of unrealistic gains in 8 months. Her threshold power went up about 35% while her HR went down >10bpm. That doesn’t happen. If you get more fit, your LTHR goes up, so when she was doing 5.75w/kg for about 40 minutes, that was a sub-maximal effort…supposedly.
It’s laughable if a national federation is backing her. I propose a simple test. Give her a nice light road bike and send her up a local climb around 30min. Heck, 20min, 15min or even 8min would be fine to get a sense of her aerobic power potential. Have independent parties follow her attempt in a car with a stopwatch.
Signed,
The guy who submitted the weight adjustment bug report to Zwift over a year ago.
link to content-cdn.zwift.com
Yeah, I’d say my general take on the whole thing is pretty much what Eugene says. And honestly, what Zwift says.
I’ve investigated – in-depth – every single one of the bans thus far. Though, I haven’t selected to write or publish on all of them. In some cases, I even spent considerable time with the ‘defendent’ understanding things, trying to peice it together, etc… And in each case, when push came to shove of me proposing a very specific test (depending on the scenario) that would exonerate them, they got quiet and stopped responding. Thus, in effect, Zwift was correct.
Which, isn’t to say I think Zwift has handled these all well. In fact, the bulk of the cases – especially the women’s cases (for whatever reason), have largely hinged on Zwift’s inability to do dual-recording internal to the app itself, which have generally caused competitors to try and fake a secondary file when something when wrong technically. Something that was totally innocent, and would have normally resulted in just a simple DQ.
Yet another year later, and Zwift still requires people have 3rd party devices record a 3rd party power meter. And hope it all works. It’s ridiculous, especially given their infrastructure already supports dual-recording.
According to a podcast with German Federal Cycling organization after the ban 2 test sessions were run (including a 20 min test) and the tests confirmed the riders performance. The data was sent to Zwift and Zwift didn’t reply.
I’m far away from all participants but in this case at least the communication from Zwift (no reply) was lousy as the requested independent check was made and data was provided.
Yeah, maybe there’s a language barrier here somewhere, but the entire test procedure is hard to understand. It implies that she rode her KICKR+Vector 3 pedals and that they were calibrated under supervision, and matched an SRM. Which, is fine, but then why wasn’t the SRM data used? And did she put her pedals on the SRM Ergometer? Or are they referring to some other pedals that were previously tested?
And then if so – why was the test on her KICKR and not just done on the SRM – if that’s trusted? Frankly, I don’t think anybody cares about a test of her KICKR+pedals. Though, tricking the V5 is substantially more difficult than previous models. Except, in this case it says the power source was Vector 3. Though, it does say those were calibrated under supervision, so that gets back to the question of whether or not they calibrated *HER* pedals, or just using a past historical. Again, the translation is wishy-washy.
Also, I don’t understand why they were doing a 20-minute VO2Max test, when in reality, Zwift and the UCI lay-out a very clear test that athletes need to do to prove things, validity-wise.
Finally, I wish they had included the actual data (with label sources). Like others have said, her numbers are insane if true. I just wish the entire claim by the federation was backed with actual data.
Like anyone else, I want to believe in brilliant out of this world cycling wattage performances. But, like most others over the last decade or so, I’ve come to the realization that they basically don’t exist in real-life. Thus, the bar for proof of those is substantially higher and more detailed.
(And without actual data and very clear outlining of a procedure, I don’t care who certifies something.)
I can answer some of the questions (by listening to a podcast which was held with one of the BDR participants of the tests and hopefully I got all details correct):
– 2 tests sessions were run on seperate days
— one seesion where a 20min FTP test was done (it sounds like some parts were run in the saddle and other parts out of the saddle and the out of the saddle numbers where significantly higher)
— another session was run where V02max was tested
– the tests were run on a SRM device AND with the Garmin pedals; the Garmin pedals were within 1% of SRM (which proved that the pedals were not the source of the problem)
– the athlet is only training for some 2 years seriously and in this case significant improvements are still possible
– the riding style (mainly out of the saddle) is “non standard” and cannot easily be compared to outdoor performance as indoor areo doesn’t play any role
– according to the podcast the full data was sent to Zwift and no reply was received from Zwift
P.S. I (like many others) would really like to understand the whole subject better and do not know why a detailed report was not made public
Thanks – appreciate the clarity there.
It’s interesting on the Zwift “not responding” comment. I’ve heard this multiple times from those accused of cheating. Actually, almost every time. Though inversely, every time I’ve asked Zwift about it, they’ve pretty clearly demonstrated that they did actually respond – it’s just that the accused didn’t like the response. I don’t really understand this either, as perhaps this is also a language barrier thing (virtually all of these were with non-native English speakers).
I do agree that newer athletes can make gains, though these are insane gains by any measure – even for a newer, undiscovered athlete. I’m also slightly surprised they’d have tested her back to back FTP test with VO2Max the next day. Given that would have hurt her case a fair bit. Maybe it was just a timing/logistical thing. The standing during an FTP test is generally considered a no-no, but honestly, that’s also kinda more for trying to normalize the data from a training threshold standpoint. At the end of the day, it doesn’t matter how you nail the wattages – as long as you can repro them.
Maybe I’ll give a casual volley poke at Zwift on it. Each time I investigate these, they end up taking a boatload of my time, and the end-resultant is that when push comes to shove, the accused/convicted always seems to stop short of providing the actual final needed data at the end of the day. Thus rending all my work a waste of time.
Thanks for your interest and actions to help all of us to better understand what really is going on!
Language is most often an issue and also culture and inexperience with technology and communication.
The 2 tests were NOT done within 2 days but on 2 seperate days. I do not know what time period was between the 2 tests; the second test was mainly focused on VO2Max and most probably can be ignored for the wattage/FTP part of the story.
This athlete (like some other woman) is using a relative low cadence; in the Women Zwift World Championship I was surprised to see this pattern (low cadence, low BPM, high wattage) at a number of riders.
Okey doke…digging complete!
Based on the info I’ve gathered, specifically regarding the German Federation test, there are some very substantial technical and non-technical concerns with the execution, data files, and oversight of her test that are being sorted out. Once things get sorted I may consider circling back on the topic, if it ends up back in the Zwift camp.
In other words, we’re gonna be in a holding pattern for a bit while peoples sort some things out. But, the topic is very much not dead.
(Oh, and welcome to being a new DCR Supporter! In fact, don’t forget to check out the latest Quarantine Corner episode that just went live about an hour or two before you joined. It’s linked in the welcome e-mail. Cheers!)
Great summery!
My take on it: Zwift has known about this for a long time but the fix isn’t easy because their code is still a giant, steaming plate of legacy spaghetti. If they were using an off-the-shelf game engine, they’d have hot-fixed it in a day.
Disclosure: I don’t find Luciano’s approach particularly mature – he comes across as “does not play well with others.” Doesn’t make Zwift’s response any better – as you put it, it sounds like the adults weren’t in the room for some of their decisions, until things blew up.
There are off-the-shelf game engines that have code to implement user inputting their real life weight? Never knew of such a thing.
The thing is Zwift is a company that deals with the public, they have to deal with all sorts of types of people all the time just as a matter of daily day-to-day business. Some of those people are going to be “does not play well with others” types. So to play this off as a 50-50 kind of deal is really a little hard to swallow.
It is a tricky question of responsible disclosure. If you find a security bug and have no reason to believe it is “in the wild” then 90 days is the standard and reasonable policy before going public (exceptions can be made in very special cases, but this wouldn’t have been one of them).
When it is being actively exploited in the wild things are a bit different – Google’s Project Zero has a fixed 7 days.
However these are security bugs where the impact is greater than impaired enjoyment for the typical user. In this case it is not a security bug. I’d probably classify it as “unintended consequences of a legitmate use” as the CA allows as part of its normal function to change weight (and height) without any sensible limitation. The underlying reason is that the platform is trusting user input without any validation or sanity checking, and this is a schoolchild error for a professional software developer.
I don’t think Luciano did anything unethical – Zwift has had loads of time to improve the defective feature (up to four years from reports in the forum thread) and has elected not to do so. Would I have published without telling Zwift? Probably not
There are also other articles online showing people how to cheat – link to zwiftinsider.com – for starters (a partner organisation). Where is the ban hammer for this? The fact that any policy is wildly inconsistent really doesn’t give them a leg to stand-on as anyone reading that article will reasonably assume that Zwift are not unhappy with such publication. While “a foolish consistency is a hobgoblin for little minds” a billion dollar company really shouldn’t be making things up on the fly either.
And as far as click-baity titles go – Nah. If we started censuring people for click-bait then half of the internet would been shut down.
I have to be honest here. At the moment we posted we knew Zwift had known at least for a year and that ZADA had raised the issue to them. We were also told that post race controls were in place for Premium League and certain WTRL events. However, we were not aware at all of the “right way to do it” or the industry standards on it. Now that we know, we understand we did nothing wrong in that sense, since Zwift knew for 4 years and did not fix.
I agree. Oh no, someone is faster in zwift is not a security problem. The servers aren’t compromised and the users aren’t compromised. This type of bug shouldn’t be held to that standard. Making a cheat more widely known can be more helpful in making a label playing field.
If the bug has been around for a few years and ignored making a click bait title makes sense to keep it from being ignored.
I would also not call this a bug, more a design flaw. Allowing users to change parameters that impact their speed? Not having logging of every time something happens that could impact their speed? Weight dopping is a very well known concept that they should have considered
This is just using functionality that is already in the game. Look at how others consider cheats in games:
link to research.nccgroup.com
thanks. I think I understand by now some differences, specially we did not manipulate the funcionality, we just used it the way it was designed for…
Yes. You basically did the equivalent of jumping on a teammates head to look over a wall. There is no doing something obscure that gets the game to do something unexpected or doing anything external to the app (intercepting network packets, ant+ packets, modifying the memory of the app, etc)
If you posted an app that took data from a power meter and rebroadcast at a higher wattage. That would be a cheat tool.
Does Luciano have a responsibility or duty to quietly disclose the exploit to Zwift before going public? I’m not in the field, but if an exploit can result in some harm because some noncomputer person hasn’t updated their Windows, then I understand. Disclosing to the public first could really harm some people. But Luciano disclosing this publicly first seems like an entirely different situation as Zwift may be able to fix this – Zwift can somewhat control/log all the companion apps out there. So I don’t see the harm from first publicly disclosing this matter. As I’m not in the field I suppose there may be other rationales for quietly disclosing first but so far I don’t see that here.
If true, the fact that it was publicized years before makes me think that Luciano did the right thing as the effect of his public disclosure will likely be that racers will not use this cheat now.
I’m not in the field either, in fact i’m not even in Zwift having given it up years ago when they made some decision I disagreed with, and I simply voted with my dollars and they never and still don’t miss me which is normal and fair). However Zwifts initial action in this case, and many of the supporting comments, at the very least irritate me and some down right anger me. What I am struggling with is WHO decided what is “reasonable” or a “standard” on what others can post using their own time, money resources and words. Nothing posted was illegal. Nothing posted was treacherous or caused irreperable harm to anyone.
The poster did not design, install, or modify the platform, the code, or anything else. All he did was say the way the GAME is designed if you do x, y is the result.
Too bad, so sad, but the only problem is something that is literally designed into the platform by the owners of the platform and code.
As far as I’m concerned he had every right to write about it in any manner he chose, to publish it at anytime he wanted and did not need to wait for ANYONE to respond to anything.
Anyone that didn’t like it thats on them.
There are probably thousands of ways to do nefarious things within the platform, I look at all the cheating that goes on in gaming. Anyone that wants to do these things do you think banning or shadowbanning an account means anything? Recording or blocking an IP? Freezing an account? People that want to cheat and / or do harm have so many ways around those things its absurd to think that it means anything.
All he did was publish a post, and terms like clickbait and standards and rules are thrown around by who exactly? Self appointed gatekeepers? There is NOTHING official on any of this.
I’m reallly old but this takes me right back to when I was growing up and the moral police wanted to ban records for lyrics or tv shows for ideas or cartoons because a coyote hit the ground. Absolute absurdity. You don’t like it don’t read it or listen or watch it but don’t for a second think that you personally are in charge of it or can make decisions for others.
“Luciano’s choice of titles was at best designed to attract cheaters to cheat, and at worst, designed to raise the profile of such an exploit just days before the biggest event of the year.”
Seems like you got these two things backwards? Raising the profile of the issue so that it can be fixed is a good thing, as it finally gets the issue fixed. Trying to help cheaters is a bad thing.
I’m referring to raising the profile such that some of the World Championship racers find it, and use it. Not racing the profile to Zwift.
A better way to title it for that purpose, that’s still clickbait friendly and almost the same would have been “The Huge Zwift Cheat that Zwift Needs To Fix Immediately”.
Don’t get me wrong, he shouldn’t be banned for this. But pretending that it was all focused on helping Zwift close the gap seems naive, given he didn’t even notify Zwift directly or wait for their response via 3rd party.
Next time I will contact you to write the title. Obviously we are amateurs. We discussed this at length and your title would have covered all our intentions.
By the way, I love the irony (and i am not being ironic here) that you say the title was not appropriate and clickbaiting and you put it in big in the image showing in your post! Magic!
Why else would I have clicked on Ray’s Facebook post?
The focus on the title seems to be a red herring. Zwift brought it up, sure it’s reasonable to mention the title, but to put so much focus on the title when the content of the article make his intention clear seems to be giving zwift’s petty reason for banning him much more coverage then it deserves.
Love it. “You should not have used the clickbaiting title I am going to use now to tell you not to use it”. Pros and amateurs. However, I mean it, DC suggestion of title was way better and would have covered the same intention. Also you have to take into account that none of the three guys with this have english as mother tongue
While I don’t disagree with your point, I think the thumbnail image I selected tried to balance the two sides as precisely as possible into a single image. Also, it was the best I could do in PowerPoint (I don’t know how to use Photoshop).
And yes, as one living in a country other than my mother tongue for the last decade, I completely understand how easy it is, on a daily basis, to get out the precise meaning of what I’m trying to say. I considered mentioning it in the post, but wasn’t 100% sure of your upbringing. Obviously, I’m envious of your language skills.
“that adults might not have been present ‘in the room’ when the initial ban decision was made”
Anyone who has the ability to ban someone from your company better be someone with enough decision making ability to do so in a reasonable manner. The fact that such petty bans are in the culture is scary, even if a “non-adult” handed out the ban.
Yeah, I’m confused by that. This isn’t some objective cheat that is easy to show, this is a ban based on a post on the web. I’m assuming these are done very infrequently so I can see a low level employee flagging it but would need someone higher up to do the ban
It’s only for 30 days. I’m sure he’ll get over it. Lesson learned on his part and for anyone else who doesn’t report these things through the proper channels. Irrelevant whether Zwift knew about it or not. As you say, it’s a big organisation now. These things get lost in streams of emails and Slack messages.
Hello. I am Luciano.
Your article grabs the essence of the issue. Thanks for that. Sorry for the long explanation coming now. I will speak from the heart more than from the head, so it might sometimes sound cheesy 😊
I am not saying we did things 100% right. In reality we did not really know how to do it. We are not bug hunters, just three guys racing in ZRL. None of us gamers. I am 45+ average Zwifter guy who discovered Cycling during the pandemic.
One of us was suggested the cheat as a way to improve his performance as if he would be stupid not to do it. That’s how all started.
The title, yes, was to bring attention to it. Two reasons.
1) Because we already had evidence that the issue had been rugged under the carpet previously at least for a year, now we know it is 4, so we wanted to raise attention on it.
2) But also we wanted cheaters to click on it so they would know Zwift is looking at it and would chase them! Like “party is over”
On the “spreading cheating” aspect. We asked ourselves the same question, and came to the conclusion that most of the people with the intention to cheat may already be aware, but above all knowing about the cheat does not make you a cheater. Using the cheat makes you a cheater. By the way I was contacted in numerous occasions by people telling me I knew! Allow me the analogy, not everybody given a weapon becomes a murderer… The perfect illustration is the friend contacting me when exposed to the cheat. He did not use it and was upset about it not being solved.
In any case, with such a long time to solve the issue, we believed it was Zwift accountability to solve it, specially when they are claiming they are promoting fair competition.
Again, not saying right or wrong here, just explaining the reasoning.
Just for clarification as some inferred some financial interest, the post was hosted to a wordpress blog that was made in three minutes for that purpose and will never be used again. No ads involved, nothing revenue related.
Our intent was all in good faith as we were seeing that other people were already using it and we could see it in some specific situations. It was simply unfair competition.
I post regularly at Zwift Insider, but I don’t even know how many people read the articles there. I don’t really know the audience reached. Even an order of magnitude. Genuinely (in a good way) we did not anticipate what it would trigger. The order of magnitude. We never thought this would grow to be considered as something relevant or even shadowing the World Champs. The only reason why we were considering the World champs in our reasoning was: hopefully it is an easy fix and they avoid people cheating. With the perspective… that was genuine in a bad way.
Since we are not bug hunters, we did not expect a reward, just the bug (I had to search for “exploit” to understand the difference with a “bug”) to be fixed.
I also believe this crystalized some longer frustrations from the community in the way they / we perceived are treated.
What I did not expect either is to be shadow banned. But hey, there are some other way more important things happening currently. Being shadow banned from Zwift is absolutely irrelevant.
If we had to do it again, yes, we may have done some slight things differently. There is a learning process in everything I guess. Now, I hope I will never ever have to publish something similar again. I am way more comfortable writing comic posts at ZwiftInsider on how much I sweat and suffer during the races!
I love Zwift community. I am (was?) an enthusiast Zwifter promoting the platform everywhere I can, contributing as much as I can and sharing as much as I can.
There is nothing I would hate more than harming the community which has been a life saver for me during the pandemic.
If anyone felt bad about this I apologize (I think the two mates embarked in this with me would say the same), however from the feedback and reaction of the community I believe we were globally right.
Regards,
Luciano
This just seems to fit within the general operating ethos of Zwift. There are flaws that, after years, just continue on.
Weight doping, in many flavors, is a continual problem in Zwift.
It’s almost like Zwift really enjoys keeping elephants in the room
“You learn something new every day,” like the words instantiation and whataboutism, and it’s only 6:15 AM here.
Once you see whatboutism, you can’t unsee it for the rest of your life.
I know Spanish and Chinese as second languages with English at my mother tongue. I’m fluent in Spanish, less so in Chinese. In Spanish, one might invent quetalsifuerasismo as a neologism translation.
I have no good idea about the Chinese. Maybe 如果事情主義, which you might shorten to 如事主義 as a neologism.
But whataboutism in English is truly non-unseeable forever.
Anyone who wants to do a both sides thing like “zwift shoulda … but also Luciano shoulda …” needs to explain why the reporting of the bug to zwift this time would be any different then all of the other times this bug has been reported to zwift over the years.
Exactly Jep. No kerfuffle no fix. Good work Luciano.
Anyone that got to Worlds using this is now too scared to use it as it may be true they will check the post race data more closely now.
You really can’t enjoy zwift if you care if other people cheat.
Great summary, as usual! Perhaps Zwift will take a moment to reflect and release an expanded apology/statement and update their internal processes for bug/issue/security fixes and any user corresponding actions.
They’ve been doing this long enough you’d think they would be better at it.
Just a quick update for those following along in the comments section – Zwift has issued an apology, rescinded the ban, and outlined next steps for a bug bounty program. I’ve added a section above: link to dcrainmaker.com
I don’t understand why it took this long to arrive at this point. I mean it was relatively quick, but once you start getting emails from DC Rainmaker and Zwift Insider wouldn’t you immediately escalate things to the “hmm, let’s reevaluate our decisions” stage? I guess I’d want to be 100% certain that the ban was going to stick before I sent a response to leading voices in the community. Otherwise, it looks makes it look like the only guy at Zwift with a level head is Eric Min.
Probably took someone from the board to slap them upside the head.
I suspect Zwift was hoping this would stay under the radar. I started digging on Thursday evening with a slate of questions for Zwift, but I didn’t quite have my details/ducks where I wanted them till very late last night when I finished the bulk of the post at 2:30AM, but don’t tend to like to publish my 2AM work until I’ve had to read it post-sleep.
By the time I woke up this morning, CyclingTips and ZwiftInsider joined the existing Road.cc fray. I think each article, in theory own way, probably pushed Zwift closer to changing their stance.
And I also suspect they didn’t want the entire chat session on their stream tonight (for the World Championships) to simply be about #FreeLuciano (which, it most certainly would have been).
There’s an update at the top of link to zwiftinsider.com new. Zwift seems to have come to their senses.
Ye, after their boss caught wind of it and gave them an earful (hopefully) …
For the the record. I was just reinstated.
Fantastic news Luciano, I’ll go unfreeze my account… keep up the fantastic work, I love your articles on ZI.
Thanks Chris!!!
Ray has The Force
The bug bounty program will probably take a while because they will want to fix all the easy to find bugs first so they don’t have to pay out for them.
Reads a bit snarky, whereas I would say that is a win-win situation in reality.
The bug bounty system once they get it up and running is a win-win situation? Yes, definitely. I’m just predicting that it won’t come in time to meet Ray’s definition of a reasonable timeline before it gets up and running. If this easy to exploit bug has been around for years what other stuff is still lurking? If I was in their position I would want to fix everything we know about and then do an audit or some probing to try to find some other problems so we don’t have to pay out as much, both so that we don’t have to pay out as much money and to save face so we don’t have a lot of bugs reported by the public. Not really snarky, just realistically what I think will happen.
Bug bounty pay out will be in drops, (or XP if you are at L50).
Do we not think this points to Zwift’s “point/deflect the blame to anyone else but Zwift” culture though?
First we had,
– “no it’s your hardware, not us” issue, then there was
– “well, instead of fixing our system and using something like ‘Verified by VISA’, we will kill off all gift card purchases”, and now,
– “how dare you make public something that has been known a long while, while we haven’t done anything about it!?”
And those are the ones I’ve known, and I’ve been widely known to live under a rock…
I wish people didn’t compare this to security exploits. This is a cheat in the game, nothing more. No one’s computer becomes unusable, no disclosure of data, no one gains access to other systems. Will this appear on link to nvd.nist.gov
No
I was pretty clear above that I didn’t see it as a security issue, and also discussed this very topic in the post.
While I agree, don’t people win prizes/$ in these official races? I don’t know, I just use Zwift to train not compete. But if there are awards, maybe not a security exploit but one that can have bigger consequences than a game.
You did mostly imply there is an obligation to notify the vendor and wait for a response before publishing online which is very much like a security vulnerability. All this is is a way to cheat by weight dopping in the middle of the event. Would it be great for a vendor if no one publicly disclosed any bugs their software had and only privately disclosed it? Sure. The reason for security vulnerabilities to be treated different is to protect innocent users of the product from a vendor not to help protect the vendor
“You did mostly imply there is an obligation to notify the vendor and wait for a response before publishing online which is very much like a security vulnerability.”
Yup, and in this situation, I think there is an obligation to at least attempt to contact the vendor before publishing.
“The reason for security vulnerabilities to be treated different is to protect innocent users of the product from a vendor not to help protect the vendor”
As noted in the post, that’s the challenge here. This is grey area. Many (including the UCI, and all the racers racing tonight in the UCI World Championships), would argue the security of the race is at stake. In terms of the validity of the prize money, the validity of the title, and the validity of each of those racers time, money, and suffering spent to get there.
So is it a breach that impacts every Zwift user’s security? Not really. But is it a breach impacts the security of the race, and ultimately, everything it stands for? Absolutely.
If Zwift truly didn’t know of this issue (which, to be be clear, I don’t believe – at least, someone there undoubtedly knew of it), then in that case, I can see why Zwift would be incredibly frustrated by what was essentially a zero-day exploit.
Zwift is a game. This is a way to cheat in the game. Cheating in zwift should be treated the same as cheating in any other game. There are other eSports tournaments. Sure the level of physical fitness is different in those that compete but the burden on cheating is the same
Great article Ray! I really enjoyed this one. I have a gut feeling that most dedicated software engineers could cheat at Zwift if they really put their mind to it.🤷🏻♂️
smokin’ gun
Oh wow! Thanks for posting. Really makes Zwift look bad with their claims that this was “relatively unknown within Zwift” when their low level support agents have a canned response for it. Either the support agent was correct that the issue is known. If the support agent messed up and confused the issue with something else, or just didn’t put any effort into it and gave you a canned response. Either looks really bad on Zwift! Not taking these problems seriously enough so that you just give back canned responses and don’t track them is bad (and for users having other non-cheating related issues it also shows that support doesn’t take them seriously and just writes back canned responses if true). Having it be widely known within Zwift directly conflicts with what they’ve said.
I wonder why this issue even exists in the first place (regardless of Zwift being involved or not being involved into official racing)?
It seems logical that whatever changes you make to your profile should only be applied after the race/ride/workout/whatever, not in the middle of it. I’m very curious, who in Zwift thought it is a good idea to allow weight changes mid-ride (I’m pretty sure there was the whole discussion with software architects and other important people that decided that this feature is logical and required by people)
“I’m very curious, who in Zwift thought it is a good idea to allow weight changes mid-ride” – probably just an oversight originally. It was fixed in the game client about 18mths ago IIRC for events (you can’t change your weight during an event). Doing it via the CA is just another example of the very poor QA and testing Zwift do
I agree. When they first created zwift this kind of issue was not even a thought. You could change bikes without even slowing down way back in the day until they stopped allowing that. So the game had the ability to make changes to pretty much anything at any time and then that had to add limits to that as they went, this is one where they knew about it and just let it be because they thought so few people knew about and they had other fish to fry (new features to make, new maps to make, etc). Of course this is the problem with trying to keep things a secret, invariable they get out, so that’s why all serious software now proactively tries to fix issues like this.
I’ve reported cheating bugs to a different company in the past. What I discovered was that, at the end of the day, they are like any other business in that they exist to sell a product and a bug has to hit them square in the wallet (in one way or another) to get their attention. In my case with this other company I was either ignored or told, very adamantly, that their users don’t cheat. They make indoor bike, ski, and rowing equipment as well as outdoor rowing equipment too. They also do online/indoor racing. If you think your users don’t cheat, or at least exploit the way the system is designed, then I’ve got some real-estate to sell you. :)
So, when Zwift fixed this, if one would say, just not update their Zwift companion app and would still be able to keep using said exploit? Asking for a friend, LOL
Does this mean we will finally see the back of Zwifters like the guy who got stripped of his title after cheating, and who is still very much a competitor. We’ll see if he actually competes in the worlds,
Why would this affect anyone who actually cheated? Totally unrelated.
I just hope they don´t fix it so I can continue to use it as an excuse to be an snail in the game because the rest is using it!
I´ll take my coat… :p
As someone from the cyber security world, why try to analyse the title (you haven’t known Luciano for years, so don’t bother) ? Step one, did it get results? Yes, If the CEO of a company has responded, that is huge, if it was initially a steaming pile of xxxx, that’s not great but the outcome has been achieved. This “win” is not the norm, so be thankful.
Agree. The title is a red herring that zwift put out there that Ray fell for, giving it way too much time and space here. “We have an exploit that we’ve known about for years without fixing? But the guy who published it used a clickbait title! He’s the real villain here!” For big security vulnerabilities (not quite the same as this as has been pointed out) making an _entire website_ is now the norm, ie link to heartbleed.com
I’m not sure how I “fell for it”, since my entire post basically concluded with:
“And ultimately, as it stands now, the longer Zwift waits for Mea Culpa, the more media attention this is going to get. And certainly, some of those media are eventually going to ask the next most logical question: “Will you ban my account the next time you don’t like our article title”?”
Which was me, pretty clearly, saying “If Zwift can ban people on a title, that seems wrong and dangerous.”
And, if folks want to compare to to the security world – that’s very much the wrong comparison to make here if you’re trying to argue for Luciano’s support. Because rule #1 of that is to notify the company. That didn’t help by Luciano.
“Fell for it” in that you gave the whole title business a lot of time and space in your article, and you continue to use the title to argue in the comments. Giving an argument time and space legitimizes it, and in my opinion gave the title thing more legitimacy then it deserved. The title doesn’t matter if his intention was to get the bug fixed, which you do agree with.
Where there are video games, there are exploits. Can’t believe they do real races on this platform! Lol. Looking forward to the next one :popcorn eating gif:
I 100% know for a fact that I turned in this bug in late 2018…..I changed my weight up Alp De Zwift and changed my weight at the top of alp de zwift and made a mockery of racing while staying in the “cat restrictions”. I found it by accident going down Alp De Zwift wanting the 100KM badge. I then changed my weight back at the bottom. That is when I used it on a couple climbs in order to make things easier for me to get other badges. I only used it in 1 race as I wanted to know if it was possible. You can also use it on group rides which I also tried a few times. I wrote up a long drawn out email and never even heard anything back. I had no idea the bug/cheat was still active. I thought for sure they would of fixed it as I gave them step by step instruction. Oh and there are more bugs out there…..I know of a couple others that help you get badges and turned them in using a burner account. No way I am ever going to turn in bugs/cheats on my account. I dont trust zwift
If any Zwift coders are reading this is it to much to ask for you to add a data field AVG WATTS for a ride.
Thanks in advance.
Instead of banning (or shadow banning) Luciano, Zwift should be offering him a job
Not to confuse this with other comments about post titles, but…the title of this post does not seem, to me, to use correct grammar: Zwift’s Bans Cheat Whistleblower: A Deeper Dive Into the Issue. Maybe drop the ‘apostrophe s’?
Ahh, yeah, I had tweaked the title at the last second and as always, screwed it up leaving the apostrophe. Thanks!
Ray, while I presume you are trying to be as non biased as you usually are…may I ask what your financial arrangement with Zwift and their partners is? I know you almost always test products and reveal this information, but in any case, would you say it’s safe to assume that you benefit from Zwift as a platform continuing to expand (or any sports tech continuing to expand)?
The second question is perhaps a bit rhetorical, but I ask it because it doesn’t feel like you took a hard stance here when from my perspective, a zwift user reported openly on a known bug that Zwift has chosen to ignore for years. Frankly, Zwift has a massive track record of ignoring such bugs (or lacks developers competent enough to do anything about it). Every time I or those I know have reached out to zwift directly regarding other users cheating or a general bug, it’s effectively been ignored, either owing to Zwift’s utter indifference or explicit desires to maintain their public image at all costs.
Eric’s emails all read like utter nonsense. He’s explicitly lying. The notion that they just heard about this bug or that they have a working process in place to address users concerns over cheating etc. is patently false.
This seems pretty cut and dry to me…Luciano posted something publicly (which last I heard he’s allowed to do, we more or less live in a country where the first amendment still applies), and I presume he did so because at this point the only way to make Zwift do anything other than market their game (seriously, it’s staggering how bad the entire engine is compared to other modern video games).
I’m a bit perplexed why you wouldn’t just be entirely on the side of the whistleblower given how much you follow the product. Kudos to Luciano for trying to incite change. He didn’t follow protocol, fine, but the protocol didn’t work and he was left with no choice, so your claiming that he is even kind of in the wrong here is a bit like claiming Snowden should have kept his mouth shut. If that’s how you feel, we can agree to disagree, but this post seems a bit out of line with your general unbiased commentary to be perfectly honest.
Cheers.
I don’t have any financial agreement with Zwift, or any other company I write about. It’s as simple as that.
“I’m a bit perplexed why you wouldn’t just be entirely on the side of the whistleblower given how much you follow the product.” .. “He didn’t follow protocol, fine, but the protocol didn’t work and he was left with no choice, so your claiming that he is even kind of in the wrong here is a bit like claiming Snowden should have kept his mouth shut.
Except, he didn’t follow protocol here. That’s my point. Not only did he not follow protocol, but he published an article with a title, subtitle, and intro all focused 100% on how this was the greatest cheat of all time. It took until the 3rd or 4th paragraph till we got to him asking Zwift to fix it.
Which isn’t to say he should have gotten a ban, but to pretend that he even tried to contact Zwift is silly. Not only did he not try, but when Eric (of ZwiftInsider) told him to wait just a second for him (Eric) to contact Zwift, he went ahead with it anyways. Eric not only knows everyone at Zwift, but could have easily escalated this to the other Eric (Min). None of that happened.
Don’t get me wrong, I disagree with how Zwift handled this – quite a bit. And I think Eric’s response hints to something when he said it “wasn’t widely known within Zwift”, which is short for “it was known, but buried/closed in a database without the right people knowing about it”. Luciano had all the tools at his disposal to ensure “the right people” knew about it. It would have taken mere hours, or at most a day.
Nobody here – NOBODY – is saying he “should have kept his mouth shut”. Instead, most of the people with an IT/security background (such as myself) are saying he should have at least tried to notify Zwift. And while Zwift generally ignores every other bug for years, in this scenario, Luciano had effectively direct connections to the CEO of Zwift via ZwiftInsider, and didn’t use it. If – after that, this wasn’t fixed. Then go forth!
“Except, he didn’t follow protocol here. That’s my point.”
The protocol didn’t work. so you’re point is moot.
“he published an article with a title, subtitle, and intro all focused 100% on how this was the greatest cheat of all time. It took until the 3rd or 4th paragraph till we got to him asking Zwift to fix it.”
Nobody has questioned (including you) that his goal was to stop cheating, so why do you single out specific parts of what he wrote? If someone was trying to make the case to get something bad fixed, why wouldn’t they make clear just how bad it is right up front? Otherwise it could just just buried again. He wanted to get all the facts out first and then ask for what he wanted. It’s a format that perfectly makes sense for someone who wants to get it fixed. Trying to dissect the format to paint him as something he is not is…I don’t know, not good.
“Which isn’t to say he should have gotten a ban, but to pretend that he even tried to contact Zwift is silly.”
He already knew at that point that other people had contacted zwifts months ago. What’s the point of him contacting zwift again and waiting for nothing to happen?
‘And I think Eric’s response hints to something when he said it “wasn’t widely known within Zwift”’
That part seems like an outright lie at this point. Many people have come forward saying that they have reported it years ago. And a prominent member of the zwift community was even told by Zwift themselves of this hack as a way to increase their weight to avoid their power numbers from tripping the “you’re too good, you must be a pro”. (I can send you the link if you like)
“And while Zwift generally ignores every other bug for years, in this scenario, Luciano had effectively direct connections to the CEO of Zwift via ZwiftInsider, and didn’t use it. If – after that, this wasn’t fixed. ”
He already knew that zwift had been notified over a month ago. You conveniently keep forgetting this part (and the other parts I mentioned above) to make your point. I know you know this part, so it’s really disingenuous to ignore it in your reply to try to bolster your position. Disappointing.
“The protocol didn’t work. so you’re point is moot.”
No. Simply no.
He didn’t file a bug. Didn’t attempt to file a bug. Didn’t even wait for arguably the most prominent member of Zwift’s community (Eric of ZI) to at least ask Zwift if the bug exists (since if Eric didn’t know it exists, that’s probably a pretty strong indicator the bulk of Zwift didn’t know it exists). Thus, it’s not that the protocol didn’t work – it’s that he didn’t even try any protocol. Given he was served up the opportunity to connect directly with Zwift’s top engineers/CEO on getting confirmation of this, and said “Nah, I’m just gonna publish” – c’mon…
“He already knew that zwift had been notified over a month ago. You conveniently keep forgetting this part (and the other parts I mentioned above) to make your point. I know you know this part, so it’s really disingenuous to ignore it in your reply to try to bolster your position”
Sigh, reading appears tough today. As I’ve said numerous times. It’s clear that Zwift knew about this. I put it in the original article, it’s been in followups, and it’s been in comments. You keep ignoring this.
What I said, because words and nuance matter, is that Zwift’s internal system is dorked up enough that this seems to not have gotten in front of the right people to fix it. As in, it came in on a customer service database and then got buried. And again, I outlined in the post that technically, there should be processes in place to fix that.
I think people want this weird black and white world where everything fits in a tidy box. That’s not the real world. The real world is full of grey. And I tried in this post to outline all the grey too.
If you want to be upset with me about illuminating all parts of the story – that’s fine. But ultimately, that’s also why you’re here. Because repeatedly across the interwebs people have linked this story and noted that it was a balanced look at things. And more notably, from someone that actually worked in one of the worlds largest software companies, and has far more experience in this type of stuff than most.
Nevermind the fact that literally the *ENTIRE* conclusion section of my post is me slamming Zwift.
“He didn’t file a bug. Didn’t attempt to file a bug. Didn’t even wait for arguably the most prominent member of Zwift’s community (Eric of ZI) to at least ask Zwift if the bug exists (since if Eric didn’t know it exists, that’s probably a pretty strong indicator the bulk of Zwift didn’t know it exists).”
I agree, factually it is what happened. We did not file it.
But where I cannot agree you is when you decorrelate it from the past behavior of the Company. We did not file it for a reason.
A company can refer to its own control processes, where it is judge and jury at the same time, as long as they stand by them and show consistency towards implementation. Meaning the process is effective. In summary you follow a process only if the process is trustable.
Eugene’s ticket (posted here through another comment)on the topic is a deal-breaker of trust in a process. We already knew this, through the Zwiftpower claim that you attached. But this one shows the materialization of the breach of trust in the process.
We had proof the bug had been reported numerous times, we had proof ZADA had reported to Zwifft the issue, and that Zwift did not take action on it.
Plus we had our own Zwifter experiences of reporting many situations of cheating, through official channels, without action taken either. For us it was a clear call.
When Eric Schlange tells me that he cannot publish it and would rather contact ZHQ, I perfectly understand him. We have already decided by then that we will publish it whether Eric contacts ZHQ or not. I send Eric all the info but continue investigating on the side with my own contacts. I do not delegate this to Eric at all, and I maintain him aware of the evolution of my findings.
A few passed and we have many conversations with many quite important stakeholders confirming they have reported the bug to Zwift in the past year without action from them.
By the way you have seen that when you asked them, they did not come transparent and denied to be aware. Which, for a fact, now I know, was not the case. It was very widely known at Zwift very high in the organization and for years and it was rugged under the carpet. I have no evidence nor indication that Eric Min was aware though.
To be fair, I came to learn this afterward, so we did not use this info at the moment to make the decision, but our analysis was 100% confirmed. The process was not trustable.
Jay, the situation at this point in time is: if representants of very important teams and clubs, if ZADA, if other users like us have already followed the official channels of reporting for at least a year and it did not work. Would you trust the process? I can have a side conversation with you if you want or need more details. But with all we had in our hands we did not.
This is only then, Wednesday morning and with all this context, that we seek a way to post it during the day. And even like this what is the channel I choose? WTRL… It is not a coincidence.
I try to post it there. I know that they control the posts because I have posted there many times in the past, plus I love these guys. An admin will look at the content for sure.
WTRL is the entity organizing the most important races for Zwift, an entity independent but extremely close to Zwift, and I give them the possibility to look at it, and revert. And the only answer I get is that my post has been declined. I contact them asking why it has been declined, and the only answer I get (I summarize) is they don’t promote cheating.
We did not come out of the blue and we decided to go rogue about it. Some people are inferring that and I want to be super clear. We are not hackers and know nothing about bug reporting, but we come from the corporate world, in my case in an executive position, and I am used to make decisions. All the time.
There was a whole context around, a lot of information, a personal experience and frustration towards the positioning of Zwift dealing with cheaters. Trust in the process was broken.
If after this, you keep thinking the same, then we will have to agree to disagree, which is a perfect outcome for a super constructive exchange of ideas. I learned a lot through those over the past days and I am also thankful for it.
Thanks for the response.
I think my wording was extraordinarily clear and you didn’t read it correctly though.
You start by quoting my statement that “he didn’t follow, protocol, fine, but…” and then immediately dive into a rebuttal about how he didn’t follow protocol and that’s “exactly your point”.
“He didn’t follow protocol, fine…” is a concession. I’m agreeing with that part. It’s kind of difficult to have any sort of constructive conversation if you lead with disagreeing with a statement that is in agreement with you. That’s called a contradiction.
Mistaking “did for didn’t” aside, my point is that the existing protocol does not work. I have not seen it work in any meaningful way and do not know anyone who has. I’ve personally filed numerous formal complaints through their “protocol” and presented explicit evidence of riders in high level races cheating. Zwift has received evidence of a top 10 ranked zwifter blatantly manipulating multiple secondary power files in addition to a video of that same person using a god damn paint roller in a race to slow down his deliberately miscalibrated flywheel so he could get in an aero tuck. During a race. On video. And that’s about 10% of the evidence.
You know what zwifts reaction was? Nothing. Zero. Why? I can only assume because it’s bad publicity to be banning top ranked zwift riders.
This is the larger point I’m making. When I say Luciano had no choice but to go public with it, I mean that for anything to actually be changed, he would have to bypass the formal “protocol”.
The length of this thread and the general massive reaction from the zwift community (this is in the front page of your website!) effectively demonstrates what he did worked. This, like every other problem with zwift, would get swept under the rug under the guise of “community fairness” and “inclusiveness”.
You know what’s actually productive? Sparking a conversation that needs to be had. Luciano did that, and I think you’re deeply misguided or haven’t dealt with the system yourself if that isn’t abundantly clear. If you think Eric Minn is some sort of saint who doesn’t lie…I’m not sure what to tell you. He has claimed naïveté on numerous occasions when he’s known precisely what the problem is.
Zwift literally employs a team full of grade A morons (led by an old guy with an Astrophysics PhD who doesn’t known how to use a computer) and they consistently fail to address known issues and ignore cheating. We should be thanking Luciano for doing their job. Lord knows Dr. George Gilbert has never done his.
Sorry if it sounds like I’m yelling, that’s kind of just what the internet makes it sound like everyone is doing. I think it’s great that this discussion is being had and that Luciano let the cat out of the bag.
Happy paint rolling.
Thanks CS.
I completely agree that Zwift ignores bugs. I’ve said it so many times over the years, that I think everyone is clear that Zwift ignores bug (and that I think that).
However, my point here is that neither you nor I are Eric of ZwiftInsider. He’s arguably the one person that offered to ask the right engineering resources Zwift (or simply Eric Min) if they were at least aware of it. It’s as simple as that.
As for thinking Eric Min is a saint or such. C’mon. Nobody from a media standpoint in this entire industry has consistently pushed back on what Zwift says, and specifically what Eric says, as much as I have. Countless articles dealing with everything from layoffs, to cheating (and Dr. Gilbert’s program – and their lack of willing to do anything except for Pros), to their new hardware plans – all covering the inconsistencies of what they’re saying. Every other publication is too busy taking their advertising dollars to write even a fraction of what I note.
What I find perplexing here though is that my entire summary section is basically me blasting Zwift. As is the vast majority of the post. Even my update section is me saying – and I quote:
“All too often we’ve seen Zwift promise things down the road, and not deliver on them. ”
I keep seeing the reference to me mentioning the title. Why? Because that’s the *explicit* quoted reason Zwift gave me for why Luciano got the ban. It’s *NOT* for the content. I included Zwift’s exact quote on that in the post.
The problem here is that, again, everyone likes to make this some weird black and white issue. Again, it’s not. Had Luciano simply notified Zwift with it, I’d be fully onboard Team Luciano. Instead, as someone with vast amounts of experience in the software development and securities fields, it’s hard for me to say he’s entirely innocent here.
If he had notified Zwift and even given Eric from ZwiftInsider a single day to get an answer, that’d been totally reasonable for Luciano to post. Else, I just don’t see the urgency if this had been there a year or more, between posting on Monday vs Tuesday.
Thanks for being a DCR Supporter, cheers.
Then you should be fully with Team Luciano.
I wrote to Eric from ZI Friday 18th February with the cheat. That is when he writes to me that he can’t publish this and should contact ZHQ. I share with him the video and he asks me to prepare a doc (the doc is what we finally post) on Saturday 19th February in the morning. I ask Eric in several occasions if he had the chance to look at the document or to share with ZHQ. The day before posting, (Tuesday) I ask again Eric is he had the chance to look at the document and tell him that I have evidence that ZADA has reported to Zwift the issue in the past. I also tell Eric that I understand he does not feel comfortable to post at ZI but in those conditions I will post before the end of the week.
This is only 23 Feb in the morning, meaning 5 days after the first contact with Eric, 4 days after I put the entire post at his disposal, that we publish. Not one or two. Five. By the way I know for a fact Eric was out of town so J am not blaming at all. But just to give the time frame here.
I appreciate most of the article is “favorable” to what we did. Thanks.
But beyond the fact Eric had super enough time to contact ZHQ, first he did not HAVE to, and second we did not delegate this to him at all. Eric is not our spokeperson.
Only one feedback on your process to get the article done from your side. The same way you have contacted Zwift to get their version or comments, I would have appreciated that at least you send me some questions to get the context. I have put at your disposal the original post and the video. But I have the feeling that you have considered Eric was our intermediary or something similar. While si have been in permanent contact with Eric and he has been of tremendous support during the process, I have shared with him maybe 25% of the details. Sometimes not to expose the people speaking to me, sometimes because I understood perfectly that the situation might be uncomfortable for him.
Let’s be clear. If it wasn’t because of you voicing out this with road.cc and cyclingtips the day of the UCI worlds, I am not sure Zwift would have reacted the same, and I am super thankful for it. We are not hackers and did not know industry standards to report this and could have done better. Now we can’t be blamed for not having asked for help and support because we tried for 5 days (not only through Eric at ZI)before we came to the conclusion that we would have to do it by ourselves.
Last one. We were highly suspicious that one team participating in WTRL TTT Worlds on Thursday was using the cheat, that is the reason we decided to contact WTRL on Wednesday and to publish on Wednesday.
Thanks Luciano-
Appreciate the details. And good to know about the gap between initial informing and final publishing. I agree, that’s long enough. As you noted though, one can’t entirely blame Eric for being out of town either.
Though, as noted above, I do think there was still an opportunity to notify Zwift directly, even if that attempt would have been for not. Perhaps nothing would have happened, and perhaps something would have (such as a forum post on Zwift’s own forums, where Zwift HQ staff would have noticed it).
Again – I appreciate all the work you did here in identifying and proving the issue. And as you said above, I think we’ll simply have to have a friendly disagreement on the precise execution of it. But I am glad it all ended well. As noted above, I never agreed with a ban for sure – and made that excruciating clear both privately as well as publically. Looking forward to your usual posts on ZI.
Cheers!
Thanks. And to be clear, I don’t blame Eric at all! He has zero obligation, or responsibility over what I do or say. He has been an immense support throughout all this and the link to Zwift HQ in order to solve it. So, if anything, thank you Eric!
As if we needed further confirmation that, if anything, TrainerRoad should be acquiring Zwift, versus any other directional acquisition, collaboration, or partnership. Leadership matters.
Why all the drama?
Surely plenty of startups have released beta software with bugs before? 😱 🤣
As for Eric Min, who actually believes anything he says?
This is pretty typical for a companies reaction to finding out they did something wrong. The underlings publish/post an immediate reaction (in this case banning Luciano, i.e. shooting the messenger) based on the letter of the law knowing full well the company itself is in the wrong. Then “after review” the head of PR or CEO or some other C level type gets to step in to make nice with the public knowing that the last action the company takes is the one that sticks in the mind of the public.
Consider the Zwift CEO response to, “Not look back.” As if not knowing is beneficial in any way. How many subscribers have used this exploit, how often, or more importantly how flagrantly? When sitting on a mountain of data, use it. Or else don’t pretend the mountain of data exists.
Messy article, Ray. Not up to your usual standards. Lots of opinion (‘Luciano’s choice of titles was at best designed to attract cheaters to cheat…’), little new fact. And ‘with the publishing designed to draw attention’.
When living in a glass house…
My articles have always included opinions. Every single one of them, since the beginning of time.
As for “little new fact”, actually, at the time of publishing, there was a number of new facts in here. Most notably Zwift’s admission that the title specifically, is why they gave the ban:
“Luciano has not been banned for highlighting an issue, it is because he chose to host a WordPress site titled ‘The Ultimate Undetectable Weight Cheat on Zwift’ promoting this exploit and shared this on forums and Zwift community groups (some of which also forbid members from sharing information on how to cheat).”
Being someone new to Zwift Racing and working hard to do a C league – it’s disheartening to think that cheating can happen. It took away the meaning, knowing that it could all be fake. The connections, the competition. Makes one feel foolish, like a kid with a game that has no meaning. I think its imperative that things are fair on Zwift so it has credibility. I just watched the UCI e on it tonight! Is it real for UCI, while its a free for all in my crit? As a UI designer, I know firsthand the ability to turn off the weight change on a simple app is ridiculously easy to accomplish, as its just a UI tweak. No database. No server. Just a simple simple if/then statement.
I hope I can believe in my Zwift experience again.
Hi Molly,
Luciano here. I have published the cheat coming exactly from the same feeling: it takes out the fun of the racing
Now, one thing there is a lot more to Zwift than racing. Racing was being ruined for me and hopefully they fix it. However enjoying Zwift for me is way more about sharing with the community.
I think that even without races I would stay a Zwift subscriber.
Silly me, here I thought pedal politics would ensure that the Wahoo Speedplay post would generate the most heat this week…
Taking the discussion up a level, it increasingly feels like no one can write anything — no matter how innocuous — about Zwift or Peloton without generating heated debate. It’s remarkable that two of the biggest sports tech success stories of modern times have created such passionate camps of defenders and dislikers. Other than perhaps Tesla, I can’t think of another global tech brand that instills such emotion (both pro and con) in people. It’s a fascinating brand challenge.
As someone who doesn’t use Zwift (or Peloton), I have zero skin in the game on this one. And yet, even through I thought the article was completely reasonable, neutral and evenhanded, I had no doubt what was about to happen in the comments.
Now I’m going to sit on the bike with TrainerRoad, a company whose primary contribution to heated debate involves air fryers and the length of podcasts. Thanks, Team TR, that’s (one of the reasons) why I love you.
Tom
I love your takeaway on TR, but it implies you don’t spend much time on the TR forum :P
We have more than a few “hot topics” over there that are long running and quite contentious. We try to keep them pointed in a respectful, constructive and useful direction and achieve that on most days. But there are some interesting times to be sure, so we aren’t free from the tough discussion for those that take part.
I’m just wondering what Nate and team (from Trainer Road) would have:
1. Done when such “bug” was “relatively unknown” in TR
2. Done or said when a post (even with “clickbaity” title) was published about it
We can’t know, but the thing is that I *trust* they would have fixed the “relatively uknown” (seriously?) bug. And of they hadn’t, Nate himself would have “owned” it.
Leadership and culture don’t (?) convince us to buy a product, but they do build (or erode) trust.
¨Leadership and culture don’t (?) convince us to buy a product, but they do build (or erode) trust.¨
Amen. That´s the key point. The root cause.
Not sure ‘bugs’ have risen to the level in TR that this incident in Z, but TR has weathered several notable storms where Nate has been present on the TR forum discussion in particular. You can review them for a glimpse of how he handles challenging times. From memory here are a few of the hotter topics:
1) Dylan Johnson’s TR Review Video
2) Recent TR Pricing Discussion (as well as any of the other prior pricing topics)
3) TR Polarized Training Plans
I am a biased source, but see a massive difference in how TR handles these issues compared to Z. It’s one reason the recent “merger” topic speculation between the two companies worries me. I simply don’t trust Z (despite using them since 2016) or have the faith in their goals and intentions that I see as much better on the TR side. The two couldn’t be more different from a “culture” and user support standpoint, IMO.
I use Zwift, but not (yet?) TR. I find I like races and informal races for training. Not sure I have the patience for just structured training.
Anyway, if I were a TR user, I think I wouldn’t want Zwift to buy TR for the reasons you outlined. You can see this issue and a bunch of long-standing but unresolved bugs as indicators that Zwift’s organizational capacity is stretched thin. If they buy TR, they are committing to fuse the databases, to fuse the user interfaces, to do a whole bunch of things that we on the outside have no idea about. I am not confident they have the organizational capacity to pull that off.
As someone who likes to race on Zwift, I also know that Zwift has considered us to be a minority of users. They have taken a very long time to really act on race integrity for the masses, and they’re just getting started – it took them until early 2022 to start to think about enforcing race categories in their software, it took them over a year after taking over Zwiftpower to even get a single sign on process (and people are still not automatically entered into Zwiftpower), etc. if they buy TR, we are still going to be low priority, but they will have a bunch of additional issues to deal with.
To some extent, I wonder if this is inevitable. as they grow, people keep demanding new content, and so maintenance and the elimination of technical debt goes by the wayside. Maybe it’s not wrong to stick to a smaller and more focused platform.
I love irony. Youve got DC Rainmaker querying why this guy didnt wait for Zwift to respond to his post and asking why he felt the need to publish so quickly….and then you have DC Rainmaker himself who often publishes reviews of new products before the actual companies publish it the same day….you couldnt make it up. Maybe he published it for the same reason you do Ray, his EGO telling him to be the first and bathe in the kudos of it…ffs
I pulled this comment out of the trash bin, where it – appropriately – automatically got sent. Simply because your latest fake e-mail address contained profanity.
Nonetheless, just to clarify, since both this post and all of your other posts have seemed misguided and confused (and also really angry). I’m not publishing “ahead” of any company. That’s laughable.
Companies have what are called “embargoes”, which is a defined time I can publish about a given product. You can read up on embargo times, but in short, they specify the date/time a media entity is allowed to publish. I typically publish at precisely that time. Just like every other media entity from The Verge to the BBC to whomever. That’s the way embargoes have worked for about 80 years. Many times, companies themselves tend to wait 15-30 minutes before they publish their own announcements/e-malis. Perhaps out of courtesy, I don’t know. Either way, the company setting the embargo doesn’t have any insight or say into what I’m publishing, just that I can’t publish until X date/time. It’s really rather simple.
The point being, as with your incorrect technical comments in recent weeks, your obvious inability to understand how fake e-mail addresses work, your incorrect understanding of embargoes, and ultimately your apparent anger, it would seem best to find another spot on the internet. Cheers!
I got shadow banned in late 2020 after a ride I did totalled over a million kms, not that I got any XPs for it but did get the drops. I responded to zwift and provided them with the info they requested plus a lot more info from some “test” rides that I had done, I recall I eventually had to go via the zendesk as they never responded to any of my emails. One of the things I did was a 45 minute Everest on zwift with only having to pedal for about 10 seconds. As a result of this Zwift tightened the minimum and max weights and heights. In short I got shadow banned for life even though I gave them a lot of info which has obviously helped limit cheating on the platform. I feel the lifetime shadow ban was harsh so I closed my account.
You are Neo then, you are the Matrix :)
Banned for life? This is absolute garbage.
“It is important for us to uphold our terms of service as they exist to protect the enjoyment of the majority of Zwifters.”
if the interpretation of the ToS extends to what you say elsewhere, does that mean writing a negative review of zwift, or complaining about a zwift race on twitter, or whatever, violates the ToS? Assuming it has a large enough audience, I guess.
I think the bigger issue is Zwift’s customer support.
I would check my badges status by logging into Zwift on another platform while biking on Zwift (because there is no easy way to check badge status) and lost data. I tried to report the problem, went back and forth with Zwift customer support as they kept thinking losing data was more of a feature or checking status is something I should do before or after I Zwift. Basically my fault I lose data. Their customer support folks knew nothing about software and somehow were unable to grasp the problem. I gave up, will be slow to adopt any of their hardware, and when something better comes along I’ll switch. I think for think Zwift is growing and making so much money that I suspect the CEO would rather make public statements than invest in competent customer support. Cheers,
wb
Yeah, their customer support can really suck sometimes. Logging in to zwift from more than one device at a time is a known problem to zwift, definitely do not do that anymore otherwise you are risking data loss. Of course this is a zwift bug that they should fix, but until they do all you can do it try to avoid it. It’s pretty annoying how customer support these days is so poor they can’t even communicate decently.