On Tuesday, White Ops exposed a sophisticated browser-based botnet called Methbot that was reportedly stealing somewhere between $3 million and $5 million of US-based video ad spend a day.
The story developed quickly after that.
Some large, well-known exchanges have come out to say they’ve had little or no exposure to Methbot. And a few prominent figures called for ad buyers to shift spend to direct channels and private exchanges.
In this follow-up, we round up the latest details in the saga, along with some perspective on how it might affect the industry’s response to fraud in 2017.
In a post on Medium, Brian O’Kelley, CEO of AppNexus, wrote that the company had hosted under $500 in transactions on the fraudulent IP addresses called out by White Ops: less than 0.1% of video spend and a drop in the bucket. Hundreds of millions of dollars’ worth of transactions take place on the AppNexus platform every year.
According to DoubleVerify, less than 1% of traffic across the SpotX platform originated from Methbot IPs.
Rubicon put out a statement that its platform was not affected, and Index Exchange pulled its logs for December and found only 14 impressions out of billions overall that were served against the almost 600,000 bogus IP addresses identified by White Ops.
Some DSPs have weighed in too. DataXu estimates that only .08% of daily spend on its platform in December, the equivalent of about $700 a day, was impacted by Methbot.
But clearly the fraud was happening somewhere, and names need to be named.
“There are quite a few high-profile exchanges that have not commented yet and I’m hopeful they will, but then, who’s left?” said Andrew Casale, CEO of Index Exchange. “It’s probably the smaller-tier players who have the same seat at the table that we all do in programmatic. But the supply chain needs to be evaluated. We can’t continue to assume that every company that calls itself an exchange is implementing the same controls.”
How Much Are We Talking?
So how much money was actually paid out to Methbot? It’s not clear, said Mike Zaneis, president and CEO of the Trustworthy Accountability Group.
“I certainly wouldn’t expect 100%, because not 100% of impressions are bid on, and not 100% of auctions go to completion,” Zaneis said.
Some of the bad traffic was also presumably caught by the anti-fraud vendors out there.
A back-of-the-envelope calculation puts the likely losses at somewhere more in the $25,000-to-$250,000-a-day range, rather than $3 million to $5 million. (The numbers were crowdsourced in response to a call from AppNexus co-founder and former CTO Mike Nolet on Medium, who was looking for some clarity from “the people who are actually writing checks.”)
But even if Methbot was only costing the industry a little over $90 million in hard cash a year at the high end and $9 million at the low end, neither is a number to sniff at.
Everybody Culpa
The details about Methbot have led to an understandable amount of garment rending and finger pointing. After all, even programmatic novices are aware by now that Russian cybercriminals require US accessories to commit their crimes.
“It’s an easy thing to blame someone in a faraway land, but there are also some people right here in the US who are very complicit,” said Ashley Ashford, SVP of audit and accuracy at Telemetry.
Advertisers were hoodwinked into paying for counterfeit inventory, while DSPs and exchanges acted as matchmaker with Methbot.
It doesn’t sit well, said Jason Kint, CEO of publisher trade org Digital Content Next.
“How could intermediaries make money off of this and pass the money along ultimately to the bad guys? Someone passed money to a bad place, right?” Kint said. “Everyone in the supply chain has to be accountable to the advertiser.”
As the programmatic connective tissue between the buy side, the sell side and, unfortunately, the bad guys, exchanges have a responsibility to soul-search, even if it takes a bite out of their bottom line in the short term.
“I look at Methbot as a call to action, because we shouldn’t be tolerating this, even among our peers, and it will damage us if it continues to happen,” Casale said. “Buyers might start to say, ‘I have no trust in ad tech now, so I’m going to put all of my money in Facebook.’ A couple of bad apples could undermine the ecosystem.”
Go Direct?
It’s true. Programmatic and automation are the growing future, but if there isn’t more accountability, advertisers and publishers could get fed up.
“I’ve used programmatic vendors, [and] it’s virtually impossible to diligence that space. It’s virtually impossible to diligence where your campaigns showed up, and almost 100% of the time, your ads ended up running on [Hong Kong-based torrent site] Megaupload,” said Jon Steinberg, CEO of Cheddar, speaking at the Business Insider Ignition conference in early December in New York City.
Steinberg, former CEO of MailOnline, North America and former CEO and president of Buzzfeed, went on to say that he would “never do a programmatic advertising buy if I was an advertiser. I don’t see why I should do anything other than Facebook, Google, maybe Snapchat, and then pick a few publishers that are doing something I want to do.”
Going direct is one solution, said DCN’s Kint, although that doesn’t mean eliminating automation entirely. It just means making the supply chain more accountable to the advertiser.
Digital Context Next launched an exchange in September called TrustX with inventory from more than 25 of its members, including premium publishers like ABC, NBCUniversal, The Washington Post, Vox Media and News Corp.
“The point is that if you go direct and know exactly where your ads and money are delivering working media, you don’t have these issues,” Kint said.
White Ops did find Methbot lurking in some private marketplaces, and it is possible for fraud to sneak into direct deals if a publisher is buying mixed traffic. But there’s no denying that direct deals are cleaner and easier to control overall.
But going direct doesn’t fix the programmatic supply chain, and the programmatic part of the supply chain isn’t going away. It’s growing dramatically.
“People remarked after this incident that they think every transaction should be direct or private and the problem would be solved, which I don’t disagree with,” Casale said. “But that’s symptomatic of a complete lack of trust. We can move a bunch of transactions to private, or we can actually solve the problem, which is that the supply chain shouldn’t be so porous.”
However, not every exchange is equipped or willing to police itself, said Ashford, who claims that Telemetry had identified and silently filtered Methbot traffic since at least April.
Telemetry was in touch with a “major exchange where this particular Methbot was largely housed,” Ashford said, “and told us themselves explicitly that they couldn’t block on IPs, even though we’d been feeding them the IPs.”
Telemetry would send this exchange the impression-level logs on a daily basis, flagging which IP addresses and user agents were invalid, to no avail.
“They’ve taken no action to invalidate the accounts of the people responsible,” Ashford said. “Where is the MRC accreditation for exchanges that follow processes and have accountability in what they do?”
Success Story
To be fair, policing fraud is a work in progress. And the industry is making progress.
The Trustworthy Accountability Group has developed an ID system that authenticates members of the supply chain and ensures that payouts are made to legit companies.
On Tuesday morning, after the Methbot news initially hit, TAG, which is working with White Ops and federal law enforcement, hosted a phone briefing with around 170 anti-fraud and technical execs from more than 130 TAG member companies with presentations by representatives from TAG, the Interactive Advertising Bureau Tech Lab and White Ops.
White Ops also made all of information it has on Methbot public, including the more than 6,000 hacked publisher domains and nearly 600,000 compromised IP addresses, so that exchanges can identify any bids that went to Methbot and trace them back to particular sellers.
That’s the real story of Methbot, said White Ops CEO Michael Tiffany.
“As an industry, we just acted extraordinarily quickly to repel an attack, [and] everyone now has the hard data they needed to identify all of the counterfeit impressions generated by Methbot,” Tiffany said. “Extrapolating our measurement of the daily volume of Methbot across some big timescale misses the point entirely, and raising a hand and saying ‘it wasn’t affecting me’ also misses the point. By acting quickly industry-wide, Methbot is getting stopped everywhere before a lot of money has gone out the door. Impressions clear in real time, but payments do not.”
The cynical view on that? As O’Kelley scornfully put it on Medium: “In the short term, White Ops has gotten a ton of free press for what amounts to yelling, “Fire!’ in a half-empty theater.”
Tiffany, obviously, takes issue with that characterization.
“The reason it’s different this time is because we released hard data so that everyone can quantify to an exact degree how much they were affected,” Tiffany said. “That’s not yelling, ‘Fire.’ That’s saying, ‘There are fires in the following coordinates and, oh, here are some buckets of water.”
