“Data-Driven Thinking” is written by members of the media community and contains fresh ideas on the digital revolution in media.
Today’s column is written by Alon Amit, co-founder and vice president of product at Origami Logic.
Facebook’s recent admission that it miscalculated the average view time of video ads for two years has sparked discussion about the relationship between marketers, publishers and agencies, and the quality of the data used to measure digital marketing performance.
The tenuous relationship and the validity of performance data have become critical issues because marketers are under intense pressure to justify their investments in digital marketing.
But while people’s frustration is understandable, it is important to recognize that these things happen. Getting the metrics right is not easy. The amount of data being processed on a daily basis by Google, Facebook, Twitter and other platform providers is overwhelming, and the people who are writing the code to calculate the metrics are human.
Although mistakes do happen, brands and agencies still need to deal with the consequences, stay abreast of other changes made by platform providers and perform periodic checks and balances.
Common Adjustments
Unlike the Facebook video metric debacle, platform providers make adjustments to their metrics regularly. This happens often, for example, with ad click metrics because over time a platform provider may conclude that certain clicks were caused by spam, some of which cannot be identified instantly, or are otherwise illegitimate. As the number of clicks an ad receives affects spend, adjustments typically happen – and should happen – right before the end of a billing cycle.
Some metrics are also redefined by a platform provider over time. In some cases, after a metric is redefined, the results may be recalculated historically according to the new definition. In some cases, the platform provider may only use the new definition and its results moving forward.
When platform providers make mistakes calculating metrics, they typically restate the results of metrics in question, as Facebook did with its flawed video metric. The restated results may be less dramatic, with some values shifted from one day to the next.
Getting the metrics right is very difficult, particularly since some of the problems can be very subtle and hard to detect.
Keep An Eye On Announcements
Typically, when a platform provider makes a change to a metric, it discloses it somewhere. For smaller, less significant changes, this is usually done in blogs or forums for the technical community.
When the change is more significant, as with the Facebook video metric restatement, the announcement may be made in one of the company’s primary communications channels.
Checks And Balances
Performing checks and balances is the most important step agencies, marketers and publishers can take, but it is also the hardest.
A simple way to start is to make sure some metrics, such as impressions, clicks and spend, add up. For example, when all values are combined for a particular dimension, such as regions, the sum should match the overall total. Not all metrics are additive so this cannot be done across the board.
Perform easy comparisons. Certain metrics should always be greater than related metrics. For example, the number of unique visits to a website should always be less than the total number of page visits.
Compare past results. On a regular basis, fetch historical data to see if results have changed over time. Fetches can be performed with different time windows. To ensure that there are no anomalies with recent results, start with the last seven days of data. The size of the time window can be decreased as you gain confidence with the results over time. Every once in awhile, fetch the data for a random historical quarter to make sure the results have not changed. Keep in mind that there may be some small variations in the results due to adjustments made by the platform providers.
Finally, inform the platform providers of any changes. When there is an anomaly that suggests the results of a particular metric are incorrect, let the platform provider know. They appreciate feedback and freely admit that it is impossible for them to catch all issues themselves.
Managing performance metrics is a lot of work. The effort, however, is needed to ensure the integrity of marketing data. Since marketers rely on this data to make important decisions about how budgets are spent, every metric matters.
Follow Origami Logic (@OrigamiLogic) and AdExchanger (@adexchanger) on Twitter.
