Design and development of a worldwide-scale measurement methodology and its application in network measurements and online advertising auditing
- Autores: Patricia Callejo Pinardo
- Directores de la Tesis: Rubén Cuevas Rumín (dir. tes.)
- Lectura: En la Universidad Carlos III de Madrid ( España ) en 2020
- Idioma: español
- Tribunal Calificador de la Tesis: Marco Mellia (presid.), Marcelo Bagnulo Braun (secret.), Andra Lutu (voc.)
- Programa de doctorado: Programa de Doctorado en Ingeniería Telemática por la Universidad Carlos III de Madrid
- Materias:
- Enlaces
- Tesis en acceso abierto en: e-Archivo
- Resumen
Advertising is in continuous growth, such that we have shifted from the traditional media, like newspapers, radio, or TV, to the big online world. The emergence of online advertising in 1994 marked a turning point in the Internet era. From the first time we saw an ad online to the present day, an ecosystem has been built capable of operating in real-time, automatically and working faster than the human eye can blink.
Online advertising is one of the most crucial factors in marketing campaigns, thanks to its ability to reach a vast customer base at a low cost. Many Ad-Tech companies make the argument that online advertisements provide an effective form of advertising and that such advertisements provide a plausible alternative to newspapers and other types of traditional advertising. As a result of this phenomenon, the revenue growth generated by online advertising surpasses the expectations yearly.
The online advertising ecosystem has access to over 4 billion users around the world on devices like mobile phones, computers, tablets, and televisions connected to both fixed and mobile networks. This ecosystem has set up one of the largest networking distributed architectures with worldwide-scale. This infrastructure may be used for other purposes than solely delivering ads.
This thesis proposes a versatile measurement methodology that leverages the aforementioned worldwide-scale distributed infrastructure set up by the online advertising ecosystem. In particular, we insert a lightweight custom measurement code in an ad, which is distributed (up) to millions of devices via advertising campaigns that leverage the online advertising infrastructure. Each time the ad is shown in a device, the code executes and runs the custom measurement. The code can be modified to conduct different types of measurements so that various campaigns can be used to measure different things. The use of online advertising infrastructure enables our methodology to run worldwide-scale measurement experiments in short periods of time (days). To showcase the great utility of the proposed methodology, in this thesis, we leverage it to address research questions in two different areas: Network measurements and transparency in online advertising.
Measuring Network Transparency, Security and Performance Tens of thousands of ISPs offer Internet access to billions of customers from all over the world. The Quality of Experience (QoE) perceived by Internet users is defined by myriad factors related to the ISPs' network design, regulatory policies, network configuration, and operational decisions. In addition, a large number of research studies have revealed application-level and end-to-end connectivity violations, including traffic discrimination and network neutrality infringements, DNS manipulations for profit, in-path TLS proxies, and traffic manipulation by in-path proxies, for example via HTTP header injection to facilitate advertising and user-tracking. Revealing these manipulations, as well as identifying the culprits, is of significant interest to researchers, regulators, and end-users alike. This has motivated both the research community and practitioners to design and deploy tools to perform network measurements from the edge of the network. The resulting tools leverage dedicated testbeds, crowdsourced measurements, and Virtual Private Network (VPN) services to gather insights into the edge view of the network. While powerful, they all possess inherent drawbacks such as limited geographical and ISP coverage, or short-term experiment lifespan.
Despite years of network measurement and other studies conducted from the edge of the network, pervasive access to the network edge in order to facilitate measurements has remained elusive. To close this gap, this thesis propose AdTag, an approach that leverages online advertising to launch network measurements at a global scale, in a time- and cost-effective manner. The nature of online advertising services make them an ideal, yet underused, distribution channel for launching rich network measurements either globally, opportunistically or focused on specific regions using the targeting mechanisms provided by online advertising service.
The goal in this work is to take a step back and consider the experimental apparatus of JavaScript-enabled ad placement and explore its broader feasibility for network measurement.
• Measure DNS Global Infrastructure Understanding the global infrastructure of recursive DNS resolvers, their behavior, and users' DNS choices is critical to identify common mistakes and inefficient deployment strategies that can degrade users' web experience, security, and privacy. The research community has devoted important efforts to study infrastructural and performance aspects of the DNS subsystem.
However, previous measurement methods failed to reach the fundamental scale, openness, global coverage, and reproducibility requirements to characterize the DNS infrastructure from the edge of the network.
For this purpose, we adapt AdTag to overcome the limitations found in previous DNS measurement methods. For that, we use the rich suite of networking APIs and capabilities offered by modern browsers to develop JavaScript-based DNS measurement scripts that trigger a DNS resolution process with an authoritative NS under control. To gather empirical data at a global scale and in a timely manner, we distribute the scripts through online advertising campaigns which also enables performing targeted experiments in regions of the world that were typically underrepresented in previous studies.
We distribute the JavaScript-based measurements using two small ad campaigns. With a $450 USD budget ---a relatively low amount for online advertising campaigns--- we could run 3.8M DNS measurements from 2.5M public IPs (including both mobile and desktop users) distributed across 1M /24 IP prefixes from 25k ASes and 178 countries.
These experiments allowed to identify 76k IP addresses hosting recursive DNS resolvers across 49k /24 IP prefixes and 14k ASes. The pool of IP addresses provides with large-scale data of the global DNS infrastructure deployed both by ISPs and third-party DNS providers, as well as unique information about end users' DNS configurations.
This analysis reveals new findings about DNS recursive resolvers not reported so far:
• A significant percentage of Internet users resort to third-party DNS providers. Namely, Google, OpenDNS, Level3, and CloudFlare are responsible for ~13% of the DNS requests. We observe a notable increase in the use of third-party DNS providers by users accessing the Internet from countries reported to implement state-level censorship and mass surveillance. These results suggest that end-users may perceive the use of third-party DNS providers as a useful resource for avoiding censorship despite the fact that regular DNS traffic is being sent in the clear.
• For users accessing the Internet from outside of Europe and North America, third-party DNS providers are more likely to assign DNS resolvers located far from the user (i.e., resolvers placed in other continents). The concentration of third-party DNS resolvers in North America and Europe may have an impact in the web experience of users accessing the Internet from other world regions.
• Most ISPs providing both mobile and fixed-line access tend to decouple the DNS infrastructures used to serve their mobile and fixed networks. However, a few ISPs deploy a single DNS infrastructure to serve both types of services.
While the number of features studied in this work is limited, this thesis demonstrate the potential of the proposed lightweight method to run global DNS measurements. Stakeholders, from regulators to researchers and industry, can benefit from this technology to survey DNS usage trends, and to identify deployment and performance problems, both at the granularity of specific ASes and at a global scale.
• Measure Browser Marketplace In the current Internet, desktop computers interact with a large number of services, including the most popular ones (Online Social Networks, Video Portals, Streaming services, etc.), through browsers. Although this affirmation is not valid for mobile devices where most popular services run through proprietary applications; it is notably the importance of browsers in the mobile ecosystem as well. Arguably, we can assert that browsers are the most common online tool on the Internet, used every day by billions of users.
The most important technology companies (Apple, Google, and Microsoft) are aware of the importance of having a strong position in the browser market so that they dedicate a large amount of resources to develop their browsers. Measuring the browser market share is, for obvious reason, relevant for these browser development companies.
A monitoring solution should account with three principal characteristics:
• Scalability: It should be able to retrieve a large scale sample with at least millions of data points in order to provide statistically representative results. • Accessibility: Any entity including small private companies, researchers, regulators should be able to use and obtain valid results from it.
• Geographical and Demographic Targeting Capability: The solution should offer the possibility to run targeted measurements on specific geographical locations (e.g., a country or a region) and for specific demographic groups of users based on their age and their gender. By doing so, for instance, a company willing to lunch a new online software targeting a specific demographic group (e.g., male between 20 and 30) in a given geographical location (e.g., France) can know the browser market share of its target in advance and chose the best development strategy to follow.
Existing solutions to monitoring the browser market landscape rely on passive measurement techniques. The monitoring company installs a tracking code in a pool of N websites to collect the browser id associated with each visit to these websites. N ranges between thousands and millions of websites. While this methodology provides the scalability property and thus it is undoubtedly valuable to provide a solid knowledge about the browser marketplace, it presents important limitations in the accessibility and targeting capabilities since (i) it is accessible to just few companies with capacity to monitor thousands of websites and (ii) its passive nature prevents targeted monitoring campaigns for specific geographical areas or browser ids/versions. In addition to commercial solutions, there are academic works that analyze the browsers’ marketplace with a focus on security. However these works do not develop any specific technique to collect browser information, instead they use logs from Google, which is obviously proprietary and not accessible.
In this work, we use AdTag to monitor the browser market landscape. This approach overcomes the described limitations of traditional passive methodologies and offers the three main functional requirements mentioned above: scalability, accessibility and targeting capabilities.
AdTag inserts a lightweight JavaScript code within display advertisements. When an impression of these instrumented ads is displayed on a website, the embedded JavaScript code collects the User-Agent and the IP address of the device. The User-Agent reveals the browser brand and its version, whereas the IP address allows to map the device to a geographical region.
There is a large number of advertising providers that can serve as an appropriate infrastructure to execute AdTag. Moreover, the cost associated with it is low. Note that the price of a thousand impressions (a.k.a. CPM) can be as low as $0.01 in some providers. For instance, the provider used in the experiments has a CPM starting at $0.10. This shows that the described technique is accessible and affordable for any institution (small private companies, researchers, regulators, etc.) interested in monitoring the browsers marketplace. Moreover, it offers the required scalability, allowing to obtain millions of measurements per day with a low investment of tens to hundreds of dollars.
In addition, this proposal can leverage the targeting capacity of the online advertising ecosystem to set up targeted measurement campaigns based on geographical location and demographic properties (age and gender). Note that other targeting parameters are also available, e.g., Operating System, device type (mobile vs. desktop vs. tablet), etc.
Finally, to prove the efficiency of the aforementioned methodology, we have run general purpose as well as targeted experiments.
• Auditing Quality Metrics of the Online Advertising With thousands of vendor companies, helping advertisers place ads on millions of sites, to target over 4 billion Internet users, the online advertising ecosystem is far from transparent. Without transparency, it is not possible to truly establish if online advertising is as effective as a form of advertising as the total dollar investment in it suggests.
In particular, the lack of transparency of this market forces advertisers to rely in reports and metrics provided by different vendors such as Ad Networks, DSPs or Agency partners to assess the quality of their advertising campaigns. Some recent works have shown that, protected by this opacity, some vendors are providing inaccurate information to advertisers about their advertising campaigns. These findings urge to define methodologies to allow advertisers to independently assess the quality of their online advertising campaigns as well as auditing the reports received from vendors.
The research community has contributed techniques to evaluate the efficiency of different vendors in the detection and filtering of fraud. Unfortunately, fraud is not the only one aspect of the transparency problem.
For filling this gap, using the measurement methodology developed in this thesis, we present Q-Tag, a lightweight and scalable methodology to audit the performance of display advertising campaigns. In essence, we propose to inject a light JavaScript code in the ads, a method which is typically used for collecting behavioral targeting data from a user that sees the ad. This code collects relevant information associated with each impression and sends it to a central server. Specifically, the JavaScript code obtains the User-Agent receiving the impression, the URL where the impression was shown and user interactions with the ad impression (mouse movements or clicks on the ad). Moreover, we use the connection established with the server to obtain the IP address of the device receiving the ad impression as well as the timestamp associated to the impression. Finally, we estimate the exposure time of the ad impression as the duration of the connection.
We have tested the proposed methodology in 8 different campaigns set up using Google AdWords. In total these campaigns delivered around 160k ad impressions across more than 7k publishers. The obtained results indicate that the information reported by AdWords to advertisers is incomplete. In particular, this auditing methodology reveals the following insights: (i) AdWords did not report 57% of the publishers where ads from the campaigns were delivered. Without a complete list of publishers, an advertiser cannot optimize its Brand Safety protection; (ii) AdWords reports a large fraction of contextually relevant ad impressions based on (non-disclosed) criteria different from the publisher's thematic context; (iii) we configure campaigns with Cost-Per-Mille (CPM) investment ranging between 0.01€ and 0.30€ and conclude that, contrary to my expectation, a higher investment does not lead to impressions delivered to more popular publishers; (iv) AdWords does not impose any default frequency cap. This leads to hundreds of cases in the campaigns where a user receives the same ad more than 100 times with inter-arrival times between two consecutive ad impressions lower than 1 minute; (v) ~10% impressions are served to IP addresses belonging to Data Centers in two of the campaigns. Note that the Ad-Tech industry considers Data Center traffic to be likely associated to fraud.
In addition, we have developed a specific version of Q-Tag tailored to measure one of the most important quality metric in online advertising campaigns, defined as viewability. The ad-tech industry, under the guidance of the IAB and accreditation entities such as the MRC and JICWEBS, has defined the viewability standard. Based on this standard, for instance, a display ad impression is considered viewed by a user only if at least 50% of the pixels of the ad are visible to the user during at least 1 second (these requirements are slightly different for other ad formats). Then, ads shown below the fold, displayed in a different tab than the one currently visible, or hidden in the background, would not be considered viewed. Unfortunately, as it occurs with other metrics, reported viewability rates also suffer from the opacity of the ad-tech industry. Significant stakeholders, such as Google, Facebook, or Yahoo, directly measure the viewability rate to report it to its customers. Indeed, these large vendors have defined pricing schemes that only charge their advertisers for those ad impressions meeting the viewability condition characterized by the standard. Conversely, smaller vendors rely on third-party companies referred to as verifiers Integral Ad Science, Moat, etc.) specialized in quality assessment of ad campaigns. All these companies use proprietary techniques to measure the viewability.
As a result, the performance and limitations of such techniques are unknown. Different studies conducted by the industry and the research community have revealed episodes of inaccurate measurements of ad impressions' viewability as well as misreporting of different quality-related metrics. These findings question the performance of these opaque techniques and claim for the necessity of transparent and auditable mechanisms to measure viewability.
For all the stated above, in this work we modify Q-Tag for assessing if an individual ad impression meets the viewability standard criteria. The methodology can be used to compute the viewability of individual ad impressions as well as the viewability rate of ad campaigns. We have performed a thorough evaluation of the proposed solution through stress tests in a lab environment that report a high measurement accuracy of 93.4%.
Q-Tag has been deployed in production by a Demand Side Platform (DSP) and its performance compared in real ad campaigns with one of the most widely used viewability measurement solution in the ad-tech ecosystem. Q-Tag can measure viewability for 93% of the ad impressions in a campaign (on average). This represents a 19 percentage points of improvement over the commercial solution analyzed, which can measure viewability for only 74% of the ad impressions (on average). This substantial enhancement in the rate of measured ads may translate into an annual revenue increase in the order of millions of dollars for mid-size DSPs serving in the order of hundreds of millions of ads per day.
In summary, Q-Tag and its application to measure general ad campaigns metrics and more specifically the viewability, contribute a novel research methodology whose application in a real use case provides solid evidences about the inconsistency of reporting the quality metrics from vendors in the online advertising market, how this may affect the interests of advertisers, and how this may impact the revenue of online advertising intermediaries.
