Cloud Bigtable, a widely-used key-value database on Google Cloud, is popular. This service offers scale elasticity, cost efficiency and excellent performance characteristics. It also provides 99.999% availability guaranteed. This has resulted in a huge adoption, with thousands of customers trusting Bigtable for a variety their mission-critical workloads.
Bigtable is in continuous production usage at Google since more than 15 years. It handles more than 5 billion requests per minute at peak, and manages more than 10 exabytes. It is Google’s largest semi-structured storage service.
Ad personalization is one of Bigtable’s key uses cases at Google. This article explains the central role of Bigtable in ad personalized.
Ad personalization
Personalization is a way to make users’ experience better by showing relevant and current ad content. YouTube has many videos about bread making. My viewing history can indicate to YouTube that baking is a topic of interest and I would be interested in ads related to baking products if ads personalization is enabled.
Personalization of ads requires large-scale data processing in real-time to allow for prompt personalization and strict control over data retention and handling. Due to the limited time frame in which decisions must be made about what ad content should be retrieved and served, system availability and serving times need to be high. Serving decisions that are not optimal (e.g. Reverting to generic ads could impact the user experience. Ad economics demands that infrastructure costs be kept as low and affordable as possible.
Google’s ad personalizedization platform allows you to create and deploy machine learning models that will rank ads and improve their relevance. It supports batch and real-time personalization. Bigtable is used to build the platform. This allows Google products to access data sources to personalize ads in a secure way that is privacy- and policy-compliant. Users can also make their own decisions about what data they wish to give to Google.
Bigtable stores the output of personalization pipelines such as advertising profiles for future consumption. These advertising profiles are retrieved by the ad stack to inform the next set of ad-serving decisions.
Some storage requirements for the personalization platform include:
High throughput batch personalization and batch processing.
Low latency (20ms at 99) Lookup for reads about the critical path to ad serving
Rapid (i.e. In a matter of seconds, incremental updates of advertising models are made in order to minimize personalization delay.
Bigtable
Bigtable’s flexibility in providing low-cost, high throughput access to data for offline personalization and consistent low-latency online data service makes it a great fit for ads workloads.
Google-scale personalization requires large storage space. These workloads are distinguished by Bigtable’s scalability and performance consistency.
Data model
Bigtable stores personalization platforms objects as serialized protobufs keyed with Object ids. Data sizes typically less than 1MB are possible. Serving latency at p99 is less than 20ms.
Data is organized in corpora. These correspond to different categories of data. A corpus is a map to a replicated Bigtable.
DataTypes are logical groups of data that organize data within a corpus. DataTypes are used to store features, embeddings and other advertising profiles. They map to Bigtable column families. DataTypes can be defined in schemas that describe the proto-structure of the data, as well as additional metadata to indicate ownership and provenance. SubTypes are free-form and map to columns in Bigtable.
A RowID is a unique identifier for each row of data. It is based on an Object ID. Personalization API allows you to identify individual values using DataType (column family), RowID (row key), SubType (column portion), and Timestamp.
Consistency
Operation’s default consistency mode is eventual. This mode retrieves data from the Bigtable replica closest to the user, with the lowest median latency and tail latency.
Consistent reading and writing to one Bigtable replica is possible. Traffic spillover between regions is likely if there are multiple Bigtable replicas in the same region. The personalization platform employs a notion called row affinity to improve read-after-write consistency. A Row ID is used to determine which replica will be preferred for a row if there are several copies in the region.
The platform requests Bigtable to return the lowest watermark (LWM), for lookups that have strict consistency requirements. The service will return a consistent response if the closest replica is the one where the write originated. If the LWMs indicate that replication has reached the required timestamp, the service will then return a consistent response. If replication is not successful, the service will issue a second lookup. This one will be directed at the Bigtable replica from which the writes originated. The request may be delayed or distant. In case the replica is not found, the platform can issue failover lookups.
Bigtable replication
Ads personalization workloads are based on a Bigtable replication topology that has more than 20 replicas spread across four continents.
Replication is a way to address high availability requirements for ad serving. Bigtable’s monthly zonal uptime percentage exceeds 99.9%. Replication combined with multi-cluster routing policies allows for availability of over 99.999%.
The global topology allows data placement close to users and minimizes serving times. It does have its challenges, such as the variability in network throughputs and costs. Bigtable employs minimum Spanning Tree-based routing algorithms, bandwidth-conserving proxy replicas, to reduce network costs.
To personalize ads, it is important to reduce Bigtable replication delay. This refers to the time between the user’s action and the time when the action has been incorporated in advertising models to show relevant ads to the user. While faster replication is preferable, we must balance serving traffic with replication traffic. We also need to ensure that low-latency user data serving is not interrupted by incoming and outgoing replication traffic flows. Bigtable uses complex flow control and priority booster mechanisms to manage traffic flows worldwide and balance the serving and replication traffic priorities.
Workload Isolation
By pinning certain workloads onto specific replicas, a set of Ad Personalization Batch workloads can be isolated from the serving workloads. Some replicas only drive personalization pipelines and others drive user-data service. This model provides a continuous, near-real-time feedback loop between offline personalization pipelines and serving systems. It also protects the workloads from each other.
Cloud Bigtable users have AppProfiles as well as cluster-routing policies that allow them to pin and confine workloads to certain replicas in order to achieve coarse-grained isolation.
Residency of data
Data is automatically replicated to all replicas, which is often spread globally. This is wasteful for data that can only be accessed locally. By limiting data to the most accessible region, regionalization reduces storage and replication costs. It is important to comply with regulations requiring that data related to specific subjects be stored in a particular geographic area.
The access location of requests can either be used to implicitly determine the location of data or via location metadata and other product signals. Once the user’s location has been determined, it is stored in location metadata tables that point to Bigtable replicas to which read requests should be directed. Data migration based on row placement policies takes place in the background without any downtime and no performance regressions.
Conclusion
This blog post examines how Bigtable is used by Google to support an important use-case: modeling user intent for ad personalizedization.
Bigtable has grown as Google’s personalization requirements have increased by orders of magnitude over the past decade. Bigtable is a low-cost storage solution that offers excellent performance and great cost for large-scale personalization tasks. With simple configurations, it seamlessly handles global traffic flows. It is an excellent choice for data processing pipelines that use lambda-style data processing.
We are continuing to invest high amounts to lower costs, improve performance and add new features to Bigtable to make it a better choice for personalization workloads.