How Meta understands data at scale

• Managing and understanding large-scale data ecosystems is a significant challenge for many organizations, requiring innovative solutions to efficiently safeguard user data. Meta’s vast and diverse systems make it particularly challenging to comprehend its structure, meaning, and context at scale.
• To address these challenges, we made substantial investments in advanced data understanding technologies, as part of our Privacy Aware Infrastructure (PAI). Specifically, we have adopted a “shift-left” approach, integrating data schematization and annotations early in the product development process. We also created a universal privacy taxonomy, a standardized framework providing a common semantic vocabulary for data privacy management across Meta’s products that ensures quality data understanding and provides developers with reusable and efficient compliance tooling.
• We discovered that a flexible and incremental approach was necessary to onboard the wide variety of systems and languages used in building Meta’s products. Additionally, continuous collaboration between privacy and product teams was essential to unlock the value of data understanding at scale.
• We embarked on the journey of understanding data across Meta a decade ago with millions of assets in scope ranging from structured and unstructured, processed by millions of flows across many of the Meta App offerings. Over the past 10 years, Meta has cataloged millions of data assets and is classifying them daily, supporting numerous privacy initiatives across our product groups. Additionally, our continuous understanding approach ensures that privacy considerations are embedded at every stage of product development. 

At Meta, we have a deep responsibility to protect the privacy of our community. We’re upholding that by investing our vast engineering capabilities into building cutting-edge privacy technology. We believe that privacy drives product innovation. This led us to develop our Privacy Aware Infrastructure (PAI), which integrates efficient and reliable privacy tools into Meta’s systems to address needs such as purpose limitation—restricting how data can be used while also unlocking opportunities for product innovation by ensuring transparency in data flows 

Data understanding is an early step in PAI. It involves capturing the structure and meaning of data assets, such as tables, logs, and AI models. Over the past decade, we have gained a deeper understanding of our data, by embedding privacy considerations into every stage of product development, ensuring a more secure and responsible approach to data management.

We embarked on our data understanding journey by employing heuristics and classifiers to automatically detect semantic types from user-generated content. This approach has evolved significantly over the years, enabling us to scale to millions of assets. However, conducting these processes outside of developer workflows presented challenges in terms of accuracy and timeliness. Delayed classifications often led to confusion and unnecessary work, while the results were difficult to consume and interpret.

Data understanding at Meta using PAI

To address shortcomings, we invested in data understanding by capturing asset structure (schematization), describing meaning (annotation), and inventorying it into OneCatalog (Meta’s system that discovers, registers, and enumerates all data assets) across all Meta technologies. We developed tools and APIs for developers to organize assets, classify data, and auto-generate annotation code. Despite significant investment, the journey was not without challenges, requiring innovative solutions and collaboration across the organization.

Walkthrough: Understanding user data for the “Beliefs” feature in Facebook Dating 

To illustrate our approach and dive into the technical solution, let’s consider a scenario involving structured user data. When creating a profile on the Facebook Dating app, users have the option to include their religious views to help match with others who share similar values.

On Facebook Dating, religious views are subject to purpose limitation requirements. Our five-step approach to data understanding provides a precise, end-to-end view of how we track and protect sensitive data assets, including those related to religious views:

Even a simple feature can involve data being processed by dozens of heterogenous systems, making end-to-end data protection critical. To ensure comprehensive protection, it is essential to apply the necessary steps to all systems that store or process data, including distributed systems (web systems, chat, mobile and backend services) and data warehouses.

Consider the data flow from online systems to the data warehouse, as shown in the diagram below. To ensure that religious belief data is identified across all these systems, we have implemented measures to prevent its use for any purpose other than the stated one.

Step 1 – Schematizing

As part of the PAI initiative, Meta developed DataSchema, a standard format that is used to capture the structure and relationships of all data assets, independent of system implementation. Creating a canonical representation for compliance tools. Understanding DataSchema requires grasping schematization, which defines the logical structure and relationships of data assets, specifying field names, types, metadata, and policies.

Implemented using the Thrift Interface Description Language, DataSchema is compatible with Meta systems and languages. It describes over 100 million schemas across more than 100 data systems, covering granular data units like database tables, key-value stores, data streams from distributed systems (such as those used for logging), processing pipelines, and AI models. Essentially, a data asset is like a class with annotated attributes. 

Let’s examine the source of truth (SoT) for a user’s dating profile schema, modeled in DataSchema. This schema includes the names and types of fields and subfields:

  - user_id (uint)
  - name (string)
  - age (uint)
  - religious_views (enum)
  - photos (array):
     - url (url)
     - photo (blob)
     - caption (string)
     - uploaded_date (timestamp)
Dating profile DataSchema

The canonical SoT schema serves as the foundation for all downstream representations of the dating profile data. In practice, this schema is often translated into system-specific schemas (source of record – “SoR”), optimized for developer experience and system implementation in each environment. 

Step 2 – Predicting metadata at scale

Building on this schematization foundation, we used annotations to describe data, enabling us to quickly and reliably locate user data, such as religious beliefs, across Meta’s vast data landscape. This is achieved through a universal privacy taxonomy, a framework that provides a common semantic vocabulary for data privacy management across Meta’s apps. It offers a consistent language for data description and understanding, independent of specific programming languages or technologies.

The universal privacy taxonomy works alongside data classification, which scans systems across Meta’s product family to ensure compliance with privacy policies. These systems use taxonomy labels to identify and classify data elements, ensuring privacy commitments are met and data is handled appropriately according to its classification.

Privacy annotations are represented by taxonomy facets and their values. For example, an asset might pertain to an Actor.Employee, with data classified as SemanticType.Email and originating from DataOrigin.onsite, not a third party. The SemanticType annotation is our standard facet for describing the meaning, interpretation, or context of data, such as user names, email addresses, phone numbers, dates, or locations. 

Below, we illustrate the semantic type taxonomy node for our scenario, Faith Spirituality:

As data models and collected data evolve, annotations can become outdated or incorrect. Moreover, new assets may lack annotations altogether. To address this, PAI utilizes various techniques to continuously verify our understanding of data elements and maintain accurate, up-to-date annotations:

Our classification system leverages machine learning models and heuristics to predict data types by sampling data, extracting features, and inferring annotation values. Efficient data sampling, such as Bernoulli sampling, and processing techniques enable scaling to billions of data elements with low-latency classifications. 

Key components include:

• Scheduling component: manages the set of data assets to scan, accommodating different data system architectures by either pulling data via APIs or receiving data pushed directly into the scanning service.
• Scanning service: processes and analyzes data from various sources by accumulating samples in memory, deserializing rows (e.g., JSON) into fields and sub-fields, and extracting features using APIs available in multiple languages (C++, Python, Hack). It ensures comprehensive data capture, even for ephemeral data.
• Classification service: utilizes heuristic rules and machine learning models to classify data types with high accuracy.
  • Heuristic rules: handle straightforward, deterministic cases by identifying specific data formats like dates, phone numbers, and user IDs.
  • Machine learning models: trained on labeled datasets using supervised learning and improved through unsupervised learning to identify patterns and anomalies in unlabeled data.
  • Ground truth calibration and verification: ensures system accuracy and reliability, allowing for model fine-tuning and improved classification performance.
• Lineage and propagation: We integrate classification rules with high-confidence lineage signals to ensure accurate data tracking and management. Our propagation mechanism enables the seamless annotation of data as needed, ensuring that exact copies of data across systems receive equivalent classification. This approach not only maintains data integrity but also optimizes the developer experience by streamlining the process of managing data classifications across our diverse systems.

Step 3 – Annotating

The integration of metadata predictions and developer input creates a comprehensive picture of a data asset’s structure (schema) and its meaning (annotation). This is achieved by attaching these elements to individual fields in data assets, providing a thorough understanding of the data.

Building on the predicting data at scale initiative (step 2), where we utilize the universal privacy taxonomy and classification systems to identify and classify data elements, the generated metadata predictions are then used to help developers annotate their data assets efficiently and correctly.