This session details how Conde Nast uses Databricks tools in real-time data pipelines for privacy compliance. Our journey started with a real-time data pipeline for customer subscriptions, and we aim to expand these practices. We share short- and long-term data privacy and compliance strategies. In our short-term approach, we leverage Delta Lake's Change Data Feed (CDF) to efficiently track data changes, ensure GDPR compliance in real-time processes, and address regulations such as 'right to be forgotten'. Our long-term plan includes separating personally identifiable information (PII) and implementing a secure data vault, which is critical to minimizing GDPR risks and improving security. The session provides practical solutions for privacy compliance in data processing, aimed at data engineers, architects and compliance officers looking to align data practices with privacy regulations using cutting-edge technology.

Lecture by: Emma Stein, Sr. Solutions Architect, Databricks; Sarah Uludag, data engineer, Condé Nast

There's more to discover here:
Big Book of Data Engineering: 2nd Edition: https://dbricks.co/3XpPgNV
The data team's guide to the Databricks Lakehouse platform: https://dbricks.co/46nuDpI

Contact us: Website: https://databricks.com
Twitter: https://twitter.com/databricks
LinkedIn: https://www.linkedin.com/company/data…
Instagram: https://www.instagram.com/databricksinc
Facebook: https://www.facebook.com/databricksinc

Please take the opportunity to connect and share this video with your friends and family if you find it helpful.