Data Platform Lab

Data Engineering platform for company built on GCP. Handles multi-source ingestion, transformation, and serving of insurance data assets.

---

Platform Summary

Aspect	Detail
Cloud	Google Cloud Platform
Orchestration	Cloud Composer (Airflow 2.x)
Processing	Apache Beam / GCP Dataflow
Storage	Cloud Storage (Avro) + BigQuery
Sources	Oracle DB, CSV files, REST APIs, PDFs
Pattern	Metadata-driven, chain-based orchestration

---

Documentation Structure

• Platform Architecture — L1 logical + L2 physical view
• Data Layers — Raw, History, Active pattern
• Orchestration Framework — Chain/Group DAG pattern
• Audit Tables — aud_* metadata model
• Operations — Day-to-day operations

---

High-Level Flow