Architect end to end data flow across ingestion storage transformation and consumption layers using Azure Synapse Data Factory Fabric Databricks and ADLS Gen2 Design logical physical and semantic data models supporting analytical operational and AI workloads Define and implement dimensional and data vault models ensuring scalability and consistency across domains Build Lakehouse and Direct Lake models optimized for Power BI and ML pipelines Maintain data design patterns reusable templates and metadatadriven orchestration frameworks for consistent delivery Collaborate with analysts and scientists to ensure data models are ML and GenAIready enabling features like automated feature extraction embeddings and contextual reasoning Performance Optimization Scalability Perform data pipeline optimization query tuning and workload balancing for largescale data processing in Synapse and Databricks Implement partitioning indexing caching and materialized views for highvolume datasets Establish performance benchmarking frameworks and continuous monitoring for ingestion transformation and serving layers Optimize Power BI and Fabric Semantic Models for realtime and adhoc analytics Design costefficient and highthroughput architectures leveraging serverless compute caching layers and autoscaling strategies Apply AIassisted insights eg anomaly detection on query latency data skew or SLA breaches for proactive performance management Data Governance Quality Lineage Implement AIdriven data governance frameworks using Microsoft Purview to automate cataloging lineage and classification Define data quality rules scoring and anomaly detection pipelines integrated with ADF or Databricks Maintain metadatadriven onboarding schema evolution handling and automated lineage visualization Drive compliance retention and data stewardship programs ensuring data accountability across domains Build governance automation using AI agents to generate lineage summaries validate schema changes and flag policy violations Security Compliance Responsible AI Architect and enforce Zero Trust data security RBAC RLS encryption at rest transit and DLP policies Integrate Microsoft Entra ID Defender for Cloud and Purview Data Policies for unified access control Embed data masking tokenization and privacypreserving design patterns in model and pipeline development Conduct AI data risk assessments ensuring fairness transparency and explainability in model training and usage Define and implement Responsible AI governance frameworks for compliant data and model operations GDPR SOC2 HIPAA CICD Automation Best Practices Establish CICD pipelines for data engineering and model deployment using Azure DevOps GitHub Actions or Fabric pipelines Automate code promotion testing and environment provisioning for data pipelines and semantic models Standardize branching version control and deployment governance for ADF Synapse and Databricks Implement DataOps principles continuous integration continuous testing and continuous monitoring for data and AI workloads Introduce metadatadriven DevOps parameterized configurations reusable scripts and dynamic environment handling Define and enforce enterprise data architecture best practices including coding standards naming conventions and change management processes Conduct design reviews audit readiness checks and performance health scans periodically AI ML Enablement Collaborate with AI and ML teams to build feature stores model training pipelines and MLOps architectures Architect AI metadata layers for lineage tracking from raw data feature model prediction Integrate GenAI automation for documentation generation error detection and rulebased pipeline recovery Enable AI observability monitor data drift model performance and bias using metadatadriven dashboards Partner with domain teams to embed AIpowered personalization and prediction engines into data products Required Skills Proven handson expertise in Azure Synapse Data Factory Fabric ADLS Gen2 and Microsoft Purview Strong command over data modeling dimensional data vault semantic SQL tuning and pipeline optimization Experience with CICD automation DevOps and metadatadriven frameworks Preferred Understanding of AI principles and governance