Key expectations Optimize Spark PySpark and SQL workloads to improve performance reliability cost efficiency and scalability Support production data pipelines through monitoring troubleshooting incident resolution and continuous improvement Implement data engineering standards CICD practices automated deployment processes unit testing and code quality expectations Partner with BI Product Engineering and clientfacing teams to translate business and reporting requirements into scalable data solutions Document technical solutions data flows pipeline logic and operational processes to support knowledge sharing and longterm maintainability Responsibilities Design build maintain and optimize data pipelines using Python SQL PySpark Databricks and AWSbased data services Develop ETLELT processes that support data warehousing analytics reporting and business intelligence use cases Build and optimize Spark jobs with a focus on performance scalability reliability and efficient resource utilization Design and implement data models for structured semistructured and NoSQL data where applicable Implement CICD practices automated deployments unit tests and code quality standards for data engineering workflows Monitor troubleshoot and support production data pipelines resolving issues and recommending improvements Collaborate with BI Analysts Product Engineering Data and clientfacing teams to understand requirements and support reporting needs Work Experience Requirements Minimum of 35 years of experience in Data Engineering preferably working with AWSbased cloud data platforms Handson experience building maintaining and supporting cloudbased data pipelines Strong knowledge of PySpark preferably on the Databricks platform Handson experience with Databricks Strong proficiency in SQL including query optimization Strong proficiency in Python Strong knowledge of data modeling data warehousing ETLELT and analytics concepts Experience with CICD practices automated deployment processes unit testing and code quality standards Experience troubleshooting monitoring and supporting production data pipelines Experience documenting technical solutions data flows and pipeline logic