• Involve in Design, development, and maintenance of scalable ETL/ELT pipelines on Azure Databricks using Apache Spark (PySpark/Spark SQL). • Design and implement both batch and real-time data ingestion and transformation processes. • Build and manage Delta Lake tables, schemas, and data models to support efficient querying and analytics. • Consolidate and process large-scale datasets from various structured and semi-structured sources (e.g., JSON, Parquet, Avro). • Write optimized SQL queries for large datasets using Spark SQL and PostgreSQL. • Develop, schedule, and monitor workflows using Databricks Workflows, Airflow or similar orchestration tools. • Design, build, and deploy cloud-native, containerized applications on Azure Kubernetes Service (AKS) and integrate with Azure services. • Ensure data quality, governance, and compliance through validation, documentation, and secure practices. • Collaborate with data analysts, data architects, and business stakeholders to translate requirements into technical solutions. • Contribute to and enforce best practices in data engineering, including version control (Git), CI/CD pipelines, and coding standards. • Continuously enhance data systems for improved performance, reliability, and scalability. • Mentor junior engineers and help evolve team practices and documentation. • Stay up to date on emerging trends, technologies, and best practices in the data engineering space. • Work effectively within an agile, cross-functional project team.