This description is a summary of our understanding of the job description. Click on 'Apply' button to find out more.

Role Description

We are seeking a visionary and hands-on Data Engineering Lead to spearhead the design, development, and optimization of our next-generation data platform. As a Lead, you will balance technical excellence with people leadership, ensuring our data architecture is scalable, resilient, and perfectly aligned with our business goals while collaborating cross-functionally to support analytics, reporting, and operational data needs.

The ideal candidate should be a PySpark expert who thrives in the Azure ecosystem and has a deep appreciation for clean, modular code and robust ETL patterns. This is an exciting opportunity to work along with a great team of data engineers, demanding technologies, and an engaging work environment to help shape our data engineering best practices.

Key Responsibilities

• Technical Leadership: Design and supervise the implementation of comprehensive data pipelines utilizing Azure Databricks and PySpark.
• Team Mentorship: Direct a team of data engineers, performing code reviews, offering technical expertise, and cultivating a culture of ongoing learning.
• Data Modeling & Optimization: Develop high-performance schemas in PostgreSQL and refine complex SQL queries for large datasets.
• ETL Strategy: Establish and apply optimal practices for data ingestion, transformation, and storage (Delta Lake/Lakehouse patterns).
• Strategic Collaboration: Collaborate closely with Data Analysts, Architects, and Product Managers to convert business requirements into technical specifications.
• Process Improvement: Promote the implementation of CI/CD, unit testing, and automated monitoring to achieve 99.9% data reliability.
• Ensure data quality, governance, and compliance through validation, documentation, and secure practices.
• Continuously improve data systems for enhanced performance, reliability, and scalability.
• Effectively engage within an agile, cross-functional project team.

Mandatory Skillset

• Azure Databricks:
  • Expert-level experience managing workspaces, clusters, and job scheduling.
  • Solid understanding of data lakehouse architectures and Delta Lake.
  • Proven experience in Performance Tuning, Spark Optimization and Cost Reduction.
• PySpark: Advanced proficiency in Spark DataFrame APIs and Spark SQL for large-scale data processing involving various data formats.
• SQL Mastery: Exceptional ability to write, tune, and troubleshoot complex queries.
• PostgreSQL: Hands-on experience with relational database design, indexing, and performance optimization.
• ETL/ELT Frameworks: Proven track record of building scalable data pipelines from scratch.

Desired Skills

• Workflow Orchestration: Experience with Apache Airflow for managing complex task dependencies.
• Containerization: Familiarity with Azure Kubernetes Service (AKS) for deploying containerized data services.
• Infrastructure as Code (IaC): Knowledge of Terraform or Bicep for managing Azure resources.

Qualifications

• 10+ years of experience in Data Engineering or Software Engineering.
• 3+ years as a formal technical Lead managing an agile team and implementing E2E solutions.
• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• Strong communication skills with the ability to explain complex technical concepts to non-technical stakeholders.
• Strong problem-solving skills and attention to detail.

Benefits

• You won't just be "maintaining" pipelines; you'll be the primary architect of a data ecosystem that powers real-time decision-making across the entire organization.