• Design and implement scalable data platform architecture on Databricks, supporting both batch and streaming ingestion • Build robust, fault-tolerant data ingestion pipelines that integrate with multiple third-party APIs and data providers • Design and implement AI-powered enrichment stages within pipelines—applying ML clustering, generative AI summarization, classification, and entity extraction to transform raw data into actionable intelligence • Build analytical systems with full-text search capabilities using Elasticsearch for rapid querying and analysis of enriched data • Work with AI/ML researchers to implement, integrate and scaling AI processing • Expose data platform capabilities as APIs and other interfaces for downstream consumption by applications and services • Optimize data lake and lakehouse architecture for performance, cost-efficiency, and scalability • Design and implement data quality frameworks, monitoring, and alerting systems • Design efficient architectures for calling external AI APIs and managing rate limits, costs, and reliability • Architect solutions with cost-efficiency as a first-class concern, implementing monitoring and optimization strategies for compute and storage • Make critical build-vs-buy decisions and establish architectural standards for the data organization • Mentor engineers and elevate the team's technical capabilities through code reviews, design discussions, and knowledge sharing