Accelerate Any Engine on Any Data on Any Hardware

Data Processing Principal Engineer

Full-stack EngineerSoftware EngineerFull TimeRemoteTeam 51-200Since 2022H1B SponsorCompany Site LinkedIn

Location

United States

Posted

142 days ago

Salary

Not specified

Bachelor Degree15 yrs expExperience acceptedEnglishApacheLinuxRustSpark

Job Description

• Lead architecture, design, and implementation of next-generation data processing engine built to exploit accelerated computing. • Work with cross-functional teams to advance the architecture at the core of Datapelago’s parallel and distributed execution engine based on accelerated computing • Lead the execution engine team in the design, implementation, and rollout of enterprise grade, highly reliable acceleration engine for data processing • Design, implement, test, and maintain major components of the execution engine • Analyze and identify areas for differentiation and improvements in the execution engine • Collaborate with other teams and team members to drive code reviews, design reviews, performance and reliability reviews, and development process and to drive continuous improvement in all of these

Job Requirements

B.S. EE/CS or equivalent with 15+ years experience or MS with 10+ years experience
10+ years of experience developing core components of an enterprise-grade database or analytics execution engine serving large-scale data processing workloads. Experience developing for platforms such as Apache Spark, Gluten, Velox, DataFusion preferred.
Experienced developing high-performance parallel implementations of data processing operators and functions, such as joins, aggregations, sorts
Experience leading 10+ teams in designing, developing, and releasing high-performance data processing engines for large production deployments
Strong programming ability in C, C++, and Rust
Strong development experience on Linux platforms