• Evaluate and tune LLM-powered features, such as prompt optimization, retrieval-augmented generation (RAG) systems, and semantic search performance • Design and execute experiments to measure model quality, reliability, and user impact—translating technical findings into product recommendations • Develop and maintain data pipelines for evaluating, tracking, and improving system performance (e.g., accuracy, latency, cost, and relevance metrics) • Analyze structured and unstructured datasets (e.g., product usage logs, document metadata, LLM outputs) to identify patterns, insights, and areas for optimization • Collaborate with product managers to translate product goals into measurable data science questions, propose next steps, and inform roadmap priorities • Provide technical guidance to data engineers who build and maintain analytics and model evaluation infrastructure • Communicate results clearly—through written reports, dashboards, and presentations—to technical and non-technical stakeholders • Stay current on emerging practices in applied NLP, LLM evaluation, and data-driven product development, and thoughtfully adapt them to our environment