Brainscape
Learning & productivity hacks for the world’s smartest humans.
AI Prompt Engineer
Location
New York
Posted
14 days ago
Salary
$40 - $100 / hour
Bachelor Degree1 yr expExperience acceptedEnglishPython
Job Description
• Migrate and test existing bulk flashcard creation prompts in an updated AI environment with newer GPT models - and plan future migrations as OpenAI retires older models
• Run test suites and manually review AI outputs for quality and correctness (fine-tune prompts)
• Analyze real user data to identify failure patterns and inform prompt improvements
• Streamline testing and evaluation workflows to make QA faster and more repeatable
• Monitor production quality post-launch and detect regressions as underlying models shift
• Build and maintain model evaluation datasets from real user inputs across all AI features
• Write new test cases for edge cases, multilingual content, and messy real-world inputs
• Document prompt changes, test results, and lessons learned
• Work with the Content Team to apply flashcard authoring quality standards
Job Requirements
- 1+ years hands-on prompt engineering experience with LLMs / OpenAI API (systematic testing and iteration, not just casual ChatGPT usage)
- Familiarity with Cursor IDE or similar AI-assisted development tools (our work is primarily Python - Cursor experience is more important than raw Python skill)
- Some experience with Git version control and collaborating via shared repositories (we use GitLab)
- A habit of documenting what you tried, what worked, and why - you don't need a formal QA background, but you naturally keep track of your process
- Clear written communication skills
- Proactive attitude; ability to work independently and manage your own time
- BONUS: Experience building prompt evals, AI quality assurance, or using GPT to grade GPT outputs
- BONUS: Experience with regression testing for AI systems or detecting model drift
- BONUS: Background in education technology (EdTech) or content creation - especially microlearning, flashcards, or other concise Q&A formats
- BONUS: A degree in Computer Science, Information Science, or a similar field