Alight Solutions

Alight Inc. (NYSE: ALIT) is a leading cloud-based human capital technology and services provider

AI Automation Engineer

QA Automation EngineerQA Automation EngineerFull TimeRemoteTeam 10,001+Since 2016H1B SponsorCompany SiteLinkedIn

Location

Illinois

Posted

12 days ago

Salary

$160K - $190K / year

Bachelor DegreeEnglishCloudJava ScriptNode.jsPython

Job Description

• Develop and maintain automated test frameworks for LLM, RAG, agentic, and classic ML systems • Implement golden/adversarial test cases (functional, safety, bias, prompt injection, refusal correctness, tool use) • Integrate evaluation tools (promptfoo, Ragas) into CI/CD pipelines and support batch quality assessments • Author and validate OpenAPI/JSON Schema contracts for model inputs/outputs; enforce schema checks in pipelines • Instrument observability metrics (latency, success rate, cost, drift) using OpenTelemetry and export to dashboards • Support data QA, lineage tracking, and drift monitoring (Evidently, OpenLineage) • Collaborate with AI developers, SDETs, and SRE to troubleshoot, optimize, and document automation processes • Contribute to red-team playbooks and security testbooks mapped to OWASP/ATLAS risks

Job Requirements

  • Experience in test automation, software engineering, or quality assurance (AI/ML experience is a plus, but not required)
  • Proficiency in Python, JavaScript/Node.js, or similar programming languages
  • Familiarity with CI/CD, version control, and cloud-based deployment environments
  • Experience with test automation tools (promptfoo, Ragas, Postman)
  • Understanding of AI model evaluation metrics (faithfulness, answer relevancy, hallucination, refusal correctness)
  • Basic knowledge of fairness, safety, and security testing for AI systems
  • Experience with AI infrastructure, orchestration, data integrity, deployments, and model registry practices
  • Interest in Responsible AI, bias mitigation, and regulatory compliance
  • Bachelor’s degree in Computer Science, Engineering, or related field, or equivalent practical experience

Benefits

  • Options include a variety of health coverage options
  • Wellbeing and support programs
  • Retirement
  • Vacation and sick leave
  • Maternity, paternity & adoption leave
  • Continuing education and training
  • A number of voluntary benefit options

Related Categories

Related Job Pages

More QA Automation Engineer Jobs

QA Automation Engineer12 days ago
Full TimeRemoteTeam 201-500

The primary duty involves gathering, sorting, and logging all requests according to contractual requirements to support the Case Processing Operation Center (CPOC). Key tasks include sorting workload, sending completed work to Federal Agencies, entering adjudication codes, and monitoring email requests.

United States
ContractRemoteTeam 11-50

We’re looking for a Robotics & Automation Engineering Consultant to support our venture team as we explore and validate automation opportunities. In this role, you will advise our venture team to help identify pain points, evaluate automation feasibility, and inform the developme...

United States

Senior Quality Automation Engineer

CINC Systems

CINC Systems is the leading provider of accounting and management software for the community association management industry. Our platform supports tens of thousands of associations and millions of homes in a highly reliable, multi-tenant SaaS environment. CINC is an Equal Opportunity Employer of women, minorities, protected veterans and individuals with disabilities.

QA Automation Engineer12 days ago
Full TimeRemote

The Senior Quality Automation Engineer is a hands-on engineer responsible for designing and building automated quality systems that scale with our architecture and teams. This role partners closely with product engineers, platform teams, and product managers to ensure that qualit...

United States + 180 moreAll locations: United States, Canada, Brazil, Colombia, Argentina, Chile, Venezuela, Bolivarian Republic Of, Bolivia, Plurinational State Of, Ecuador, French Guiana, Guyana, Paraguay, Peru, Suriname, Uruguay, Mexico, Costa Rica, El Salvador, Guatemala, Honduras, Nicaragua, Panama, Dominican Republic, Puerto Rico, Bahamas, Guadeloupe, Haiti, Jamaica, Martinique, Montserrat, United Kingdom, Germany, France, Estonia, Portugal, Hungary, Poland, Ukraine, Romania, Bulgaria, Czech Republic, Slovakia, Belarus, Moldova, Republic Of, Sweden, Greece, Belgium, Italy, Ireland, Switzerland, Netherlands, Finland, Malta, Denmark, Lithuania, Croatia, Spain, Austria, Bosnia And Herzegovina, Iceland, Luxembourg, Macedonia, The Former Yugoslav Republic Of, Montenegro, Norway, Serbia, Slovenia, Albania, Cyprus, Latvia, Monaco, South Africa, Egypt, Algeria, Angola, Benin, Botswana, Burkina Faso, Burundi, Cameroon, Cape Verde, Central African Republic, Chad, Congo, Côte D'ivoire, Congo, The Democratic Republic Of The, Equatorial Guinea, Eritrea, Ethiopia, Gabon, Gambia, Ghana, Guinea, Guinea-bissau, Kenya, Lesotho, Liberia, Libyan Arab Jamahiriya, Madagascar, Malawi, Mali, Mauritania, Mauritius, Mayotte, Morocco, Mozambique, Namibia, Niger, Nigeria, Réunion, Rwanda, Senegal, Seychelles, Sierra Leone, Somalia, Sudan, Swaziland, Tanzania, United Republic Of, Togo, Tunisia, Uganda, Zambia, Zimbabwe, Georgia, Turkey, Israel, United Arab Emirates, Armenia, Azerbaijan, Bahrain, Iraq, Jordan, Kuwait, Lebanon, Oman, Qatar, Saudi Arabia, Palestinian Territory, Occupied, Yemen, India, Japan, Philippines, Pakistan, Thailand, Singapore, Viet Nam, Taiwan, Province Of China, Indonesia, Cambodia, Lao People's Democratic Republic, Malaysia, Myanmar, Korea, Republic Of, China, Afghanistan, Bangladesh, Bhutan, Kazakhstan, Kyrgyzstan, Maldives, Mongolia, Nepal, Sri Lanka, Tajikistan, Turkmenistan, Uzbekistan, Australia, Papua New Guinea, Kiribati, Palau, French Polynesia, Tuvalu, New Zealand
QA Automation Engineer12 days ago
Full TimeRemoteTeam 51-200

This role is responsible for evaluating the quality of patient care, monitoring compliance with laws, and implementing clinical best practices, which includes reviewing all staff nursing notes and ensuring timely submission and accuracy.

United States
$38 - $40 / hour