QA Engineer

Remote (United States)

Job Details

Location: United States

Workplace: Remote

Employment Type: Full-Time

Experience: 7+ years in Software QA Engineering, including 2+ years testing AI or ML systems

Core Areas: AI/LLM Evaluation, Test Automation, Python, CI/CD Quality Gates, API and Integration Testing, Regression and End-to-End Testing, Performance and Load Testing, SOC 2 Compliance

Compensation: $110,000 - $135,000 per year, plus equity

Travel: Occasional travel to client sites and team offsites

About the Role

This opportunity is for a QA Engineer who will build and establish a formal quality assurance function for a growing AI software environment. The role is a hands-on individual contributor position focused on writing test cases, building AI evaluation frameworks, developing automated test coverage, implementing CI/CD quality gates, triaging production issues, and creating reliable regression baselines.

The position combines software QA engineering with AI and ML testing, release reliability, and regulatory quality requirements. The successful candidate will work extensively with Python, pytest, Playwright or Selenium, REST APIs, cloud infrastructure, LLM evaluation tooling, automated test pipelines, and SOC 2 documentation while helping establish a scalable quality engineering practice.

What You'll Do

AI Evaluation

• Design and execute evaluation frameworks for LLM and agentic AI outputs across AI development environments and client-deployed instances.
• Write assertions, define behavioral contracts, and establish regression baselines for model behavior.
• Develop evaluation approaches for non-deterministic AI systems where traditional deterministic assertions are insufficient.
• Apply statistical concepts such as distributions and confidence intervals when evaluating AI behavior.
• Build tooling that supports reliable evaluation of complex AI-generated outputs.

Test Coverage

• Write and maintain automated test suites covering end-to-end, integration, and regression scenarios.
• Build automated coverage for backend APIs, document ingestion pipelines, AI inference workflows, and frontend surfaces.
• Own performance and load testing for latency-sensitive AI inference paths.
• Implement and enforce quality gates within CI/CD pipelines.
• Participate directly in production bug triage and investigate failures across application and infrastructure layers.
• Create reproducible test cases for production defects.
• Build regression tests that prevent resolved production issues from recurring.

Compliance and Client Quality

• Produce test artifacts, audit logs, and QA process documentation that meet SOC 2 Type II requirements.
• Ensure quality documentation and testing evidence can withstand regulatory and audit scrutiny.
• Work directly with Forward Deployed Engineering on client-side validation.
• Reproduce production issues affecting client environments and validate resulting fixes.

QA Function Development

• Assess existing test coverage and identify gaps in quality assurance processes.
• Build the QA function through hands-on implementation of test cases, evaluation frameworks, automated test suites, and release quality controls.
• Document the quality engineering practice so it can scale as the organization and customer base grow.
• Help establish the foundation for additional Quality Engineering hires once the QA practice is stable.

Qualifications

Required Experience

• 7+ years of professional experience in Software Quality Assurance Engineering.
• At least 2 years of hands-on experience testing AI or machine learning systems.
• Experience writing test cases for LLM outputs and evaluating non-deterministic AI behavior.
• Proven experience building AI evaluation pipelines from scratch.
• Experience building and maintaining automated test suites in production software environments.
• Experience integrating QA quality gates into CI/CD pipelines and owning the process end to end.

Required Skills

• Strong Python programming skills for test automation, evaluation tooling, and quality engineering workflows.
• Hands-on experience building automated test suites using pytest, Playwright, Selenium, or comparable testing frameworks.
• Practical experience with AI evaluation tools such as RAGAS, DeepEval, LangSmith, or comparable platforms.
• Ability to distinguish flaky automated tests from genuinely non-deterministic system behavior.
• Strong understanding of REST APIs and the ability to test and troubleshoot API-driven applications.
• Ability to trace failures from the application layer through infrastructure without relying on another engineer to perform the investigation.
• Working knowledge of Azure and asynchronous software systems.
• Experience with end-to-end, integration, regression, performance, and load testing.
• Ability to establish regression baselines and reliable quality gates for AI and traditional software workflows.
• Ability to create defensible QA documentation, test artifacts, and audit evidence for regulated environments.

Preferred Qualifications

• Experience working in fintech, banking, or another regulated software environment.
• Familiarity with document processing pipelines.
• Familiarity with multi-agent architectures.
• Experience with Retrieval-Augmented Generation (RAG) validation.
• Familiarity with observability tools such as Arize or Langfuse.

What Success Looks Like

• Within the first 90 days, complete a diagnostic assessment of current test coverage and share the findings with engineering leadership.
• Within the first 90 days, build and deploy an evaluation framework for at least one AI-powered workflow.
• Within the first 90 days, implement live quality gates within CI/CD.
• Within the first six months, establish regression baselines for model behavior.
• Within the first six months, document SOC 2 test artifacts and ensure they are audit-ready.
• Within the first six months, establish automated test execution on every release without manual intervention.
• Within the first year, help scale the QA function with additional Quality Engineering capacity.
• Build a quality framework in which test coverage scales with product releases and quality becomes a core input to deployment decisions.