QA Engineer, Voice AI Agents

Remote (United States)

Job Details

Location: United States

Workplace: Remote

Employment Type: Full-Time

Experience: 3–5 years in QA, support engineering, technical operations, or a related role

Core Areas: Software Quality Assurance, Voice AI Agents, Production Debugging, Conversation QA, Root Cause Analysis, Regression Testing, SQL, Test Automation

Compensation: $180,000–$200,000 annually

About the Role

This opportunity is for a QA Engineer, Voice AI Agents responsible for investigating production issues, validating conversational AI workflows, and improving software quality across live voice AI campaigns. The role focuses on identifying defects, analyzing call behavior, tracing technical failures, and ensuring reliable agent responses, scheduling outcomes, escalation paths, and system integrations. Responsibilities include hands-on debugging, pre-launch testing, ongoing quality monitoring, and development of structured QA processes.

Reporting to the Head of Product, this position works with software engineers, deployment managers, and clinical and nurse operations teams to resolve platform issues and prevent recurring failures. The work involves analyzing call recordings, conversation transcripts, system logs, campaign configurations, and integration data, while using SQL, APIs, JSON or YAML, and automation techniques to improve testing efficiency and production reliability.

What You'll Do

• Investigate reported and proactively detected production issues across live voice AI campaigns, including dropped calls, incorrect agent responses, failed handoffs, scheduling errors, and data mismatches.
• Analyze call recordings, conversation transcripts, system logs, campaign configurations, and integration data to identify the underlying causes of technical and conversational failures.
• Perform root cause analysis to distinguish between AI agent behavior, configuration errors, data inconsistencies, integration problems, and software platform defects.
• Document defects through detailed, reproducible bug reports that include severity, affected members, business impact, supporting evidence, and actionable information for engineering teams.
• Triage incoming production issues, assess severity and priority, and route defects to the appropriate engineering, deployment, or operational owners.
• Coordinate with engineers, deployment managers, and clinical and nurse operations staff to investigate issues, support resolution, and improve system reliability.

• Execute pre-launch quality assurance testing for new campaigns, workflow updates, and configuration changes before production deployment.
• Validate test scripts, workflow scenarios, edge cases, escalation procedures, and handoffs to nurses or live support staff.
• Conduct ongoing conversation QA for active campaigns, evaluating agent interactions against established quality standards and identifying recurring performance issues.
• Verify campaign targeting, sequencing, disposition logic, and outcome capture to ensure workflows operate according to defined requirements.
• Identify quality trends, emerging defects, and potential operational risks before they develop into customer-facing issues.

• Develop and maintain structured QA processes, including test cases, quality scoring rubrics, regression test suites, and release readiness checklists.
• Identify opportunities to automate repetitive quality checks, including transcript analysis, campaign configuration validation, and outcome reconciliation.
• Support the implementation and improvement of automated QA checks to increase testing consistency and reduce manual investigation effort.
• Track quality metrics, analyze recurring defects, and communicate findings to product, engineering, and deployment teams.
• Use quality trends and investigation findings to recommend improvements that address the root causes of production issues.

Qualifications

Required Experience

• 3–5 years of professional experience in software quality assurance, support engineering, technical operations, or a closely related role supporting a production software platform.
• Experience creating and executing test plans and documenting reproducible software defects through clear, actionable bug reports.

Required Skills

• Strong technical debugging and root cause analysis skills, including the ability to investigate complex production issues and validate findings using supporting evidence.
• Ability to read and interpret system logs, query data using SQL, and work with APIs and JSON or YAML configuration files.
• Practical knowledge of software QA processes, test planning, defect identification, bug reporting, and issue prioritization.
• Ability to write clear technical documentation, communicate investigation findings accurately, and provide engineers with reproducible information.
• Sound judgment when assessing defect severity, affected users, and competing priorities across multiple active production issues.
• Strong analytical skills and a systematic approach to investigating failures beyond initial assumptions.

Preferred Qualifications

• Experience with voice AI, conversational AI, interactive voice response (IVR), or contact center platforms.
• Ability to write Python scripts or use similar scripting languages to automate quality checks, technical investigations, and data analysis.
• Exposure to large language model (LLM) evaluation, prompt testing, or conversational quality scoring methodologies.
• Experience in healthcare technology or healthcare operations, particularly payer or provider outreach workflows.
• Familiarity with HIPAA requirements and the secure handling of protected health information (PHI).
• Experience using observability platforms or call analytics tools to investigate production behavior and system performance.

Benefits

• Competitive base salary and equity compensation package.
• Generous paid time off (PTO).
• Comprehensive health, dental, and vision insurance.
• Life and disability insurance coverage.
• Simple IRA matching.
• Professional development budget for conferences, courses, and certifications.
• Wellness programs supporting physical and mental well-being.