Senior QA Automation Engineer

About the role

We are seeking a Senior QA Automation Engineer to own quality for an AI‑driven voice virtual agent. You will work on testing non‑deterministic conversational flows that span large language models, speech recognition, text‑to‑speech, and telephony layers.

Key responsibilities

• Embed in a Scrum team, own the end‑to‑end test strategy and maintain test artifacts.
• Design and automate complex multi‑turn dialogue scenarios, including interruptions, barge‑in, and escalation handling.
• Build an evaluation framework for non‑deterministic outputs using golden datasets, semantic rubrics, and regression detection.
• Provision SIP devices, drive programmatic calls, and diagnose failures across LLM, ASR, TTS, media path, and routing layers.
• Integrate automated suites into CI/CD pipelines as release gates and track quality metrics such as intent accuracy and response latency.

Required profile

• 5+ years of QA experience with hands‑on ownership of complex integrated products.
• Practical experience testing SIP/VoIP systems and troubleshooting with SIP traces or packet captures.
• Proficiency in Python to build and maintain automation frameworks.
• Experience testing non‑deterministic or probabilistic systems and defining robust quality metrics.
• Professional fluency in English and ability to work overlapping US Eastern business hours.

Required skills

• Python
• pytest
• Playwright
• JsSIP / WebRTC
• SIP softphones and devices
• Allure TestOps
• Jira
• Jenkins & GitHub Actions
• Grafana
• LLM APIs (evaluation tooling)

What we offer

• Remote work opportunity with B2B employment.
• Competitive salary with annual performance review.
• High‑end hardware and compensation for professional training.
• Company‑supported English classes and a referral program.
• Continuous learning and career growth in a cutting‑edge AI‑voice environment.