Applied AI Engineer

Remote (United States)

Job Details

Location: United States

Workplace: Remote

Employment Type: Full-Time

Experience: 5+ years of backend software engineering, with hands-on experience deploying LLM-powered applications and services in production

Core Areas: Large Language Models (LLMs), AI Inference Infrastructure, LLM Gateway Development, Python, LLM Evaluation Pipelines, AWS & Infrastructure as Code, CI/CD & Model Deployment, AI Observability

Compensation: $150,000–$175,000 annually (base salary)

About the Role

The Applied AI Engineer will design, build, and operate production infrastructure for large language model (LLM) applications. This hands-on engineering position focuses on developing a centralized LLM gateway, managing model integrations, implementing automated evaluation and deployment pipelines, and ensuring AI-powered services operate reliably, securely, and cost-effectively at scale.

Working within platform engineering and collaborating with AI/ML researchers, engineering teams, and Security and Compliance, this role transforms validated AI prototypes into scalable production services. Key technologies include Python, commercial LLM APIs, AWS, containers, infrastructure as code, CI/CD, and observability tooling. The position supports AI evaluation, red teaming, dataset generation, and annotation systems, including applications involving sensitive healthcare and life sciences data.

What You'll Do

• Design, develop, and operate a centralized LLM gateway that routes production AI services to approved model providers.
• Implement request logging, cost attribution, rate limiting, provider failover, and protected health information (PHI) controls across LLM inference services.
• Replace direct model-provider integrations with secure gateway-based access, eliminating exposed provider API keys from individual services.
• Identify and document existing production LLM integrations, including providers, models, application features, and data usage.
• Support reliable, scalable, and cost-efficient execution of LLM-powered applications.

• Maintain an approved catalog of AI models, providers, API endpoints, and supported capabilities, including structured outputs, tool use, caching, and batch processing.
• Document model-specific data retention policies, Business Associate Agreement (BAA) coverage, and version pinning requirements.
• Maintain decision records for newly approved models, providers, and capabilities.
• Evaluate changes in the model-provider ecosystem and assess their implications for production infrastructure.

• Develop standardized deployment workflows for LLM model configurations and prompt changes.
• Maintain version-controlled prompts and model configurations alongside application code.
• Integrate AI/ML evaluation frameworks and acceptance thresholds into CI/CD pipelines as automated release gates.
• Implement staged deployments, regression evaluation checks, and rollback mechanisms for model and prompt releases.
• Provide reusable templates and self-service workflows that allow engineers and researchers to experiment in sandbox and staging environments while maintaining controlled production releases.
• Apply consistent testing, reproducibility, and auditability standards to AI application changes.

• Convert validated AI/ML prototypes, evaluation harnesses, and acceptance criteria into scalable, observable production services.
• Collaborate with AI/ML researchers to preserve validated model behavior and evaluation requirements throughout production implementation.
• Advise engineering and research teams on model selection considerations, including inference costs, latency, and availability.
• Develop production infrastructure supporting LLM-powered features, AI evaluation, red teaming, dataset generation, and annotation workflows.
• Work with platform engineering, research, and domain specialists to translate technical and scientific requirements into reliable applications.

• Implement end-to-end observability for LLM-powered services using OpenTelemetry and related monitoring platforms.
• Track inference latency, failures, token consumption, operating costs, and model output quality.
• Monitor model behavior and quality drift against established AI/ML evaluation benchmarks.
• Participate in on-call operations and lead incident response for production LLM services.
• Apply security practices covering secrets management, access control, and sensitive-data handling.
• Partner with Security and Compliance to generate audit evidence supporting SOC 2, ISO 27001, HIPAA, and FDA 21 CFR Part 11 requirements.
• Maintain traceable and auditable release records for model, prompt, and configuration changes.

• During the first 60–90 days, assess the existing LLM infrastructure, document production model integrations, establish an approved model catalog, and launch an initial LLM gateway with at least one production service migrated to it.
• During the first year, consolidate production LLM traffic through the gateway, establish automated model and prompt release controls, productionize validated AI/ML prototypes, implement operational monitoring, and deliver compliance-ready audit evidence.

Qualifications

Required Experience

• 5+ years of professional backend software engineering experience, including development and operation of production APIs and multi-tenant SaaS services.
• Demonstrated experience building and deploying production applications using commercial LLM APIs, such as Anthropic, OpenAI, AWS Bedrock, or Azure OpenAI.
• Hands-on experience implementing prompt management, structured outputs, and tool use in LLM-powered applications.
• Practical experience designing LLM evaluation pipelines and integrating automated evaluation checks into CI/CD deployment workflows.
• Proven ability to take LLM-powered products beyond prototypes and manage production challenges involving evaluation regressions, latency, cost, outages, and audit requirements.
• Experience translating validated AI/ML research prototypes into reliable production software services.

Required Skills

• Strong Python programming skills for backend services, API integrations, and production AI infrastructure development.
• Working knowledge of commercial LLM providers, model APIs, prompt configuration, structured outputs, and tool-based AI functionality.
• Ability to design and maintain secure, scalable LLM gateway and inference infrastructure.
• Understanding of modern AI evaluation methodologies, including LLM-as-judge, red teaming, and regression evaluations, and the ability to implement automated release gates.
• Experience with AWS, containerized applications such as Amazon ECS or similar platforms, and infrastructure as code using Pulumi, Terraform, or AWS CDK.
• Proficiency with observability systems such as OpenTelemetry, Datadog, or comparable tools for tracking application performance, reliability, and AI inference behavior.
• Familiarity with software delivery tools and practices, including version control, CI pipelines, feature flags, and issue tracking systems such as Linear.
• Working knowledge of application security, including secrets management, access controls, and secure handling of sensitive information.
• Ability to apply reproducibility, reliability, and auditability standards to model configurations, prompt changes, and production releases.
• Ability to communicate technical decisions through design documentation, decision records, and architectural diagrams.
• Strong collaboration skills for working with researchers, clinicians, engineers, and security professionals to translate validated prototypes into production systems.
• Comfort working with sensitive subject matter involving healthcare, safety, and wellbeing.
• Ability to establish engineering standards and reusable infrastructure that enable other teams to deploy AI-powered features safely and efficiently.

Preferred Qualifications

• Experience developing software in regulated healthcare or life sciences environments, including systems subject to HIPAA, SOC 2, ISO 27001, or FDA 21 CFR Part 11 requirements.
• Experience building internal developer platforms, reusable engineering infrastructure, or standardized deployment workflows used across multiple teams.
• Experience with LLM-as-judge evaluation, AI red teaming, or conversational analytics.
• Experience training or fine-tuning machine learning models from scratch.
• Experience working with Node.js in backend engineering environments.