Job Description

Test Engineering Analyst (ML/AI Chatbot)

We are seeking a Machine Learning, AI Chatbot Test Engineering Analyst to our client's team to drive innovation in AI-powered division leveraging Chatbots within their proprietary platform for external information. This is a great opportunity to work with cutting-edge AI and automation technologies.

Key Responsibilities:
- Conduct deep technical analysis and testing of AI/ML and Generative AI systems, including validation and verification of model outputs, prompt behavior, and end-to-end application workflows.
- Participate in Business Requirements reviews and collaborate with systems analysts and developers to fully test application changes, including AI/ML model integrations and Gen AI features.
- Understand functional specifications and scope of releases to design robust testing harnesses that ensure quality delivery of AI-powered applications.
- Write complex SQL queries for retrieval of data from multi-database environments, including querying ML feature stores and model metadata. (Strong knowledge of Oracle database required).
- Apply expert systems analysis and design techniques to complex enterprise systems, with a focus on validating Gen AI outputs, LLM responses, and ML model predictions.
- Maintain broad knowledge of data sources/flow, interactions of complex systems, and the capabilities and limitations of AI/ML systems, including model drift, hallucination detection, and bias evaluation.
- Work closely with Product Owners, developers, Data Scientists, and stakeholders to understand AI/ML product specifics and create corresponding test strategies, including evaluation frameworks for generative outputs.
- Execute manual and exploratory testing on frontend (UI) and backend (API) systems, including AI/ML inference endpoints and Gen AI prompt pipelines.
- Develop and maintain testing suites for a large-scale Salesforce environment integrated with AI/ML capabilities such as Salesforce Einstein and external LLM services. Responsible for front-end and back-end testing for applications built on the Salesforce platform, including AI-driven features and automated decision systems.

This position is a hybrid opportunity (1-2x onsite per week in Maryland and/or Wash DC office).-

Required Skills

- Bachelor's degree in Computer Science, IT, or related field is preferred
- 3+ years of experience in software testing and test automation, leveraging Selenium Java / BDD Framework
- Proven experience with AI tools for test generation, refactoring, and self-healing
- Experience building and scaling testing agents
- Knowledge of context engineering for AI-driven testing
- Experience with modern testing frameworks beyond Selenium
- Background in machine learning or AI applications
- Experience with microservices and containerized applications

Required Technical Skills
- Databases: Snowflake and Databricks
- Programming Languages: Python, Java/Selenium, SQL
- AI/ML Frameworks: LangChain, Pandas, and other popular Machine Learning libraries
- Test Automation: Selenium, Playwright, API testing frameworks
- Clouds: Any Cloud platform will suffice, AWS preferred (Bedrock, Sagemaker)
- CI/CD: CloudBees or Jenkins

Soft Skills
- Strong analytical and problem-solving abilities
- Excellent communication and collaboration skills
- Adaptability to rapidly evolving AI and automation technologies
- Strategic thinking with focus on testing efficiency and coverage
- Self-motivated with ability to work independently

Apply Now

Job Tags

Contract work, Work at office

Similar Jobs