Your work days are brighter here.
At Workday, it all began with a conversation over breakfast. When our founders met at a sunny California diner, they came up with an idea to revolutionize the enterprise software market. And when we began to rise, one thing that really set us apart was our culture. A culture which was driven by our value of putting our people first. And ever since, the happiness, development, and contribution of every Workmate is central to who we are. Our Workmates believe a healthy employee-centric, collaborative culture is the essential mix of ingredients for success in business. That's why we look after our people, communities and the planet while still being profitable. Feel encouraged to shine, however that manifests: you don't need to hide who you are. You can feel the energy and the passion, it's what makes us unique. Inspired to make a brighter work day for all and transform with us to the next stage of our growth journey? Bring your brightest version of you and have a brighter work day here.
At Workday, we value our candidates' privacy and data security. Workday will never ask candidates to apply to jobs through websites that are not Workday Careers.
Please be aware of sites that may ask for you to input your data in connection with a job posting that appears to be from Workday but is not.
In addition, Workday will never ask candidates to pay a recruiting fee, or pay for consulting or coaching services, in order to apply for a job at Workday.
About the Team
You'll be joining the Machine Learning Serving Infrastructure team, the team responsible for building and maintaining Workday's LLM Gateway. This critical infrastructure powers the seamless integration of large language models across Workday's products and services. Our team is at the forefront of LLM deployment, ensuring scalability, reliability, and performance for our cutting-edge AI capabilities. We work with the latest advancements in LLM technology, and this role will be instrumental in evaluating and optimizing the models that drive our platform. You'll be part of a collaborative and innovative environment, working alongside experienced engineers who are passionate about pushing the boundaries of what's possible with LLMs in enterprise applications. Your contributions will directly impact the efficiency and effectiveness of our LLM Gateway, ensuring Workday remains a leader in AI-powered solutions.About the Role
We're seeking a highly skilled and motivated Senior Machine Learning Engineer to join a team and drive ground breaking experimentation with Large Language Models (LLMs). In this role, you'll be the primary architect and executor of experiments designed to rigorously compare and evaluate the performance of various LLMs, particularly focusing on the impact of model updates on our feature set. You'll be responsible for crafting robust experimental frameworks, implementing automated evaluation pipelines, analyzing sophisticated datasets, and communicating your findings to both technical and non-technical collaborators.
Key Responsibilities:
Design and implement comprehensive experimental frameworks for comparing LLM performance across defined features.
Develop automated pipelines for data collection, model inference, and performance evaluation.
Conduct thorough statistical analysis of experimental results to identify significant performance differences.
Build detailed reports and presentations summarizing experimental findings and providing actionable insights.
Collaborate with other ML engineers and product teams to integrate experimental results into product development.
Stay up-to-date with the latest advancements in LLM research and evaluation methodologies.
Optimize and maintain existing evaluation tools and infrastructure.
Define and track key performance indicators (KPIs) for LLM performance.
Document all experiments, results, and methodologies for reproducibility.
About You
We're looking for a seasoned ML Engineer with a strong passion for LLMs and a proven track record of designing and executing complex experiments. You thrive in a fast-paced environment and are eager to contribute to the advancement of LLM technology.
Basic Qualifications:
Master's or Ph.D. in Computer Science, Machine Learning, or equivalent experience.
5+ years of experience in machine learning engineering, with a focus on natural language processing (NLP).
1+ year experience with LLM architectures, training methodologies, and evaluation metrics.
3+ years' experience in Python and experience with relevant machine learning libraries (e.g., TensorFlow, PyTorch, scikit-learn).
Experience with cloud computing platforms (e.g., Google Cloud Platform, AWS, etc).
2+ years' experience with version control systems such as git.
Other Qualifications:
Solid understanding of statistical analysis and experimental design principles.
Excellent written and verbal communication skills.
Experience with LLM evaluation frameworks and concepts (e.g., LLM-as-a-judge, ragas).
Experience with prompt engineering and fine-tuning LLMs.
Familiarity with containerization and orchestration tools (e.g., Docker, Kubernetes).
Experience with large-scale data processing and analysis.
Experience with building and maintaining automated testing and CI/CD pipelines.
Publications or contributions to the field of LLM research.
Experience with the Gemini, Anthropic, and Nova family of models.
Experience with A/B testing methodologies within an LLM environment.
Familiarity with ethical considerations in LLM development and deployment.
Posting End Date: 4/2/2025
If hired in Colorado, click here for information about Workday's comprehensive benefits in Colorado: https://workdaybenefits.com/us/welcome-to-workday-benefits/prospective-workmates.
The application deadline for this role is the same as the posting end date stated.
Workday Pay Transparency Statement
The annualized base salary ranges for the primary location and any additional locations are listed below. Workday pay ranges vary based on work location. As a part of the total compensation package, this role may be eligible for the Workday Bonus Plan or a role-specific commission/bonus, as well as annual refresh stock grants. Recruiters can share more detail during the hiring process. Each candidate's compensation offer will be based on multiple factors including, but not limited to, geography, experience, skills, job duties, and business need, among other things. For more information regarding Workday's comprehensive benefits, please click here.
Primary Location: USA.CO.Boulder
The application deadline for this role is the same as the posting end date stated as below:
Our Approach to Flexible Work
With Flex Work, we're combining the best of both worlds: in-person time and remote. Our approach enables our teams to deepen connections, maintain a strong community, and do their best work. We know that flexibility can take shape in many ways, so rather than a number of required days in-office each week, we simply spend at least half (50%) of our time each quarter in the office or in the field with our customers, prospects, and partners (depending on role). This means you'll have the freedom to create a flexible schedule that caters to your business, team, and personal needs, while being intentional to make the most of time spent together. Those in our remote "home office" roles also have the opportunity to come together in our offices for important moments that matter.
Pursuant to applicable Fair Chance law, Workday will consider for employment qualified applicants with arrest and conviction records.
Workday is an Equal Opportunity Employer including individuals with disabilities and protected veterans.
Are you being referred to one of our roles? If so, ask your connection at Workday about our Employee Referral process!
