AI Product Manager (Evaluation) - Flow
Apply or review the details on the original posting.
Apply for this roleOriginal posting on LinkedIn
Responsibilities
About the Team
Flow is ByteDance's AI-native innovation business team, focusing on developing products in emerging fields such as LLM. Our product portfolio includes Doubao (China) and Dola (Global). The Model Evaluation Team plays a key role within Flow, providing accurate and timely assessment signals for product iterations. We are responsible for the entire evaluation process, from designing methodology to executing evaluations, identifying issues, and proposing feasible improvement suggestions.
Join us to explore innovative opportunities in AI, grow in a dynamic team environment, and create real value for users!
Responsibilities
- Evaluation Framework Establishment: Design evaluation frameworks for the Dola App including standards, test cases, and workflows. Create metrics that measure performance across key dimensions and enable data-driven decisions in existing and upcoming use scenarios
- Human-Machine Evaluation Integration: Build workflows combining human expert and automated evaluation. Ensure automated processes align with human judgment, identify performance gaps, and increase testing coverage to supplement human evaluation
- Evaluation Results Analysis: Interpret and analyze evaluation findings with depth, transforming nuanced case analysis into actionable insights. Develop data production and product design strategies aimed at enhancing model capabilities and/or user experience
- Cross-functional Collaboration: Align product, engineering and evaluation teams across cultural and professional boundaries. Facilitate communication, resolve differences, and build consensus around overall evaluation methodologies
- Evaluation Methodology Research: Stay at the forefront of LLM evaluation by researching industry-leading assessment methodologies and reports. Integrate insights with practical applications to continuously improve our evaluation framework
Qualifications
Minimum Qualifications
- Bachelor's degree or above in Computer Science, Data Science, Business Analytics, or related fields
- 3+ years of experience as a Product Manager, Business Analyst, or Operations Specialist in technology companies or other highly dynamic environment
- Strong data analysis skills with experience in designing metrics for product performance
- Project management capabilities with the ability to balance priorities and handle ambiguous requirements
- Quick adaptation to new tools, technologies and concepts, with demonstrated learning agility
- Genuine passion for AI/LLM technologies
- Professional proficiency in both English and Mandarin is required for collaboration with key stakeholders and businesses based in China and across APAC. This collaboration is necessary to support the development of new technology and drive improvements to the company's platforms
Preferred Qualifications
- Experience in AI/ML product management, particularly with conversational AI
- Hands-on experience with LLM evaluation methodologies
- Background in developing metrics frameworks for measuring product performance
- Excellent stakeholder management and communication skills
About Us
Founded in 2012, ByteDance's mission is to inspire creativity and enrich life. With a suite of more than a dozen products, including TikTok, Lemon8, CapCut and Pico as well as platforms specific to the China market, including Toutiao, Douyin, and Xigua, ByteDance has made it easier and more fun for people to connect with, consume, and create content.
Why Join ByteDance
Inspiring creativity is at the core of ByteDance's mission. Our innovative products are built to help people authentically express themselves, discover and connect – and our global, diverse teams make that possible. Together, we create value for our communities, inspire creativity and enrich life - a mission we work towards every day.
As ByteDancers, we strive to do great things with great people. We lead with curiosity, humility, and a desire to make impact in a rapidly growing tech company. By constantly iterating and fostering an "Always Day 1" mindset, we achieve meaningful breakthroughs for ourselves, our Company, and our users. When we create and grow together, the possibilities are limitless. Join us.
Diversity & Inclusion
ByteDance is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At ByteDance, our mission is to inspire creativity and enrich life. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too.