Responsibilities

About the Team The livestream industry has seen tremendous growth in recent years and has become the next growth driver for TikTok global business. It also brings great joy to users globally and it creates a new channel for our creators to show themselves. We are seeking a AI/ ML Data Operations Manager who owns the end-to-end quality and operating system of prompts, policy skills, and evaluation data that power multimodal LLM and Agent moderation. The role translates policies and expert judgment into model-executable strategies, runs closed-loop iteration from bad cases through release, and establishes standards so that delivery across multiple queues remains consistent, measurable, and scalable. This role sits at the intersection of Prompt Engineering, Data Operations, Evaluation Design and Cross functional Program Delivery, and works closely with Algorithm, Product, Engineering, Data Science, Operations and Labelling team. Responsibilities: - Lead Prompt Engineering and Data Operations. Own the full lifecycle of prompts, skills, and policy provisions for multimodal LLM Agents across video, image, for TTLive scenarios, including Prompt and skill generation, Version management, Release management , Online maintenance and Quality governance. - Design Training and Evaluation Data Strategies. Define data and iteration strategies for model training and evaluation, including: - Sampling strategies, Golden-set and benchmark construction, Blind-label protocols, Offline and online evaluation packages, A/B measurement , Substitute evaluation designs when instrumentation is incomplete - Align Models with Policies. Convert policy provisions and expert judgment into executable prompts and Agent skills, establishing: Prompt-writing standards, Writer principles, Admission criteria, Standard operating procedures, Quality and consistency requirements , Explainability requirements , Chain-of-thought or title and sub-title-level decision structures - Partner with Labeling Teams. Drive the delivery of high-quality training and evaluation data through: Labeling specifications , Quality-control SOPs , Ground-truth alignment , Edge-case mining, Throughput and quality SLAs, Error analysis, Dataset refresh, Model and strategy iteration - Define Tooling Requirements. Work with Product and Engineering to define requirements for: Case retrieval, Batch testing, Prompt debugging, Online-result backfill, Guidance and policy tools, Self-service RCA tools, Analysis assistants, Data-iteration workflows - Own Evaluation, Iteration, and Rollout. Diagnose false positives, false negatives, and metric movements across indicators such as: Accuracy , Risk, False Rejection, Coverage - Prioritize remediation actions, prepare Launch Review evidence, and support: Canary testing, A/B testing, Rollout decisions, Cross-queue launches - Build and Scale the PE Operating System. Standardize the full operating workflow: Intake → Sampling → Labeling → QC → Evaluation → Release - Build program-tracking and multi-queue delivery mechanisms so that new prompts and maintenance cycles are delivered predictably and at scale. - Drive Continuous Improvement from Production Signals. Convert online bad cases, RCA findings, and hard-policy constraints into: Prompt updates, Skill updates, Data updates, Reusable playbooks, Clear separation between confirmed conclusions and hypotheses

Qualifications

Minimum Qualification(s): - Bachelor’s degree or above. - 3+ years of experience in AI/ML data operations, prompt engineering, content-moderation strategy, annotation operations, technical program management, strategy, or related fields. - Hands-on experience designing or operating prompts and LLM workflows for production systems, including training, evaluation, or online inference. - Proven experience building evaluation frameworks, including golden sets, benchmarks, labeling SOPs, quality control, offline and online metrics, and experiment-ready evaluation packages; strong skills with a proactive approach to identifying and driving process improvements. - Strong experience working with labeling vendors or internal annotation teams. - Experience in taxonomy and specification design. - Experience in quality control and delivery management. Preferred Qualification(s): - Experience in content governance, Trust & Safety, or other high-stakes decision systems. - Exposure to multimodal model evaluation across video, image, text, URL, or webpage content. - Exposure to Agent systems, including planners, skills, tool use, and chain-of-thought or structured decision-making. - Familiarity with A/B experiments, metric decomposition, numerator and denominator analysis, mix-shift analysis, and case-level RCA. - Experience productizing internal tools for prompt debugging, case replay, and data-iteration platforms. - Experience owning zero-to-one operating systems, including SOPs, workflow tools such as Meego or Jira, dashboards, and scalable delivery processes. - Basic SQL and data-analysis capability. - Python scripting experience for evaluation pipelines is a plus. - PMP or equivalent program-management experience is a plus.

Job Information