Volunteer AI Data Annotator (Benchmarks & Communication Rubrics) — AI Apps for Human Communication

Review, label, and score conversational AI data to help establish gold-standard benchmarks for tools designed to reduce political polarization and foster constructive dialogue.

Crossing Party Lines is seeking a thoughtful, detail-focused AI Data Annotator to join our Apps and Agents team. Our conversational agents are built to handle emotionally sensitive, politically nuanced interactions. Standard automated evaluators often miss subtle conversational missteps, which makes human ground-truth data indispensable. In this role, you will analyze simulated dialogues, evaluate AI responses against qualitative communication rubrics, and generate labeled benchmark datasets used to train, evaluate, and align our models.

About the Apps and Agents Team

Our team integrates generative AI with Crossing Party Lines’ proprietary frameworks developed through nearly a decade of real-world dialogue facilitation. As we prepare an application to Y Combinator and expand our suite of digital role-play and coaching tools, high-quality human data is essential to ensure our agents remain objective, emotionally safe, and genuinely depolarizing.

What You’ll Do

• Review multi-turn conversational transcripts between users and AI agents across diverse, high-friction social and political topics.
• Annotate and score AI outputs using structured rubrics covering active listening, perspective-taking, neutrality, de-escalation, tone calibration, and defensiveness triggers.
• Identify and tag failure modes such as subtle ideological bias, sycophancy, tone-policing, preachy phrasing, and hallucinations.
• Write and curate high-quality "golden responses" (ideal model outputs) to serve as few-shot exemplars and ground-truth references for evaluation pipelines.
• Provide qualitative feedback to AI engineers, prompt designers, and subject-matter experts on recurring model behaviors and annotation edge cases.
• Participate in inter-annotator calibration sessions to maintain high data consistency and label reliability across the team.

You May Be a Good Fit If You

• Have strong reading comprehension, analytical judgment, and close attention to language nuance and subtext.
• Bring an objective, open-minded approach capable of evaluating dialogue across a wide spectrum of political and social viewpoints without personal bias.
• Have an interest in linguistics, psychology, communication studies, political science, philosophy, or AI safety.
• Enjoy repetitive, analytical tasks that require high precision, consistency, and adherence to labeling guidelines.
• Prior experience with data annotation platforms (Label Studio, Prodigy, or custom spreadsheets) is a plus, but not required.

What You’ll Gain

• Practical experience in human-in-the-loop (HITL) AI development and LLM benchmarking methodologies.
• A direct role in setting the safety and alignment standards for civic technology applications.
• Collaboration with AI engineers, data specialists, and conflict-resolution experts.
• Professional references recognizing your attention to detail, analytical rigor, and contributions to production AI evaluation datasets.