Evaluate AI search responses for realistic failures involving relevance, source quality, freshness, language, completeness, and regional applicability. Develop search queries, test multiple scenarios, document model shortcomings, and write clear instructions and evaluation criteria. The role requires Chinese language expertise, strong research and source-verification skills, critical thinking, attention to detail, problem-solving, clear writing, and deep familiarity with local Australian cultural and regulatory context.
Overview
Are you passionate about languages, writing, and quality evaluation? We are launching an exciting language evaluation project and are looking for Chinese-speaking experts to join our team.
What you will do:
Identify realistic search-related failures where an AI model provides inadequate responses. Create search queries, test them, document what went wrong, and write instructions and evaluation criteria for correct answers.
Common issues: local relevance, source quality, outdated information, language mismatches, incomplete answers, and region-specific inapplicability. Testing multiple queries before finding failures is expected.
Project Details:
- Start Date: ASAP
- Duration: Ongoing Project.
- Commitment: 20 hours per week.
- Job Type: Freelance contract.
- Location: Remote, Australia
- Rate: $40 per hour.
Requirements:
- Research skills — ability to find and verify information from multiple sources; comfort navigating different websites, databases, and documentation
- Critical thinking — Can spot contradictions between sources, identify outdated information, and recognize when something doesn't apply to a specific locale/context
- Attention to detail — Can catch nuances in language, local variations (prices, regulations, requirements), and edge cases
- Problem-solving mindset — Comfortable iterating and testing when things don't work; doesn't get frustrated by trial- and-error
- Domain knowledge — Ideally someone with background in research, quality assurance, compliance, or similar work that requires precision and source verification
- Writing clarity — Can articulate instructions and rubrics clearly so others understand the logic behind their reasoning
Locale knowledge — Deep familiarity with their specific locale (local regulations, business practices, cultural nuances) is valuable for spotting region-specific failures
#L1-CC1
Similar Jobs
Software • Hospitality
Provides friendly and efficient food and beverage service in hotel dining outlets. Welcomes guests, takes orders, serves food and drinks, handles feedback, maintains outlet readiness, supports teamwork and revenue goals, follows food safety and hygiene standards, and participates in ongoing training.
Cloud • Information Technology • Productivity • Security • Software • App development • Automation
Lead enterprise post-sales engagements for Atlassian’s Strategy Collection, including Jira Align, Focus, Talent, and Rovo. Conduct executive discovery, design solution adoption architectures, plan phased rollouts, establish governance, define value measures, and guide customers toward sustained operating-model adoption. Partner with cross-functional teams, create reusable playbooks, provide thought leadership, and translate customer feedback into product improvements. The role involves up to 15% domestic and international travel.
Top Skills:
Atlassian FocusAtlassian GoalsAtlassian ProjectsAtlassian TalentConfluenceJIRAJira AlignRovo
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Drive new business and expand existing customer accounts across the New Zealand market. Manage the full sales cycle from prospecting through closure, build pipelines with SDRs, collaborate with Sales Engineers, forecast performance, and sell complex SaaS cybersecurity solutions to executive and technical stakeholders. Support subscription renewals, cross-selling, and up-selling while exceeding quotas and developing expertise in CrowdStrike products and the endpoint security market.
Top Skills:
Ai TechnologiesEndpoint SecuritySaaS
What you need to know about the Melbourne Tech Scene
Home to 650 biotech companies, 10 major research institutes and nine universities, Melbourne is among one of the top cities for biotech. In fact, some of the greatest medical advancements were conceptualized and developed here, including Symex Lab's "lab-on-a-chip" solution that monitors hormones to predict ovulation for conception, and Denteric's vaccine for periodontal gum disease. Yet, the thousands of people working in the city's healthtech sector are just getting started, to say nothing of the tech advancements across all other sectors.
