Researchers present CALICO, a human-centered system that helps domain experts turn their annotation codebooks into prompts for large language models. The work targets gaps in existing pipelines, which offer little support for producing prompts that stay reliable, easy to revise, and auditable. It is described in a paper posted to arXiv under the cs.CL category.
A new preprint evaluates whether automatically generated narrative feature labels would be endorsed by human annotators. The authors compare LLM-based and rule-based annotation against human judgments across three studies using the Turkish-language Objective Projection corpus. The work contributes inter-rater reliability evidence for datasets that ship machine-generated annotations.
A new paper studies how human labelers disagree about the precise onset and boundaries of laughter when annotating audio. The authors show that this disagreement follows systematic patterns rather than acting as random noise, challenging the common practice of scoring temporal laughter localization against a single reference annotation. They argue that evaluation protocols should instead account for the structured nature of annotator disagreement.
A new arXiv paper introduces TimeCues Studio, a workspace for placing precise annotations such as points, segments, and loops on music recordings. The tool supports both hand labeling and algorithmic prototyping, targeting the shortage of annotated training data that limits machine-learning approaches in multimedia applications.
A new arXiv preprint argues that industrial systems matching user queries to AI agents often confuse topical relevance with whether an agent can actually execute the request, especially for rare or ambiguous cases. The authors frame annotation as capability-bound process supervision and introduce a method for labeling this data. The work targets more reliable agent selection in production settings.
A new arXiv paper examines whether clinical annotation projection between languages can be treated as a document-level generative task that keeps the original text intact. The work spans six languages and aims to output character-level annotations that can be verified automatically. The stated goal is to support the construction of multilingual clinical corpora.