CAICCAIC: Center for Artificial Intelligence Challenge on Conversational AI Correctness

Challenges for Natural Language Processing (CNLPS)

The goal of this competition is to develop Natural Language Understanding models that are robust against speech recognition errors. Regardless of near-human accuracy of Automatic Speech Recognition in general-purpose transcription tasks, speech recognition errors can significantly deteriorate the performance of a Natural Language Understanding model that follows the speech-to-text module in a virtual assistant. The problem is even more apparent when an ASR system from an external vendor is used as an integral part of a conversational system without any further adaptation. The contestants are expected to develop NLU models that maintain satisfactory performance regardless of the presence of ASR errors in the input.

The data for the competition consist of natural language utterances along with semantic frames that represent the commands targeted at a virtual assistant. The detailed specification of the task and the description of the evaluation procedure are available here.

Submissions

The solutions for the task are to be submitted via the Gonito platform challenge. In order to be included in the final ranking the participants are expected to provide the report that describes their solution. The reports should conform to the requirements for papers submitted to FedCSIS conference and should not exceed 4 pages.

Presentations and publication

The authors of selected submissions will be invited to prepare the extended versions of their reports for publication in the conference proceedings and presentation at FedCSIS 2023. The selection will be made by a Jury on the basis of final evaluation results, quality of the submitted reports and originality of the presented methods.

Important Dates

Feb 13, 2023: Training data available (English)
Feb 20, 2023: Training data available (other languages)
May 17, 2023: Test data available
June 21, 2023: Deadline for submitting the results
June 23, 2023: Announcement of the final results, sending invitations for submitting papers
July 9, 2023: Deadline for submitting invited papers
July 16, 2023: Author notification
July 31, 2023: Final paper submission, registration
Sept 20, 2023: FedCSIS conference