The Winograd schema challenge composes tasks with syntactic ambiguity, which can be resolved with logic and reasoning.
Motivation
The dataset presents an extended version of a traditional Winograd challenge (Levesque et al., 2012): each sentence contains unresolved homonymy, which can be resolved based on commonsense and reasoning. The Winograd scheme is extendable with the real-life sentences filtered out of the National Corpora with a set of 11 syntactic queries, extracting sentences like "Katya asked Masha if she..." (two possible references to a pronoun), "A change of scenery that..." (Noun phrase & subordinate clause with "that" in the same gender and number), etc. The extraction pipeline can be adjusted to various languages depending on the set of ambiguous syntactic constructions possible.
An example in English for illustration purposes:
{
‘text’: ‘But then I was glad, because in the end the singer from Turkey who performed something national, although in a modern version, won.’,
‘answer’: ‘singer’,
‘label’: 1,
‘options’: [‘singer’, ‘Turkey’],
‘reference’: ‘who’,
‘homonymia_type’: ‘1.1’,
episode: [15],
‘perturbation’ : ‘winograd’
}
Data Fields
Data Splits
The dataset consists of a training set with labeled examples and a test set in two configurations:
The train and test sets are disjoint with respect to the sentence-candidate answer pairs but may include overlaps in individual sentences and homonymy type.
Test Perturbations
Each training episode in the dataset corresponds to six test variations, including the original test data and five adversarial test sets, acquired through the modification of the original test through the following text perturbations:
Paper | Code | Results | Date | Stars |
---|