A Collection of Winograd Schemas
schema is a pair of sentences that differ in only one or two words and
that contain an ambiguity that is resolved in
opposite ways in the two sentences and requires the use of world knowledge
and reasoning for its resolution. The schema takes its name from a
well-known example by Terry Winograd (1972)
The city councilmen refused the demonstrators a permit because
they [feared/advocated] violence.
If the word is ``feared'', then ``they'' presumably refers to the city council;
if it is ``advocated'' then ``they'' presumably refers to the demonstrators.
In his paper, ``The Winograd Schema Challenge'' Hector
Levesque (2011) proposes to assemble a set of such Winograd schemas that are
- easily disambiguated by the human reader (ideally, so easily that the
reader does not even notice that there is an ambiguity);
- not solvable by simple techniques such as selectional restrictions
- Google-proof; that is, there is no obvious statistical test over
text corpora that will reliably disambiguate these correctly.
The set would then be presented as a challenge for AI programs, along the
lines of the Turing test. The strengths of the challenge are that it is
clear-cut, in that the
answer to each schema is a binary choice; vivid, in that it is obvious to
non-experts that a program that
fails to get the right answers clearly has serious gaps in its understanding;
and difficult, in that it is far beyond the current state of the art.
This online document presents a collection of 114 Winograd schemas.
Item (1) is the original schema due to Terry Winograd;
items (2) --- (19) were constructed by Hector Levesque;
(20) --- (109) by Ernest Davis;
and (110)---(113) by Ray Jackendoff. Item (114) is adapted by Ernest Davis
from a sentence in Emma by Jane Austen.
We have made a deliberate effort to cover a wide range of world knowledge
and a wide range of linguistic features. At the same time, many of the
schemas were designed in closely related pairs; these are generally
adjacent in the schema. There did not seem to be any serious drawback
to including pairs of closely related schemas.
In some cases where we were uncertain whether the schema was Google-proof,
we have done some experiments with searches using Google's count of result
pages. These counts, however, are notoriously unreliable (Lapata and Keller,
2005) so these ``experiments''
should be taken with several grains of salt.
For the ease of machine reading, the text of each example is
bracketed with the HTML tag < text > < /text >. The question is
bracked with < question > < /question >. The answer is bracketed with
< answer > < /answer >. Comments are bracketed with < remark >
< /remark >. These tags are invisible in a browser.
The collection is open, and readers are invited to submit contributions,
which will be added if the editor considers that they statisfy the constraints
and are meaningfully different from the schemas already in the collection.
Email submissions to email@example.com.
Acknowledgements:Thanks to Hector Levesque, Leora Morgenstern,
and Ray Jackendoff.
M. Lapata and F. Keller,
Web-based models for natural language
processing, ACM Transactions on Speech and Language Processing
The Winograd Schema Challenge, Commonsense-2011.
Hector Levesque, Ernest Davis, and Leora Morgenstern,
The Winograd Schema Challenge,
submitted to KR-2012. An expanded version of the previous item.
Understanding Natural Language,
Academic Press, 1972.
Collection of Winograd Schemas
Created 9/8/2011 by Ernest Davis.
The city councilmen refused the demonstrators a permit because
they [feared/advocated] violence.
Who [feared/advocated] violence?
Answers: The city councilmen/the demonstrators.
The trophy would not fit into the brown suitcase because it
was too [small/large].
What was too [small/large]?
Answers:The suitcase/the trophy.
Joan made sure to thank Susan for all the help she had [given/received].
Who had [given/received] help?
Paul tried to call George on the phone, but he was not
Who was not [successful/available]
The lawyer asked the witness a question, but he was reluctant to
[answer/repeat] it .
Who was reluctant to [answer/repeat] the question?
Answers: The witness/the lawyer.
The delivery truck zoomed by the school bus because it was going so
What was going so [fast/slow]?
Answers: The truck/the bus
Frank felt [vindicated/crushed] when his longtime rival Bill revealed that
he was the winner of the competition.
Who was the winner of the competition?
The man could not lift his son because he was so [weak/heavy].
Who was [weak/heavy]?
Answers: The man/the son.
The large ball crashed right through the table because it was made of
What was made of [steel/styrofoam]?
Answers: The ball/the table.
John could not see the stage with Billy in front of him because he is so
[short/tall]. Who is so [short/tall]?
Tom threw his schoolbag down to Ray after he reached the [top/bottom] of the
Who reached the [top/bottom] of the stairs?
Although they ran at about the same speed, Sue beat Sally because she had such
a [good/bad] start.
Who had a [good/bad] start?
The sculpture rolled off the shelf because it was not [anchored/level].
What was not [anchored/level]?
Answers: The sculpture/the shelf.
Sam's drawing was hung just above Tina's and it did look much better with
another one [below/above] it.
Which looked better?
Answers: Sam's drawing/Tina's drawing.
Anna did a lot [better/worse]
than her good friend Lucy on the test because she had
studied so hard.
Who studied hard?
The firemen arrived [after/before] the police because they were coming from so
Who came from far away?
Answers: The firemen/the police.
Frank was upset with Tom because the toaster he had [bought from/sold to]
him did not work.
Who had [bought/sold] the toaster.
Jim [yelled at/comforted] Kevin because he was so upset.
Who was upset?
The sack of potatoes had been placed [above/below]
the bag of flour, so it had to be
What had to be moved first?
Answers: The sack of potatoes/the bag of flour.
Pete envies Martin [because/although] he is very successful.
Who is very successful?
I was trying to balance the bottle upside down on the table, but I
couldn't do it because it was so [top-heavy/uneven].
What was [top-heavy/uneven]?
Answers: the bottle/the table.
I spread the cloth on the table in order to [protect/display] it.
To [protect/display] what?
Answers: the table/the cloth.
The older students were bullying the younger ones, so we [rescued/punished]
Whom did we [rescue/punish]?
Answers: The younger students/the older students.
I poured water from the bottle into the cup until it was [full/empty].
What was [full/empty]?
Answers: The cup/the bottle.
Susan knows all about Ann's personal problems because she is
Who is [nosy/indiscreet]?
Sid explained his theory to Mark but he couldn't [convince/understand] him.
Who did not [convince/understand] whom?
Answers: Sid did not convince Mark/Mark did not understand Sid.
Susan knew that Ann's son had been in a car accident, [so/because] she told her
Who told the other about the accident?
Joe's uncle can still beat him at tennis, even though he is 30 years
Who is [older/younger]?
Comment: Note the following results of Google search (1/9/11).
"uncle older" 14,500,000
"uncle younger" 9,630,000
Thus "uncle" is not particularly associated with "older".
The police left the house and went into the garage, [where/after] they found
the murder weapon.
Where did they find the murder weapon?
Answers: In the garage/in the house.
Comment: "after" is perhaps slightly awkward here.
The painting in Mark's living room shows an oak tree.
It is to the right of [the bookcase/a house].
What is to the right of [the bookcase/a house]?
Answers: The painting/the tree.
There is a gap in the wall. You can see the garden [through/behind] it.
You can see the garden [through/behind] what?
Answers: The gap/the wall.
Comment. The syntax of the question is strictly incorrect, but
``[Behind/through] what can you see the garden?'' is
The drain is clogged with hair. It has to be [cleaned/removed].
What has to be [cleaned/removed]?
Answers: The drain/the hair.
My meeting started at 4:00 and I needed to catch the train at 4:30,
so there wasn't much time. Luckily, it was [short/delayed], so it worked out.
What was [short/delayed]?
Answers: The meeting/the train.
There is a pillar between me and the stage, and I can't [see/see around]
What can't I [see/see around]
Answers: The stage/the pillar.
They broadcast an announcement, but a subway came into the
station and I couldn't [hear/hear over] it.
What couldn't I [hear/hear over]?
Answers:} The announcement/the subway.
In the middle of the outdoor concert, the rain started falling, [and/but] it
continued until 10.
What continued until 10?
Answers: The rain/the concert.
I used an old rag to clean the knife, and then I put it in the [drawer/trash].
What did I put in the [drawer/trash]?
Answers: The knife/the rag.
Ann asked Mary what time the library closes, [but/because] she had forgotten.
Who had forgotten?
I took the water bottle out of the backpack so that it would be [lighter/handy].
What would be [lighter/handy]?
Answers: The backpack/the bottle.
I couldn't put the pot on the shelf because it was too high/tall.
What was too [high/tall].
Answers: The shelf/the pot.
The Google query ``high pot" gives about 10 times as many pages
as ``tall pot" (search 9/8/11)
for reasons unconnected with the meanings in this sentence.
I'm sure that my map will show this building; it is very [famous/good].
What is [famous/good]?
Answers: The building/the map.
Comment: ``Detailed'' would of course be a better word to describe the
map, but it would certainly be Googlable, and probably solvable by
Bob paid for Charlie's college education. He is very [generous/grateful].
Who is [generous/grateful]?
Bob paid for Charlie's college education, but now Charlie acts as though
it never happened. He is very [hurt/ungrateful].
Who is [hurt/ungrateful]?
Bob was playing cards with Adam and was way ahead. If Adam hadn't had a
sudden run of good luck, he would have [won/lost].
Who would have [won/lost]?
Adam can't leave work here until Bob arrives to replace him. If Bob had
left home for work on time, he would be [here/gone] by this time.
Who would be [here/gone]?
If the con artist has succeeded in fooling Sam, he would have [gotten/lost] a
lot of money.
Who would have [gotten/lost] the money?
Answers: The con artist/Sam.
It was a summer afternoon, and the dog was sitting in the middle of the
lawn. After a while, it got up and moved under the tree, because it was
What was [hot/cooler]?
``What?'' question, but the phrasing of the text seems natural enough,
and the antecedent of ``it'' has got to be ``under the tree.''
Answers: The dog/under the tree.
Admittedly a prepositional phrase is an odd answer to a
The cat was lying by the mouse hole waiting for the mouse, but it was too
What was too cautious/impatient?
Answers: The mouse/the cat.
Anne gave birth to a daughter last month. She is a very charming
Who is a very charming [woman/baby]?
Answers: Anne/Anne's daughter.
At the party, Amy and her friends were [chatting/barking];
her mother was frantically
trying to make them stop. It was very strange behavior.
Who was behaving strangely?
Answers: Amy's mother/Amy and her friends.
Admittedly, a strange example, but I think a valid WS. Note that
the question does not directly ask about the antecedent of ``It'', but phrasing
it so that it does requires a clumsy answer: ``What was strange behavior?''
``Amy's mother trying to stop her and her friends from chatting/Amy
and her friends barking.''
I saw Jim yelling at some guy in a military uniform with a huge red
beard. I don't know [who/why] he was, but he looked very unhappy.
Who looked very unhappy?
Answers: The guy in the uniform/Jim.
If you stop the second sentence at "I don't know [who/why] he was" you
still have the same ambiguity, but it becomes very difficult to ask the
question. "I don't know who who was?'' is perhaps best.
Of course, either way there is a reading in
which the two ``he'' refer to different people, but it seems to me that that's
considerably less preferred.
The fish ate the worm. It was [tasty/hungry].
What was [tasty/hungry]?
Answers: The worm/the fish.
I was trying to open the lock with the key, but someone had filled the
keyhole with chewing gum, and I couldn't get it [in/out].
What couldn't I get [in/out]?
Answers: The key/the chewing gum.
The dog chased the cat, which ran up a tree. It waited at the [top/bottom]
Which waited at the [top/bottom]?
Answers: The cat/the dog.
In the storm, the tree fell down and crashed through the roof of my
house. Now, I have to get it [removed/repaired]
What has to be [removed/repaired]?
Answers: The tree/the roof.
The customer walked into the bank and stabbed one of the tellers. He was
immediately taken to the [emergency room/police station].
Who was taken to the [emergency room/police station]?
Answers: The teller/the customer.
John was doing research in the library when he heard a man humming and
whistling. He was very [annoyed/annoying].
Who was [annoyed/annoying]?
John was jogging through the park when he saw a man juggling watermelons.
He was very [impressed/impressive]
Who was [impressed/impressive]?
Answers: John/the juggler.
Bob collapsed on the sidewalk. Soon he saw Carl coming to help. He was
Who was [ill/concerned]?
Sam and Amy are passionately in love, but Amy's parents are unhappy
about it, because they are [snobs/fifteen].
Who are [snobs/fifteen]?
Answers: Amy's parents/Sam and Amy.
Mark told Pete many lies about himself, which Pete included in his book.
He should have been more [truthful/skeptical].
Who should have been more [truthful/skeptical]?
Joe has sold his house and bought a new one a few miles away. He will
be moving [out of/into] it on Thursday.
Which house will he be moving [out of/into]?
Answers: The old house/the new house.
Many people start to read Paul's books and can't put them down. They
are [gripped/popular] because Paul writes so well.
Who or what are gripped/popular?
Answers: The readers/the books.
A lot of synonyms for ``gripped" (e.g. ``fascinated") can be resolved
by selectional restrictions. ``Gripped" cannot because it is polysemic with
Mary took out her flute and played one of her favorite pieces. She
has [loved/had] it since she was a child.
What has Mary [loved/had] since she was a child?
Answers: The piece/the flute.
Sam pulled up a chair to the piano, but it was broken, so he
had to [stand/sing] instead.
What was broken?
Answers: The chair/the piano.
Since it was raining, I carried the newspaper [over/in] my backpack to
keep it dry. What was I trying to keep dry?
Answers: The backpack/the newspaper.
The form of the question is quite far from the text, but there
doesn't seem to be a better way to formulate this question.
Sara borrowed the book from the library because she needs it for an
article she is working on. She [reads/writes] it when she gets home from work.
What does Sara [read/write] when she gets home from work/
Answers: The book/the article.
This morning, Joey built a sand castle on the beach, and put a toy
flag in the highest tower, but this afternoon the [wind/tide] knocked it down.
What did the [wind/tide] knock down?
Answers: The flag/the sand castle.
Comment: Perhaps a little overly delicate.
Jane knocked on Susan's door, but there was no answer. She was
Who was [out/disappointed]?
Comment: Note that Jane is also "out" (of the house or room);
The disambiguation in this case involves issues of textual coherence. A
simplified version is given in #115.
Jane knocked on the door, and Susan answered it. She invited her to come
Who invited whom?
Answers: Jane invited Susan/Susan invited Jane.
Sam took French classes from Adam, because he was [eager/known] to
speak it fluently.
Who was [eager/known] to speak French fluently?
The path to the lake was blocked, so we couldn't [reach/use] it.
What couldn't we [reach/use]?
Answers: The lake/the path.
The sun was covered by a thick cloud all morning, but luckily, by the time
the picnic started, it was [gone/out].
What was [gone/out]?
Answers: The cloud/the sun.
We went to the lake, because a shark had been seen at the ocean beach, so
it was a [dangerous/safer] place to swim.
Which was a [dangerous/safer] place to swim?
Answers: The beach/the lake.
Sam tried to paint a picture of shepherds with sheep, but they ended up
looking more like [dogs/golfers].
What looked like [dogs/golfers]?
Answers:the sheep/the shepherds.
Mary tucked her daughter Anne into bed, so that she could [sleep/work].
Who is going to [sleep/work]?
Fred and Alice had very warm down coats, but they were not [enough/prepared]
for the cold in Alaska.
Who or what were not [enough/prepared] for the cold?
Answers: The coats/Fred and Alice.
Thomson visited Cooper's grave in 1765. At that date he had been
[dead/travelling] for five years.
Who had been [dead/travelling] for five years?
Jackson was greatly influenced by Arnold, though he lived two centuries
Who lived [earlier/later]?
Tom's daughter Eva is engaged to Dr. Stewart, who is his partner.
The two [doctors/lovers] have known one another for ten years.
Which two people have known one another for ten years?
Answers: Tom and Dr. Stewart / Eva and Dr. Stewart.
I can't cut that tree down with that axe; it is too [thick/small].
What is too [thick/small]?
Answers: The tree/the axe.
The foxes are getting in at night and attacking the chickens. I shall have
to [guard/kill] them.
What do I have to [guard/kill]?
Answers: The chickens/the foxes.
The foxes are getting in at night and attacking the chickens. They have
gotten very [bold/nervous].
What has gotten [bold/nervous]?
Answers: The foxes/the chickens.
Fred covered his eyes with his hands, because the wind was blowing sand
around. He [opened/lowered] them when the wind stopped.
What did Fred [open/lower]?
Answers: His eyes/his hands.
The actress used to be named Terpsichore, but she changed it to Tina a
few years ago, because she figured it was [easier/too hard] to pronounce.
Which name was [easier/too hard] to pronounce?
This question can be answered without seeing the text, just seeing
the question and the choice of possible answers. I don't see that this is
a defect, but it is certainly an anomaly.
Fred watched TV while George went out to buy groceries. After an hour he
Who got [up/back]?
Fred was supposed to run the dishwasher, but he put it off, because
he wanted to watch TV. But the show turned out to be boring, so he
changed his mind and turned it [on//off].
What did Fred turn [on/off]?
Answers: The dishwasher/the television.
Fred is the only man still alive who remembers my great-grandfather. He
[is/was] a remarkable man.
Who [is/was] a remarkable man?
Answers: Fred/my great-grandfather.
Fred is the only man alive who still remembers my father as an infant.
When Fred first saw my father, he was twelve [years/months] old.
Who was twelve [years/months] old?
Answers: Fred/my father.
In July, Kamtchatka declared war on Yakutsk. Since Yakutsk's army was much
better equipped and ten times larger, they were [victorious/defeated]
Who was [victorious/defeated]
Elizabeth moved her company from Sparta to Troy to save money on
taxes; the taxes are much [higher/lower] there.
Where are the taxes [higher/lower]?
Answers: In Sparta/In Troy
Esther figures that she will save shipping costs if she builds her
factory in Springfield instead of Franklin, because [most/none]
of her customers live there.
In which town do [most/none] of Esther's customers live?
Look! There is a [shark/minnow] swimming right below that duck! It had better
get away to safety fast!
What needs to get away to safety?
Answers: The duck/the minnow.
There are too many deer in the park, so the park service brought in a
small pack of wolves. The population should [increase/decrease] over the
next few years.
Which population will [increase/decrease]?
Answers: The wolves/the deer.
Archaeologists have concluded that humans lived in Laputa 20,000 years
ago. They hunted for [deer/evidence] on the river banks.
Who hunted for [deer/evidence]?
Answers: The prehistoric humans/the archaeologists.
The scientists are studying three species of fish that have recently
been found living in the Indian Ocean. They [appeared/began] two years ago.
Who or what [appeared/began] two years ago?
Answers: The fish/the scientists.
The journalists interviewed the stars of the new movie. They were very
[cooperative/persistent], so the interview lasted for a long time.
Who was [cooperative/persistent]?
Answers: The stars/the journalists
The police arrested all of the gang members. They were trying to [run/stop]
the drug trade in the neighborhood.
Who was trying to [run/stop] the drug trade?
Answers: The gang/the police.
Comment: Hopefully the reader is not too cynical.
I brought the flower pot out to the garage. It is full of [dirt/tools].
What is full of [dirt/tools]?
Answers: The flower pot/the garage.
I put the cake away in the refrigerator. It has a lot of [butter/leftovers]
What has a lot of [butter/leftovers]?
Answers: The cake/the refrigerator.
Sam broke both his ankles and he's walking with crutches. But a month
or so from now they should be [better/unnecessary].
What should be [better/unnecessary]?
Answers: The ankles/the crutches.
When the sponsors of the bill got to the town hall, they were surprised
to find that the room was full of opponents. They were very much in the
Who were in the [majority/minority]?
Answers: The sponsors/the opponents.
Everyone really loved the oatmeal cookies; only a few people liked the
chocolate chip cookies. Next time, we should make [more/fewer] of them.
Which cookie should we make [more/fewer] of, next time?
Answers: The oatmeal cookies/the chocolate chip.
We can't put a copy of the program on every chair in the theater.
There are [too many/not enough] of them.
There are [too many/not enough] of what?
Answers: chairs/copies of the program.
I stuck a pin through a carrot. When I pulled the pin out, it [left/had]
What [left/had] a hole?
Answers: The pin/the carrot.
Note: You might think this is Googlable, but in fact, ``pin left a hole"
finds two results and ``pin leaves a hole" finds 66, whereas ``pin has a hole"
has 11,800 (because of equipment pins, not sewing pins). The phrase
"carrot has a hole" has 2 results and "carrot left a
hole" has none. (8/17/10)
I couldn't find a spoon, so I tried using a pen to stir my
coffee. But that turned out to be a bad idea, because it got full of
What got full of [ink/coffee]?
Answers:The coffee/the pen.
Comment: The statistical associations give the backward answer here:
``ink'' is associated with ``pen'' and ``coffee'' is associated with
``coffee''. Of course, a contestant could use a backward rule here: Since
the challenge designers have excluded examples where statistics give
the right answer, if you find a statistical relation, guess that
the answer runs opposite to it. But that seems very risky.
Steve follows Fred's example in everything. He [admires/influences] him hugely.
Who [admires/influences] whom?
Answers: Steve admires Fred/Fred influences Steve.
The table won't fit through the doorway because it is too [wide/narrow].
What is too [wide/narrow]?
Answers: The table/the doorway.
Grace was happy to trade me her sweater for my jacket. She thinks it looks
[great/dowdy] on her.
What looks [great/dowdy] on Grace?
Answers: The jacket/the sweater.
Bill thinks that calling attention to himself was rude [to/of] Bert.
Who called attention to himself?
John [hired/hired himself out to] Bill to take care of him.
Who is taking care of whom?
Answers: Bill is taking care of John/John is taking care of Bill.
John [promised/ordered] Bill to leave, so an hour later he left.
Sam Goodman's biography of the Spartan general Xenophanes
conveys a vivid sense of the difficulties he faced in his
Who faced difficulties?
It is quite possible that "biography" is correlated with "research".
But even if that correlation is detected, there is another non-trivial
step to realize that the research is associated with the author rather
than the subject of the biography.
Emma's mother had died long ago, and her [place/education] had been
[taken/managed] by an excellent woman as governess.
Whose [place/education] had been [taken/managed]?
Answers: Emma's mother/Emma.
The first version is adapted from a sentence in the second paragraph
of Jane Austen's Emma. The original is
Her mother had died too long ago for her to have more than an indistinct
remembrance of her caresses; and her place had been taken by an excellent
woman as governess, who had fallen little short of a mother in affection.
Note that the original presents two further difficult disambiguation problems,
with the second and third "her"'s in the sentence.
Jane knocked on Susan's door but she did not [answer/get an answer].
Who did not [answer/get an answer]?
Simplified version of #69. Like #69, it relies on
textual coherence for disambiguation; it is, after all, equally true that Jane
did not answer, and that Susan did not get an answer.
Last update: 11/27/2011