Coreference Resolution Tools : A First Look
Coreference is where two or more noun phrases refer to the same entity. This is an integral part of natural languages to avoid repetition, demonstrate possession/relation etc.
Eg: Harry wouldn’t bother to read “Hogwarts: A History” as long as Hermione is around. He knows she knows the book by heart.
The different types of coreference includes:
Noun phrases: Hogwarts A history <- the book
Pronouns : Harry <- He
Possessives : her, his, their
Demonstratives: This boy
Coreference resolution or anaphor resolution is determining what an entity is refering to. This has profound applications in nlp tasks such as semantic analysis, text summarisation, sentiment analysis etc.
In spite of extensive research, the number of tools available for CR and level of their maturity is much less compared to more established nlp tasks such as parsing. This is due to the inherent ambiguities in resolution.
The following are some of the tools currently available. Almost all tools come with bundled sentence deducters, taggers, parsers, named entity recognizers etc as setting them up all would be tedious.
Let us try using the following sentence from one of the presentations on BART as input. I'm using the demo app wherever possible and where not, I'm installing the same on my local machine.
Queen Elizabeth set about transforming her husband, King George VI, into a viable monarch. Lionel Logue, a renowned speech therapist, was summoned to help the King overcome his speech impediment.
The following equivalence sets need to be identified.
QE: Queen Elizabeth, her
KG: husband, King George VI, the King, his
LL: Lionel Logue, a renowned speech therapist
The results are as follows.
| Tools | Result | Comments |
| Illinois Coreference Package | Lionel Logue(0) a renowned speech |therapist|(2) his(8) Queen Elizabeth(3) the |King|(5) King(4) King |George VI|(7) transforming her |husband|(6) her(1) a viable |monarch|(9) |
The ‘his’ and ‘her’ are wrongly matched to the wrong entities. his is matched to Logue and her is matched to King George |
| CherryPicker | <COREF ID=”1″>Queen</COREF> <COREF ID=”2″>Elizabeth</COREF> set about transforming <COREF ID=”3″ REF=”2″>her</COREF> <COREF ID=”4″>husband</COREF>, <COREF ID=”5″>King</COREF> <COREF ID=”6″ REF= “5″>George VI</COREF>, into a viable monarch. <COREF ID=”9″>Lionel Logue</COREF>, a renowned speech <COREF ID=”10″>therapist</COREF>, was summoned to help the <COREF ID=”7″ REF=”5″>King</COREF> overcome <COREF ID=”8″ REF=”5″>his</COREF> sp eech impediment. Queen |
It is mostly ok, except that Queen Elizabeth is split into two Entities Queen and Elizabeth . Other than that, it is one of the best results. Notably it matches King to King George VI and hence, his is correctly mapped to King George VI |
| Natural Language Synergy Lab | 0 Queen Elizabeth set about transforming 0 her husband , King George VI , into a viable monarch. Lionel Logue , a renowned speech therapist , was summoned to help the King overcome 0 his speech impediment . | |
| BART |
{person Queen Elizabeth } set about transforming {np {np her } husband } , {person King George VI } , into {np a viable monarch } . {person Lionel Logue } , {np a renowned {np speech } therapist } , was summoned to help {np the King } overcome {np {np his } {np speech } impediment } .
Coreference chain 1{person Queen Elizabeth }
{np her }
{np a viable monarch }
{np the King }
{np his }
Coreference chain 2{person Lionel Logue }
{np a renowned {np speech } therapist }
Coreference chain 3{np speech }
{np speech }
|
|
| JavaRAP | ********Anaphor-antecedent pairs***** (0,0) Queen Elizabeth <– (0,5) her, (1,12) the King <– (1,15) his ********Text with substitution***** Queen Elizabeth set about transforming <Queen Elizabeth’s> husband, King George VI, into a viable monarch. Lionel Logue, a renowned speech therapist, was summoned to help the King overcome <the King’s> speech impediment. |
It has attempted only the pronoun resolution and that has been done well. |
| GuiTAR | Failed | |
| OpenNLP | (TOP (S (NP#6 (NNP Queen) (NNP Elizabeth)) (VP (VBD set) (PP (IN about) (S (VP (VBG transforming) (NP (NP (NML#6 (PRP$ her)) (NN husband)) (, ,) (NP (NNP King) (person (NNP George) (NNP VI)))) (, ,) (PP (IN into) (NP (DT a) (JJ viable) (NN monarch))))))) (. .)) ) (TOP (S (NP#1 (NP (person (NNP Lionel) (NNP Logue))) (, ,) (NP#1 (DT a) (JJ renowned) (NN speech) (NN therapist))) (, ,) (VP (VBD was) (VP (VBN summoned) (S (VP (TO to) (VP (VB help) (S (NP (DT the) (NNP King)) (VP (VBN overcome) (NP (NML#1 (PRP$ his)) (NN speech) (NN impediment))))))))) (. .)) )
Lionel Logue |
|
| Reconcile | <NP NO=”0″ CorefID=”1″>Queen Elizabeth</NP> set about transforming <NP NO=”2″ CorefID=”3″><NP NO=”1″ CorefID=”1″>her</NP> husband</NP>, <NP NO=”3″ CorefID=”3″>King George VI</NP>, into <NP NO=” 4″ CorefID=”4″>a viable monarch</NP>. <NP NO=”5″ CorefID=”6″>Lionel Logue</NP>, <NP NO=”6″ CorefID=”6″>a renowned speech therapist</NP>, was summoned to help <NP NO=”7″ CorefID=”6″>the King</NP > overcome <NP NO=”9″ CorefID=”9″><NP NO=”8″ CorefID=”6″>his</NP> speech impediment</NP>.
Queen Elizabeth |
the king has been wrongly attributed to Lionel Logue, which resulted in his also to be wronlt atttributed. |
| ARKref | [Queen Elizabeth]1 set about transforming [[her]1 husband , [King George VI]2 ,]2 into [a viable monarch] . [Lionel Logue , [a renowned speech therapist]6 ,]6 was summoned to help [the King]8 overcome [[his]8 speech impediment] . |
One of the best results. only info lacking is linking ‘the King’ to “king George VI” |
_As a side note, cherry picker fails with the following error.
cherrypicker1.01/tools/crf++/.libs/lt-crf_test: error while loading shared libraries: libcrfpp.so.0: cannot open shared object file: No such file or directory
To proceed, we need to download CRF++ . Install it.
Then we need to modify line no 18 in cherrypicker.sh file
tools/crf++/crf_test -m modelmd $1.crf > $1.pred
to
crf_test -m modelmd $1.crf > $1.predopen a new terminal:
run
sudo ldconfig_now cherrypicker should work
ARKref and Cherrypicker seem to be the best options available right now.
Are there any other coreference resolution systems that have not been looked at? Can we add more about the above tools? Please post your comments.