Event detection in Chinese news sentences

1、Task description

2、Data set

3、Evaluation methodology

4、Copyright and license issue

Word sense disambiguation (WSD) and semantic role labeling (SRL) are two important semantic analyzing techniques for NLP applications. To improve their performance, many evaluation tasks have been designed in recent years, including senseval-1/2/3 ([1], [2], [3]), CoNLL-2004/2005 ([4], [5]) and SemEval-2007 [7]. But almost all these evaluation tasks were designed to test one of these techniques, such as WSD tasks in senseval-1/2, SRL tasks for Propbank corpora in CoNLL-2004/2005 and SRL tasks for FrameNet corpora in senseval-3 and SemEval-2007. Only few tasks are designed for test the interaction of these two techniques, such as the frame semantic structure extraction task in SemEval-2007 [8]. Meanwhile, almost all these tasks are designed for the English language, only few tasks are designed for other languages, such as Chinese [9], Spanish [10], etc.
We think the event detection is an important semantic analysis task for real world sentences. We use a situation description formula to represent the content of an event. For example, for the ‘buy’ event, we use such a situation description formula: DO(x, P(x,y)) CAUSE (have(x,y) & NOT have(z,y)), where P, x, y and z are different situation arguments, ‘DO’, ‘CAUSE’, ‘&’ and ‘NOT’ are meta logical predicates, ‘have’ is a primitive predicate specially designed for ownership relation description.
In the task, we only focus on the following two event descriptions:

Ownership relations and their changing, such as possession transferring;
Existence states and their changing in different location and time situation, such as moving, living, died, etc.

We think they are the basic event units to describe other complex natural phenomena in natural language texts.
The goal of the task is to detect and analyze these event contents in real world Chinese news texts. It consists of finding key verbs or verb phrases to describe these events in the Chinese sentences after word segmentation and part-of-speech tagging, selecting suitable situation description formula for them, and anchoring different situation arguments with suitable syntactic chunks in the sentence. Three main sub-tasks are as follows:

Target verb WSD: to recognize whether there are some key verbs or verb phrases to describe two focused event contents in the sentence, and select suitable situation description formula for these recognized key verbs (or verb phrases), from a situation network lexicon.
The input of the sub-task is a Chinese sentence annotated with correct word-segmentation and POS tags. Its output is the sense selection or disambiguation tags of the target verbs in the sentence.
Sentence SRL: to anchor different situation arguments with suitable syntactic chunks in the sentence, and annotate suitable syntactic constituent and functional tags for these arguments.
Its input is a Chinese sentence annotated with correct word-segmentation, POS tags and the sense tags of the target verbs in the sentence. Its output is the syntactic chunk recognition and situation argument anchoring results.
Event detection: to detect and analyze the special event content through the interaction of target verb WSD and sentence SRL.
Its input is a Chinese sentence annotated with correct word-segmentation and POS tags. Its output is a complete event description detected in the sentence (if it has a focused target verb).

The following is a detailed example to explain the above procedure:
For such a Chinese sentence after word-segmentation and POS tagging:

今天/n(Today) 我/r(I) 在/p(at) 书店/n(bookstore) 买/v(buy) 了/u(-ed) 三/m(three) 本/q 新/a(new) 书/n(book) 。/w (Today, I bought three new books at the bookstore.)

After the first processing stage: target verb WSD, we find there is a possession-transferring verb ‘买/v(buy)’ in the sentence and select the following situation description formula for it:

买/v(buy): DO(x, P(x,y)) CAUSE have(x,y) AND NOT have(z,y) [P=buy]

Then, we anchor four situation arguments with suitable syntactic chunks in the sentence and obtain the following sentence SRL result:

今天/n(Today) [S-np 我/r(I) ]x [D-pp 在/p(at) 书店/n(bookstore) ]z [P-vp 买/v(buy) 了/u(-ed) ]Tgt [O-np 三/m(three) 本/q 新/a(new) 书/n(book) ]y 。/w

Finally, we can get the following situation description for the sentence:

DO(x, P(x,y)) CAUSE have(x,y) AND NOT have(z,y) [x=我/r(I), y=三/m(three) 本/q 新/a(new) 书/n(book), z=书店/n(bookstore), P=买/v(buy)]

We design the following two types of measures to evaluate the analysis performance:

Micro-measures, including Micro-Precision, Micro-Recall and Micro-F-measure, which are used to evaluate the analysis performance of each event token (key verb or verb phrase) in test set.
- Micro-P = Number of correctly-analyzed tokens / Number of all recognized tokens * 100%
- Micro-R = Number of correctly-analyzed tokens / Number of gold-standard tokens * 100%
- Micro-F = (Micro-P + Micro-R) / (Micro-P * Mirco-R)
Macro-measures, including Macro-Precision, Macro-Recall and Macro-F-measure, which are used to evaluation the analysis performance of overall events in test set.
- Macro-P/R/F =  Micro-P/R/F * wi, wi = frequency of the token in test set / total token frequency in test set

For the target verb WSD subtask, the evaluation measure is counted on the key verbs or verb phrases. The correct results should match the following conditions:

The selected event types and situation description formula of the tokens will be same with the gold-standard codes.

For the sentence SRL subtask, the evaluation measure is counted on the anchored argument chunks. The correct results should match all the following conditions:

The recognized chunks should have the same boundaries with the gold-standard argument chunks of the key verbs or verb phrases.
The recognized chunks should have the same syntactic constituent and functional tags with the gold-standard ones.
The recognized chunks should have the same situation argument tags with the gold-standard ones.

For the event detection subtask, the evaluation measure is counted on the complete event descriptions. The correct results should match all the following conditions:

The event type and situation description formula of the target verb should be same with the gold-standard ones.
All the argument chunks of the event descriptions should be same with the gold-standard ones.
The number of the recognized chunks should be same with the gold-standard one.