The goal of ED consists of identifying event triggers (trigger identification) and classifying them into corresponding event types (trigger classification). According to the ACE 2005 dataset, an event is defined as a specific occurrence involving one or more participants. The event trigger is the main word or phrase that can most clearly express the occurrence of an event. As shown in Table 1, the ACE 2005 dataset defines 8 event types and 33 subtypes. Each event subtype has its specific semantic information and different event subtypes have certain semantic correlations.

Formally, given a training set $D = \{ {d_1},{d_2}, \cdots $$ {d_i}, \cdots {d_l}\}$, where $ l $ is the number of documents in training set, and a document $d = \left\{ {{s_1},{s_2}, \cdots {s_j}, \cdots {s_m}} \right\} $, where $ m $ is the number of sentences in the document $ d $, the j-th sentence can be represented as ${s_j} = \{ {w_{j1}}, $$ {w_{j2}}, \cdots {w_{jk}}, \cdots {w_{jn}}\} $, where $ n $ is the number of words in sentence $ {s_j} $.

We formalize ED as a multi-label sequence tagging problem. We assign a tag for each word to indicate whether it triggers the specific event. We adopt the “BIO” tags schema. Tags “B” and “I” represent the position of the word in a trigger to solve the problem that a trigger contains multiple words such as “take away”, “go to” and so on.

Figure 1 describes the architecture of DENSS, which primarily involves the following four components: 1) Event embedding, which learns correlations between event types through BERT; 2) Word embedding, which exploits BERT and gated attention to gain semantic information of words; 3) Trigger identification, which identifies the event triggers; 4) Trigger classification, which classifies the event triggers to corresponding types.

To enrich the contextual information of event type words, we replace each trigger word in the sentence with the corresponding event type word. For instance, sentence S1 is transformed into another sentence “He has die of his wounds after being attack”. Sentence S3 is converted into a new sentence “Another veteran war correspondent is being end-position for his controversial conduct in Iraq”. Contextualized embedding produced by pre-trained language models^[7] has been proved to be capable of modeling context beyond the sentence boundary and improving performance on a variety of tasks. Pre-trained bi-directional transformer models such as BERT can better capture long-distance dependencies as compared with Recurrent Neural Network (RNN) architecture. These newly replaced sentences are fed into BERT, and the last layer’s hidden vectors of BERT are set as the words’ embedding. Let $ {E_i} $ be the event embedding corresponding to the i-th event type word. For simplicity, we calculate the average of all representations to get the final representation for the event type word, which appears many times in the training sentences. The ACE 2005 dataset defines 33 subtypes. According to “BIO” tags schema, we finally obtain 67 representations of the event type words, as $ E = \{ {E_1},{E_2}, \cdots {E_y}, \cdots {E_{67}}\} $, and map the feature vectors $ {E_y} $ into a shared semantic space.

Figure 1.  The Architecture of the DENSS Model

To give an intuitive illustration, the different event correlations are shown in Figure 2.

Figure 2.  Event Correlation

In this figure, the solid circle denotes the event type and event type vector. The empty circle denotes the event trigger and event trigger vector.

Given a document $ d = \{ {s_1},{s_2}, \cdots {s_j}, \cdots {s_m}\} $, the j-th sentence can be represented as token sequence $ {s_j} = \{ {w_{j1}},{w_{j2}}, \cdots {w_{jk}}, \cdots {w_{jn}}\} $. Special tokens [CLS] and [SEP] are placed at the start and end of the sentence, as $\{ [{\rm{CLS}}],{w_{j1}},{w_{j2}}, \cdots {w_{jk}}, \cdots {w_{jn}},[{\rm{SEP}}]\} $. BERT can create token embedding, segment embedding, position embedding automatically, and concatenate these embeddings as the input of the next layer. For each word $ {w_{jk}} $, we select the feature vector from the last layer of BERT as word embedding $ {v_{jk}} $. The sentence $ {s_j} $ is represented as $\{ {v_{j1}},{v_{j2}}, \cdots {v_{jk}}, \cdots {v_{jn}}\} $. By considering the embedding of token [CLS] as the sentence embedding, simultaneously we obtain the sentence embedding $ {v_{j0}} $, which corresponds to token [CLS].

The sentence-level attention mechanism is utilized to extract the important clues at sentence level. For each word $ {w_{jk}} $, we employ the attention mechanism to calculate its relatedness with all other words in the sentence. $ r_s^t $ is the relatedness between the k-th word representation $ {v_{jk}} $ and the t-th word representation $ {v_{jt}} $:

where $ {\boldsymbol{W}_{sa}} $ is the weight matrix and ${b_{sa}} $ is the bias term. $ {\boldsymbol{r}}_s^t $ is then normalized to obtain scalar attention weight $ \alpha _s^t $:

For each word $ {w_{jk}} $, its sentence-level semantic information $ {\boldsymbol{s}_{jk}} $ is calculated by:

Similar to sentence-level attention, we utilize the document-level attention mechanism for capturing the significant clues at the document level. For each sentence $ {s_j} $, we employ the attention mechanism to calculate its relatedness with all other sentences in the document. $ {\boldsymbol{r}}_d^t $ is the relatedness between the j-th sentence representation $ {v_j} $ and the t-th sentence representation $ {v_t} $:

where $ {\boldsymbol{W}_{da}} $ is the weight matrix and ${b_{da}}$ is the bias term. ${\boldsymbol{r}}_d^t$ is then normalized to obtain scalar attention weight $\alpha _d^t$:

For each sentence ${s_j}$, its document-level semantic information ${d_j}$ is calculated by:

Inspired by the gated multi-level attention mechanisms^[5], we apply a fusion gate to dynamically incorporate sentence-level information ${s_{jk}}$ and document-level information ${d_j}$ for the k-th word $ {w_{jk}} $ in the j-th sentence $ {s_j} $ of the document $d$. The fusion gate $ {g_k} $ is designed to control how information should be integrated, which is calculated by:

where $ {{\boldsymbol{W}}_g} $ is the weight matrix, $ {b_g} $ is the bias term, and $ \sigma $ is the sigmoid function. Hence, the contextual representation of the word $ {w_{jk}} $ with both sentence-level information and document-level information is calculated by:

where $ \otimes $ denotes element-wise multiplication. We concatenate the contextual representation $ {c_{jk}} $ and word embedding $ {v_{jk}} $ to acquire the final word representation $ {e_{jk}} = [{c_{jk}},{v_{jk}}]$.

We model trigger identification task as a binary classification problem and annotate the trigger with label 1 while the others with label 0. The final word representation $ {e_{jk}} $ is fed into a binary classifier to decide whether it is a trigger.

The bidirectional long short term memory (Bi-LSTM) has been proven effective to capture the semantic information of words^[8]. To learn the correlations of different triggers, we filter out the words that are not triggers and assign all triggers of the document into a sequence. Let $\{ {e_{1t}},{e_{2t}}, \cdots {e_{jt}}, \cdots {e_{zt}}\}$ refer to the trigger representation sequence in document $ d $, where $ {e_{jt}} $ is the real-value vector. Then, we feed the sequence into Bi-LSTM to fuse the contextual information of the triggers with document-level information. The forward LSTM generates the forward hidden vector sequence $\{ \overrightarrow {{h_{1t}}} ,\overrightarrow {{h_{2t}}} , \cdots \overrightarrow {{h_{jt}}} , \cdots \overrightarrow {{h_{zt}}} \}$ and the backward LSTM generates the backward hidden vector sequence $\{ \overleftarrow {{h_{1t}}} ,\overleftarrow {{h_{2t}}} , \cdots \overleftarrow {{h_{jt}}} , \cdots \overleftarrow {{h_{zt}}} \}$. Thus, we acquire the trigger feature sequence $\{ {e_1},{e_2}, \cdots {e_x}, \cdots {e_z}\}$ where $ {e_x} = [\overrightarrow {{h_x}} ,\overleftarrow {{h_x}} ]$ by concatenating the forward and backward hidden states from the Bi-LSTM. A fully connected layer behind Bi-LSTM is added to map the feature vector $ {e_x} $ of the trigger into the antecedent semantic space. Inspired by Refs. [9-10], we exploit cosine similarity to measure the distance between the current trigger and all event types, and choose the label of the closest event type as the label of the trigger.

We adopt cross-entropy loss as the loss function in trigger identification and hinge loss in trigger classification. The hinge loss which is widely used for maximum-margin classification, aims to separate the correct and incorrect predictions with a margin larger than a pre-defined constant. For each trigger $ x $, we name the corresponding event type $ y $ as positive and the other types as negative. We construct the hinge ranking loss:

where $ y $ is the corresponding event type of $ x $, $ Y $ is the event type set, $ i $ is the other event type for $ x $ from $ Y $, and $b$ is the margin. The function cos calculates the cosine similarity between the feature vector $ {e_x} $ of the trigger $ x $ and the feature vector $ {E_y} $ of the event type $ y $.

We conduct experiments on the ACE 2005 dataset. For comparison, we create the same test set with 40 documents, the development set of 30 documents, and the training set of the remaining 529, the same as previous works^[2-3]. We adopt the formal ACE evaluation criteria with Precision (P), Recall (R) and F measure (F₁) to evaluate the model.

Hyper-parameters are tuned on the development set. We employ BERT-base model, which generates 768 dimensional word embedding. We set the dimension of hidden vector as 768, the dimension of semantic space as 768, and margin b as 0.1. We adopt the Adam optimizer for training with a learning rate of 2×10⁻⁵.

In order to evaluate our model, we compare it with a comprehensive set of baselines and representative models, including:

1) DMCNN builds the dynamic multi-pooling convolutional neural network to learn sentence-level features^[2].

2) JRNN exploits the bidirectional RNN to capture sentence-level features for event extraction^[3].

3) GCN-ED applies Graph Convolutional Network (GCN) to model dependency tree for extracting event information^[11].

4) DEEB-RNN utilizes document embedding and hierarchical supervised attention mechanism^[4].

5) HBTNGMA uses hierarchical and bias tagging networks to detect multiple events^[5].

6) PLMEE employs BERT to create labeled data for promoting event extraction^[12].

7) DMBERT+Boot utilizes BERT to generate more training data for ED^[13].

8) EE-GCN exploits syntactic structure and typed dependency label information to perform ED^[14].

Experimental results are shown in Table 2. From the table, we can observe that our proposed DENSS model achieves the best F₁ score for trigger classification among all the compared methods.

Compared with DMCNN and JRNN, our method significantly outperforms them. The reason is that DMCNN and JRNN only extract sentence-level information, while our method exploits multi-level information. It indicates that the document-level information is indeed beneficial to ED task. In contrast to DEEB-RNN and HBTNGMA, our method gains great improvement. It’s because that DEEB-RNN and HBTNGMA learn document-level information but do not capture rich semantic information. However our method applies pre-trained language model BERT to acquire semantic information of words and employs the semantic space to represent the semantic correlations of different event types. As compared with PLMEE and DMBERT+Boot, our method achieves more desirable performance. PLMEE and DMBERT+Boot use BERT to create training data and promote event extraction, whereas our method fuses multi-level information to represent features of words with rich semantic information. Compared with GCN-ED and EE-GCN, our method is also superior. GCN-ED and EE-GCN adopt GCN with the syntactic information to capture the event information, but the syntactic information is still limited at sentence level. Our method learns the embedding of the document through the hierarchical attention mechanisms, which indicates that multi-level semantic information is conducive to ED task.

In this section, we focus on the effectiveness of crucial components in our DENSS model with the ablation study. We examine the following models: 1) EE: to study whether the event embedding contributes to improving the performance, we substitute the one-hot label for the event embedding. As a result, the F₁ score drops by 6.4% absolutely, which demonstrates that the event embedding is beneficial to represent the semantic correlations. 2) SATT: to prove the contribution of the sentence-level attention, we remove it. As can be seen from Table 3, the F₁ score drops by 2.7%, which verifies that the sentence-level information provides important clues. 3) DATT: removing the document-level attention update model hurts the performance by 2.1%, which proves that the document-level information is helpful to enhance the performance. 4) GATE: when we calculate the average of sentence-level information and document-level information instead of the fusion gate, the F₁ score decreases by 1.5%, which indicates that the fusion gate dynamically incorporates multi-level semantic information. 5) Bi-LSTM: Bi-LSTM is removed from the model and the result score decline by 1.8%, which again verifies the effectiveness of the document-level information.

From these ablations, we have the following observations: 1) All crucial components are beneficial to the DENSS model, as removing any component degrades the performance significantly. 2) Compared with others, DENSS-EE substituting the one-hot label for the event embedding hurts the performance deeply. We inference that semantic correlations among event types can propagate more knowledge. 3) As compared with DENSS-DATT, DENSS-SATT has greater performance degradation. It illustrates that the sentence-level information provides more signals than the document-level information commonly. 4) The sentence-level information and document-level information are complementary to the feature representation, and the semantic correlation information is conducive to enhancing ED.

In the section, we present the visualization for the role of the attention mechanism, to validate whether the attention works as we designed. Figure 3 shows the example of the scalar attention weight α learned by our model. In this case, “delivered” triggers a “Phone-Write” event. Our model captures the clue “couriers delivered the letters” and assigns it with a large attention weight. The contextual information plays important role in disambiguating “delivered”, and the words “couriers” and “letters” provide the evidence to predict that “delivered” triggers a “Phone-Write” event.

Figure 3.  Visualization for the Role of the Sentence-Level Attention Mechanism. The heat map expresses the contextual attention weight, which represents the relatedness of the corresponding word pair.

Figure 4 shows that the document-level information contributes to improving the performance. We observe that some sentences with the triggers in Table 4 obtain greater attention weight than others. The triggers “convicted”, “killed” and “murdering” in the same document tend to be semantically coherent. It indicates that document-level attention can capture the significant clues at the document level to alleviate semantic ambiguity.

Figure 4.  Visualization for the Role of the Document-Level Attention Mechanism. The heat map expresses the contextual attention weight, which represents the relatedness of the corresponding sentence pair.

ED is one of the important tasks in NLP. Many methods have been proposed for this task. In earlier ED studies, researchers focused on employing feature-based methods^[15-16], which depended on the quality of artificially designed features. Most recent works have concentrated on the representation-based neural network methods, which automatically capture the feature representations by the neural network. These methods can be roughly divided into two classes. One class is to improve ED through different learning techniques, such as CNN^[2], RNN^[3], GCN^[11,14,17], and pre-trained models^[7,12]. The other class is to enhance ED through introducing extra resources, such as document information^[4-5], argument information^[18], semantic information^[9] and syntactic information^[19-20].

Document information plays an important role in ED. Ref. [4] employed document embedding and hierarchical supervised attention mechanism to enhance event detection. Ref. [5] utilized hierarchical and bias tagging networks to model document information. The attention mechanism widely used in NLP has also been applied to ED. Ref. [18] proposed to encode argument information via supervised attention mechanisms. MOGANED^[21] improved GCN with aggregative attention to model multi-order syntactic representations.

In this work, we propose a novel approach to integrate document-level and sentence-level information to enhance ED task. A hierarchical attention network is devised to automatically capture contextual information. Each event type has specific semantic information and different event types have certain semantic correlations. We deploy a shared semantic space to represent the event types and event triggers, which minimizes the distance between each event trigger and its corresponding type such that the classification of the latter is more informative and precise. Experiments on the ACE 2005 dataset verify the effectiveness of the proposed method.