An Ion Exchange Mechanism Inspired Story Ending Generator for Different Characters

In this section, we formulate the task of customizing the ending for each character in a story. Given a story content $x=(x_1,...,x_l)$, which contains $l$ sentences, and $m$ characters $(c_1,...,c_m)$. The task is to predict customized endings $y=(y_1,...,y_m)$ for all the $m$ characters.

A story generally corresponds to only one ground truth ending, which may consist of the actions or opinions of a particular character; the endings of other characters are unavailable. To tackle the issue, in the training stage, we extract the experience sequence of the character in the ground-truth ending, then train our proposed model to generate an ending related to the extracted character (ground-truth ending). In the generating stage, we extract the experience sequences of all characters who appear in the story and then apply the proposed model to generate an ending for the characters.

The architecture of our proposed CoSEG model is depicted in Figure 3. Our model consists of three modules—a CMM module, a VBF module and a C-CA module—as well as an encoder to encode the story context and a decoder to generate the story ending. As shown in Figure 3, the CMM module generates the character representation $cc$ for each character $c_i$ by modeling the character’s experiences; the VBF module learns the interaction between the character representation and the story context through multiple information exchanges and generates multiple interaction results, namely product candidates $({\text{p}}_0^c,...,{\text{p}}_n^c)$; the C-CA module generates a character-related story context representation by picking out the most effective product candidates. The context representation is further used as the initial state of the decoder to predict an ending for the character. The following sections present the details of each module.

We construct the experience sequence for each character in the story; the experience sequence is further fed into CMM to generate the character representation.

We take the sentence She knew a discount store near her sold socks as an example to illustrate how to extract a character experience with the following four steps:

1) Construct a dependency tree for the given sentence, and get the headword knew of the character She.

2) Extract the context words, namely knew, discount, store, sold, socks, as the first part of the character experience. This part is the background of the story.

3) Extract the entity words, namely the character She, the headword knew and the corresponding object store, as the second part of the character experience.

4) Connect the above two parts of character experience with a token OBJ to obtain the final character experience $[$ knew, discount, store, sold, socks, OBJ, knew, store, She $]$. The token OBJ is utilized to separate the two parts, explicitly telling the model which words are context information and which are directly related to the character.

In this way, we can extract each character’s experience from each sentence. Subsequently, we build the experience sequence $(e_1^i,...,e_s^i)$ of the character $c_i$ via organizing the character’s experience in chronological order, where $s$ is the number of the sentences that contain $c_i$.

We generate the character representation by modeling the character experience sequence. Formally, given a character $c_i$ and a corresponding character experience sequence $(e_1^i,...,e_S^i)$, the module encodes the token sequence $(w_1^s,...,w_{T_s}^s)$ inside each experience $e_s^i$ to obtain the hidden states $({\text{h}}_1^s,...,{\text{h}}_{T_s}^s)$, where $T_s$ is the length of $e_s^i$, and the superscript $s$ of h or $w$ represents it is for the $s^{th}$ character experience.

We choose the final hidden state ${\text{h}}_{T_s}^s$ as the representation of the character $c_i$ who went through the character experience $e_s^i$. The character representation ${\text{h}}_{T_s}^s$ is then used as the initial state of the next encoder to generate further enriched character representation ${\text{h}}_{T_{s+1}}^{s+1}$ as follows:

where $T_{s+1}$ is the length of experience $e_{s+1}^i$, $w_t^{s+1}$ is the $t^{th}$ token in the sequence $(w_1^{s+1},...,w_{T_{s+1}}^{s+1})$ inside the experience $e_{s+1}^i$, ${\text{w}}_t^{s+1}$ denotes the embedding of $w_t^{s+1}$ and the superscript $s+1$ of h or $w$ represents it is for the $(s+1{)}^{th}$ character experience.

Finally, the CMM Module will generate the $S^{th}$ character representation ${\text{h}}_{T_S}$, which has gone through all the character experiences $(e_1^i,...,e_S^i)$.

Since both character and context are represented with high-dimensional vectors, we propose a novel VBF module to learn the interaction between the character and the story context representation based on the information exchange procedure (cf. Figure 2) and then generate multiple product candidates.

Vector Breaking: Assume that there are invisible bonds between the adjacent elements of a vector, and VBF breaks the bonds between adjacent elements. For a vector of size $n$, there are $n+1$ potential bond-breaking positions. For example, given a vector $v_1=[0.1,0.2]$, the size of $v_1$ is 2 and we have 3 bond-breaking positions, as follows:

where $Vec{B}_k$ represents breaking the bond in the position $k$, and the superscripts $l$ and $r$ of the breaking results represent the left and right parts of $v_1$, respectively.

Vector Forming: The two interaction vectors break at each position respectively. To keep the size of interaction results constant, for each bond-breaking position, VBF integrates the left part of the first vector and the right part of the second vector to generate a product candidate¹¹1There is no order between the two interaction vectors, which vector as the first one has little influence on the experimental results.. In this way, two vectors of size $n$ can interact to obtain a total of $n+1$ product candidates. For example, let $v_1$ interact with $v_2=[0.3,0.4]$, and we can obtain such three product candidates ${\text{p}}_0^c=[0.3,0.4],{\text{p}}_1^c=[0.1,0.4],{\text{p}}_2^c=[0.1,0.2]$. As shown in Figure 3 part (3), the character representation and the encoder final state ${\text{h}}_t$ (i.e., story context representation) interact in the VBF module to generate the product candidates $({\text{p}}_0^c,...,{\text{p}}_n^c)$.

As shown in Figure 3 part (4), the C-CA Module aims to pick out the most effective product candidates. Specifically, we utilize the $s^{th}$ character representation ${\text{h}}_{T_s}$ and the encoder final state ${\text{h}}_t$ to obtain the attention weight of each product candidate ${\text{p}}_k^c$:

where $\sigma$ is the softmax function, ${\text{W}}_a$ is the weight matrix, ${\text{b}}_a$ is the bias, $att$ is the attention weight, and ${\text{a}}^s$ stands for the character-related story context representation of the $s^{th}$ character. As shown in Figure 3 part (5), the ${\text{r}}^s$ is further used as the initial state of the decoder (note that we omit the superscript $s$ in the following for simplicity):

where ${\text{h}}_t^y$ is the $t^{th}$ hidden state of the decoder, which is further utilized to generate the $t^{th}$ word $w_t^y$, ${\text{w}}_{t-1}^y$ is the embedding of word $w_{t-1}^y$.

We evaluated our model on the ROCStories corpus [21]. This corpus contains 98,162 five-sentence stories. Our task is to generate an ending for each character that appears in a given four-sentence story²²2We identify characters in a macro way. We extract the subject of each sentence in the story. We regard name entity or noun as the character of the sentence. In principle, in this way, we can generate an ending for any noun. Since there are few endings regarding nouns as the characters in the training data (such as the ending with "car" as the character), the proposed model is difficult to generate high-quality endings for those characters.. For each story, we extract the experience sequence for the character who appears in the ground truth ending. We select 66,881 stories in which the length of the ground-truth character’s experience sequence is no less than 2 and treat these stories as the training set. We elaborately design two test sets, each with 3073 stories. Specifically, the two sets are called sufficient test set and inadequate test set. In the sufficient test set, the length of the ground-truth character’s experience sequence is no less than 2 for all stories, while the length is less than 2 for all stories in the inadequate test set. The two test sets are applied to evaluate the performance of our proposed model when the character information is sufficient and inadequate, respectively.

We use the GloVe.6B [24] pre-trained word embedding, and the number of dimensions is 200. The hidden size of the LSTM cell is 512. Since the size of the character representation and the encoder’s final state both are 512, the number of product candidates will be $512+1=513$. A larger dimension size brings large computation costs for the model and the device. In summary, the detailed experimental settings are provided as follows:

• We use 66,881 stories for training.

• We use 3073 stories for validation, which is shared by the two test sets.

• We have two test sets with 3073 stories each. We refer to the two test sets as the sufficient test set and the inadequate test set respectively. In the sufficient test set, the character in the ground truth ending also appears multiple times in the story context, and in the inadequate test set is the opposite. These two test sets evaluate the performance of our proposed model when the character information is sufficient and inadequate, respectively.

• We use Momentum Optimizer to update parameters when training and empirically set the momentum to be 0.9.

• The size of the character representation and the encoder hidden state are 512.

• We select the product candidates generated using $Vec{B}_0$, $Vec{B}_{128}$, $Vec{B}_{256}$, $Vec{B}_{384}$ and $Vec{B}_{512}$ five breaking operation.

• The number of product candidates will be $512+1=513$.

Note that the reason for the selection of the product candidates is analyzed in Combination Analysis on Product Candidates in Section 4.5.

We compared our model with the following state-of-the-art baseline methods:

Seq2Seq [17]: A vanilla encoder-decoder model with an attention mechanism. The model treats the story context as a single sentence.

HAN [33]: A hierarchical attention architecture is applied to encode text information so as to generate the context representation.

IE [10]: It adopts an incremental encoding scheme to represent context clues and applies commonsense knowledge by multi-source attention.

T-CVAE [29]: It proposes a conditional variational autoencoder based on Transformer for missing plot generation.

MGCN-DP [11]: It leverages multi-level graph convolutional networks over dependency parse trees to capture dependency relations and context clues.

In addition, we introduce two variations of the proposed CoSEG model:

CoADD: We replace the VBF Module in CoSEG with an element-wise summation.

CoCAT: We concatenate the character representation and the encoder final state, and pass the concatenated vector through a linear layer to obtain the character-related story context representation.

We evaluate our model from two perspectives: the quality of the generated endings and the ability to customize endings.

1) In both the sufficient and inadequate test sets, our model has lower perplexity and higher BLEU scores than the baselines. Specifically, in terms of perplexity, CoSEG outperforms MGCN-DP, T-CVAE, IE, HAN and Seq2Seq by 1.02/ 1.22/ 2.09/ 3.44/ 3.27 respectively in the sufficient test set, and by 6.01/ 5.63/ 5.45/ 12.81/ 10.36 respectively in the inadequate test set. In addition, in terms of BLEU-1, CoSEG outperforms MGCN-DP, T-CVAE, IE, HAN and Seq2Seq by 1.38%/ 1.56%/ 2.2%/ 2.85%/ 2.82% respectively in the sufficient test set, and by 5.17%/ 3.96%/ 7.66%/ 8.91%/ 8.93% respectively in the inadequate test set.

2) Our CoSEG model has the smallest performance gap between the two test sets, which illustrates the performance of our model is not easily affected by the amount of information. Specifically, the perplexity increased by 1.55 in the inadequate test set based on the sufficient test set, and the BLEU-1 increased by 0.78%.

3) In both the sufficient and inadequate test set, the CoSEG model outperforms the CoADD and the CoCAT a lot, which illustrates the interaction ability of the VBF Module is much stronger than the addition and concatenation.

Manual Evaluation The manual evaluation results for the sufficient and inadequate test sets are shown in Table 2, where we can observe:

In both the sufficient and inadequate test set, the CoSEG model obtains the best coherency score and the best grammar score. Specifically, in terms of Coherency, CoSEG outperforms CoCAT, CoADD, MGCN-DP,T-CVAE, IE, HAN and Seq2Seq by 0.025/ 0.19/ 0.12/ 0.13/ 0.52/ 0.72/ 0.485 respectively in sufficient test set, and by 0.655/ 0.4/ 0.305/ 0.18/ 0.41/ 1.02/ 0.715 respectively in inadequate test set. Moreover, in terms of Grammar, CoSEG outperforms CoCAT, CoADD, MGCN-DP, T-CVAE, IE, HAN and Seq2Seq by 0.2/ 0.03/ 0.025/ 0.02/ 0.045/ 0.12/ 0.15 respectively in sufficient test set, and by 0.13/ 0.075/ 0.04/ 0.02/ 0.015/ 0.05/ 0.055 respectively in inadequate test set.

1) In the sufficient test set, the success rate (SucR) and the discrimination degree (DiscD) are 85.5% and 75.5% respectively, which indicates that our model is able to identify the differences between characters. The SucR and DiscD in the inadequate test set are lower than there in the sufficient test set, which illustrates that the amount of character information has a certain influence on distinguishing different characters.

2) In both the sufficient and inadequate test sets, the rationality of the customized endings is close to 2.0 on the premise that the maximum score is 3.0. It indicates that our model has a high probability of 66% to predict a reasonable ending for each character.

The experimental results are shown in Table 4. We can observe that CoSEG (0-128-256-384-512) achieves the best performance. The result explains that we finally selected the product candidates generated using $Vec{B}_0$, $Vec{B}_{128}$, $Vec{B}_{256}$, $Vec{B}_{384}$ and $Vec{B}_{512}$ five breaking operations and utilize the selected five product candidates as inputs to C-CA module. In addition, the result is attributed to the fact that CoSEG (0-128-256-384-512) involves more and smaller candidate which facilitate generating fine-grained semantic elements and providing more semantic combinations.

We present several examples of ground-truth story endings generated by baselines and our model in Table 5 to demonstrate that our model is able to generate more natural and more character-related endings than the baselines. Specifically, in the first story in Table 5, the ending generated by baseline T-CVAE makes the reader feel that the character is not smart enough, it has begun to rain, and it should not be very useful to look at the sky at this time; the baseline IE only describes He had to go home, without other details. And other baselines predicted wrong endings. By contrast, our model CoSEG allows the character to take an appropriate action, get an umbrella, according to the given story context, started raining.

In addition, the second story in Table 5 is derived from the inadequate test set. In this example, the context is a story about a wrong lamp. The baselines HAN, IE and MGCN-DP describe She/We had to clean it up, and IE further describes get a new one. Our model CoSEG not only allows the character to have an appropriate emotion, I was so upset, but also let the character take a reasonable action, decided to get her a new one. Obviously, the ending generated by CoSEG takes the character’s emotions (upset), behaviors (decided…get…), and relationships (get her a…) into account, which illustrates the ability of our model to obtain character’s personality.

The third example in Table 5 is sampled from the sufficient test set. In this example, the context is a story about a man’s bills mount up and he needs a job. Most of the baselines describe He get a job, as well as the ground truth and our proposed model. In addition, different from all baselines and the ground truth endings, our model further describes the man’s purpose of looking for a job, paid bills, which also demonstrates that our model is able to generate a more character-related ending.

In this section, we present three examples with character-orient endings (including the ground-truth endings) generated by our method in Table 6, to illustrate the ability of our model to customize endings for different characters.

In the first example in Table 6, our model customizes endings for the characters I, driver and son. The context is a story about a car accident that happened to a father and son. The endings generated by our model are that the father had to go to the hospital, the driver ran away, and the son was upset. These three endings generated by our model describe the behavior of the father and the driver and the mood of the son, which demonstrate the effectiveness of our proposed model to customize endings for different characters. The second example in Table 6 is a story in that a man has a cavity because he uses the toothpaste recommended by the dentist, and our model customizes endings for the characters I and dentist. The third example in Table 6 is a story in that Rich is stranded on a highway and he calls John to pick him up. Our model identifies the differences between Rich and John, generating endings, He drove to the mall and bought a new car, for Rich, and John and his friends went to the park and had a great time, for John. Rich need a new car, because his car is stranded on the highway.