结合多注意力和条件变分自编码器的宋词生成模型

Song Ci Generation Model Based on Multi-attention and Conditional Variational Auto-encoder

关键词:  条件变分自编码器  宋词风格  宋词生成  Bi-GRU  自注意力机制

Key words:  conditional variational auto-encoder  Song Ci style  generation of Song Ci  Bi-GRU  self-attention mechanism

摘要:

现有的诗词生成方法忽略了风格的重要性。另外，由于宋词大部分词牌词句较多，逐句生成宋词的过程中容易产生上下文缺乏连贯性的现象，在上下文连贯性方面仍存在提升空间。针对这两个问题，在编码解码的文本生成框架基础上，引入自注意力机制的句子表示算法计算多注意力权重矩阵，用于提取词句的多种重要语义特征，让模型更多地关注上文的显著信息来提高上下文连贯性。引入条件变分自编码器(CVAE)将每条宋词数据转化为隐空间中不同风格特征的高维高斯分布，从各自的分布中采样隐变量来控制宋词的风格。由于自构建的宋词语料库缺少完整风格分类标签，使用具有风格标签的宋词微调BERT模型，将其作为风格分类器标注全部的宋词数据。在上述关键技术的基础上实现了宋词生成模型，生成上下文连贯的婉约词以及豪放词。通过与其他4种基准方法进行对比实验，结果表明引入自注意力机制的句子表示算法和条件变分自编码器，在上下文连贯性和风格控制方面有一定的提升。

Abstract:

The existing poetry generation methods ignore the importance of style.In addition,since there are the large number of Cipai and phrases in most of Song Ci,it is easy to produce the phenomenon that the context lacks coherence in the process of generating Song Ci sentence by sentence,and there is still space for improvement in context coherence.To solve these two problems,based on the encoding and decoding text generation framework,a sentence representation algorithm based on self-attention mechanism is introduced to calculate multi-attention weight matrix,which is used to extract a variety of important semantic features of words and sentences,so that the model can pay more attention to the significant information above to improve context coherence.The conditional variational self-encoder is introduced to transform each Song Ci data into a high-dimensional Gaussian distribution with different style features in the hidden space,and the hidden variables are sampled from the respective distributions to control the style of Song Ci.Since the self-constructed corpus of Song Ci lacks a complete style classification label,a fine-tuning BERT model of Song Ci with style label is used as a style classifier to label all the data of Song Ci.Based on the above key technologies,the generation model of Song Ci is realized,which generates euphemistic poetry and bold poetry with coherent context.Compared with the other four benchmark methods,the results show that the sentence representation algorithm with self-attention mechanism and the conditional variation self-encoder have certain improvements in context coherence and style control.