PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization

Paper Link: arXiv

Code is available on Github: https://github.com/google-research/pegasus

Post by Google AI Blog: https://ai.googleblog.com/2020/06/pegasus-state-of-art-model-for.html

Abstract

Recent work pre-training Transformers with self-supervised objectives on large text corpora has shown great success when fine-tuned on downstream NLP tasks including text summarization. However, pre-training objectives tailored for abstractive text summarization have not been explored. Furthermore there is a lack of systematic evaluation across diverse domains. In this work, we propose pre-training large Transformer-based encoder-decoder models on massive text corpora with a new self-supervised objective. In PEGASUS, important sentences are removed/masked from an input document and are generated together as one output sequence from the remaining sentences, similar to an extractive summary. We evaluated our best PEGASUS model on 12 downstream summarization tasks spanning news, science, stories, instructions, emails, patents, and legislative bills. Experiments demonstrate it achieves state-of-the-art performance on all 12 downstream datasets measured by ROUGE scores. Our model also shows surprising performance on low-resource summarization, surpassing previous state-of-the-art results on 6 datasets with only 1000 examples.

State-of-the-art Results on 12 Datasets

The “Mixed & Dynamic” model has the following changes:

• Trained on both C4 and HugeNews (dataset mixsure is weighted by their number of examples).
• Trained for 1.5M instead of 500k (we observe slower convergence on pretraining perplexity).
• The model dynamicly choose 15%-45% important sentences to generate
• Importance sentences are sampled instead of using a fixed strategy (This is done by adding a 20% noise to importance scores.)
• The sentencepiece tokenizer is updated to be able to encode newline character.

(*) The numbers of wikihow and big_patent datasets are not comparable because of change in tokenization and data:

• Wikihow dataset contains newline characters which is useful for paragraph segmentation, the C4 and HugeNews model’s sentencepiece tokenizer doesn’t encode newline and loose this information.
• We update the BigPatent dataset to preserve casing, some format cleanings are also changed, please refer to change in TFDS.

Citation

@misc{zhang2019pegasus,
    title={PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization},
    author={Jingqing Zhang and Yao Zhao and Mohammad Saleh and Peter J. Liu},
    year={2019},
    eprint={1912.08777},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}