What Makes A Good Story? Designing Composite Rewards for Visual Storytelling

Implemented by Junjie Hu

Contact: [email protected]

If you use the codes in this repo, please cite our AAAI2020 paper.

@inproceedings{hu20aaai,
    title = {What Makes A Good Story? Designing Composite Rewards for Visual Storytelling},
    author = {Junjie Hu and Yu Cheng and Zhe Gan and Jingjing Liu and Jianfeng Gao and Graham Neubig},
    booktitle = {Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI)},
    address = {New York, USA},
    month = {February},
    url = {https://arxiv.org/abs/1909.05316},
    year = {2020}
}

Installation

Please use the following Anaconda environment.

• python=3.6
• pytorch=1.0.1
• pytorch_pretrained_bert=0.6.2
• spacy, nltk, numpy, scipy, h5py, json, pickle
• cython
• pycocoevalcap
• spice

conda env create --name reco --file conda-env.txt
conda activate reco
# Install cython, pycocoevalcap
pip install cython
pip install pycocoevalcap
cd src
bash get_stanford_models.sh

Downloads

The preprocessed data and pre-trained models can be found here. Extract data.zip under the ReCo-RL/data directory. Extract reco-rl-model.zip under the ReCo-RL/outputs/rl/ directory. Extract bert-base-uncased.zip under ReCo-RL/bert-weight/ directory.

• data.zip: train/dev/test data including image features, VIST captions and entities preprocessed by spacy.
• reco-rl-model.zip: model file (model.bin) and vocab file (vocab.bin).
• bert-base-uncased.zip: BERT's next sentence predictor model and its vocab file.
• decode-beam5-len30.tsv: predicted outputs by our ReCO-RL model. Each line contains the sequence ID and the generated story. Please consider using this file for a fair comparison in your human evaluation.

# Set REPO to the path of this repository
REPO=/home/junjieh/ReCo-RL
mkdir -p $REPO/outputs/rl/ $REPO/downloads

# Download data.zip to $REPO/downloads
cd $REPO/downloads & unzip data.zip

# Download reco-rl-model.zip to $REPO/outputs/rl/
cd $REPO/outputs/rl/ & unzip reco-rl-model.zip

Demo

• Decode the test set using pre-trained model. We recommend to use our pre-trained model for further comparison in your paper. After decoding, you would expect to see the decoding automatic scores as follows (improved scores over those reported in the paper). You can find the decoding output by our model here.

Score	SPICE	BLEU-4	ROUGE-L	CIDEr	METEOR
ReCo-RL	11.2	13.5	29.5	11.8	35.8

# Go to the folder of this repository.
# Replace [GPU id] by an integer, indicating which GPU to run on a multi-GPU machine.
bash scripts/test.sh [GPU id]

• Train a MLE model from scratch

bash scripts/train_mle.sh [GPU id]

• Train the model initialized by the MLE-trained model

bash scripts/train_rl.sh [GPU id]