Skip to content

Latest commit

 

History

History
70 lines (54 loc) · 2.63 KB

README.md

File metadata and controls

70 lines (54 loc) · 2.63 KB
Raw

SMDN: SofterMax and Deep Novelty detection

This repository contains the implementation of the research paper A post-processing framework for detecting unknown intent of dialogue system via pre-trained deep neural network classifier, submitted to Knowledge-based Systems by Tingen Lin, Hua Xu

In this paper, we propose SofterMax and deep novelty detection (SMDN), a simple yet effective post-processing method for detecting unknown intent in dialogue systems based on pre-trained deep neural network classifiers . We calibrate the confidence of DNN classifier via temperature scaling to make more reasonable/conservative prediction. Here is an example illustrating the difference between Softmax and SofterMax:

softermax

The architecture of the proposed SMDN method for the single-turn dialogue system.

Multi-turn

The architecture of the proposed SMDN method for the multi-turn dialogue system.

Single-turn

Usage

  1. Install all required library
pip install -r requirements.txt
  1. Unzip and preprocess SwDA dataset (and download your GloVe embedding & change the path in file)
unzip data/swda.zip
python preprocessing_swda.py
  1. Get the pre-trained intent classifiers with different settings
python train.py

or you can get a single pre-trained classifier with

python train_BiLSTM.py <dataset> <proportion>
python train_BiLSTM-DOC.py <dataset> <proportion>
python train_HCNN.py <proportion>
python train_HCNN-DOC.py <proportion>
  1. Evaluate the results with different detection method (DOC, SofterMax, LOF, SMDN, ...)
python eval.py

or you can get a single result with

python eval_BiLSTM.py <dataset> <proportion>
python eval_BiLSTM-DOC.py <dataset> <proportion>
python eval_HCNN.py <proportion>
python eval_HCNN-DOC.py <proportion>

Result

% of known intents 25% 50% 75% 25% 50% 75% 25% 50% 75%
Datasets SNIPS ATIS SwDA
Softmax (t=0.5) - 6.15 8.32 8.14 15.3 17.2 19.3 18.4 8.36
DOC 72.5 67.9 63.9 61.6 63.8 37.7 25.4 19.6 7.63
DOC (Softmax) 72.8 65.7 61.8 63.6 63.3 39.7 23.6 18.9 7.67
SofterMax 78.8 70.5 67.2 67.2 65.5 40.7 28.0 20.7 7.51
LOF 76.0 69.4 65.8 67.3 61.8 38.9 21.1 12.7 4.50
SMDN 79.8 73.0 71.0 71.1 66.6 41.7 20.8 18.4 8.44

Confusion matrix for SMDN experiment results on three different datasets

SNIPS ATIS SwDA