How are metrics computed in sequential models? Session-based recsys

[elloza]

Hi RecBole developers,

What exactly is the evaluation scheme that RecBole follows when evaluating sessions?

Suppose the sessions are 5 like this example I have seen in other comments as in #1060

E.g. Suppose u1 interacts with i1,i2,i3,i4,i5 in time and MAX_ITEM_LIST_LENGTH is 3. After the data augmentation.The sequential dataset becomes {(u1,<i1,0,0>,i2),(u1,<i1,i2,0>,i3),(u1,<i1,i2,i3>,i4),(u1,<i2,i3,i4>,i5)}.In the LS strategy of valid_and_test mode,the train dataset, valid dataset and test dataset are {(u1,<i1,0,0>,i2),(u1,<i1,i2,0>,i3)}, {(u1,<i1,i2,i3>,i4)} and {(u1,<i2,i3,i4>,i5)} respectively.

• What happens to the metrics when there are zeros a as padding?
• Given a list with only one item and the rest zeros, only the next item is provided? or all of them until filling 5?

I am trying to fit it into one of these evaluation protocols that can be found in the literature but it is not clear to me.

Source: https://github.com/mquad/sars_tutorial/blob/master/slides/TheWebConf2019_03_Evaluation.pdf

If you could give me a small example or indicate me the code in which this procedure is performed, I would be very thankful.

Thank you very much for your fantastic library, well done!

[Ethan-TZ]

@elloza Hello, thanks for your attention to RecBole!
The evaluation scheme of RecBole follows "Revel incrementally". i.e. given a session [1,2,3,4,5], we utilize [1] to predict 2; [1,2] to predict 3; [1,2,3] to predict 4; [1,2,3,4] to predict 5. To keep the sequence length consistent, zeros is used as padding item. In evaluation, we only utilize the reveal items of session and we record the real sequence length by item_seq_len.
For example, in GRU4Rec, the output of the sequence is defined as:

  item_seq_emb = self.item_embedding(item_seq)
  item_seq_emb_dropout = self.emb_dropout(item_seq_emb)
  gru_output, _ = self.gru_layers(item_seq_emb_dropout)
  gru_output = self.dense(gru_output)
  seq_output = self.gather_indexes(gru_output, item_seq_len - 1)
  return seq_output 

[elloza]

@chenyuwuxin Hello, thanks for your quick answer. Just a couple of questions:

• When you calculate the metrics, what is the ground truth used for comparison? Is it the same approach as in the next slide?

• Are these metrics subsequently averaged across all users?

Thank you so much!

[Ethan-TZ]

@elloza Hi! As discussed in #1060, we adopt the "Leave one out" evaluation strategy each user. In the above example, we only untilize [1,2,3,4] to predict 5 when calculate the metrics and others for training. Therefore, the evaluation protocols of RecBole is "Compare top-n recommendations only with immediate next item in the current session". And these metrics are subsequently averaged across all users.

[elloza]

Perfect! @chenyuwuxin thank you so much for your support!