Training LightSANs and it gives terrible results. A way to reproduce the paper's performance?

[Jeriousman]

I trained lightSANs model with code below.

import argparse
from recbole.quick_start import run_recbole
import logging
from logging import getLogger
import torch
import pickle
import recbole
from recbole.config import Config
from recbole.data import create_dataset, data_preparation, save_split_dataloaders, load_split_dataloaders, create_samplers
from recbole.utils import init_logger, get_model, get_trainer, init_seed, set_color
recbole.__file__




config_dict = {
                'neg_sampling' : None, 
                'data_path': '/home/Python/Models/Recommender_systems/RecBole/run_example/dataset/',
               'load_col':
                   {'inter': ['user_id', 'item_id', 'rating', 'timestamp'], ##'rating'
                    'item': ['item_id','movie_title', 'release_year', 'genre'],
                    'user': ['user_id', 'age', 'gender', 'occupation']},
                   'USER_ID_FIELD': 'user_id',
                   'ITEM_ID_FIELD': 'item_id',
                   'TIME_FIELD': 'timestamp',
                    'LABEL_FIELD': 'rating',
                    'threshold': {'rating':4},
                    'save_dataset': True,
                    'save_dataloaders': True,
                    'normalize_all': True,
                    ##training
                    'epochs':20,
                    'train_batch_size': 1024,
                    'learner':'adam',
                    'learning_rate': 0.003,
                    'eval_step': 1,
                    'stopping_step':10, ##early-stopping
                    'clip_grad_norm': None,
                    'loss_decimal_place': 5,
                    'weight_decay': 0.01,
                    'require_pow': False,
                    ##evaluation
                    'eval_args': {'group_by': 'user', 'order':'TO', 'split': {'RS': [0.8, 0.1, 0.1]}, 'mode':'uni10'},
                    'metrics': ['Hit', 'NDCG'],
                    'topk': 10,
                    'valid_metric': 'NDCG@10', #['Hit@10', 'NDCG@10'],
                    'eval_batch_size': 1024,
                    'metric_decimal_place': 5,
                    
                    ##model specific
                    'dropout_prob': 0.3,
                    'n_layers': 10,
                    'n_heads': 8,
                    'hidden_size': 128,    
                   }


# configurations initialization
config = Config(model='LightSANs', dataset = 'ml-1m', config_dict= config_dict) ##, config_file_list=config_file_list, config_dict=config_dict)

init_seed(config['seed'], config['reproducibility'])

# logger initialization
init_logger(config)
logger = getLogger()
logger.info(config)

# dataset filtering
dataset = create_dataset(config)

logger.info(dataset)

# dataset splitting
train_data, valid_data, test_data = data_preparation(config, dataset)



# model loading and initialization
init_seed(config['seed'], config['reproducibility'])
model = get_model(config['model'])(config, train_data.dataset).to(config['device'])
logger.info(model)

# trainer loading and initialization
trainer = get_trainer(config['MODEL_TYPE'], config['model'])(config, model)

# model training
best_valid_score, best_valid_result = trainer.fit(
    train_data, valid_data, saved=True, show_progress=config['show_progress']
)

# model evaluation
test_result = trainer.evaluate(test_data, load_best_model=True, show_progress=config['show_progress'])

logger.info(set_color('best valid ', 'yellow') + f': {best_valid_result}')
logger.info(set_color('test result', 'yellow') + f': {test_result}')

The result shows me this

test_result
Out[92]: OrderedDict([('hit@10', 0.99735), ('ndcg@10', 0.68633)])

Screenshot below is performance reported by official paper of lightSANs

Here are my questions.

1. First, based on my analysis of my model prediction result, it seems that my model is not really learning anything. Above code I shared, is there anything crucial I am missing or making mistakes of?

2. Why the result of my model is 'hit@10', 0.99735, 'ndcg@10', 0.68633 in decimals but papers always have like Hit@10 22.84, ndcg@10 11.45? Is it because my model is merely too terrible? (It is actually terrible) or is it because Recbole always normalizes values between 0 and 1 whereas papers normally use % so they multiply 100 to result of recbole? But even if I multiply to recbole's result by 100 to 0.99735 hit rate, it is 99.735 which is unrealistically promising whereas my model is actually rubbish.

Because it seems terrible, I tried an analysis of the result. Taking one batch from valid_data, and compared ground truth and predicted values. Code is as below.

for batch_idx, interaction in enumerate(valid_data):
    interaction = interaction
    break

interaction[0]['item_id_list'] #item_sequence
interaction[0]['item_id'] #groud truth item for next item prediction

scores = model.full_sort_predict(interaction[0].to('cuda'))
argmaxidx = scores.argmax(-1)

interaction[0]['item_id'] #groud truth item for next item prediction is as below.

argmaxidx = scores.argmax(-1) is as below

I see basically the model is predicting rubbish. This means I was wrong very much somewhere when I trained. So my questions are,

3. Is my way of analysis right to compare predicted values and actual ground truth values? I guess I am right, but I wanna be sure.

4. Am I correctly saying based on my analysis? If I am, like I mentioned in question 1, where should be fixed? What is critically so wrong in my code? (very much a duplicated question to question 1 though)

Thank you very much in advance.

[Wicknight]

1. It seems that you change the parameters too much. According to our test results, you can get performance close to the paper report. Therefore, we recommend you to use the following configuration：

#dataset config
field_separator: "\t"
seq_separator: " "
USER_ID_FIELD: user_id
ITEM_ID_FIELD: item_id
RATING_FIELD: rating
TIME_FIELD: timestamp
NEG_PREFIX: neg_
ITEM_LIST_LENGTH_FIELD: item_length
LIST_SUFFIX: _list
MAX_ITEM_LIST_LENGTH: 50
POSITION_FIELD: position_id
load_col:
  inter: [user_id, item_id, rating, timestamp]
val_interval:
  rating: "[3,inf)"

#training and evaluation
epochs: 500
train_batch_size: 4096
eval_batch_size: 2000
valid_metric: recall@10
eval_args:
  split: {'LS': 'valid_and_test'}
  mode: full
  order: TO
neg_sampling: ~

#model
embedding_size: 64 
learning_rate: 0.003
k_interests: 15

2. As you said, Recbole always normalizes values between 0 and 1 whereas papers normally use %.
3. You can compare in the way you say.
4. We recommend you to use the configuration mentioned in first point.

[Jeriousman]

Thank you for sharing the config. I will try and report you back.