You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi,
Thanks for sharing the code. I have done a small-scale experiment to make sure that the code is working on my end. I followed the instruction mentioned under "Label Embedding" in the readme.txt. I used the following setting:
DATASETS=(arc fnc ibmcs)
TARGET=arc
In the generated test_predictions.csv, I see lines like the following:
Since I set the target dataset to be the arc dataset, I expected the predicted labels to be also from the arc dataset (here, it's fnc1_agree). Can you kindly explain this? Also, I checked the confusion matrix, it seems there are rows corresponding to labels from the fnc and ibmcs datasets, but not from the arc dataset. Can you also kindly explain this?
In the generated test_metric.json, I see a very low accuracy score. I expected that, whenever the "agree" label is predicted (same goes for other labels too), no matter whether it is fnc1_agree or arc_agree, it will be treated as a correct prediction because they both belong to POSITIVE _LABELS. However, I don't think that's how the accuracy score is being computed. Can you kindly clarify this part too?
The text was updated successfully, but these errors were encountered:
Hi,
Thanks for sharing the code. I have done a small-scale experiment to make sure that the code is working on my end. I followed the instruction mentioned under "Label Embedding" in the readme.txt. I used the following setting:
In the generated test_predictions.csv, I see lines like the following:
Since I set the target dataset to be the arc dataset, I expected the predicted labels to be also from the arc dataset (here, it's fnc1_agree). Can you kindly explain this? Also, I checked the confusion matrix, it seems there are rows corresponding to labels from the fnc and ibmcs datasets, but not from the arc dataset. Can you also kindly explain this?
In the generated test_metric.json, I see a very low accuracy score. I expected that, whenever the "agree" label is predicted (same goes for other labels too), no matter whether it is
fnc1_agree
orarc_agree
, it will be treated as a correct prediction because they both belong toPOSITIVE _LABELS
. However, I don't think that's how the accuracy score is being computed. Can you kindly clarify this part too?The text was updated successfully, but these errors were encountered: