Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explanation of the test_prediction.csv #6

Open
rudra0713 opened this issue Jun 19, 2023 · 0 comments
Open

Explanation of the test_prediction.csv #6

rudra0713 opened this issue Jun 19, 2023 · 0 comments

Comments

@rudra0713
Copy link

rudra0713 commented Jun 19, 2023

Hi,
Thanks for sharing the code. I have done a small-scale experiment to make sure that the code is working on my end. I followed the instruction mentioned under "Label Embedding" in the readme.txt. I used the following setting:

DATASETS=(arc fnc ibmcs)
TARGET=arc

In the generated test_predictions.csv, I see lines like the following:

14,"[0.4692992568016052, 0.11499807983636856, 0.10301849246025085, 0.07154946774244308, 0.2411346733570099]",0,0,fnc1_agree,14,arc,arc__disagree

Since I set the target dataset to be the arc dataset, I expected the predicted labels to be also from the arc dataset (here, it's fnc1_agree). Can you kindly explain this? Also, I checked the confusion matrix, it seems there are rows corresponding to labels from the fnc and ibmcs datasets, but not from the arc dataset. Can you also kindly explain this?

In the generated test_metric.json, I see a very low accuracy score. I expected that, whenever the "agree" label is predicted (same goes for other labels too), no matter whether it is fnc1_agree or arc_agree, it will be treated as a correct prediction because they both belong to POSITIVE _LABELS. However, I don't think that's how the accuracy score is being computed. Can you kindly clarify this part too?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant