The Napolab is your go-to collection of Portuguese datasets for the evaluation of Large Language Models.
A format of Napolab specifically designed for researchers experimenting with Large Language Models (LLMs) is now available. This format includes two main fields:
- Prompt: The input prompt to be fed into the LLM.
- Answer: The expected classification output label from the LLM, which is always a number between 0 and 5.
The dataset in this format can be accessed at https://huggingface.co/datasets/ruanchaves/napolab. If you’ve used Napolab for LLM evaluations, please share your findings with us!
The Open PT LLM Leaderboard incorporates datasets from Napolab.
Napolab adopts the following guidelines for the inclusion of datasets:
- 🌿 Natural: As much as possible, datasets consist of natural Portuguese text or professionally translated text.
- ✅ Reliable: Metrics correlate reliably with human judgments (accuracy, F1 score, Pearson correlation, etc.).
- 🌐 Public: Every dataset is available through a public link.
- 👩🔧 Human: Expert human annotations only. No automatic or unreliable annotations.
- 🎓 General: No domain-specific knowledge or advanced preparation is needed to solve dataset tasks.
Napolab currently includes the following datasets:
| assin | assin2 | rerelem |
| hatebr | reli-sa | faquad-nli |
| porsimplessent |
💡 Contribute: We're open to expanding Napolab! Suggest additions in the issues. For more information, read our CONTRIBUTING.md.
🌍 For broader accessibility, all datasets have translations in Catalan, English, Galician and Spanish using the facebook/nllb-200-1.3B model via Easy-Translate.
We've made several models, fine-tuned on this benchmark, available on Hugging Face Hub:
| Datasets | mDeBERTa v3 | BERT Large | BERT Base |
|---|---|---|---|
| ASSIN 2 - STS | Link | Link | Link |
| ASSIN 2 - RTE | Link | Link | Link |
| ASSIN - STS | Link | Link | Link |
| ASSIN - RTE | Link | Link | Link |
| HateBR | Link | Link | Link |
| FaQUaD-NLI | Link | Link | Link |
| PorSimplesSent | Link | Link | Link |
For model fine-tuning details and benchmark results, visit EVALUATION.md.
To reproduce the Napolab benchmark available on the Hugging Face Hub locally, follow these steps:
- Clone the repository and install the library:
git clone https://github.com/ruanchaves/napolab.git
cd napolab
pip install -e .- Generate the benchmark file:
from napolab import export_napolab_benchmark, convert_to_completions_format
input_df = export_napolab_benchmark()
output_df = convert_to_completions_format(input_df)
output_df.reset_index().to_csv("test.csv", index=False)Our research is ongoing, and we are currently working on describing our experiments in a paper, which will be published soon. In the meanwhile, if you would like to cite our work or models before the publication of the paper, please use the following BibTeX citation for this repository:
@software{Chaves_Rodrigues_napolab_2023,
author = {Chaves Rodrigues, Ruan and Tanti, Marc and Agerri, Rodrigo},
doi = {10.5281/zenodo.7781848},
month = {3},
title = {{Natural Portuguese Language Benchmark (Napolab)}},
url = {https://github.com/ruanchaves/napolab},
version = {1.0.0},
year = {2023}
}
The HateBR dataset, including all its components, is provided strictly for academic and research purposes. The use of the HateBR dataset for any commercial or non-academic purpose is expressly prohibited without the prior written consent of SINCH.