Refactor NgramTokenizer

[mhabedank]

The NgramTokenizer is using torchtext. We want to remove torchtext as a dependency so this Tokenizer has to be refactored not using it.

[nqbao]

if you can provide an example i can help with the rest

[mhabedank]

If we decide to replace the dependency, this would be about 5 lines of code: https://pytorch.org/text/stable/_modules/torchtext/data/utils.html#ngrams_iterator

torchtext is used here:

https://github.com/ludwig-ai/ludwig/blob/00c51e0a286c3fa399a07a550e48d0f3deadc57d/ludwig/utils/tokenizers.py#L142C1-L145C60

[nqbao]

can we just copy the code over?

[mhabedank]

yeah that would probably be the solution for this tokenizer.