Decompounding for German (and other languages)

[danielstucky]

Hi,
I'm new to sparkNLP and I could not find any functionality for decompounding in German. Is there anything planned?
In the meantime I wonder if it possible to create a custom pipeline step for decompounding, something like this:

• tokenize the input text
• decompound the tokens (some tokens may not have compounds, others 2, 3 or even more)
• apply Lemmatizer on the decompounding results (not on the initial token, but the compounds)

I did some experimentation with creating my own Decompounder (an extension of AnnotatorModel) but I am stuck on how to handle multiple compounds for a single token.

Feedback is most welcome.

Bye,
Daniel

[maziyarpanahi]

Hi @danielstucky

I am not familiar with decompounding task, is there any example to demonstrate this task just to understand what it means? (Or if this feature already exists in other library, an example of that could also help)

[danielstucky]

Sure,
in German it's commonplace to combine words into new words, especially nouns. These word chains are not seperated by space or hyphen but they are one word. Examples are "Spracherkennung" (language detection) or "Glasflächenreinigung" (Glass surface cleaning) or "Gießkanne" (watering can). The number of words you can combine is not limited (mostly it's 2 or 3 but more is possible)

With decompounding one wants to extract the base words from these combined words

• "Sprach" + "Erkennung"
• "Glas" + "flächen" + "reinigung"
• "Gieß" + "kanne"
  and then the Lemmas/Stems of these words
• "Sprache" + "Erkennung"
• "Glas" + "Fläche" + "Reinigung"
• "gießen" + "Kanne"

Commercial tools like Rossette Base Linguistic supports this feature, not sure about any open source projects.

Hope this helps to understand what I'am looking for ;-)

[maziyarpanahi]

Thanks for the examples. I think this could be equivalent to slang in English. We do have an annotator called Normalizer which accept a slang dictionary and then it will replace those with the given forms:

lol, laugh out loud
whatsup, how are you
Spracherkennung, Sprach Erkennung

I think this feature can be used here as well for decompounding German combined words:

An example in our unit test: https://github.com/JohnSnowLabs/spark-nlp/blob/master/src/test/scala/com/johnsnowlabs/nlp/annotators/NormalizerBehaviors.scala

Python APIs: https://nlp.johnsnowlabs.com/api/python/reference/autosummary/sparknlp.annotator.Normalizer.html?highlight=normalizer (under slangDictionary)