The Nordic Token Tax

vpaulus

Finnish is in a totally different language family than Swedish-Norwegian-Danish. However Viking performs the same result. I am wondering the outcome when using Viking for Hungarian, as it is a near relative to Finnish.

ahartmetz

Would have been interesting to include German and Dutch. The German training corpus must be much bigger while the language is roughly the same "distance" from English as other Germanic languages except Dutch. Dutch is most similar to English, but likely has a smaller training corpus than German.

Frieren

Very interesting, now I really wonder about Spanish, and Mandarin Chinese. There is so much written literature in Spanish that I would guess that much of it has been used to train models. For Chinese is a similar case but it has the interesting side effect to be a very different language than English. (English has germanic roots, but most of its words are latin based so not so far away from Spanish).

barrell

I’d be curious to see this ran through cohere’s tokenizer - they seem to be the largest lab with a focus on multilingualism.