Arabic Word Tokenisation Scheme
The Arabic Word Tokenisation Scheme v0.1.0, a modality-agnostic, script-agnostic and dialect-agnostic morphological tokenisation framework for tokenising Arabic words into constituent word units for rhetorical density calculations.
The Scheme was originally developed for use with the BALAGHA Score, an implementation of rhetorical density measurement for Arabic-language texts. However, it has been published on Github and Zenodo with version control for use in any other digital humanities applications in Arabic, and as an exemplar for tokenisation in other languages.
Why is tokenisation required?
Rhetorical density is defined as the number of rhetorical devices per 100 orthographic words (for isolating languages such as English and Chinese), or per 100 word units (for fusional languages such as Arabic and Spanish, and agglutinative languages such as Turkish and Finnish.) Arabic is a fusional language where compound words are formed by the addition of meaningful components to other words.
As an example, a meaning which needs 2 words in English – such as “my house” would be represented by only one word in Arabic – “بيتي” because the words for “my” and “house” are joined into one word.
This difference between English and Arabic affects the “word count” denominator in the rhetorical density calculation, making it impossible to compare the rhetorical densities of texts from different languages.
More importantly, this behaviour can affect “word count” comparison between Arabic texts as well, because the “word count” is affected by the syntax of words being joined together, and is no longer a reflection of how much information is contained within the text.
As an example, the word فأسقيناكموه consists of 4 individual units of meaning. A text with lots of words like this would have fewer overall words and an artificially high rhetorical density, compared to a text with the same amount of information but with simpler – and more numerous – words. The rhetorical density is therefore altered due to stylistic factors within the text – which become a confounding variable – rather than the density of rhetorical devices itself.
The way to solve this problem is to tokenise the Arabic text – break the words down to expose smaller units of meaning – and to count these word units instead of the words. It is crucial to tokenise different texts in a consistent and reproducible way so that rhetorical densities for different texts can be compared.
The tokenisation strategy
Arabic tokenisation has no universally correct or official standard. The appropriate level of tokenisation depends entirely on the intended analytical use case such as machine translation, sentiment analysis or morphological analysis. Existing tokenisation schemes such as lemmatisation or morphological parsing create either overly large tokens which conceal meaningful rhetorical structure, or overly granular tokens which inflate word unit counts and distort density measures. These schemes were not intended for rhetorical density analysis.
The Arabic Word Tokenisation Scheme therefore defines the “word unit” in a way optimised specifically for rhetorical density calculation. The underlying philosophy reflects the need to consistently enumerate the amount of semantic and linguistic information in the text, as a denominator for the rhetorical density calculation. The philosophy consists of three pillars.
Pillar 1: Syntactic layer – compound words will be segmented into their syntactically independent components for the reasons explained above. For example, “بيتي”is broken into 2 word units: “بيت” and “+ي”.
Pillar 2: Morphologic layer – morphologically discrete words will be preserved because further tokenisation increases word unit counts for every text, across the whole spectrum of texts, without yielding useful information about rhetorical density. While more granular tokenisation is required for other use cases, the additional computational complexity associated with sub-atomic splitting does not benefit rhetorical density calculation.
- Root and pattern decomposition is not performed. For example, ذهبنا is not broken into ذهب +نا, and أكبر is not broken into كبير + أفعل.
- Etymological decomposition is not performed. For example, words such as لماذا are retained as a single word unit, despite their etymological origin ل + ماذا.
Pillar 3: Orthographic layer – orthographic variations – which do not represent variation in the amount of information conveyed – will be normalised into the simplest and most efficient forms. For example:
- Writing a number in digits conveys the same amount of information as writing it in long form. However, the former is one word, while the latter may be more than five words.
- A text with diacritics can be tokenised into more words than a text without diacritics. This is merely a reflection of orthographic style rather than the amount of information conveyed, and is hence a confounding factor when comparing rhetorical densities.
Scope of this tokenisation scheme
This tokenisation scheme can be used to tokenise Arabic text in any orthographic form (Arabic script, Romanized, Arabizi) and in any modality (written text, transcribed speech). The scheme handles Modern Standard Arabic, regional dialects, and mixed varieties. Code-switching with non-Arabic languages is handled through the rules for non-Arabic words.
The primary requirement is that the annotator or computer script applying the tokenisation scheme can identify morphological boundaries – where clitics attach, and how conjugation is encoded, for example. The surface form (written, spoken, or in any script) does not affect the tokenization rules. The only limitation is the annotator’s or computer script’s ability to recognize the underlying structure. An annotator or computer script untrained with Levantine Arabic, for example, may not correctly tokenise Levantine-specific forms.
Publication
The tokenisation scheme has been published with versioning in external, independent repositories – Github and Zenodo – so that it can be adopted for the tokenisation of Arabic words for rhetorical density measurement and other use cases.[1] This ensures transparency, reproducibility, and consistency across datasets and analyses.
The tokenisation scheme can be implemented manually or computationally. An online version is available on the BALAGHAScore.com website.
[1] Cite as: Marathe, Mandar. BALAGHAScore.com Arabic Word Tokenisation Scheme. V0.1.0. Zenodo. 10 December 2025. https://doi.org/10.5281/zenodo.17877973
Version history
Download the Arabic Word Tokenisation Scheme
The Arabic Word Tokenisation Scheme is available as a single PDF file from the project’s Github page. It is also formally archived on Zenodo.