Skip to content
PLEN
All work

NLP tooling2025

Crimson Tokenizer

A BPE tokenizer trained on a Polish text corpus from SpeakLeash.

Author and developer: Dawid Bartczak

My role
Experimenting with tokenisation and language data preparation.
Technologies
Python / NLP
Source code
NLP tooling
Crimson Tokenizer
NLP tooling

Context

Language models start with data and tokenisation. I wanted to understand this stage practically.

The project

A BPE tokenizer trained on a Polish text corpus.

My contribution

Preparing the experiment and working with tokenisation mechanics.

Approach

The project focuses on text representation preparation before model training.

Challenges

  • Working with a text corpus and Polish language.
  • Understanding BPE as a mechanism, not a buzzword.