NedoTurkishTokenizer / nedo_turkish_tokenizer

774 kB

Ctrl+K

6 contributors

History: 11 commits

Ethosoft

Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper

edec8b7 2 days ago

data
use zemberek-python and add regression tests 8 days ago
__init__.py

647 Bytes
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper 2 days ago
_acronym_table.py

5.11 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper 2 days ago
_domain_vocab.py

7.56 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper 2 days ago
_suffix_table.py

16.4 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper 2 days ago
apostrophe.py

4.3 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper 2 days ago
engine.py

6.22 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper 2 days ago
morphology.py

6.99 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper 2 days ago
normalization.py

2.01 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper 2 days ago
resources.py

3.71 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper 2 days ago
segmentation.py

17.7 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper 2 days ago
special_spans.py

10.7 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper 2 days ago
tokenizer.py

5.81 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper 2 days ago
types.py

3.52 kB
Refactor to standalone v2.0: zero dependencies, internal engine, removed zemberek/hf wrapper 2 days ago