view reply Nice work on the tokenizer optimization! Have you compared it with https://github.com/marcelroed/gigatoken ?
opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte Feature Extraction • 0.1B • Updated Jul 22, 2025 • 14.6k • 18