Set Ripple preview tokenizer context to 1Mi tokens
#7
by ebrahim-ricursive - opened
tokenizer_config.json declares a 131,072-token default, which can make tokenizer-based clients validate or truncate at 128Ki despite the advertised 1M context window.
Raise the tokenizer default to 1,048,576 tokens and state the exact 1Mi limit in the model card. Keep config.json.max_position_embeddings unchanged at 2,097,152.
Validation: tokenizer JSON parses with the 1Mi default; config.json is byte-for-byte identical to the base revision; only the tokenizer field and model-card wording differ.
ebrahim-ricursive changed pull request title from Set Ripple preview context metadata to 1Mi tokens to Set Ripple preview tokenizer context to 1Mi tokens
ebrahim-ricursive changed pull request status to merged