bench-labs/slop-classification
Viewer • Updated • 364 • 387 • 11
♾️ Thinking
Wop
wop
AI & ML interests
AI research
AGI
Recent Activity
updated a dataset about 9 hours ago
bench-labs/slop-classification updated a model 3 days ago
wop/littlechat-ultra-50M-Instruct liked a model 3 days ago
wop/littlechat-ultra-50M-InstructOrganizations
updated a
dataset about 9 hours ago
updated a
model 3 days ago
liked a
model 3 days ago
published a
model 3 days ago
liked a
model 3 days ago
updated a
model 3 days ago
published a
model 3 days ago
updated a
model 3 days ago
reacted to FlameF0X's post with 👍 3 days ago
Post
2594
Hello HuggingFace!
I would like to share a small preview of a language model that I have been experiencing for a while. In the first attached image there is a small sample of the Myosotis-1, an attempt to make a small 100m parameter flagship model that is built on my bizarre architecture that is somewhat similar to an S4/S5 model with WKV added. I call it FWKV (Feed-Forward WKV). The model is currently still in training because of the nature of RNN-like models. It can also be seen that the model has insane prompt processing and token generation speed (evaluation done on a 2x Titan XP); even for its small size, some similar Transformer models do struggle to get the same results without custom kernels (some Transformer models can achieve this level of throughput on cheap hardware).
In the second image, you can see checkpoint 20k of the model in its next token prediction state (this means it can't chat), ranking in the top 100 on AxiomicLabs/Open_SLM_Leaderboard (the results have not been submitted since the model is not done training).
- Why not just use Transformers?
Have you seen any pure non-Transformers SLMs besides RWKV and Mamba?
- Should you expect this project to become the next LFM or another very fast language model thing on some Raspberry Pi?
No, the model is still an experiment; it's very sensible and prone to collapse (by the time of this post, it can be seen in the 1st image).
- Should you use it?
Maybe not yet; the architecture itself is still very "naive"—that's how I could call it at its current level. If you just want to play with it and see what you could do or how fast the model is on your hardware, then you can do it.
Once the training is finished and I feel satisfied with the model next token prediction (the base model) and "assistants" (the instruction-tuned model) capabilities, I will make open weights at
FWKV with full support of the 🤗 Transformers.
I would like to share a small preview of a language model that I have been experiencing for a while. In the first attached image there is a small sample of the Myosotis-1, an attempt to make a small 100m parameter flagship model that is built on my bizarre architecture that is somewhat similar to an S4/S5 model with WKV added. I call it FWKV (Feed-Forward WKV). The model is currently still in training because of the nature of RNN-like models. It can also be seen that the model has insane prompt processing and token generation speed (evaluation done on a 2x Titan XP); even for its small size, some similar Transformer models do struggle to get the same results without custom kernels (some Transformer models can achieve this level of throughput on cheap hardware).
In the second image, you can see checkpoint 20k of the model in its next token prediction state (this means it can't chat), ranking in the top 100 on AxiomicLabs/Open_SLM_Leaderboard (the results have not been submitted since the model is not done training).
- Why not just use Transformers?
Have you seen any pure non-Transformers SLMs besides RWKV and Mamba?
- Should you expect this project to become the next LFM or another very fast language model thing on some Raspberry Pi?
No, the model is still an experiment; it's very sensible and prone to collapse (by the time of this post, it can be seen in the 1st image).
- Should you use it?
Maybe not yet; the architecture itself is still very "naive"—that's how I could call it at its current level. If you just want to play with it and see what you could do or how fast the model is on your hardware, then you can do it.
Once the training is finished and I feel satisfied with the model next token prediction (the base model) and "assistants" (the instruction-tuned model) capabilities, I will make open weights at
- 10 replies
liked a
model 5 days ago
reacted to appvoid's post with 🔥 7 days ago
Post
2627
Nobody knows what is doing, when you train a model, you are experimenting to advance the frontier, so keep failing 🫵
- 17 replies
liked 2
models 7 days ago
reacted to salma-remyx's post with 🔥🔥🔥 7 days ago
Post
2701
If you can’t explain why your AI-generated contribution belongs in the repo, don’t put it in a maintainer’s queue 🙅🏻♀️
Before asking for review, you should be able to answer:
1. What project need does it address?
2. Where does it fit, and does it duplicate existing work?
3. What evidence shows it works, and will you own it through review?
If you can’t answer those, you haven’t saved anyone time. You’ve passed the buck to the maintainer.
Outrider has made us better contributors by doing more of this work before upstream review by
* reading contribution rules, accepted PRs, and open issues
* finding needs and integration points
* drafting the code, tests, and context.
We still decide what deserves to go upstream, verify the claims, coordinate with maintainers and contributors, and stay involved through review.
On huggingface/peft, only 4 of 20 Outrider runs opened draft PRs. Two contributions have now merged:
✅ Riemannian-preconditioned LoRA: https://github.com/huggingface/peft/pull/3382
✅ Super-Tuning: https://github.com/huggingface/peft/pull/3518
We have more contributions in review and far more ideas were filtered out before they reached a maintainer.
Full case study: https://remyx.ai/case-study
Outrider: https://github.com/remyxai/outrider
Before asking for review, you should be able to answer:
1. What project need does it address?
2. Where does it fit, and does it duplicate existing work?
3. What evidence shows it works, and will you own it through review?
If you can’t answer those, you haven’t saved anyone time. You’ve passed the buck to the maintainer.
Outrider has made us better contributors by doing more of this work before upstream review by
* reading contribution rules, accepted PRs, and open issues
* finding needs and integration points
* drafting the code, tests, and context.
We still decide what deserves to go upstream, verify the claims, coordinate with maintainers and contributors, and stay involved through review.
On huggingface/peft, only 4 of 20 Outrider runs opened draft PRs. Two contributions have now merged:
✅ Riemannian-preconditioned LoRA: https://github.com/huggingface/peft/pull/3382
✅ Super-Tuning: https://github.com/huggingface/peft/pull/3518
We have more contributions in review and far more ideas were filtered out before they reached a maintainer.
Full case study: https://remyx.ai/case-study
Outrider: https://github.com/remyxai/outrider
- 3 replies
liked 2
Spaces 9 days ago
liked 2
Spaces 11 days ago