9bow/webtorch-models
Text Generation • Updated • 1
Junghwan Park
9bow
webbrain-one-780104's profile picture
ecrire06's profile picture
webbrain-one-612959's profile picture·
AI & ML interests
time-series forecasting
Recent Activity
updated a model 27 days ago
9bow/webtorch-models published a model about 2 months ago
9bow/webtorch-models upvoted a paper 11 months ago
KORMo: Korean Open Reasoning Model for EveryoneOrganizations
updated a
model 27 days ago
published a
model about 2 months ago
upvoted a paper 11 months ago
liked a
Space almost 2 years ago
upvoted a paper almost 2 years ago
liked 2
models over 2 years ago
liked a
Space over 2 years ago
reacted to thomwolf's post with 👍❤️🔥 over 2 years ago
Post
5,561
A Little guide to building Large Language Models in 2024
This is a post-recording of a 75min lecture I gave two weeks ago on how to train a LLM from scratch in 2024. I tried to keep it short and comprehensive – focusing on concepts that are crucial for training good LLM but often hidden in tech reports.
In the lecture, I introduce the students to all the important concepts/tools/techniques for training good performance LLM:
* finding, preparing and evaluating web scale data
* understanding model parallelism and efficient training
* fine-tuning/aligning models
* fast inference
There is of course many things and details missing and that I should have added to it, don't hesitate to tell me you're most frustrating omission and I'll add it in a future part. In particular I think I'll add more focus on how to filter topics well and extensively and maybe more practical anecdotes and details.
Now that I recorded it I've been thinking this could be part 1 of a two-parts series with a 2nd fully hands-on video on how to run all these steps with some libraries and recipes we've released recently at HF around LLM training (and could be easily adapted to your other framework anyway):
*
*
*
Here is the link to watch the lecture on Youtube: https://www.youtube.com/watch?v=2-SPH9hIKT8
And here is the link to the Google slides: https://docs.google.com/presentation/d/1IkzESdOwdmwvPxIELYJi8--K3EZ98_cL6c5ZcLKSyVg/edit#slide=id.p
Enjoy and happy to hear feedback on it and what to add, correct, extend in a second part.
This is a post-recording of a 75min lecture I gave two weeks ago on how to train a LLM from scratch in 2024. I tried to keep it short and comprehensive – focusing on concepts that are crucial for training good LLM but often hidden in tech reports.
In the lecture, I introduce the students to all the important concepts/tools/techniques for training good performance LLM:
* finding, preparing and evaluating web scale data
* understanding model parallelism and efficient training
* fine-tuning/aligning models
* fast inference
There is of course many things and details missing and that I should have added to it, don't hesitate to tell me you're most frustrating omission and I'll add it in a future part. In particular I think I'll add more focus on how to filter topics well and extensively and maybe more practical anecdotes and details.
Now that I recorded it I've been thinking this could be part 1 of a two-parts series with a 2nd fully hands-on video on how to run all these steps with some libraries and recipes we've released recently at HF around LLM training (and could be easily adapted to your other framework anyway):
*
datatrove for all things web-scale data preparation: https://github.com/huggingface/datatrove*
nanotron for lightweight 4D parallelism LLM training: https://github.com/huggingface/nanotron*
lighteval for in-training fast parallel LLM evaluations: https://github.com/huggingface/lightevalHere is the link to watch the lecture on Youtube: https://www.youtube.com/watch?v=2-SPH9hIKT8
And here is the link to the Google slides: https://docs.google.com/presentation/d/1IkzESdOwdmwvPxIELYJi8--K3EZ98_cL6c5ZcLKSyVg/edit#slide=id.p
Enjoy and happy to hear feedback on it and what to add, correct, extend in a second part.
- 2 replies
liked a
dataset over 2 years ago
liked a
model over 2 years ago
reacted to wvaneaton's post with 👍 over 2 years ago
Post
We fine-tuned 25 mistral-7b models that outperform GPT-4 on task-specific use cases!
Check them out at LoRA Land: https://pbase.ai/3uFh7Qc
You can prompt them and compare their results to mistral-7b-instruct in real time!
They're also on HF for you to play with:
predibase
Let us know what you think!
Check them out at LoRA Land: https://pbase.ai/3uFh7Qc
You can prompt them and compare their results to mistral-7b-instruct in real time!
They're also on HF for you to play with:
Let us know what you think!
upvoted a paper over 3 years ago
liked 5
models over 3 years ago
Salesforce/xgen-7b-8k-inst
Text Generation • Updated • 741 • 95
Salesforce/xgen-7b-8k-base
Text Generation • Updated • 1.26k • 317
tiiuae/falcon-40b-instruct
Text Generation • Updated • 11k • 1.19k
tiiuae/falcon-40b
Text Generation • 42B • Updated • 9.48k • 2.45k
tiiuae/falcon-7b-instruct
Text Generation • 7B • Updated • 29.8k • 1.04k