gonzo-обзоры ML статей страница 6. Показано 5 статей из 305
2023-07-11 15:14:08
Dataset Mixture
They trained on 13T tokens.
CommonCrawl & RefinedWeb are both 5T.
Remove the duplication of tokens from multiple epochs and we get to a much reasonable number of "unaccounted for" tokens: The "secret" data.
Which by this point we already get rumors that parts of it came from twitter, reddit & youtube.
[Rumors that start to become lawsuits]
Some speculations are:
- LibGen (4M+ books)
- Sci-Hub (80M+ papers)
- All of GitHub
My own opinion:
The missing dataset it a custom dataset of college textbooks collected by hand for as much courses as possible.
This is very easy to convert to txt file and than with self-instruct into instruction form.
This creates the "illusion" that GPT-4 "is smart" no matter who use it.
Computer scientist? sure! it can help you with your questions about P!=NP
Philosophy major? It can totally talk to you about epistemology.
Don't you see?
It was trained on the textbooks. It is so obvious.
There are also papers that try to extract by force memorized parts of books from GPT-4 to understand what it trained on.
There are some books it knows so well that it had seen them for sure.
Moreover, If i remember correctly: It even know the unique ids of project Euler exes.
2023-07-05 17:56:03
Something interesting:
Introducing Superalignment
We need scientific and technical breakthroughs to steer and control AI systems much smarter than us. To solve this problem within four years, we’re starting a new team, co-led by Ilya Sutskever and Jan Leike, and dedicating 20% of the compute we’ve secured to date to this effort. We’re looking for excellent ML researchers and engineers to join us.
https://openai.com/blog/introducing-superalignment
2023-07-04 06:56:21
An interesting topic
https://www.nytimes.com/2023/07/02/science/ai-mathematics-machine-learning.html
2023-06-29 22:02:15
Не про ML (хотя он может там и есть), но не могу не расшарить.
Gravitational-wave background is here.
“What we’ve essentially done is hack the entire galaxy to make a giant gravitational wave antenna”
https://www.quantamagazine.org/an-enormous-gravity-hum-moves-through-the-universe-20230628/
2023-06-29 16:49:23
За последнее время неожиданно поучаствовал в двух подкастах вокруг LLM и современного AI. Если вдруг кто по мне соскучился, вот:
1. Подкаст Вани Ямщикова:
https://www.youtube.com/watch?v=5ioSqLspbAE
2. Conversations:
https://www.youtube.com/watch?v=t0G4ZjTqkLg