It was really lucky for me to join...
It was really lucky for me to join Google Brain in 2012 where there were believers in large scale neural networks who allowed me to work on word2vec to demonstrate the potential. But I don't want to give the impression everything was always perfect - as a follow up project after word2vec, I wanted to popularize neural language models by improving Google Translate. I did start collaboration with Franz Och and his team, during which time I proposed a couple of models that could either complement the phrase-based machine translation, or even replace it. I came up (actually even before joining Google) with a really simple idea to do end-to-end translation by training a neural language model on pairs of sentences (say French - English), and then use the generation mode to produce translation after seeing the first sentence. It worked great on short sentences, but not so much on the longer ones. I discussed this project many times with others in Google Brain - mainly Quoc and Ilya - who took over this project after I moved to Facebook AI. I was quite negatively surprised when they ended up publishing my idea under now famous name "sequence to sequence" where not only I was not mentioned as a co-author, but in fact my former friends forgot to mention me also in the long Acknowledgement section, where they thanked personally pretty much every single person in Google Brain except me. This was the time when money started flowing massively into AI and every idea was worth gold. It was sad to see the deep learning community quickly turn into some sort of Game of Thrones. Money and power certainly corrupts people...
Anyhow, the interest in language models was growing maybe slowly over the years, but with the explosion of interest since ChatGPT was released it is really cool to see so many people finally making connection between AI and language. We're not there yet, and I personally believe we need to make new discoveries to push through generalization limits of neural models. We're certainly living in exciting times. But let's not put too much faith into individuals who want to monopolize technology that is based on the hard work of dozens, or even hundreds of scientists while making claims it's all for the good of humanity."
https://www.facebook.com/1533402400/posts/pfbid0ao3fqoznHoprc8FawH6p84bctobvpTPrrbwxtGUXmBz92CzWoG63U6VSjcWJCJJTl/
Anyhow, the interest in language models was growing maybe slowly over the years, but with the explosion of interest since ChatGPT was released it is really cool to see so many people finally making connection between AI and language. We're not there yet, and I personally believe we need to make new discoveries to push through generalization limits of neural models. We're certainly living in exciting times. But let's not put too much faith into individuals who want to monopolize technology that is based on the hard work of dozens, or even hundreds of scientists while making claims it's all for the good of humanity."
https://www.facebook.com/1533402400/posts/pfbid0ao3fqoznHoprc8FawH6p84bctobvpTPrrbwxtGUXmBz92CzWoG63U6VSjcWJCJJTl/
Источник: gonzo-обзоры ML статей
2023-12-14 00:34:55