François Chollet on Compression vs Intelligence:
While intelligence leverages compression in important ways in representation learning, intelligence and compression are by nature opposite in key aspects.
Because intelligence is all about generalization to future data (out of distribution) while compression is all about efficiently fitting the distribution of past data. If you’re optimal at the latter, you’re terrible at the former.
If you were an optimal compression algorithm, the behavior policy you would develop during the first 10 years of your life (maximizing your extrinsic rewards such as candy intake, while forgetting all information that appears useless as per past rewards) would be entirely inadequate to handle the next 10.
Intelligence is about generating adequate behavior in the presence of high uncertainty and constant change. If you could have full information and if your environment were static, then there would be no need for intelligence – instead, compression would give you an optimal solution to the problem of behavior generation. Evolution would simply find the optimal behavior policy for your species and would encode it in your genes, in a compressed, optimally efficient form.
But that’s not our reality. And that’s why intelligence had to emerge. So you can adapt to situations you’ve never seen before, and that none of your ancestors has ever seen before.
The way children learn is via play, driven by intrinsic curiosity. 90% of what they learn appears entirely useless when judged through the lens of their past. But part of that useless stuff is exactly what will enable them to make sense of the future. And it’s impossible to tell in advance which part.
| Subscribe To Our Free Newsletter |
