Exploring Program Synthesis: Francois Chollet, Kevin Ellis, Zenna Tavares

This episode of MLST features Francois Chollet, Zenna Tavares, and Kevin Ellis discussing the limitations of deep learning and the potential of program synthesis. Chollet shares his journey from deep learning to program synthesis, driven by the realization that neural networks struggle with discrete algorithmic tasks. The conversation explores the importance of learning mechanisms and representations, with Chollet emphasizing the limitations of gradient descent. Tavares proposes integrating neural networks into programming language frameworks. The discussion highlights insights from the Abstraction and Reasoning Corpus (ARC), including the potential of test-time training and program synthesis for handling novel situations. The panel touches on the distinction between compositional novelty and pattern recognition, noting transformers' struggles with function composition. Finally, they look forward to ARC 2.0, designed to test strong generalization capabilities. Chollet emphasizes ARC's value as a benchmark that focuses on core abstraction and generalization challenges. TRANSCRIPT+REFS: https://www.dropbox.com/scl/fi/etoehoii53p6zbkhadsff/CHOLLETPANEL.pdf?rlkey=tk5w03tun2h4asud7hriektxf&dl=0 SPONSOR MESSAGES: *** Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich. Goto https://tufalabs.ai/ *** Zenna Tavares: Zenna is the co-founder and president of the Basis Research Institute, focusing on developing universal reasoning systems to address complex scientific and societal challenges. https://www.zenna.org/ https://www.basis.ai/ Kevin Ellis: Kevin is an Assistant Professor in the Computer Science department at Cornell University. His research focuses on artificial intelligence, program synthesis, and the intersection of AI and cognitive science. Ellis earned his Ph.D. from MIT, where he worked under the guidance of Joshua B. Tenenbaum and Armando Solar-Lezama. He famously wrote the Dreamcoder paper. https://www.cs.cornell.edu/~ellisk/ Francois Chollet: https://x.com/fchollet https://ndea.com/ https://arcprize.org/ TOC 1. Deep Learning Limitations [00:00:00] 1.1 Deep Learning Limitations in Theorem Proving [00:03:27] 1.2 Learning Mechanisms vs Representations [00:06:41] 1.3 Continuous vs Discrete Problem Spaces 2. Neural-Symbolic Integration [00:11:18] 2.1 Integration of Neural Networks with Program Semantics [00:13:05] 2.2 Neural-Symbolic Integration Approaches [00:14:56] 2.3 Historical Evolution of Program Synthesis [00:16:40] 2.4 Computational Resources and Infrastructure 3. Knowledge Representation and Reasoning [00:19:00] 3.1 Knowledge Representation: CYC vs Neural Networks [00:21:22] 3.2 Novel Approaches to Abstract Reasoning in AI Systems [00:25:13] 3.3 Limitations and Challenges in Transformer Architectures 4. ARC Benchmark [00:27:40] 4.1 Development and Features of ARC2 Dataset [00:29:15] 4.2 ARC's Unique Value as Knowledge-Light Benchmark REFS: [00:01:10] HolStep Dataset, Kaliszyk, Chollet, Szegedy https://openreview.net/pdf?id=ryuxYmvel [00:05:30] Abstraction and Reasoning Corpus (ARC), Chollet https://arxiv.org/abs/1911.01547 [00:07:10] Neural Turing Machines, Graves https://arxiv.org/pdf/1410.5401 [00:07:30] Manifold Hypothesis, De Bortoli https://arxiv.org/abs/2208.05314 [00:14:40] Keras, Chollet https://keras.io/ [00:15:00] Armando Solar-Lezama https://www.csail.mit.edu/news/solar-lezama-wins-robin-milner-young-researcher-award [00:15:10] Kevin Ellis https://www.cs.cornell.edu/~ellisk/ [00:19:00] The CYC Project, Lenat https://en.wikipedia.org/wiki/Cyc [00:21:55] Test-Time Training, Akyürek et al. https://ekinakyurek.github.io/papers/ttt.pdf [00:22:40] AlphaZero-style program synthesis, Laurent https://arxiv.org/abs/2205.14229 [00:26:25] ARC Prize https://arcprize.org/