#71 - ZAK JOST (Graph Neural Networks + Geometric DL) [UNPLUGGED]

Special discount link for Zak's GNN course - https://bit.ly/3uqmYVq Patreon: https://www.patreon.com/mlst Discord: https://discord.gg/ESrGqhf5CB Pod: https://anchor.fm/machinelearningstreettalk/episodes/71---ZAK-JOST-Graph-Neural-Networks--Geometric-DL-UNPLUGGED-e1g8dvr Want to sponsor MLST!? Let us know on Linkedin / Twitter. [00:00:00] Preamble [00:03:12] Geometric deep learning [00:10:04] Message passing [00:20:42] Top down vs bottom up [00:24:59] All NN architectures are different forms of information diffusion processes (squashing and smoothing problem) [00:29:51] Graph rewiring [00:31:38] Back to information diffusion [00:42:43] Transformers vs GNNs [00:47:10] Equivariant subgraph aggregation networks + WL test [00:55:36] Do equivariant layers aggregate too? [00:57:49] Zak's GNN course References; Welcome AI Overlords YT channel https://www.youtube.com/channel/UCxw9_WYmLqlj5PyXu2AWU_g Author Interview - Equivariant Subgraph Aggregation Networks https://www.youtube.com/watch?v=VYZog7kbXks https://arxiv.org/abs/2110.02910 Geometric Deep Learning Grids, Groups, Graphs, Geodesics, and Gauges https://arxiv.org/abs/2104.13478 Joan Bruna sources of error in learning https://cims.nyu.edu/~bruna/ https://www.youtube.com/watch?v=4RmpSvQ2LL0 Blind men and an elephant https://en.wikipedia.org/wiki/Blind_men_and_an_elephant Geometric Deep Learning From Learning ODE Dynamics towards Graph Neural Diffusion [Brune] https://bathicmsworkshop.github.io/ChristophBrune.pdf The Road to Reality: A Complete Guide to the Laws of the Universe https://www.amazon.co.uk/Road-Reality-Complete-Guide-Universe/dp/0099440687 Lenia - Mathematical Life Forms [Cellula Automata] https://www.youtube.com/watch?v=iE46jKYcI4Y Graph Neural Networks - a perspective from the ground up [Alex Foo] https://www.youtube.com/watch?v=GXhBEj1ZtE8 SEMI-SUPERVISED CLASSIFICATION WITH GRAPH CONVOLUTIONAL NETWORKS [Kipf] https://arxiv.org/pdf/1609.02907.pdf Convolution theorem https://en.wikipedia.org/wiki/Convolution_theorem https://en.wikipedia.org/wiki/Graph_Fourier_transform Wavelets on Graphs via Spectral Graph Theory [Hammond] https://arxiv.org/pdf/0912.3848.pdf Growing Neural Cellular Automata https://distill.pub/2020/growing-ca/ Rediscovering the power of pairwise interactions [William Bialek] https://www.princeton.edu/~wbialek/rome/refs/bialek+ranganathan_07.pdf UNDERSTANDING OVER-SQUASHING AND BOTTLENECKS ON GRAPHS VIA CURVATURE [Topping12, inc Bronstein] https://arxiv.org/pdf/2111.14522.pdf https://towardsdatascience.com/over-squashing-bottlenecks-and-graph-ricci-curvature-c238b7169e16 GRAND: Graph Neural Diffusion [Chamberlain, inc Bronstein] http://proceedings.mlr.press/v139/chamberlain21a/chamberlain21a.pdf https://blog.twitter.com/engineering/en_us/topics/insights/2021/graph-neural-networks-as-neural-diffusion-pdes Dr. Daniele Grattarola https://danielegrattarola.github.io/ ON THE UNREASONABLE EFFECTIVENESS OF FEATURE PROPAGATION IN LEARNING ON GRAPHS WITH MISSING NODE FEATURES [Rossi + Bronstein et al] https://arxiv.org/pdf/2111.12128.pdf A Spline Theory of Deep Learning [_**Balestriero**_] https://proceedings.mlr.press/v80/balestriero18b.html COMBINING LABEL PROPAGATION AND SIMPLE MODELS OUT-PERFORMS GRAPH NEURAL NETWORK (Correct and smooth) [Huang] https://arxiv.org/pdf/2010.13993.pdf Review: Deep Learning on Sets [Fuchs] https://fabianfuchsml.github.io/learningonsets/ Transformers are Graph Neural Networks https://thegradient.pub/transformers-are-graph-neural-networks/ The Weisfeiler-Lehman Isomorphism Test https://davidbieber.com/post/2019-05-10-weisfeiler-lehman-isomorphism-test/ How Powerful are Graph Neural Networks? [Xu, Stefanie Jegelka] https://arxiv.org/abs/1810.00826