Hey there, this is the fifth edition of this newsletter where I share new things I find interesting in technology and AI (paper summaries, open-source, more) straight to your inbox!
My updates on Open source 🧑‍💻
I implemented the recent paper Compositional Attention: Disentangling Search and Retrieval by MILA. This is an easy drop-in replacement for standard multi-head attention and outperforms it on a variety of tasks.
My writings đź“ť
This year I got a chance to attend my first in-person KubeCon + CloudNativeCon in Valencia, Spain under the generous Dan Kohn scholarship. I share my experiences and how I try to make the most of conferences.
Great reads from the communityđź“–
A comparison of existing deep learning approaches and provides theoretical explanations for their performances. They also suggest novel methods that have shown to be more robust theoretically and numerically. I personally found this to be quite a heavy read.
A new state-of-the-art autoregressive text-to-image model that generates photorealistic images. Interestingly the model treats this as a sequence-to-sequence modeling problem and all the components are transformers: any technique that works for a large language model is applicable!
A very interesting blog about how dissipative HNNs improve over HNNs. HNNs and related models (like Lagrangian and symplectic) only handle data where some quantity is exactly conserved (mass, energy, etc.). Dissipative HNNs use a Neural network to parameterize both Hamiltonian (conservative component) and Rayleigh dissipation functions (dissipative component). Does require some pre-physics knowledge to best understand.
The 2nd annual Kubernetes report is out, a summary of the Kubernetes project’s contributor community and activities. This report documents both quantitative measures of community health as well as qualitative measures of the community.
GitHub Copilot finally goes GA and I have been using copilot for a while now in the technical preview. They do have a quite generous trial period of 60 days and are free for students which really appeals to me. What I also like is a checkbox if you want your code to be used to train the model which we did not have in TP.
A network trained to play Minecraft on a massive unlabelled dataset of human play and a small amount of labeled data. They show that VPT can directly learn large-scale behavioral priors in non-language domains too!
Imagen builds on the power of large transformer language models in understanding text and hinges on the strength of diffusion models in high-fidelity image generation. They find that generic large-scale language models pretrained on text are effective at this. The results just look way too good!
Open-source from community đź‘Ź
The new large-scale image dataset, HaGRID for hand gesture recognition systems. Includes 552,992 images divided into 18 hand gestures.
That’s all, hope you liked this. Stay tuned for more updates. Feel free to submit any links for the next issue.
Regards,
Rishit Dagli