Zach Mueller
@TheZachMueller

I stream things occasionally https://t.co/UpZIajLBBl | https://t.co/ceuhjZulbp | Deep Learing Software Engineer | https://t.co/Y32cxGX95d




Zach Mueller    @TheZachMueller
If you have any open source software, DO THIS! You never know

Zach Mueller    @TheZachMueller
Small update, considering most of what I do is building ML/DL software libraries (such as adaptnlp and lib2nbdev), I'm now officially a Deep Learning Software Engineer 😁

Zach Mueller    @TheZachMueller
TIL einops can be faster than raw PyTorch 🤯 On stream we're reimplementing @fastdotai's AdaptiveConcatPool2d, which just calls Average and Max pooling, and concatentates. We found that doing the same with einops can be a *little* faster!

Zach Mueller    @TheZachMueller
Here's a great resource I found converting torch code to use it. Later this week I'd like to stream, going through this and converting some fastai models to use einops. https://t.co/f8SYmy4J4F

Zach Mueller    @TheZachMueller
Thanks everyone for joining me on the stream, we got TONS done! What are the fruits of our labor? We've documented all of the layers inside of the model py from SAINT, and we have live documentation for our little library! Check it out here: https://t.co/XEk3k2AWuU 1/

Zach Mueller    @TheZachMueller
If you're ever questioning what to do because you don't have enough time between meetings etc to get something big done, work on documentation. Even just 5 minutes added is a great value.

Zach Mueller    @TheZachMueller
Those @huggingface model names will get you sometimes if you're not careful. Accidently had 'bert-base-cased' for the tokenizer, and 'bert-base-uncased' for the actual model. Fought it for a bit trying to figure out why the embeddings didn't match. Oops 🤦‍♂️

Zach Mueller    @TheZachMueller
That was great! In tonights stream we were able to generate and recreate their preprocessing with fastai, and successfully pass it into the model. Next stream (TBD) we'll be looking at training the arch in @fastdotai 1/ https://t.co/VOWLTs2ONa

Zach Mueller    @TheZachMueller
Whelp that's a resounding yes! At 8:30 PST (30 minutes) we'll be exploring @gowthami_s and @tomgoldsteincs's SAINT implementation, seeing how it fits in the @fastdotai framework, and trying to train a model! https://t.co/VOWLTs2ONa

Zach Mueller    @TheZachMueller
I’ve received some notes on why I tend to reinvent the wheel rather than use existing tools or frameworks on some projects. (Or do a mix of both) For me it’s two-fold: * I learn how to do it, and I become knowledge in something new! * I can put my own spin on things.

Zach Mueller    @TheZachMueller
Perusing the paper this is everything I'd ever want from a tabular NN model by todays standards: - CutMix - MixUp - Attention - SSL/Pretraining - Embedding continuous data is important and can boost model performance - Non-Batch size deterministic (scales up)

Zach Mueller    @TheZachMueller
Why looking in the activations of your network is super important. There is a *noticeable* difference when sending a B/W image through a CNN designed for 3 channel (pretrained) and accounting for that in 3 different ways (shown below)

Zach Mueller    @TheZachMueller
fasterai, a @fastdotai library aimed at network distillation with sparse neural training (and knowledge distillation) https://t.co/pgVZJ2PQcH
 Reply      Retweet   24      Like    120   











 








Zach Mueller

TensorFlow

hardmaru

Facebook AI

Aran Komatsuzaki

PyTorch Lightning