Trick or ResNet Treat

A small 🎃 treat, I just uploaded a few small ResNets trained like they've never been trained before. A user request, I threw a recent hparam set (MobileNet-v4 Conv Small x ResNet Strikes Back / timm, ra4 in the tables) at the 'Basic Block' ResNet-18 & 34, including V2 (pre-activation) variants.

The results were good! ResNet-18s at 73-74% and 34s at 77-78%, oh my!

See the table below for some context. I included some past 'best' ResNet results,

• ResNet Strikes Back (a1, a1h)
• torchvision 'Batteries Included' Resnets (tv2) that followed RSB
• O.G. torchvision (tv) ResNets in 18-50 range

I did actually train a D variant ResNet50 w/ similar ra4 hparams, but they didn't improve upon past quite as much, likely need further hparam tweaks, more augreg.

The new weights all scale quite nicely to higher resolutions at inference time. Some points of interest here. The tv2 and a1h ResNet50 were trained at 176x176 resolution and by evaluating at 224x224 they were attempting to hit the 'peak' in the train-test resolution discrepancy (https://arxiv.org/abs/1906.06423). When I was working on the RSB recipe I did not want to sacrifice higher res scaling by trying to bag the peak for 224x224 eval, only the low-cost a3 trained at lower res. You can see that in this 288x288 table, the a1 RSB has more to give, the tv2 is already on the downslope, and a1h just past peak. These new ra4 are res scaling champs and have a bit more to give.