Post
216
STRAW: Sample-Tuned Rank-Augmented Weights
Dropped a write-up on a weird direction I’ve been exploring lately.
It’s basically a lightweight, low-rank hypernetwork that rewrites a model’s weights per sample instead of running everything through a frozen graph. I’m calling it STRAW (Sample-Tuned Rank-Augmented Weights), a tiny modulator that generates dynamic LoRA-style updates on the fly, so the model morphs its own parameters based on the input’s “vibe.”
The good part is it actually trains stably and outperforms a static baseline in low-data regimes. Not SOTA (obviously), but the fact that real-time weight modulation didn’t collapse was the whole point. Next step is making the modulator actually understand geometry instead of just reacting to pixels.
This is still early more “research log” than polished result but it opens up a fun direction toward liquid-ish, context-shaped networks instead of rigid ones.
Blog: https://teendifferent.substack.com/p/sample-tuned-rank-augmented-weights
Dropped a write-up on a weird direction I’ve been exploring lately.
It’s basically a lightweight, low-rank hypernetwork that rewrites a model’s weights per sample instead of running everything through a frozen graph. I’m calling it STRAW (Sample-Tuned Rank-Augmented Weights), a tiny modulator that generates dynamic LoRA-style updates on the fly, so the model morphs its own parameters based on the input’s “vibe.”
The good part is it actually trains stably and outperforms a static baseline in low-data regimes. Not SOTA (obviously), but the fact that real-time weight modulation didn’t collapse was the whole point. Next step is making the modulator actually understand geometry instead of just reacting to pixels.
This is still early more “research log” than polished result but it opens up a fun direction toward liquid-ish, context-shaped networks instead of rigid ones.
Blog: https://teendifferent.substack.com/p/sample-tuned-rank-augmented-weights