MoH: Multi-Head Attention as Mixture-of-Head Attention
Paper
•
2410.11842
•
Published
•
20
Multiplexer
Note The mixture of head stuff can be done by leveraging the categorical distribution and gumble max.
Note The mixture of head stuff can be done by leveraging the categorical distribution and gumble max.