

TITLE
[activations] pytorch-1.11+ Tanh Gelu Approximation 

COMMENTS
14

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
# 🚀 Feature request

As kindly flagged by @vadimkantorov pt-1.11 will have a fast Tanh Gelu Approximation as implemented here https://github.com/pytorch/pytorch/pull/61439 so we could replace our manual implementation with the fast one when pt>=1.11 is detected.

for additional context please see this thread: https://github.com/pytorch/pytorch/issues/39853