

TITLE
Longformer inference speed is slower than bert of the same length

COMMENTS
7

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
I used bert-base as the basic model and retrained my long-bert with a length of 1536. Then I compared the difference in inference speed of the original bert-base-1536. After a lot of testing, I found that long-bert-1536 and bert-base-1536 are basically the same in inference speed. I see a similar problem [https://github.com/allenai/longformer/issues/106] , but the length of my test data is all greater than 1000. I think window attention should be faster than self-attention because the amount of calculation is smaller, but why does this problem occur? Here are some settings:
attention windows (each layer is the same): 512
Global attention: only used for cls token
Inference device: cpu
task: text classification

By the way, does the size of the attention window affect the speed of inference? I tested different window sizes, but the speed is basically the same.