Update
Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers
Google DeepMind posted “Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers , Google DeepMind.” A quick update for the record.
Quick note: Google DeepMind posted “Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers , Google DeepMind.” Logging it now. Context can come later if this turns into something bigger.