Vision Transformer

self attention

multiplicative - Effective Approaches to Attention-based Neural Machine Translation

additive - Neural Machine Translation by Jointly Learning to Align and Translate

scaled dot product - Attention Is All You Need

Rethinking Attention with Performers

Linformer: Self-Attention with Linear Complexity

Vision Transformer with Quadrangle Attention

positional encoding

feature embedding

local window attention

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Vision Transformer with Quadrangle Attention

Conditional Positional Encodings for Vision Transformers

Absolute positional encoding

Self-Attention with Relative Position Representations

Data efficient

Training data-efficient image transformers & distillation through attention - DeIT

distributed machine learning

Federated Split Vision Transformer for COVID-19
CXR Diagnosis using Task-Agnostic Training

FeSViBS: Federated Split Learning of Vision
Transformer with Block Sampling

Emerging Properties in Self-Supervised Vision Transformers - DINO

Communication-Efficient Learning of Deep Networks from Decentralized Data - FL

Distributed learning of deep neural network over multiple agents - SL