Varformers — Benchmarking Efficient Transformer Variants
Varformers is an open-source library that implements and benchmarks memory-efficient variants of the Transformer against the original architecture, built primarily as a way to understand — concretely, in code — how these efficient-attention designs actually work and where they pay off. The standard Transformer's self-attention scales quadratically with sequence length, which becomes the bottleneck on long inputs; the project puts two of the best-known responses to that problem side by side with the baseline so the trade-offs are measurable rather than theoretical. I ideated this project and guided it through, with the goal of building a hands-on, from-first-principles understanding of efficient transformer variants.
The library covers three models, with the two variants implemented by modifying only the self-attention mechanism of a base Transformer so the comparison stays clean. The base Transformer follows the Annotated Transformer implementation of Attention Is All You Need. The Performer replaces softmax attention with FAVOR+ (Fast Attention Via Orthogonal Random features), a kernel-based approximation using random feature maps that brings attention down to linear O(N) complexity. The Longformer instead uses a sliding-window local attention combined with dilated windows and a handful of global-attention tokens (like a CLS token) — again reducing complexity from O(N²) to O(N) while preserving the ability to capture long-range context where it's needed.
To make the comparison meaningful, all three were evaluated on the same task: binary sentiment classification using only the encoder, since the decoder isn't needed there. The dataset was 120k fashion product reviews from Amazon Reviews 2023 (80k train / 20k validation / 20k test), labelled positive or negative, with the three variants trained under identical hyperparameters and compared on validation loss and accuracy across epochs. The repository ships the full training pipeline — dataset handling, tokenizer, configurable model definitions, and trainer — so the whole benchmark is reproducible from a config file, and it's accompanied by a companion blog post walking through the theory behind each variant.