PatternBoost: Constructions in Mathematics with a Little Help from AI

STT: Stateful Tracking with Transformers for Autonomous Driving

Retrieval Head Mechanistically Explains Long-Context Factuality

Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

Does Transformer Interpretability Transfer to RNNs?

DSI++: Updating Transformer Memory with New Documents

Faith and Fate: Limits of Transformers on Compositionality

CodeTF: One-stop Transformer Library for State-of-the-art Code LLM

Bytes Are All You Need: Transformers Operating Directly On File Bytes

The Impact of Positional Encoding on Length Generalization in Transformers

Can Transformers Learn to Solve Problems Recursively?

RWKV: Reinventing RNNs for the Transformer Era

Scaling Transformer to 1M tokens and beyond with RMT

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Scaling Vision Transformers to 22 Billion Parameters

Exphormer: Sparse Transformers for Graphs