Attention · 论文

论文总览 · 基础笔记

注意力机制、稀疏 Attention、高效内核、线性 Attention 与记忆。

共 42 个清单条目；已读 14，待读 28。保留原有跨分类重复条目与阅读状态。

已下载 PDF 与标签目录

本地笔记

稀疏注意力部分包含架构、选择算法与训练方法，MSA 和 DSA 是其中的具体案例。

LLM Inference / Long Context Inference

Status	Paper	Venue	Why It Matters	Links
✅	Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference	ICML'24	根据当前 query 和 KV 页元数据选择重要页面，减少长上下文解码中的 KV 读取	Paper / 归档标签
✅ 📝	DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads	ICLR'25	区分 retrieval heads 与 streaming heads，为不同 head 分配不同 KV 保留策略	Paper / Note
✅	MiniMax Sparse Attention	arXiv'26	按 GQA group 选择 KV 块并协同设计稀疏 kernel，将块级稀疏转化为长上下文加速	Paper / Note / 归档标签
✅ 📝	IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse	arXiv'26	跨层复用 Top-k 位置，用 LM-loss 搜索或多层 KL 蒸馏减少 indexer 计算，并区分索引缓存与主 KV 缓存	Paper / Note / 归档标签
⬜	Efficient Context Scaling with LongCat ZigZag Attention	arXiv · 2026	标签：Attention / 稀疏 Attention	Paper / PDF / Zotero / 归档标签

Deep Learning / Attention Mechanisms & Variants

Status	Paper	Venue	Why It Matters	Links
✅ 📝	Attention Is All You Need	NeurIPS'17	用多头自注意力替代循环与卷积，建立可并行训练的 Transformer 架构	Paper / Note
✅	Big Bird: Transformers for Longer Sequences	NeurIPS'20	结合局部、随机与全局连接实现线性稀疏注意力，并分析其表达能力	Paper / Note / 归档标签
✅ 📝	FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness	NeurIPS'22	通过分块、算子融合与重计算减少 HBM 读写，实现不近似的高效 attention	Paper / Note / 历史笔记 / 归档标签
✅	FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning	ICLR'24	优化线程块与 warp 间工作划分，减少非矩阵计算和共享内存通信，提高 attention 吞吐	Paper / Note / 归档标签
✅	Flash-Decoding for Long-Context Inference	Blog'23	沿 KV 序列切分并合并部分 attention 结果，提高小 batch 长上下文 decode 的 GPU 利用率	Paper / Note
✅	A Survey of Efficient Attention Methods: Hardware-efficient, Sparse, Compact, and Linear Attention	Tech report'25	统一梳理硬件优化、稀疏、KV 压缩与线性 attention，区分计算和缓存优化的作用范围	Paper / 归档标签

Deep Learning / Sparse Attention Algorithms & Training / Architecture and Modern Sparse Attention

Status	Paper	Venue	Why It Matters	Links
✅ 📝	Fast Transformer Decoding: One Write-Head is All You Need	arXiv'19	MQA 原始论文；理解所有 Query heads 共享 KV 与 decode 带宽瓶颈	Paper / Note
⬜	GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints	EMNLP'23	建立 MHA–GQA–MQA 的连续关系；理解 MSA 为什么按 GQA group 独立选择	Paper
⬜	DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model	arXiv'24	MLA、latent KV compression、decoupled RoPE，以及 `uk_proj` / `uo_proj` 的矩阵吸收	Paper
⬜	SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs	NeurIPS'25	可学习的 block gate、自蒸馏与 block-sparse kernel；最接近 MSA Indexer 的对照之一	Paper
⬜	Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention	ACL'25	同时研究压缩、选择、局部窗口与硬件对齐；理解 DSA 的算法–kernel 协同背景	Paper / 归档标签
⬜	MoBA: Mixture of Block Attention for Long-Context LLMs	NeurIPS'25	MoE 风格的 block routing；适合对比 MSA 的 block max-pooling 与 group-specific selection	Paper / 归档标签
⬜	MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention	NeurIPS'24	无需重新训练的动态稀疏 prefill；对比学习式 Indexer 与预设 attention pattern	Paper
⬜	SpargeAttn: Accurate Sparse Attention Accelerating Any Model Inference	ICML'25	两阶段在线过滤与 softmax-aware pruning；理解推理期稀疏化的另一条路线	Paper / 归档标签
⬜	BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding	MLSys'26	复用 online softmax 统计量跳过低贡献 blocks；无需训练并覆盖 MHA/GQA/MQA/MLA	Paper
⬜	Block Sparse Attention with Log-Linear Complexity	arXiv · 2026	标签：Attention / 稀疏 Attention	Paper / PDF / Zotero / 归档标签
⬜ 📝	DoSA: Dots Sparse Attention	arXiv · 2026	阅读中；标签：Attention / 稀疏 Attention	Paper / PDF / Zotero / Note / Annotations / 归档标签
⬜	FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel	arXiv · 2026	标签：Attention / 稀疏 Attention	Paper / PDF / Zotero / 归档标签
✅ 📝	PageWeaver: KV-Guided Query Unions for Sparse Attention	arXiv · 2026	根据 KV page 重叠在线重组 Query，以双 CTA Union8 共享搬运，并计入分组开销	Paper / PDF / Zotero / Note / Annotations / 归档标签

Deep Learning / Sparse Attention Algorithms & Training / Algorithm and Training Foundations

Status	Paper	Venue	Why It Matters	Links
⬜	Generating Long Sequences with Sparse Transformers	arXiv'19	固定 factorized sparse pattern 与早期 block-sparse kernel；理解“规则但不自适应”的稀疏性	Paper
⬜	Longformer: The Long-Document Transformer	arXiv'20	local window + global token；理解局部先验和全局信息通路	Paper
⬜	Reformer: The Efficient Transformer	ICLR'20	LSH attention、bucket、排序和 reversible layers；理解动态候选集带来的数据重排成本	Paper
⬜	Efficient Content-Based Sparse Attention with Routing Transformers	TACL'21	online k-means 内容路由；连接固定 sparse pattern 与 learned Indexer	Paper
⬜	Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers	arXiv'24	SparseK scoring network + differentiable Top-k；直接对比 MSA/DSA 的 hard Top-k + KL 路线	Paper
⬜	Differentiable Top-k Operator with Optimal Transport	NeurIPS'20	用熵正则 Optimal Transport 平滑 Top-k，理解连续松弛及其梯度	Paper
⬜	Fast, Differentiable and Sparse Top-k: A Convex Analysis Perspective	ICML'23	从凸优化构造可微且真正稀疏的 Top-k，并讨论 GPU/TPU-friendly 算法	Paper
⬜	Categorical Reparameterization with Gumbel-Softmax	ICLR'17	离散采样的经典连续松弛；理解 temperature、annealing 与 soft-train/hard-inference 差异	Paper
⬜	From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification	ICML'16	在概率 simplex 上产生精确零值并保留可计算 Jacobian	Paper
⬜	Adaptively Sparse Transformers	EMNLP-IJCNLP'19	使用可学习的 $α$ -entmax 让不同 attention heads 自适应选择稠密或稀疏分布	Paper

Deep Learning / New Architectures

Status	Paper	Venue	Why It Matters	Links
✅	Gated Linear Attention Transformers with Hardware-Efficient Training	ICML'24	用数据依赖门控增强线性 attention，并以 IO 感知实现兼顾表达能力和训练吞吐	Paper / Note / 归档标签
✅ 📝	Kimi Linear: An Expressive, Efficient Attention Architecture	arXiv'25	用 Kimi Delta Attention 与混合架构改善线性 attention 的表达能力、长上下文效率和缓存成本	Paper / 归档标签 / Note
⬜	Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models	来源待补全	标签：Attention / 线性 Attention 与记忆	PDF / Zotero / 归档标签
⬜	Fast Weight Attention for Continual Learning	arXiv · 2026	标签：Attention / 线性 Attention 与记忆	Paper / PDF / Zotero / 归档标签
⬜	Memory Attention	arXiv · 2026	标签：Attention / 线性 Attention 与记忆；推理服务与 KV Cache	Paper / PDF / Zotero / 归档标签

Attention / 高效 Attention 内核

Status	Paper	Venue	Why It Matters	Links
⬜	FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision	arXiv · 2024	标签：Attention / 高效 Attention 内核	Paper / PDF / Zotero / 归档标签
⬜	FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling	arXiv · 2026	阅读中；标签：Attention / 高效 Attention 内核	Paper / PDF / Zotero / Annotations / 归档标签
⬜	FlashDecoding++: Faster Large Language Model Inference on GPUs	arXiv · 2024	标签：Attention / 高效 Attention 内核	Paper / PDF / Zotero / 归档标签

Zotero 阅读材料

PageWeaver：原始批注