模型架构与推理能力 · 论文
神经网络基础、模型架构、优化与对齐、Agent、能力与安全评测。
共 108 个清单条目；已读 4，待读 104。保留原有跨分类重复条目与阅读状态。
本地笔记
- Agent 系统
- SkVM: Revisiting Language VM for Skills across Heterogenous LLMs and Harnesses
- DeepSeek-V4: 迈向高效百万级上下文智能
- Distilling the Knowledge in a Neural Network
LLM Training / Fine-Tuning & Alignment
| Status | Paper | Venue | Why It Matters | Links |
|---|---|---|---|---|
| ⬜ | Fine-Tuning Language Models from Human Preferences | arXiv'19 | 将 reward model 与 PPO 用于语言模型偏好优化的早期完整方案 | Paper |
| ⬜ | Learning to Summarize from Human Feedback | NeurIPS'20 | 展示偏好数据、reward model 和 RL 在真实生成任务中的规模化 | Paper |
| ⬜ | Prefix-Tuning: Optimizing Continuous Prompts for Generation | ACL'21 | 冻结主模型，仅训练可学习 prefix 的参数高效微调 | Paper |
| ⬜ | The Power of Scale for Parameter-Efficient Prompt Tuning | EMNLP'21 | soft prompt 与模型规模关系，适合理解 PEFT 的表达能力 | Paper |
| ⬜ | LoRA: Low-Rank Adaptation of Large Language Models | ICLR'22 | 冻结权重并注入低秩更新，连接矩阵秩、训练显存和多租户推理 | Paper |
| ⬜ | Proximal Policy Optimization Algorithms | arXiv'17 | RLHF 所需的最低限度策略优化基础：ratio、advantage 与 clipping | Paper |
| ⬜ | Finetuned Language Models Are Zero-Shot Learners | ICLR'22 | FLAN 与 instruction tuning，说明任务混合如何产生泛化能力 | Paper |
| ⬜ | Training Language Models to Follow Instructions with Human Feedback | NeurIPS'22 | InstructGPT 的 SFT → reward model → PPO 三阶段链路 | Paper |
| ⬜ | Constitutional AI: Harmlessness from AI Feedback | arXiv'22 | self-critique、revision 与 RLAIF，把安全原则引入后训练 | Paper |
| ⬜ | Self-Instruct: Aligning Language Models with Self-Generated Instructions | ACL'23 | 自生成、过滤和扩增指令数据的经典流程 | Paper |
| ⬜ | The Flan Collection: Designing Data and Methods for Effective Instruction Tuning | ICML'23 | 系统研究任务混合、模板、CoT 数据与 instruction tuning 配方 | Paper |
| ⬜ | QLoRA: Efficient Finetuning of Quantized LLMs | NeurIPS'23 | NF4、double quantization、paged optimizer 与 LoRA 的组合 | Paper |
| ⬜ | LIMA: Less Is More for Alignment | NeurIPS'23 | 少量高质量监督数据与大规模预训练知识之间的分工 | Paper |
| ⬜ | Direct Preference Optimization: Your Language Model is Secretly a Reward Model | NeurIPS'23 | 将显式 reward model + PPO 化为稳定的 pairwise classification loss | Paper |
| ⬜ | RRHF: Rank Responses to Align Language Models with Human Feedback | NeurIPS'23 | 用候选排序损失统一多种反馈来源 | Paper |
| ⬜ | A General Theoretical Paradigm to Understand Learning from Human Preferences | AISTATS'24 | IPO 与偏好优化的理论视角，理解 DPO 类方法的过拟合和正则化 | Paper |
| ⬜ | KTO: Model Alignment as Prospect Theoretic Optimization | ICML'24 | 只需 desirable/undesirable 标签的非成对偏好学习 | Paper |
| ⬜ | ORPO: Monolithic Preference Optimization without Reference Model | EMNLP'24 | 将 SFT 与偏好约束合并，移除独立 reference model | Paper |
| ⬜ | SimPO: Simple Preference Optimization with a Reference-Free Reward | NeurIPS'24 | reference-free、长度归一化 reward 与 margin objective | Paper |
| ⬜ | DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models | arXiv'24 | GRPO、数学数据与可验证奖励，是 reasoning RL 的关键前置 | Paper |
| ⬜ | DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning | Nature'25 | 大规模 reasoning RL、冷启动数据与蒸馏路线 | Paper / 归档标签 |
| ⬜ | Forking: Sudden Overfitting Under Replay | arXiv · 2026 | 标签：模型架构与推理能力 | Paper / PDF / Zotero / 归档标签 |
Deep Learning / Foundations & Optimization
| Status | Paper | Venue | Why It Matters | Links |
|---|---|---|---|---|
| ⬜ | Learning Representations by Back-Propagating Errors | Nature'86 | 从局部算子 backward 上升到计算图链式法则、梯度流和 activation 保存 | Paper |
| ⬜ | Understanding the Difficulty of Training Deep Feedforward Neural Networks | AISTATS'10 | Xavier initialization 与方差传播，理解初始化为何影响深层网络稳定性 | Paper |
| ⬜ | Dropout: A Simple Way to Prevent Neural Networks from Overfitting | JMLR'14 | 经典正则化、train/eval 行为差异与随机 mask | Paper |
| ⬜ | Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift | ICML'15 | 对比 batch statistics 与 LayerNorm，理解同步 BN 和训练/推理差异 | Paper |
| ⬜ | Deep Residual Learning for Image Recognition | CVPR'16 | 残差连接与深层梯度传播，是 Transformer residual stream 的结构前置 | Paper |
| ⬜ | Adam: A Method for Stochastic Optimization | ICLR'15 | 一阶/二阶矩、bias correction 与 optimizer state 显存 | Paper |
| ⬜ | Decoupled Weight Decay Regularization | ICLR'19 | 区分 L2 regularization 与 AdamW 的 decoupled weight decay | Paper |
| ⬜ | Layer Normalization | arXiv'16 | 单样本归一化、训练稳定性与 Transformer 中的 reduction/fusion | Paper |
| ⬜ | Root Mean Square Layer Normalization | NeurIPS'19 | 省去 re-centering 的 RMSNorm，连接现代 LLM 结构与高效 kernel | Paper |
| ⬜ | Adafactor: Adaptive Learning Rates with Sublinear Memory Cost | ICML'18 | 对二阶矩做 factored approximation，理解 optimizer memory 优化 | Paper |
| ⬜ | Large Batch Optimization for Deep Learning: Training BERT in 76 Minutes | ICLR'20 | LAMB 的 layer-wise scaling 与超大 batch 训练 | Paper |
| ⬜ | Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer | NeurIPS'21 | μP 与跨模型规模超参数迁移，连接 scaling experiment 和训练配方 | Paper |
| ⬜ | Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training | ICLR'24 | 低成本二阶曲率估计与 per-coordinate clipping | Paper |
Deep Learning / Language Modeling, Tokenization & Modern LLMs
| Status | Paper | Venue | Why It Matters | Links |
|---|---|---|---|---|
| ⬜ | A Neural Probabilistic Language Model | JMLR'03 | embedding、条件概率与 next-token language modeling 的早期完整形式 | Paper |
| ⬜ | Sequence to Sequence Learning with Neural Networks | NeurIPS'14 | 自回归分解、teacher forcing、EOS 与 beam search 的基础 | Paper |
| ⬜ | Neural Machine Translation by Jointly Learning to Align and Translate | ICLR'15 | additive attention 与 encoder–decoder alignment，理解 Transformer 之前的问题 | Paper |
| ⬜ | Neural Machine Translation of Rare Words with Subword Units | ACL'16 | BPE、词表大小、序列长度和输出 softmax 成本之间的关系 | Paper |
| ⬜ | SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing | EMNLP'18（System Demonstrations） | 从 raw text 训练 BPE/unigram tokenizer，理解现代 tokenizer pipeline | Paper |
| ⬜ | Improving Language Understanding by Generative Pre-Training | OpenAI Technical Report'18 | GPT-1 与 decoder-only 预训练–微调范式 | Paper |
| ⬜ | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding | NAACL'19 | encoder-only、masked LM 与 pretrain–finetune 范式 | Paper |
| ⬜ | Language Models are Unsupervised Multitask Learners | OpenAI Technical Report'19 | GPT-2、zero-shot transfer 与 WebText 数据路线 | Paper |
| ⬜ | Language Models are Few-Shot Learners | NeurIPS'20 | GPT-3、in-context learning 与 decoder-only scaling | Paper |
| ⬜ | Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer | JMLR'20 | T5、span corruption、C4 与统一 text-to-text interface | Paper |
| ⬜ | GLU Variants Improve Transformer | arXiv'20 | SwiGLU/GEGLU 与 gated FFN，解释现代 LLM 中三路投影和逐元素乘 | Paper |
| ⬜ | RoFormer: Enhanced Transformer with Rotary Position Embedding | Neurocomputing'24 | RoPE 的旋转与相对位置语义，以及 Q/K 融合实现 | Paper |
| ⬜ | Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation | ICLR'22 | ALiBi 与无需显式位置 embedding 的长度外推 | Paper |
| ⬜ | PaLM: Scaling Language Modeling with Pathways | JMLR'23 | 大规模 dense LLM、Pathways 并行与训练行为分析 | Paper |
| ⬜ | LLaMA: Open and Efficient Foundation Language Models | arXiv'23 | 将 RMSNorm、SwiGLU、RoPE、tokenizer、数据与训练配方串成现代 LLM | Paper |
| ⬜ | Llama 2: Open Foundation and Fine-Tuned Chat Models | arXiv'23 | 预训练、SFT、RLHF、安全评测与 chat model 的完整技术报告 | Paper |
| ⬜ | Mistral 7B | arXiv'23 | sliding-window attention、GQA 与 rolling buffer KV cache | Paper |
| ⬜ | DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models | ACL'24 | shared expert、细粒度 expert segmentation 与稀疏计算配比 | Paper |
| ⬜ | Mixtral of Experts | arXiv'24 | 实用 sparse MoE LLM 的结构、路由和质量–计算权衡 | Paper |
| ⬜ | OLMo: Accelerating the Science of Language Models | ACL'24 | 开放数据、训练代码、checkpoint、日志和评测的端到端案例 | Paper |
| ⬜ | Qwen3 Technical Report | arXiv · 2025 | 标签：模型架构与推理能力 | Paper / PDF / Zotero / 归档标签 |
Deep Learning / Scaling Laws, Data & Pretraining
| Status | Paper | Venue | Why It Matters | Links |
|---|---|---|---|---|
| ⬜ | Scaling Laws for Neural Language Models | arXiv'20 | 建立 loss 与参数量、数据量、训练算力之间的幂律关系 | Paper |
| ⬜ | Training Compute-Optimal Large Language Models | NeurIPS'22 | Chinchilla scaling 与 compute-optimal 参数/token 配比 | Paper |
| ⬜ | Scaling Data-Constrained Language Models | NeurIPS'23 | 数据受限和重复 epoch 下的 scaling behavior | Paper |
| ⬜ | The Pile: An 800GB Dataset of Diverse Text for Language Modeling | arXiv'21 | 多域预训练语料组成、治理与 benchmark contamination | Paper |
| ⬜ | Deduplicating Training Data Makes Language Models Better | ACL'22 | 去重对记忆、评测污染、训练效率和质量的影响 | Paper |
| ⬜ | Data Selection for Language Models via Importance Resampling | NeurIPS'23 | DSIR：用目标分布重要性重采样选择预训练数据 | Paper |
| ⬜ | DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining | NeurIPS'23 | 将多域数据配比转化为 group DRO 优化问题 | Paper |
| ⬜ | The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only | NeurIPS'23 (D&B) | 大规模网页过滤与去重的生产级案例 | Paper |
| ⬜ | Textbooks Are All You Need | arXiv'23 | 合成高质量数据、数据质量与小模型能力的交换关系 | Paper |
| ⬜ | Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research | ACL'24 | 开放语料的来源、过滤、去重、PII 与治理流程 | Paper |
| ⬜ | DataComp-LM: In Search of the Next Generation of Training Sets for Language Models | NeurIPS'24 (D&B) | 受控比较过滤、去重和数据混合策略 | Paper |
| ⬜ | The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale | NeurIPS'24 (D&B) | 现代网页清洗、质量过滤与 FineWeb-Edu 配方 | Paper |
Deep Learning / Sparse Attention Algorithms & Training / Algorithm and Training Foundations
| Status | Paper | Venue | Why It Matters | Links |
|---|---|---|---|---|
| ✅ 📝 | Distilling the Knowledge in a Neural Network | NeurIPS DL Workshop'15 | soft target、temperature 与 KL；理解主 attention 如何作为 Indexer teacher | Paper / Note |
Deep Learning / New Architectures
| Status | Paper | Venue | Why It Matters | Links |
|---|---|---|---|---|
| ⬜ | Hyena Hierarchy: Towards Larger Convolutional Language Models | ICML'23 | 交替使用隐式长卷积与输入控制门控，探索低于二次复杂度的语言序列建模 | Paper |
| ⬜ | RWKV: Reinventing RNNs for the Transformer Era | EMNLP Findings'23 | 将线性 attention 与循环计算结合，实现并行训练及常量状态的逐 token 推理 | Paper |
| ⬜ | Retentive Network: A Successor to Transformer for Large Language Models | arXiv'23 | 用同一 retention 机制统一并行、循环和分块计算，兼顾训练效率与低成本解码 | Paper |
| ⬜ | Mamba: Linear-Time Sequence Modeling with Selective State Spaces | COLM'24 | 以输入依赖的选择性状态空间和硬件感知扫描，提高线性时间模型的内容选择能力 | Paper |
| ⬜ | Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality | ICML'24 | 建立 attention 与结构化状态空间的对偶关系，导出 Mamba-2 及高效分块算法 | Paper |
| ⬜ | xLSTM: Extended Long Short-Term Memory | NeurIPS'24 | 通过指数门控与标量或矩阵记忆扩展 LSTM，探索可扩展的循环语言模型 | Paper |
| ✅ | DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models | arXiv'25 | 以 DSA 降低长上下文 attention 成本，并结合可扩展强化学习提升推理与工具使用能力 | Paper / 归档标签 |
| ✅ 📝 | DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence | Tech report'26 | 组合压缩稀疏与高度压缩 attention，降低百万 token 上下文的计算和 KV 缓存开销 | Paper / Note |
| ⬜ | DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression | arXiv · 2026 | 标签：模型架构与推理能力 | Paper / PDF / Zotero / 归档标签 |
| ⬜ | mHC: Manifold-Constrained Hyper-Connections | arXiv · 2026 | 标签：模型架构与推理能力 | Paper / PDF / Zotero / 归档标签 |
| ⬜ | SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer | arXiv · 2026 | 标签：模型架构与推理能力 | Paper / PDF / Zotero / 归档标签 |
| ⬜ | Scaling Latent Reasoning via Looped Language Models | arXiv · 2026 | 标签：模型架构与推理能力 | Paper / PDF / Zotero / 归档标签 |
| ⬜ | Virtual Width Networks | arXiv · 2025 | 标签：模型架构与推理能力 | Paper / PDF / Zotero / 归档标签 |
LLM Evaluation & Safety / Capability & Quality Evaluation
| Status | Paper | Venue | Why It Matters | Links |
|---|---|---|---|---|
| ⬜ | Measuring Massive Multitask Language Understanding | ICLR'21 | MMLU：用多学科考试题衡量知识与问题求解能力 | Paper |
| ⬜ | Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models | TMLR'23 | BIG-bench：异构任务与能力随规模变化 | Paper |
| ⬜ | Holistic Evaluation of Language Models | TMLR'23 | HELM：统一准确率、校准、鲁棒性、公平性、毒性和效率 | Paper |
| ⬜ | TruthfulQA: Measuring How Models Mimic Human Falsehoods | ACL'22 | 将事实真实性与普通知识准确率区分开 | Paper |
| ⬜ | Training Verifiers to Solve Math Word Problems | arXiv'21 | GSM8K 与 verifier 路线，连接推理生成和可验证结果 | Paper |
| ⬜ | Measuring Mathematical Problem Solving With the MATH Dataset | NeurIPS'21 (D&B) | 竞赛数学、分步推理与严格答案评测 | Paper |
| ⬜ | Evaluating Large Language Models Trained on Code | arXiv'21 | HumanEval 与 pass@k，理解代码生成评测 | Paper |
| ⬜ | GPQA: A Graduate-Level Google-Proof Q&A Benchmark | COLM'24 | 专家级科学问答，即使借助网络搜索仍具挑战；用于评估高难度推理能力 | Paper |
| ⬜ | Instruction-Following Evaluation for Large Language Models | arXiv'23 | IFEval：用可验证约束衡量 instruction following | Paper |
| ⬜ | Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena | NeurIPS'23 (D&B) | LLM judge、位置偏差、冗长偏差和 pairwise evaluation | Paper |
| ⬜ | Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference | ICML'24 | 真实用户盲测、成对偏好与 Elo/Bradley–Terry 排名 | Paper |
| ⬜ | RewardBench: Evaluating Reward Models for Language Modeling | NAACL Findings'25 | 对话、推理、安全等场景的 reward model 基准 | Paper |
| ⬜ | SWE-bench: Can Language Models Resolve Real-World GitHub Issues? | ICLR'24 | 真实仓库、issue、patch 与测试驱动的软件工程评测 | Paper |
| ⬜ | LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code | ICLR'25 | 持续更新题目与时间切分，降低代码评测污染 | Paper |
| ⬜ | Lessons from the Trenches on Reproducible Evaluation of Language Models | arXiv'24 | prompt、tokenizer、版本和实现细节对评测复现的影响 | Paper |
LLM Evaluation & Safety / Safety, Robustness & Red Teaming
| Status | Paper | Venue | Why It Matters | Links |
|---|---|---|---|---|
| ⬜ | RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models | EMNLP Findings'20 | 开放式生成中的 toxicity 测量与 prompt 条件效应 | Paper |
| ⬜ | Red Teaming Language Models with Language Models | EMNLP'22 | 用模型自动生成和筛选攻击，建立可扩展 red teaming | Paper |
| ⬜ | Universal and Transferable Adversarial Attacks on Aligned Language Models | arXiv'23 | GCG 对抗后缀与可迁移 jailbreak 的基础工作 | Paper |
| ⬜ | XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models | NAACL'24 | 同时衡量合理拒绝与过度拒绝 | Paper |
| ⬜ | SafetyBench: Evaluating the Safety of Large Language Models | ACL'24 | 多类别、多语言的安全知识与行为评测 | Paper |
| ⬜ | Towards Understanding Sycophancy in Language Models | ICLR'24 | 研究模型迎合用户观点的表现及训练信号来源 | Paper |
| ⬜ | Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training | arXiv'24 | 研究条件触发的欺骗策略能否熬过安全训练 | Paper |
| ⬜ | HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal | ICML'24 | jailbreak、攻击方法与稳健拒绝的标准化评测 | Paper |
| ⬜ | A StrongREJECT for Empty Jailbreaks | NeurIPS'24 (D&B) | jailbreak 评测必须衡量实际危害、完成度和拒绝质量 | Paper |
| ⬜ | The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning | ICML'24 | 高风险双用途知识评测及其与 unlearning 的关系 | Paper |
Agent Systems
| Status | Paper | Venue | Why It Matters | Links |
|---|---|---|---|---|
| ✅ 📝 | SkVM: Revisiting Language VM for Skills across Heterogenous LLMs and Harnesses | arXiv'26 | 按模型与 harness 能力编译 skills，并用运行时代码固化和自适应重编译提高可移植性与执行效率 | Paper / Note |