MATHEMATICS & MACHINE LEARNING
I am a mathematics undergraduate at Peking University. I study how Transformers compute and generalize, with a focus on the mechanisms of looped Transformers.
How Looped Transformers Route What Each Loop Does
We study how a changing hidden state selects different computations in a looped Transformer with fixed weights. Graph-walk experiments and attention interventions test how this selection works.
QUESTION
Shared weights do not require successive loops to perform the same operation.
FINDING
In the studied graph-walk models, changing the entering hidden state redirects a frozen loop. Attention interventions test the route of this control.
SCOPE
Training changes which transitions can be selected. The evidence is task- and model-specific; longer compositions remain a limitation.
Archived results support figure and statistic reproduction; full training requires additional dependencies and checkpoints.
Research note
I studied how Transformers compute multiplication in S₅. Representation analysis and activation interventions identified different internal implementations of the same function.
The experiments identified three families of product representations. Models computed these representations through different MLP, attention, and residual paths. Optimizer preferences varied across seeds.
The study uses a small set of S₅ models. It does not establish a deterministic mapping from optimizer to circuit.
School of Mathematical Sciences · Undergraduate
GPA: 3.79 / 4.00
Mathematical Analysis I–III; Geometry; Advanced Algebra I–II; Probability Theory; Mathematical Statistics; Data Structures and Algorithms; Introduction to Mathematical Machine Learning; Ordinary Differential Equations; Deep Learning Theory; Multi-Agent Foundations; Applied Stochastic Processes.
Early admission to the PKU Mathematics Talent Program
Course project
Predicting WSD loss curves from a cosine training curve, with a frozen MPL baseline and source-only calibration.
The reported correction improves all 15 same-scale targets. Evaluation uses existing public curves, not new model training.
Team course project
A joint study with Kairui Li and Kehan Huang of delayed generalization on modular arithmetic and group operations.
Shared training and evaluation tools for Transformers, MLPs, LSTMs, and GRUs; code and a course report are available.
Campus software
A home for the SMS Student Union Academic Department's problems, course resources, and events.
React and TypeScript, with mathematical typesetting and configuration-based content updates.
Lingjun Quant Challenge project. Built a causal Transformer for A-share high-frequency microstructure data with 500 stocks, 239 intraday minutes, and 384 features, predicting ten-minute-ahead returns with strict time-split validation and leakage-aware preprocessing.
Project notes ↗Midterm project for Mathematical Modeling. Built a source-attribution system for mathematical solutions from humans and major LLMs including DeepSeek, GLM, Kimi, and Qwen, combining feature-based baselines with neural text classification.
Project notes ↗Final project for Multi-Agent Foundations. Built StrictCoop-Sokoban and studied intent-grounded recurrent communication for partially observable cooperative multi-agent reinforcement learning.
Project notes ↗Teaching-assistant project design for Data Structures and Algorithms B. Developed and maintained a Flask-based Avalon agent battle platform where submitted Python agents compete under partial observability, communication constraints, referee-controlled game phases, automatic matching, ELO ranking, and replay-oriented logs.
Project notes ↗