Data Systems for Efficient LLM Training and Inference

Large language models push data systems to their limits at every stage of their lifecycle: training corpora reach trillions of tokens, long-context decoding is dominated by attention over an ever-growing KV cache, and serving personalized models to many users strains GPU memory. This project applies ideas from compact, GPU-accelerated, and external-memory data structures to make LLM training and inference faster and more scalable.

On the training side, we built Cascade (Khan & Pandey, 2026), a GPU-accelerated, out-of-core suffix array construction algorithm that scales full-text indexing to LLM-sized corpora that far exceed GPU memory. On the inference side, Iceberg Attention (Gerami et al., 2026) achieves sublinear long-context decoding by summarizing attention with a Mixture-of-Gaussians, and InferScale (Li & Pandey, 2026) enables personalized LLM serving through GPU-native KV injection. This work draws on our broader effort in GPU data structures, memory management, and vector indexes.