Our efficient LLM serving framework drastically accelerate long-context Transformers inference in a plug-and-play manner using our novel sub-quadratic complexity attention mechanism, Hierarchical Pruned Attention (HiP).