Shopify's engineering team has introduced gisting, a technique that compresses long LLM system prompts into a smaller set of learned "gist" tokens, improving throughput and reducing inference costs. This approach could make LLM deployment more efficient.
SynthePulse Insight · AI deep reading
Version 1 · 1 source
Shopify's engineering team introduces Gisting, a technique that compresses lengthy LLM system prompts into a small number of learnable 'gist' tokens, significantly reducing latency, improving throughput, and decreasing GPU allocation without modifying the model's core weights.
The core idea of Gisting is to compress a lengthy system prompt into a set of shorter 'gist' tokens, which are learnable embedding representations of the model. Unlike traditional prompt summarization, gist tokens are not a summary of the original prompt but are trained to make the model's behavior as close as possible to when it sees the original prompt.
The training process consists of two steps: in the teacher phase, the model runs with the real prompt to obtain teacher logits for the response; in the student phase, the model runs with gist tokens to obtain student logits. The gist tokens are then trained by minimizing the KL divergence between teacher and student logits until the student's predictions closely match the teacher's.
Shopify reports that after applying Gisting to its Sidekick GraphQL agent, the system prompt was compressed from about 6,000 tokens to 1,500 gist tokens, a 4:1 reduction in context size, without sacrificing prediction quality.
At a load of 350 RPM, median TTFT dropped from 438ms to 354ms, median end-to-end request latency from 6.8s to 4.2s, and throughput increased from 20.2 QPS to 23.4 QPS. These improvements allowed Shopify to reduce the number of allocated GPUs.
After training, the gist embeddings are written directly into the model's embedding matrix, and the new gist tokens are registered as special tokens in the tokenizer. During inference, the model loads and runs as usual, requiring no custom attention masks, extra encoders, or special serving paths.
Gisting complements optimization techniques like prefix caching. Prefix caching avoids recomputing KV tensors for cached prompt sequences, but the model still needs to process these cached tensors during the decoding phase; Gisting further reduces this overhead by replacing long prompts with shorter sequences of gist tokens. Shopify combines both for additive effects.
The information in this article primarily comes from InfoQ's coverage of Shopify's engineering blog, making it a secondary source. The report cites official Shopify statements and specific data but does not provide a link to the original blog post or independent verification.
The report mentions that Gisting is based on the 2022 paper 'Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models' but does not provide paper details. Additionally, the report mentions the role of 'autosearch' in tuning Gisting but does not elaborate; specific implementation details require reference to the original article.
This report is based on InfoQ's retelling of Shopify's engineering blog. All performance data and claims come from Shopify officially and have not been independently verified by third parties. The data are measurements under specific load conditions; actual results may vary by scenario.
Gisting compresses long prompts into learnable tokens, significantly reducing inference latency and cost while maintaining model quality, and is compatible with existing optimization techniques, offering a practical solution for LLM inference efficiency.
Primary report
Primary source