LinkedIn Unveils Multi-Teacher Distillation Training Infrastructure, Speeding Up Job Search Ranking Model by 8x

What LinkedIn revealed is not a new distillation algorithm, but a training infrastructure that makes distillation 'run fast enough.' As described by the source, training a small language model to improve relevance and engagement objectives such as clicks and applications requires querying one or more large teacher models for every training sample; and serving those teacher models can slow down the entire process. For a ranking system that must handle hundreds of thousands of queries per second, this becomes a bottleneck.

This is precisely the starting point of LinkedIn's trade-off: distillation itself is feasible, but the serving overhead of teacher models limits iteration speed. LinkedIn's contribution is described as creating a custom SGLang framework that enables teacher models to accept real-time queries during training.

LinkedIn used SGLang to create a multi-teacher distillation framework that can load and serve teacher models of different sizes while managing tensor parallelism and data parallelism configurations. During training, an asynchronous client sends requests to teacher models, processes their outputs, and adds the results to the distillation loss — what the team calls online multi-teacher distillation.

To keep latency low, the system deploys local teacher model replicas across multiple nodes. The source says this approach speeds up the distillation process by 3x while maintaining low latency to support rapid iteration. It should be noted that this 3x figure comes from the LinkedIn team's own statement and lacks independent third-party verification.

The online mode is flexible, but serving overhead is always present. To reduce overhead and avoid redundant computation, LinkedIn introduced offline multi-teacher distillation: precomputing teacher model outputs, storing them on HDFS or NFS, and using those results directly during training instead of querying teacher models in real time.

The switching logic given by the source is: online querying suits the early stage when teacher model choices are still changing frequently; once teacher models stabilize and query volume increases, switch to offline caching. However, the article does not specify the exact switching thresholds or criteria between the two modes in production.

Around this combined online-and-offline approach, LinkedIn also stacked multiple training-layer optimizations. Those listed by the source include: adopting LiGer to reduce memory usage and expanding batch size to 2x the original; achieving up to 3.5x further speedup through multi-node training; using FSDP2 to gain another 20% performance improvement; and adopting an H200 multi-node cluster to add up to another 30% on top of that.

The source says it is precisely these optimizations stacked together that produced the roughly 8x training speedup mentioned in the article's title. Uncertainty must be flagged here: the roughly 8x is the result of multiple optimizations stacked together, the precise breakdown of each individual contribution is not given, and all the above speedup figures come from LinkedIn's own research or team statements.

The team points out that they had evaluated FP8 mixed precision but found it does not help models with fewer than 8 billion parameters, because type conversion overhead exceeds the computational savings.

The source further distills this into a tip for teams: for models with fewer than 8 billion parameters, none of the above optimizations require using FP8. For teams considering adopting lower precision by default, this detail is worth noting.

At the model level, this 600M-parameter student model draws its knowledge from an 8B-parameter relevance prediction model and a 1.7B-parameter engagement teacher model. The source says LinkedIn's research shows the student model improved job search results, raising job search NDCG@10 by 24.48%, from 0.7583 to 0.9432.

The same study also adopted structured pruning and context compression on the inference side, increasing ranking throughput per GPU from about 290 items per second to more than 2,000. These figures likewise come from LinkedIn's own research and lack independent third-party verification; the baseline comparison details between the 600M-parameter student model and the 8B and 1.7B teacher models are also not fully disclosed in the article.

The system is already in production, supporting natural-language job search for LinkedIn's U.S. users. It is built on the open-source large language model serving engine SGLang, which LinkedIn had previously invested in for ranking workloads, rather than using a proprietary serving technology stack.

The source frames this work as a guide for teams: helping them achieve ranking quality close to cross-encoders while meeting real-time latency requirements, without having to call frontier large language models for inference on every request and thus avoid high costs. The boundary that must be preserved is: the system's deployment outside the United States is not mentioned; and the comparability of the data from the Pinterest article and the frontier model distillation survey cited as background comparisons with LinkedIn's approach is also unclear.