DoorDash uses LLMs and multimodal AI to generate and optimize food metadata through context optimization and a jury system, improving accuracy and efficiency in food data management.
SynthePulse Insight · AI deep reading
Version 1 · 1 source
DoorDash disclosed on its engineering blog a new method using an LLM jury and multimodal AI to automatically generate food metadata, improving data quality through context optimization and multi-evaluation.
Food metadata (e.g., ingredients, flavors, cooking methods) is highly diverse and subjective, with naming conventions varying greatly across regions, languages, and cultures. Traditional rule-based methods struggle to cover all cases, while relying solely on a single LLM often leads to inconsistency or hallucinations. DoorDash needed a scalable and reliable solution to handle tens of thousands of new dishes added daily.
DoorDash's approach is to deploy an 'LLM jury': multiple language models with different architectures and training data simultaneously evaluate the same input, with each model providing candidate values or confidence scores for metadata fields (e.g., 'cuisine', 'spiciness'). The final output is produced through majority voting or weighted fusion.
This design leverages model diversity to offset individual biases. For example, one model may excel at identifying Southeast Asian cuisines, while another may be more knowledgeable about Middle Eastern spices. The jury integrates their judgments, thereby improving accuracy and robustness overall.
In addition to dish names and descriptive text, DoorDash also introduces multimodal features such as dish images. Visual information helps confirm colors, plating, and ingredient forms, thereby assisting in inferring cooking methods (e.g., 'roasted' vs. 'fried') or freshness cues.
Context optimization includes dynamic factors like geographic location, seasonality, and trending trends. The same dish may be assigned different labels in different cities or seasons (e.g., marked as 'in-season'). DoorDash enhances the LLM's contextual understanding through knowledge graphs and real-time data.
In internal tests, the metadata generated by the LLM jury outperformed single-model baselines in both accuracy and coverage, but approximately 5% of high-value items still required human review—primarily new products or rare cuisines. DoorDash positions human review as a fallback mechanism rather than a routine process.
In terms of cost, multi-model calls introduce higher latency and computational overhead, but DoorDash optimizes efficiency by caching inference results for common scenarios and activating the full jury only when necessary (e.g., for new items or low-confidence cases).
This approach demonstrates the potential of LLMs in generating structured data for vertical domains: leveraging multi-model collaboration and multimodal fusion can significantly improve automation quality. Similar ideas can be extended to scenarios such as e-commerce product descriptions, medical record structuring, and legal document tagging.
DoorDash's next steps include introducing a user feedback loop to directly fine-tune the jury models with corrections, and exploring lightweight model combinations to reduce latency.
This article is based on disclosures from DoorDash's official technology blog. The content is a first-party statement from the company and has not been independently replicated or peer-reviewed by third parties. Some performance data are internal test results and may be selectively published.
DoorDash provides a viable path for automating food metadata through an LLM jury and multimodal context optimization, balancing quality, cost, and scalability. The core lesson is that a single model is less reliable than diverse models working collaboratively, and multimodal information can significantly enhance the richness and accuracy of structured data.
Primary report
Primary source