A research paper on real-time voice large language model (LLM) systems by the team of Associate Professor Yifei Zhu at Shanghai Jiao Tong University Global College (SJTUGC, abbreviated as GC hereafter), has been accepted by the 32nd ACM SIGOPS Symposium on Operating Systems Principles (SOSP 2026), a leading international conference in computer systems. Titled “Scalable Context Orchestration for Serving LLMs Over Voice,” the paper proposes llmovoice, a context-management middleware designed to improve the efficiency, naturalness, and robustness of voice LLM services.
Voice AI applications are becoming increasingly accessible as advances in LLMs enable more natural spoken interaction. Compared with text-based interaction, voice interfaces can provide a more convenient way for users to interact with AI systems and can also support applications in settings where conventional text input is less accessible. However, serving voice LLM applications presents challenges beyond those encountered in text-based systems. In addition to understanding what users say, voice systems need to account for how users speak and the conditions under which their audio is captured and transmitted. Existing voice LLM services face challenges in handling long-running conversations, adapting responses to users’ speaking styles, and maintaining reliable interaction under unstable network conditions. As audio history accumulates, the cost of processing context can increase substantially. Existing systems may also have difficulty adapting response timing to users’ speaking rates. Meanwhile, network jitter and packet loss can introduce abnormal pauses that may be mistaken for the end of a user’s utterance, causing the system to respond prematurely.
Architecture of llmovoice
To address these challenges, Zhu’s team developed llmovoice, a context-management middleware between streaming audio and model inference. The system explicitly represents not only semantic information, but also paralinguistic information such as speaking rate and environmental information such as network conditions. It then uses an LLM-driven context orchestration mechanism to determine how different types of context should be used and to generate runtime directives for the voice LLM service. The system introduces VoicePage and VoiceThread abstractions to organize voice conversations and dynamically manage the information provided to the model. Depending on the available context budget, llmovoice can select audio, transcripts, summaries, or other forms of information from the conversation history, enabling more efficient use of the model’s context window.
The research team evaluated llmovoice using benchmark workloads, real-world voice data, and end-to-end applications. Compared with an OpenAI Realtime baseline that retains the complete voice history, llmovoice reduced per-turn model usage cost by up to 24.9 times while retaining up to 98.7% of baseline answer quality. It also reduced speaking-rate alignment error by 52.4%. Under packet-loss conditions, the system reduced the false-interruption rate from 46.0% to 0.9% and reduced related model usage cost by 79.2%. These results demonstrate the potential of llmovoice to improve the cost efficiency, interaction quality, and network robustness of long-running voice LLM services. The approach has potential applications in intelligent customer service, healthcare, online education, in-vehicle assistants, and smart home systems, where long-running and reliable voice interaction is required.
Linyi Jiang, a 2024-entry Ph.D. student in Information and Communication Engineering at GC, is the first author of the paper. Yifei Zhu is the corresponding author, and Silvery D. Fu from AgenticSys is a co-author.
The ACM SIGOPS Symposium on Operating Systems Principles (SOSP) is a leading international conference in computer systems, covering research areas including operating systems, distributed systems, cloud computing, mobile and edge systems, reliable systems, and systems for artificial intelligence. SOSP 2026 will be held from September 29 to October 2 in Prague, Czech Republic. A total of 62 papers were accepted for the conference.
Personal Introduction
Linyi Jiang is a 2024-entry Ph.D. student in Information and Communication Engineering at Shanghai Jiao Tong University Global College. Her research focuses on model inference optimization for dynamic environments. She has published three first-author papers in CCF-A conferences in the fields of computer networks and systems.
Yifei Zhu is an Associate Professor at Shanghai Jiao Tong University Global College. He received his Ph.D. in Computer Science from Simon Fraser University in Canada. His research interests include edge computing, multimedia networks and systems, and distributed machine learning systems. He has published more than 90 papers in leading international conferences and journals, including ACM SIGCOMM, ACM MobiCom, ACM SOSP, IEEE INFOCOM, and ACM Multimedia. His research has received the first prize in the Telus-Juniper Software Defined Network Challenge in Canada and four IEEE conference paper awards. He also serves as an editor for the IEEE Internet of Things Journal, an area chair for ACM Multimedia, a guest editor for IEEE Network, and a technical program committee member for several international conferences.
