Continual Visual and Verbal Learning Through a Child’s Egocentric Input

Xiaoyang Jiang, Yanlai Yang, Kenneth A. Norman, Brenden Lake, Mengye Ren

• We introduce BabyCLIP, a continual multimodal learning framework that learns grounded word-referent mappings from a child’s egocentric video stream in a single chronological pass. By combining streaming visual self-supervision, temporal event segmentation, replay-based continual learning, and image-text contrastive learning, BabyCLIP substantially outperforms naive streaming baselines on the SAYCam Labeled-S benchmark, demonstrating that meaningful multimodal semantic structure can emerge under cognitively realistic online learning conditions.
• Paper accepted at CVPR 2026 Workshop on Computer Vision for Children & under review at NeurIPS 2026.

Fully Spiking Neural Network for Legged Robots

Xiaoyang Jiang, Qiang Zhang, Jingkai Sun, Jiahang Cao, Jingtong Ma, Renjing Xu

Project

• In this paper, we successfully apply a lightweight population coded Spiking Neural Network (SNN) to process legged robots, achieving outstanding results across a range of simulated terrains. This study presents a highly efficient SNN for legged robots that can be seamless integrated into other learning models.
• Paper accepted for oral presentation at ICASSP 2025.

Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learning

Jingkai Sun, Qiang Zhang, Yiqun Duan, Xiaoyang Jiang, Chong Cheng, Renjing Xu

• We present a novel approach that combines adversarial imitation learning with large language models (LLMs). This innovative method enables the agent to learn reusable skills with a single policy and solve zero-shot tasks under the guidance of LLMs. To the best of our knowledge, this is the first framework that controls humanoid robots using a single learning policy network and LLM as a planner.
• Accepted at ICRA 2024.

Saliency-Guided Hidden Associative Replay for Continual Learning

Guangji Bai, Qilong Zhao, Xiaoyang Jiang, Yifei Zhang, Liang Zhao

• This paper presents the Saliency Guided Hidden Associative Replay for Continual Learning. SHARC primarily archives salient data segments via sparse memory encoding. Importantly, by harnessing associative memory paradigms, it introduces a content focused memory retrieval mechanism, promising swift and near-perfect recall, bringing CL a step closer to authentic human memory processes.
• Accepted at NeurIPS 2023 Workshops.