{"id":175,"date":"2025-05-22T05:48:22","date_gmt":"2025-05-22T12:48:22","guid":{"rendered":"https:\/\/scienceblog.com\/neuroedge\/?p=175"},"modified":"2025-05-22T05:48:22","modified_gmt":"2025-05-22T12:48:22","slug":"ai-learns-to-connect-sight-and-sound-like-humans-do","status":"publish","type":"post","link":"https:\/\/scienceblog.com\/neuroedge\/2025\/05\/22\/ai-learns-to-connect-sight-and-sound-like-humans-do\/","title":{"rendered":"AI Learns to Connect Sight and Sound Like Humans Do"},"content":{"rendered":"

Artificial intelligence systems are getting better at mimicking how humans naturally connect what they see with what they hear.<\/p>\n

MIT researchers have developed a new machine-learning approach that helps AI models automatically match corresponding audio and visual information from video clips<\/a>\u2014without needing human labels to guide the process. The breakthrough could eventually help robots better understand real-world environments where sound and vision work together.<\/p>\n

The research builds on how humans instinctively learn by linking different senses. When you watch someone play the cello, you naturally connect the musician’s bow movements with the music you’re hearing. The MIT team wanted to recreate this seamless integration in artificial systems.<\/p>\n

Fine-Tuning Audio-Visual Connections<\/h2>\n

The researchers improved upon their earlier work by creating a method called CAV-MAE Sync, which learns more precise connections between specific video frames and the audio occurring at exactly those moments. Previously, their model would match an entire 10-second audio clip with just one random video frame\u2014like trying to sync a whole song with a single photograph.<\/p>\n

“We are building AI systems that can process the world like humans do, in terms of having both audio and visual information coming in at once and being able to seamlessly process both modalities,” explains Andrew Rouditchenko, an MIT graduate student and co-author of the research.<\/p>\n

The new approach splits audio into smaller windows before processing, creating separate representations that correspond to each smaller time segment. During training, the model learns to associate individual video frames with the audio happening during just those frames\u2014a much more granular and realistic approach.<\/p>\n

Solving Competing Objectives<\/h2>\n

The team tackled a fundamental challenge in AI training: balancing two different learning goals that can conflict with each other. The model needs to both reconstruct missing audio and visual information (like filling in blanks) and learn to associate similar sounds with similar images.<\/p>\n

These objectives were competing because they required the same internal representations to do double duty. The researchers solved this by introducing specialized “tokens”\u2014dedicated components that handle different aspects of learning without interfering with each other.<\/p>\n

Key improvements in the new system include:<\/p>\n