Google DeepMind will be able to generate music and sounds for silent videos

Google DeepMind has unveiled an innovative technology that can generate background music and sound effects for silent videos. This “video-audio” system is designed to simplify the video editing process, especially for content creators.

Google DeepMind has introduced an innovative technology that can generate background music and sound effects for silent videos.

Process of the technology

• Introduction of information from the user. Content creators upload their silent video and can provide keywords or phrases to guide the AI in creating a soundtrack. For example, for a video of a person walking in the dark, cues such as “movies, horror, music, suspense, footsteps on concrete” can be used to help the AI understand the mood and environment.
• AI Performance. DeepMind’s AI model first analyzes the visual content of the video. This data is then combined with the user’s textual cues. Using a diffusion model, the AI iteratively processes this information and eventually generates background sounds that complement the video.

.

• Customization of the audio track. The model can create different audio options for a single video, allowing creators to choose the best option for their project. The DeepMind system can also take into account the emotional tone of the cues. For example, cues emphasizing “tension” can produce tense background music, whereas cues like “joyful celebration” will create more upbeat sounds.

.

Future Development

DeepMind is actively improving the technology. Future plans are for the AI to automatically generate sounds based solely on video content, eliminating the need for user prompts. Work is also underway to improve the synchronization of generated dialogue with the lip movements of characters in the video.

DeepMind is also working on improving the synchronization of generated dialogue with the lip movements of characters in the video.

This “video-audio” technology has the potential to revolutionize video editing, especially for content creators who don’t have access to professional audio tools or expertise. DeepMind is taking steps to make the video creation process more accessible and efficient.

DeepMind is taking steps to make the video creation process more accessible and efficient.