How a Photo-Recognition AI Found Whales Using Earthquake Data

Quick Take

• Earthquake sensors were never meant to track whales, yet the accidental recordings they've been quietly collecting could unlock decades of hidden whale behavior. Explore the repurposed sensors →
• An AI built to look at photos, having never been trained on a single whale sound, can identify whale calls more reliably than methods designed specifically for the job. See how the image AI listens →
• The AI flagged whale calls that expert researchers had completely missed, and the reason why reveals something unexpected about how whale songs look. Discover the missed calls →
• Whale call timing patterns exposed a behavioral shift scientists didn't fully anticipate, a finding that changes how researchers think about when and why whales communicate. See the behavioral shift →

In the pitch-black depths along the ocean floor, scientific instruments constantly listen to the Earth move. Seismometers scattered across the seabed are designed to record the grinding of tectonic plates, but their archives have captured an unexpected sound: the deep, booming songs of Bryde’s, blue, and fin whales. These low-frequency whale calls (between 10 and 100 Hz) are so powerful that they physically shake the ocean floor. Rather than signaling earthquakes, these recordings reveal the movements of some of the largest animals on Earth.

Now, researchers are using these accidental recordings to study whale behavior with help from an unlikely tool: Meta’s Segment Anything Model (SAM). Originally designed to recognize objects in photos, this vision-based AI has been repurposed to “hear” the whales. By converting the audio recordings into visual charts (spectrograms), SAM can scan them like images to identify whale calls. This clever adaptation is giving scientists a powerful new way to map and understand these elusive marine giants.

Teaching an Image AI to “Hear” Whales

Marine scientists face an overwhelming challenge. Millions of hours of underwater recordings sit largely untouched because manually reviewing them takes so long. Traditional AI hasn’t solved the problem either. Building a custom model requires experts to manually label tens of thousands of whale calls just to teach the AI how to distinguish a whale’s voice from background noise.

However, a new study led by Zhuo Xiao of Guangxi Minzu University took a different approach: instead of training an AI to recognize whale sounds, the researchers transformed the sounds into images.

To do this, researchers used a mathematical tool called a Short-Time Fourier Transform (STFT). This converts the continuous audio recordings into spectrograms — visual charts that display sound frequency over time.

Once the sound is converted into an image, detecting whales becomes an image segmentation problem rather than an audio problem, which is exactly what Meta’s Segment Anything Model (SAM) was designed to solve.

Because SAM is already adept at identifying shapes and objects in images, it can spot the visual patterns of whale calls on the spectrogram. In fact, SAM achieved this with zero-shot learning — meaning it successfully identified the whale calls without ever having been trained on underwater audio or whale data beforehand.

Why AI Works Without Learning Whale Calls

SAM’s success doesn’t come from understanding whales — it comes from recognizing shapes. Meta originally trained the AI on over 11 million images and 1.1 billion object outlines, giving it an incredible ability to spot edges, boundaries, and textures. This means SAM doesn’t need to know what a whale is. It simply analyzes the spectrogram (the sound image), helping to identify the unique lines and repeating geometric patterns created by whale songs and separating them from random ocean noise.

Instead of training the AI with thousands of labeled examples, researchers used simple prompt engineering. They provided SAM with basic visual cues—such as clicking a point or drawing a bounding box around a specific shape—to indicate exactly which patterns it should isolate.

The team also took advantage of the predictable nature of baleen whale vocalizations. These calls occur within very specific frequency ranges and follow strict, rhythmic patterns called inter-pulse intervals (IPI). The researchers also used a multi-step cleaning process to filter out background noise from earthquakes and the environment. Since the remaining whale patterns are so rhythmic and predictable, SAM was able to distinguish them from unrelated sounds with remarkable accuracy.

The system achieved precision and recall rates exceeding 96 percent. This demonstrates that a general image-recognition AI can detect whale calls in seismic data with accuracy comparable to that of specialized models designed for this task.

A Global Tool for Every Ocean

To demonstrate that this method was not a one-off success, researchers tested it across different locations and whale species to assess its effectiveness in diverse real-world conditions. They started with Bryde’s whales using seismic data from Xieyang Island in the South China Sea. Then, without changing the core workflow, they applied the same system to fin whale recordings from off the coast of Ireland and blue whale data from Canada.

Despite the differences in species and environments, the AI model maintained its high accuracy without requiring major adjustments. In fact, it was so precise that it identified additional candidate whale calls that were missed during the initial manual review.

These results suggest that baleen whale songs leave incredibly consistent “visual signatures” on spectrograms. Because these patterns look so similar regardless of the species or location, a general image-recognition AI could be used globally. This would provide marine scientists with a ready-to-use tool for monitoring whale populations across the world’s oceans.

What Whale Songs Reveal

By unlocking these seismic archives, scientists can study whale behavior without disturbing the animals. As a test case, researchers analyzed data from January and July of 2021, tracking the exact timing between whale calls (the inter-pulse intervals). They discovered clear seasonal shifts in how Bryde’s whales communicate.

During winter, the gaps between calls were shorter, suggesting that the whales were engaging in more coordinated, group communication. In summer, the gaps became longer, indicating that the whales had shifted to more solitary, individual calling. In the Beibu Gulf, these seasonal differences are helping scientists map out exactly how these whales use this vital, shallow-water feeding ground year-round.

This breakthrough does more than simply decipher whale songs. The ability to continuously track when and where whales are calling could help scientists in many other ways. They could monitor migration routes, track changing population distributions, and better understand how whales respond to environmental change.

Turning Seafloor Sensors Into Conservation Tools

One of the most exciting aspects of this research is that it doesn’t require building an entirely new monitoring network. Instead, researchers are repurposing earthquake-monitoring equipment that has already been on the ocean floor for decades.

This could eventually help conservationists identify important whale habitats and establish ship-strike mitigation zones using data that is already being collected.

However, the system isn’t perfect. Background ocean noise can still produce occasional false positives or missed detections. Researchers are continuing to update and refine their noise filters.

Looking ahead, Zhuo Xiao and the team plan to develop a specialized AI foundation model specifically tuned to recognize whale and dolphin vocalizations. Future versions may combine seismic data with underwater microphones (hydrophones) and ocean current data. They’re also exploring how to integrate data from multiple monitoring systems — including island stations, ocean-bottom seismometers, and fiber-optic distributed acoustic sensing arrays — to further improve detection.

By transforming existing seismic sensors into whale-monitoring tools, researchers are demonstrating how AI can uncover decades of hidden history buried beneath the ocean floor. This brings scientists one step closer to mapping, understanding, and protecting the secret lives of Earth’s most elusive giants.