Mastering Sound: Frequency Features and Spectrograms

In this article, we delve into the detailed realm of Spectrogram features—a crucial component in sound analysis for machine learning applications. Following our previous guide, where we explored the fundamental properties of sound, we now narrow our focus to shed light on the importance of Spectrogram features. From theoretical foundations to practical applications, our goal is to provide a comprehensive understanding for effectively harnessing their power. Join us on this journey through the spectral landscape, building upon the groundwork laid in our earlier discussion of sound properties.

In this article we’ll cover the following topics:

• Different domain features
• Fourier transform (FT)
• Discrete Fourier transform (DFT)
• Fast Fourier transform (FFT)
• Short-time Fourier transform (STFT)
• Spectrogram
• When to use spectrogram features
• Common applications

Spectrogram domain features

Sound features can be broadly categorized into two distinct domains

• Time domain features:
  • Where the features are extracted from the raw audio itself in the time domain.
• Frequency domain features:
  • Where the features are calculated and extracted in the frequency domain instead of the time domain.
  • Frequency Domain refers to the analytic space in which mathematical functions or signals are conveyed in terms of frequency, rather than time.
  • Frequency domain allows us to operate and diagnose signals more freely, with less limitations and more capabilities.
  • Discussing these features, or some of them will be the main target of this article.
  • We will start from the Fourier transform of the raw audio from time domain to frequency domain.
  • Moving to the spectrogram features used in many applications right now.

Fourier transform (FT)

The Fourier Transform (FT) stands as a pivotal algorithm in signal processing, facilitating the conversion of signals from the time to the frequency domain. At its essence, the FT accomplishes this transformation by representing the original signal as a sum of sinusoidal waves, each characterized by unique amplitudes and frequencies. This decomposition enables a comprehensive understanding of the signal’s frequency components. In the frequency domain, each sinusoidal wave is elegantly represented by a corresponding pin, offering a visual and analytical tool to navigate the intricate landscape of the signal’s frequency composition. The Fourier Transform not only provides a powerful method for signal analysis but also proves instrumental in a wide array of applications, ranging from telecommunications to image processing.

Discrete Fourier transform (DFT)

The Discrete Fourier Transform (DFT) is specifically tailored for the discrete representation of signals in the frequency domain. Functioning as a discrete counterpart to the continuous Fourier Transform, the DFT transforms a sampled signal into a spectrum of discrete frequencies. This process involves expressing the original signal as a sum of discrete sinusoidal components, each characterized by unique amplitudes and frequencies.

The resulting frequency domain is navigated through the discrete representation of sinusoidal waves, providing insights into the signal’s frequency composition. With widespread applications across various domains, the DFT proves instrumental in digital signal processing, communication systems, and numerous other fields where discrete data analysis is paramount.

Fast Fourier transform (FFT)

Enter the Fast Fourier Transform (FFT), an ingenious algorithm designed to expedite the computation of the Discrete Fourier Transform (DFT). As an optimized version of the DFT, the FFT dramatically accelerates the process of transforming a sampled signal into its frequency domain representation. By efficiently decomposing the signal into discrete sinusoidal waves with varying amplitudes and frequencies, the FFT empowers rapid analysis and interpretation of complex signals. This acceleration in computation has positioned the FFT as a cornerstone in numerous applications, from real-time signal processing to audio and image compression. In essence, the FFT streamlines the formidable task of frequency analysis, making it a versatile and indispensable tool in the digital signal processing toolkit.

Short-time Fourier transform (STFT)

While the Fast Fourier Transform (FFT) proves highly efficient in frequency analysis, it does come with a notable drawback—it sacrifices the crucial time property of a signal. The need to pinpoint when and where specific events occur in a signal is paramount, and this is where the Short-Time Fourier Transform (STFT) emerges as a valuable solution. STFT strategically addresses this limitation by incorporating windowing in the time domain, breaking the signal into overlapping segments. By converting each window into its frequency domain representation, STFT ensures that identified features correspond to specific time intervals. A commonly employed technique involves using an overlapping Hanning window, which centers on the primary portion of interest while mitigating edge effects. This windowed segmentation, followed by FFT application, effectively preserves the temporal information, allowing for a comprehensive transformation of the signal into the frequency domain without loss of critical timing details.

The Spectrogram

The Spectrogram is essentially a representation of the power derived from the Short-Time Fourier Transform (STFT). In this process, we calculate the magnitude of the STFT for a given signal and then square the result. By doing so, we intentionally discard the phase information and concentrate solely on the magnitude.

The Spectrogram proves to be a versatile tool with applications in various domains. It finds utility in tasks such as sound super resolution, where detailed frequency information is essential. Additionally, it serves as a crucial component in sound noise rejection systems, aiding in the identification and mitigation of unwanted background noise. The Spectrogram, with its emphasis on magnitude characteristics, emerges as a valuable asset for diverse applications within the realm of sound analysis and processing.

When to use spectrogram features

• Frequency Analysis in Time: Spectrogram features are valuable when detailed insights into how the frequency content of a signal evolves over time are essential.
• Transient Event Detection: They prove effective in identifying and analyzing transient events, such as sudden changes or peaks in a signal, crucial for various applications like fault detection.
• Speech and Voice Recognition: Spectrogram features are widely employed in speech and voice recognition systems, aiding in the extraction of distinctive vocal patterns.
• Environmental Sound Classification: When categorizing sounds in the environment, such as identifying specific noises or events, spectrogram features provide a comprehensive representation.
• Musical Signal Processing: In music analysis and processing, spectrograms help reveal nuances in musical compositions, assisting in tasks like instrument recognition and melody extraction.
• Anomaly Detection: Spectrogram features are useful for detecting anomalies in time-series data, making them valuable in applications like industrial machinery monitoring.
• Biomedical Signal Processing: In the biomedical field, spectrograms are employed to analyze signals like electroencephalograms (EEG) or heart sounds, aiding in diagnostics and research.
• Sonar and Radar Signal Processing: They find application in analyzing sonar and radar signals, assisting in target detection and classification in maritime and aerospace domains.
• Communication Signal Processing: Spectrogram features play a role in communication systems, aiding in the analysis and processing of signals in areas such as modulation recognition.
• Localization of Sound Sources: They are beneficial for determining the spatial distribution of sound sources, aiding in applications like audio scene analysis and surveillance.

Common applications

• Speech Recognition: Spectrograms are widely used to extract and analyze vocal patterns in speech recognition systems.
• Sound Quality Assessment: Spectrogram features find application in assessing and enhancing the quality of audio signals, crucial in fields like telecommunications.
• Music Analysis: In music processing, spectrograms reveal details in compositions, aiding in tasks like instrument identification and music genre classification.
• Environmental Sound Classification: Spectrograms play a key role in categorizing environmental sounds, facilitating applications in areas like surveillance and monitoring.
• Fault Detection: Spectrogram features are effective in identifying transient events, making them valuable for fault detection in machinery and systems.

Resources & References

• Sound Properties: All You Need to Know
• Discrete Fourier transform – Wikipedia
• Short-time Fourier transform (STFT) overview
• Vibration Analysis: FFT, PSD, and Spectrogram Basics.