MusicClassification Evaluation: Metrics, Datasets, and Benchmarks

MusicClassification Techniques — From Feature Extraction to Deep Learning

Overview

Music classification assigns labels (genres, moods, instruments, tags) to audio. Techniques range from handcrafted feature extraction with classical ML to end-to-end deep learning that learns directly from raw audio or spectrograms.

1. Feature extraction (traditional)

• Time-domain features: Zero-crossing rate, RMS energy, tempo estimates.
• Frequency-domain features: Spectral centroid, bandwidth, roll-off, spectral flux.
• Cepstral features: Mel-frequency cepstral coefficients (MFCCs) — widely used for timbre.
• Harmonic/percussive separation: Extract harmonic features (chroma, tonnetz) and percussive onset features.
• Statistical summaries: Mean, variance, skewness, percentiles over frames to form fixed-length vectors.

2. Classical ML models

• k-NN, SVM, Random Forests, Gradient Boosting: Trained on extracted features; effective for smaller datasets and interpretable setups.
• HMMs/GMMs: Useful for modeling temporal sequences in certain tasks (e.g., instrument onset patterns).

3. Time–frequency representations

• Short-Time Fourier Transform (STFT) spectrograms
• Mel-spectrograms: Better perceptual alignment; common input to ML/DL models.
• Constant-Q Transform (CQT): Better for music pitch resolution.

4. Deep learning approaches

• Convolutional Neural Networks (CNNs): Applied to spectrogram images to learn local time–frequency patterns. Architectures: simple CNNs, ResNet, DenseNet.
• Recurrent Neural Networks (RNNs) / LSTM / GRU: Model temporal dependencies over frames or feature sequences. Often combined with CNN front-ends (CNN→RNN).
• Temporal Convolutional Networks (TCNs): Efficient alternative to RNNs for sequence modeling.
• Transformers: Self-attention models for long-range dependencies; used on frame embeddings or patchified spectrograms.
• End-to-end raw-audio models: 1D CNNs (WaveNet-like, SampleCNN) that learn filters from raw waveform.

5. Training strategies & tricks

• Data augmentation: Time-stretching, pitch-shifting, noise injection, SpecAugment on spectrograms.
• Transfer learning: Pretrained audio models (e.g., VGGish, YAMNet, OpenL3) or ImageNet CNNs fine-tuned on spectrograms.
• Multi-task learning: Jointly predict related labels (genre + mood + instruments) to improve shared representations.
• Class imbalance handling: Weighted loss, focal loss, oversampling, or mixup.

6. Evaluation & datasets

• Common metrics: Accuracy, F1-score, precision/recall, mean average precision (mAP) for multi-label tasks.
• Datasets: GTZAN (genres), MagnaTagATune (tags), Million Song Dataset (features/metadata), FMA (Free Music Archive) for balanced/large-scale experiments.

7. Deployment considerations

• Latency vs. accuracy: Lightweight models or distilled networks for real-time inference.
• Streaming inputs: Frame-wise prediction aggregation (voting, averaging, attention pooling).
• Explainability: Saliency maps on spectrograms, class activation maps to interpret model decisions.

8. Practical pipeline (concise)

1. Collect/label audio and pick task (single-label vs. multi-label).
2. Preprocess: resample, trim/pad, normalize.
3. Extract representations: mel-spectrogram or raw waveform.
4. Choose model: classical ML for small data, CNN/Transformer for large data.
5. Train with augmentation and appropriate loss.
6. Evaluate on held-out test set; iterate.
7. Optimize and deploy (quantization, pruning, streaming).

Further reading (keywords)

• MFCC, Mel-spectrogram, SpecAugment, VGGish, YAMNet, GTZAN, MagnaTagATune, Million Song Dataset.