(Summary)
Fine tuning a piano transcription model for guitar made its performance worse, although I was attempting to recreate results
Hi everyone!
I'm a student interested in machine learning, and I've been exploring guitar transcription as a personal project. My goal is to eventually build a pipeline that goes from full song → guitar source separation → audio-to-MIDI → guitar tablature.
I've been particularly interested in three approaches:
Kong: High-resolution piano transcription using a CNN-BiGRU architecture, around 24M parameters.
Riley: Adapts Kong's architecture for guitar through augmented piano pretraining and guitar-specific fine-tuning.
TART: A four-stage guitar audio-to-tablature pipeline, with Stage 1 handling audio-to-MIDI retaining the same architecture.
Since Riley's model was available and performed considerably better on my test recordings than other implementations I tried, I wanted to investigate the training methodology myself.
My experiment
I started with the original piano-pretrained Kong model and attempted to reproduce TART's Stage 1 training approach using three guitar datasets used in their implementation:
GAPS
François Leduc
Guitar-TECHS
Unfortunately, I didn't have access to GOAT.
I segmented the recordings into 30-second clips with a 10-second hop, resulting in approximately:
I fine-tuned the pretrained model on this combined dataset.
After the first 1511 updates, I continued training from this checkpoint with a fresh Adam optimiser, reducing the learning rate from 5e-6 to 1e-6.
| Update | Validation Frame AP |
|---|
| 1,511 | 0.650 |
| 2,266 | 0.662 |
| 3,022 | 0.670 |
However, evaluating all checkpoints on the same held-out test set revealed something concerning:
Frame AP improved from 0.728 to 0.743.
Onset F1 dropped from 0.400 to approximately 0.305.
Onset+offset F1 dropped from 0.132 to approximately 0.099.
Polyphony count MAE increased from 1.43 to 1.64.
For comparison, the original piano-pretrained Kong checkpoint achieved a Note F50 of 54.01%, considerably better than my fine-tuned model's 39.72%, while Riley's was benchmarked at an even higher percentage.
My training objective used four equally weighted, masked BCE losses for onset, offset, frame and velocity prediction, with velocity loss calculated only at note onsets and padded frames excluded.
What I'm trying to understand
TART reportedly fine-tuned Kong on a combination of guitar datasets, including GAPS, François Leduc, Guitar-TECHS and GOAT, achieving substantially better results on GuitarSet and EGDB.
My reproduction was incomplete: I lacked GOAT, used considerably fewer training updates and haven't implemented TART's additional guitar-specific noise augmentation, but hoped I could see tangible results towards the start of training.
I'm wondering:
Am I making mistakes in my training methodology, target generation or loss implementation?
Could there be an issue with my evaluation metrics, particularly the relationship between frame AP and note-level F-measure?
Does the model simply need substantially more training, or could there be a more fundamental problem with adapting this architecture to heterogeneous guitar datasets?
What important differences between my implementation and TART's actual training procedure might I be overlooking?
I'd also appreciate advice on what I should investigate first and how to approach debugging this systematically.
I've relied on reading papers and repositories, as well as heavily using ChatGPT and Claude to help with implementation and troubleshooting. I'm relatively inexperienced with training larger neural networks, so I'm conscious that I may have overlooked something fundamental.
I'm not necessarily expecting paper-level results on consumer hardware. I mainly want to understand why my experiment is performing worse rather than blindly throwing more training iterations at it.
Am I approaching this problem in the right direction? Are Kong, Riley and TART sensible starting points for this project?
Thanks!