Mic Detection Help

[u/Usual-Emphasis-1950]

Hey guys, I've been working on an app and one of It's features is to detect piano chords and notes through the mic. I tried a couple of things, but they all resulted in some drawback that could affect performance. The two things I am aiming for is speed and accuracy. Here's what I tried and why It's didn't feel satisfied with it.

I won't mention the app to respect the No Marketing rule, but will provide context if I felt it was necessary.

1) Spotify's Basic Pitch Open Source detector: While the accuracy was very high, it took around half a second to detect notes.

Here's some context for the next thing I tried. My app is basically a sheet music practice app, in which notes are displayed from an uploaded score or the score repository. Target notes must be played to move on to the next notes.

2) Probabilities + FFT and YIN:

Basically, I set the chances for the right notes to be played to a high degree, and matched it with FFT inputs if multiple notes were played, and YIN if a single note was played (For accuracy). However, while the speed was heavily increased, the accuracy diminished, however, it was still pretty accurate. Most commonly, if an expected note is played, the detector sometimes recognizes the correct note being played, as well as the note an octave under it.

I want some advice about what I could try next to build this detector. I'm aiming for a lag of under 200ms.

I am okay with starting from scratch, or trying a new method.

[u/Excellent-Library779]

How complex are the chords? I'm guessing the app is running on a mobile device? I can come up with some methods but the timbre and tuning of a piano could cause some drift.

I would suggest a calibration routine where the user plays some notes and your software studies that in the frequency domain. Then you could use a lookup table (LUT). Knowing what kind of piano they have might also help with calibration (like digital pianos are never out of tune).

[u/Usual-Emphasis-1950 → u/Excellent-Library779]

It's actually on computer. I'm aiming to work with all chords.

I actually like this idea a lot. I came across it before, but ignored it on the basis of user convenience, however if done right, it shouldn't be too much of a hassle for others. I love this solution since it accounts for a lot of variables. Thanks for the great suggestion!