Search code examples
javaaudiojavasound

How to detect certain sounds in a sound file and perform actions in sync with them?


I am working on an application for a class (as part of a team) where the idea is to help you learn to play the piano. An image of the instrument is displayed on screen, and we have some class Octave where you are in a certain octave of the piano so that you may highlight the appropriate key. For example, you would tell the highlight method which note it is, and it knows which part of the image to highlight since there is always an active Octave object to check, that finds the key by the distance from the beginning of the image. There is no issue with this part so far that I know of (it is my team member's responsibility).

What I am trying to accomplish is to have a sound file played (such as a scale or a song) after the user selects an appropriate option, but the keys should be highlighted on screen in sync with the correct notes being played. This is exactly my problem; I have no idea how to approach this, and googling for similar cases has not helped.

So what I am asking is if someone can suggest how I should try to do this? Ideally, it should be generic (take a quality sound file and highlight the correct notes, rather than coding for each file to be added), and as such it would be nice to have some way to perhaps detect the notes and pitch that are being played to determine which note in which octave it is, and then I would be able to use that information with the other classes easily. That is just my idea on how I think it might be solved, but I am open to other solutions you think may be better. Thanks in advance for any advice!


Solution

  • What you are asking can be really difficult, especially in the general case.

    In fact, what you are asking for is called Automatic Music Transcription. Nowadays, Automatic music transcription accuracy is similar to text-to-speech: not very good.

    What you should probably do

    For a music learning application, you should probably consider using an already written transcription. For example, you could just read a midi file describing the notes in the song, and use the information in the midi file to highlight the notes of the piano.

    If you really want Automatic Music Transcription

    Then, you should probably start by reading papers of folks who did it, and accept an accuracy of about 80%....