I've been playing around with my home Karaoke system. This has included decoding my Songken DKD disk. Once I did that, I wanted to emulate the behaviour of my Malata Karaoke player: playing the Midi files, showing the lyrics and also showing in a bar graph the notes that should be sung and the notes the performer is actually singing.
This requires processing files of Midi data, handling the soundcard microphone input and speaker output and using a GUI to show everything. Java Sound looks like a perfect choice for this as it can do all these things. (Although Oracle's custodianship of Java and their outrageous API copyright claims makes it increasingly difficult to justify starting a new project using Java.)
Playing a file of Midi data is easy:
try {
Sequence sequence = MidiSystem.getSequence(midiFile);
Sequencer sequencer = MidiSystem.getSequencer();
sequencer.open();
sequencer.setSequence(sequence);
sequencer.start();
} catch (Exception e) {...}
Copying sound from the microphone to the speaker is a bit harder. You have to set up TargetDataLine to read bytes from the microphone, set up a SourceDataLine to send bytes to the speaker and then copy bytes from the target to the source (yes, that's the correct way though the nomenclature is strange, copying from the target of the input mixer to the source of the output mixer) [based on code by Matthias Pfisterer]:
private static AudioFormat getAudioFormat(){
float sampleRate = 44100.0F;
//8000,11025,16000,22050,44100
int sampleSizeInBits = 16;
//8,16
int channels = 1;
//1,2
boolean signed = true;
//true,false
boolean bigEndian = false;
//true,false
return new AudioFormat(sampleRate,
sampleSizeInBits,
channels,
signed,
bigEndian);
}//end getAudioFormat
public void playAudio() throws Exception {
AudioFormat audioFormat;
TargetDataLine targetDataLine;
audioFormat = getAudioFormat();
DataLine.Info dataLineInfo =
new DataLine.Info(
TargetDataLine.class,
audioFormat);
targetDataLine = (TargetDataLine)
AudioSystem.getLine(dataLineInfo);
targetDataLine.open(audioFormat,
audioFormat.getFrameSize() * FRAMES_PER_BUFFER);
targetDataLine.start();
playAudioStream(new AudioInputStream(targetDataLine));
} // playAudioFile
/** Plays audio from the given audio input stream. */
public void playAudioStream( AudioInputStream audioInputStream ) {
// Audio format provides information like sample rate, size, channels.
AudioFormat audioFormat = audioInputStream.getFormat();
// Open a data line to play our type of sampled audio.
// Use SourceDataLine for play and TargetDataLine for record.
DataLine.Info info = new DataLine.Info( SourceDataLine.class,
audioFormat );
if ( !AudioSystem.isLineSupported( info ) ) {
System.out.println( "Play.playAudioStream does not handle this type of audio on this system." );
return;
}
try {
SourceDataLine dataLine = (SourceDataLine) AudioSystem.getLine( info );
dataLine.open( audioFormat,
audioFormat.getFrameSize() * FRAMES_PER_BUFFER);
// Allows the line to move data in and out to a port.
dataLine.start();
// Create a buffer for moving data from the audio stream to the line.
int bufferSize = (int) audioFormat.getSampleRate() *
audioFormat.getFrameSize();
bufferSize = audioFormat.getFrameSize() * FRAMES_PER_BUFFER;
// See http://docs.oracle.com/javase/6/docs/technotes/guides/sound/programmer_guide/chapter5.html
// for recommendation about buffer size
byte [] buffer = new byte[bufferSize / 5];
// Move the data until done or there is an error.
try {
int bytesRead = 0;
while ( bytesRead >= 0 ) {
bytesRead = audioInputStream.read( buffer, 0, buffer.length );
if ( bytesRead >= 0 ) {
int framesWritten = dataLine.write( buffer, 0, bytesRead );
}
} // while
} catch ( IOException e ) {
e.printStackTrace();
}
dataLine.drain();
dataLine.close();
} catch ( LineUnavailableException e ) {
e.printStackTrace();
}
} // playAudioStream
Now both of those work okay, picking up default devices, mixers, data lines, etc. You have to be careful running the copy code from microphone to speaker - you can set up a howling feedback loop between your laptop's microphone and speaker if you don't use, say, headphones.
There is a detectable latency (delay between the sounds) between talking/singing into the microphone and getting sound out of the speaker, but it is acceptable. But when you put the two pieces of code in the same program - even in different threads - then the latency blows out and the result isn't acceptable after all. There is a distinct delay between the input and the output sounds. Processing the Midi data somehow interferes with processing the sampled data and introduces additional delays which make it unusable.
I looked around on the Web, and read all the Sun/Oracle documentation that I could find, but couldn't find anything talking about this problem in the context of the Java Sound API. I've now found a solution (even if it isn't totally portable) so that is why I'm writing this blog.
The above code leaves almost everything to defaults. So the Midi code must be re-setting some default used by the sampled data code. The most likely candidate is the output Mixer, but you can't get from the SourceDataLine to its Mixer, and the Midi API nowhere gives you access to things like Mixers. Digging around in the OpenJDK source code showed lots of interesting things such as the Midi code setting its Midi-processing thread loop to a very high priority but I ran out of steam before finding the link between the two processing streams. The com.sun.media.sound package has a bunch of software mixers - the answer is probably in there somewhere.
So I looked at the mixers available. The following function shows how:
public void listMixers() {
try{
Mixer.Info[] mixerInfo =
AudioSystem.getMixerInfo();
System.out.println("Available mixers:");
for(int cnt = 0; cnt < mixerInfo.length; cnt++){
System.out.println(mixerInfo[cnt].getName());
}//end for loop
} catch(Exception e) {
}
}
On my laptop running Fedora 16 this lists
Available mixers:
PulseAudio Mixer
default [default]
PCH [plughw:0,0]
NVidia [plughw:1,3]
NVidia [plughw:1,7]
NVidia [plughw:1,8]
Port PCH [hw:0]
Port NVidia [hw:1]
There's a default mixer which I don't want, several hardware mixers and a PulseAudio one. PulseAudio is the audio system on most current Linux systems so I get the best (Linux) portability by choosing that one, while avoiding whatever default Java Sound gives me.
Do these mixers support source lines and target lines? This shows the full list
System.out.println("Available mixers:");
for(int cnt = 0; cnt < mixerInfo.length;
cnt++){
System.out.println(mixerInfo[cnt].
getName());
Mixer mixer = AudioSystem.getMixer(mixerInfo[cnt]);
Line.Info[] sourceLines = mixer.getSourceLineInfo();
for (Line.Info s: sourceLines) {
System.out.println(" Source line: " + s.toString());
}
Line.Info[] targetLines = mixer.getTargetLineInfo();
for (Line.Info t: targetLines) {
System.out.println(" Target line: " + t.toString());
}
}//end for loop
This shows results like
PulseAudio Mixer
Source line: interface SourceDataLine supporting 42 audio formats, and buffers of 0 to 1000000 bytes
Source line: interface Clip supporting 42 audio formats, and buffers of 0 to 1000000 bytes
Target line: interface TargetDataLine supporting 42 audio formats, and buffers of 0 to 1000000 bytes
(Note that you have to ask for mixer.getSourceLineInfo() - asking for mixer.getSourceLines() only shows the open lines and there will be none of those till you open them!)
I leave the Midi code alone. I don't need to mess with it. The sampled data I handle this way:
Mixer.Info[] mixerInfo = AudioSystem.getMixerInfo();
Mixer mixer = null;
for(int cnt = 0; cnt < mixerInfo.length; cnt++){
if (mixerInfo[cnt].getName().equals("PulseAudio Mixer")) {
mixer = AudioSystem.getMixer(mixerInfo[cnt]);
break;
}
}//end for loop
if (mixer == null) {
System.out.println("can't find a PulseAudio mixer");
} else {
Line.Info[] lines = mixer.getSourceLineInfo();
if (lines.length >= 1) {
try {
dataLine = (SourceDataLine) AudioSystem.getLine(lines[0]);
System.out.println("Got a Pulse Audio source line");
} catch(Exception e) {
}
} else {
System.out.println("no source lines for this mixer " +
mixer.toString());
}
}
And that's it! I can now write to this SourceDataLine and my sampled data is going straight to the Linux sound mixer, bypassing whatever the Java Sound Midi system is doing. Latency problem solved.
Now on to the next steps...
No comments:
Post a Comment