Found low latency for Karaoke with PulseAudio

As a followup to the last post: I can now play Midi files and sing along with no noticeable latency using PulseAudio. To direct the (default) microphone to the (default) speaker load the module_loopback (thanks to rusty0101):

    pactl load-module module-loopback latency_msec=1

Everything you speak/sing/holler will then be played on the speaker.

For Midi file playback I discovered FluidSynth. The following program will play a Midi file:

// See http://fluidsynth.sourceforge.net/api/

// Run by ./PlayFile /usr/share/soundfonts/FluidR3_GM.sf2 ../54150.mid

int main(int argc, char** argv)
{
    int i;
    fluid_settings_t* settings;
    fluid_synth_t* synth;
    fluid_player_t* player;
    fluid_audio_driver_t* adriver;
    settings = new_fluid_settings();
    synth = new_fluid_synth(settings);
    player = new_fluid_player(synth);

    fluid_settings_setstr(settings, "audio.driver", "pulseaudio");
    // Use paman to find output device's name
    fluid_settings_setstr(settings, "audio.pulseaudio.device",
              "alsa_output.pci-0000_00_1b.0.analog-stereo");

    adriver = new_fluid_audio_driver(settings, synth);
    /* process command line arguments */
    for (i = 1; i < argc; i++) {
        if (fluid_is_soundfont(argv[i])) {
           fluid_synth_sfload(synth, argv[1], 1);
        }
        if (fluid_is_midifile(argv[i])) {
            fluid_player_add(player, argv[i]);
        }
    }
    /* play the midi files, if any */
    fluid_player_play(player);
    /* wait for playback termination */
    fluid_player_join(player);
    /* cleanup */
    delete_fluid_audio_driver(adriver);
    delete_fluid_player(player);
    delete_fluid_synth(synth);
    delete_fluid_settings(settings);
    return 0;
}


And whadda-you-know? It all works fine.

There's just the matter of hooking up a GUI, and a few thousand lines of code. But at least I'm starting from a good base, and I now realise the Java Sound framework can't give me that, sad to say.

In search of (low) latency

This is a followup to my investigations into playing my Songken DVD DKD files on my laptop. In an earlier blog I described how to decode the DKD files into Midi or Midi+WMA files. The intent was then to build a Midi player that would also show the notes of the melody and also the notes the singer was singing.

Well, I did all that. Java Sound has a Midi player. Java Sound has a Sampled API to handle sounds from the microphone to the loudspeaker. Java has a GUI for showing stuff. TarsosDSP by Joren Six has implemented a number of pitch detection algorithms such as YIN and they can be pulled in to give an estimate of the pitch sung. Java can convert characters from language encodings such as GB2312 to Unicode and display them so I can see Chinese and other characters.  So it's all there....

... but latency still kills it. The Midi player introduces latency somehow into the sampled sounds, but even if you work around it - even if you just do sampled data alone - then there is still that little delay. Here are my Java source files. Maybe I will write up an explanation of what I was doing with them later. I'm going to stop work on them right now till I get the latency sorted out.

The standard audio system for (consumer) sound on Linux is Pulse Audio. But as Lennart Poettering explained at the Linux Audio Conference 2010, pro audio has different aims to consumer audio, and this project is closer to pro audio than consumer audio (although to think of Karaoke singers as pros is stretching it a bit :-). In consumer audio, latencies of upto 2 seconds may be permissible, while pro audio sets an upper limit of 20 milli-seconds.

Java Sound is estimated to have a 50msec delay: "These measurements suggest that the latency introduced by buffers in the "Java Sound Audio Engine" is about 50 ms, independant of the sample rate." Now that's on old equipment, but it means there is an uphill struggle.

The sound quality of the builtin soundcard HDA Intel PCH (STAC92xx) on my Dell laptop is appalling. That has to be overcome too. This laptop doesn't have a microphone input, so I started looking at USB sound cards. My first attempt was with a AnPu Portable USB 3D Virtual 5.1 Audio Sound Card Adapter Blue  from Dino Direct. Dino was good: delivery post-free within 2 weeks. But the card was cheap (A$4) and broke when I inadvertently yanked it out of the USB slot.


My second attempt was with Swamp Industries for an XLR to USB Adapter. That was about A$20 but I got it with a microphone as well. The service was good again. Well, the card's okay for input, but still has to go out through the onboard soundcard.


The third attempt was with a Sound Blaster X-Fi Surround 5.1 Pro at A$70. It's a USB 1.1 device (Linux still has issues with USB 2 devices, apparently).  Pulse Audio only recognises it as an input device, not as an output device, so it didn't seem to improve things.


Pulse Audio is an audio layer above Alsa (OSS was used previously to Alsa). Alsa could see the device fine:

$arecord -l
**** List of CAPTURE Hardware Devices ****
card 0: PCH [HDA Intel PCH], device 0: STAC92xx Analog [STAC92xx Analog]
  Subdevices: 0/1
  Subdevice #0: subdevice #0
card 2: Pro [SB X-Fi Surround 5.1 Pro], device 0: USB Audio [USB Audio]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
 
and

$aplay -l
...
card 2: Pro [SB X-Fi Surround 5.1 Pro], device 0: USB Audio [USB Audio]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 2: Pro [SB X-Fi Surround 5.1 Pro], device 1: USB Audio [USB Audio #1]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
 
Now about this time I went off on what turned out to be a wild goose chase (at least so far) by looking at Jack: "JACK is [a] system for handling real-time, low latency audio (and MIDI)". Jack currently also uses Alsa. Now that looks good - but Java Sound and Jack don't play together.

Java Sound has a couple of weird bits where "obviously equivalent" things aren't. I hit this first with volume control in playing a Midi file: you can't set the volume on the default device but you can if you iterate through the devices and select the default one. Then you can set the volume on it. Huh? Thanks to Greg Donahue for solving that one. You hit similar problems trying to find the sound cards and you end up either with

• Java Sound not playing to your default card; or
• When you explicitly select the default card then Java Sound throws an exception saying that its PulseAudio drivers can't find it.

So after all that, where are we?

• Java Sound has latency problems
• The inbuilt soundcard is crap
• Pulse Audio can't properly find the USB soundcard
• Java Sound uses Pulse Audio
• Pulse Audio has latency issues
• Jack is ignored by Java Sound
• Alsa and Jack can find the USB soundcards

Is it possible to have latency-free sound on Linux? Well, Jack claims to be latency-free, but then it has to go through the Alsa layer. Can the Alsa layer be latency-free? Not completely, but I finally figured out the following test:

    arecord  -f dat -B 4  -D hw:0| aplay -B 4 -D hw:2 -f dat -

i.e record at DAT standard (16 bits, 48k samples) from the builtin mike (hw:0) played on the USB soundcard (hw:2), with 4msec buffer time. And hey! It works! No latency that my poor ear can hear. This simple pipeline isn't perfect: any overrun introduces latency into the pipeline, but that can be handled in code by dropping samples. The sample size can be increased and it still sounds okay - 4ms was the lowest I could take it.

Conclusion:

• the top-down approach through Java works but has latency issues
• the bottom-up approach through Alsa handles latency

I just need to combine the two...

java sound: midi and sampled streams played together

I've been playing around with my home Karaoke system. This has included decoding my Songken DKD disk. Once I did that, I wanted to emulate the behaviour of my Malata Karaoke player: playing the Midi files, showing the lyrics and also showing in a bar graph the notes that should be sung and the notes the performer is actually singing.

This requires processing files of Midi data, handling the soundcard microphone input and speaker output and using a GUI to show everything. Java Sound looks like a perfect choice for this as it can do all these things. (Although Oracle's custodianship of Java and their outrageous API copyright claims makes it increasingly difficult to justify starting a new project using Java.)

Playing a file of Midi data is easy:

    try {
        Sequence sequence = MidiSystem.getSequence(midiFile);
        Sequencer sequencer = MidiSystem.getSequencer();
        sequencer.open();
        sequencer.setSequence(sequence);
        sequencer.start();
    } catch (Exception e) {...}

Copying sound from the microphone to the speaker is a bit harder. You have to set up TargetDataLine to read bytes from the microphone, set up a SourceDataLine to send bytes to the speaker and then copy bytes from the target to the source (yes, that's the correct way though the nomenclature is strange, copying from the target of the input mixer to the source of the output mixer) [based on code by  Matthias Pfisterer]:

    private static AudioFormat getAudioFormat(){
        float sampleRate = 44100.0F;
        //8000,11025,16000,22050,44100
        int sampleSizeInBits = 16;
        //8,16
        int channels = 1;
        //1,2
        boolean signed = true;
        //true,false
        boolean bigEndian = false;
        //true,false
        return new AudioFormat(sampleRate,
                   sampleSizeInBits,
                   channels,
                   signed,
                   bigEndian);
    }//end getAudioFormat

    public  void playAudio() throws Exception {
        AudioFormat audioFormat;
        TargetDataLine targetDataLine;
   
        audioFormat = getAudioFormat();
        DataLine.Info dataLineInfo =
            new DataLine.Info(
                  TargetDataLine.class,
                  audioFormat);
        targetDataLine = (TargetDataLine)
            AudioSystem.getLine(dataLineInfo);
   
        targetDataLine.open(audioFormat,
                audioFormat.getFrameSize() * FRAMES_PER_BUFFER);
        targetDataLine.start();
   
        playAudioStream(new AudioInputStream(targetDataLine));
    } // playAudioFile
    
    /** Plays audio from the given audio input stream. */
    public  void playAudioStream( AudioInputStream audioInputStream ) {
        // Audio format provides information like sample rate, size, channels.
        AudioFormat audioFormat = audioInputStream.getFormat();
    
        // Open a data line to play our type of sampled audio.
        // Use SourceDataLine for play and TargetDataLine for record.
        DataLine.Info info = new DataLine.Info( SourceDataLine.class,
             audioFormat );
        if ( !AudioSystem.isLineSupported( info ) ) {
            System.out.println( "Play.playAudioStream does not handle this type of audio on this system." );
            return;
        }
    
        try {
             SourceDataLine dataLine = (SourceDataLine) AudioSystem.getLine( info );

            dataLine.open( audioFormat,
               audioFormat.getFrameSize() * FRAMES_PER_BUFFER);
        
            // Allows the line to move data in and out to a port.
            dataLine.start();
    
            // Create a buffer for moving data from the audio stream to the line.
            int bufferSize = (int) audioFormat.getSampleRate() *
            audioFormat.getFrameSize();
            bufferSize =  audioFormat.getFrameSize() * FRAMES_PER_BUFFER;
            // See http://docs.oracle.com/javase/6/docs/technotes/guides/sound/programmer_guide/chapter5.html
            // for recommendation about buffer size
            byte [] buffer = new byte[bufferSize / 5];
    
            // Move the data until done or there is an error.
            try {
                int bytesRead = 0;
                while ( bytesRead >= 0 ) {
                    bytesRead = audioInputStream.read( buffer, 0, buffer.length );
                    if ( bytesRead >= 0 ) {
                        int framesWritten = dataLine.write( buffer, 0, bytesRead );
                    }
                } // while
            } catch ( IOException e ) {
                e.printStackTrace();
            }
            dataLine.drain();
    

            dataLine.close();
        } catch ( LineUnavailableException e ) {
            e.printStackTrace();
        }
    } // playAudioStream

Now both of those work okay, picking up default devices, mixers, data lines, etc. You have to be careful running the copy code from microphone to speaker - you can set up a howling feedback loop between your laptop's microphone and speaker if you don't use, say, headphones.

There is a detectable latency (delay between the sounds) between talking/singing into the microphone and getting sound out of the speaker, but it is acceptable. But when you put the two pieces of code in the same program - even in different threads - then the latency blows out and the result isn't acceptable after all. There is a distinct delay between the input and the output sounds. Processing the Midi data somehow interferes with processing the sampled data and introduces additional delays which make it unusable.

I looked around on the Web, and read all the Sun/Oracle documentation that I could find, but couldn't find anything talking about this problem in the context of the Java Sound API. I've now found a solution (even if it isn't totally portable) so that is why I'm writing this blog.

The above code leaves almost everything to defaults. So the Midi code must be re-setting some default used by the sampled data code. The most likely candidate is the output Mixer, but you can't get from the SourceDataLine to its Mixer, and the Midi API nowhere gives you access to things like Mixers. Digging around in the OpenJDK source code showed lots of interesting things such as the Midi code setting its Midi-processing thread loop to a very high priority but I ran out of steam before finding the link between the two processing streams. The com.sun.media.sound package has a bunch of software mixers - the answer is probably in there somewhere.

So I looked at the mixers available. The following function shows how:

  public void listMixers() {
    try{
        Mixer.Info[] mixerInfo =
            AudioSystem.getMixerInfo();
        System.out.println("Available mixers:");
        for(int cnt = 0; cnt < mixerInfo.length; cnt++){
            System.out.println(mixerInfo[cnt].getName());  
        }//end for loop
     } catch(Exception e) {
     }
  }
On my laptop running Fedora 16 this lists

Available mixers:
PulseAudio Mixer
default [default]
PCH [plughw:0,0]
NVidia [plughw:1,3]
NVidia [plughw:1,7]
NVidia [plughw:1,8]
Port PCH [hw:0]
Port NVidia [hw:1]

There's a default mixer which I don't want, several hardware mixers and a PulseAudio one. PulseAudio is the audio system on most current Linux systems so I get the best (Linux) portability by choosing that one, while avoiding whatever default Java Sound gives me.

Do these mixers support source lines and target lines? This shows the full list

            System.out.println("Available mixers:");
            for(int cnt = 0; cnt < mixerInfo.length;
                cnt++){
                System.out.println(mixerInfo[cnt].
                                   getName());
               
                Mixer mixer = AudioSystem.getMixer(mixerInfo[cnt]);
                Line.Info[] sourceLines = mixer.getSourceLineInfo();
                for (Line.Info s: sourceLines) {
                    System.out.println("  Source line: " + s.toString());
                }
                Line.Info[] targetLines = mixer.getTargetLineInfo();
                for (Line.Info t: targetLines) {
                    System.out.println("  Target line: " + t.toString());
                } 
            }//end for loop

This shows results like
  PulseAudio Mixer
    Source line: interface SourceDataLine supporting 42 audio formats, and buffers of 0 to 1000000 bytes
    Source line: interface Clip supporting 42 audio formats, and buffers of 0 to 1000000 bytes
    Target line: interface TargetDataLine supporting 42 audio formats, and buffers of 0 to 1000000 bytes

(Note that you have to ask for mixer.getSourceLineInfo() - asking for mixer.getSourceLines() only shows the open lines and there will be none of those till you open them!)

I leave the Midi code alone. I don't need to mess with it. The sampled data I handle this way:

            Mixer.Info[] mixerInfo = AudioSystem.getMixerInfo();
            Mixer mixer = null;
            for(int cnt = 0; cnt < mixerInfo.length; cnt++){
                if (mixerInfo[cnt].getName().equals("PulseAudio Mixer")) {
                    mixer = AudioSystem.getMixer(mixerInfo[cnt]);
                    break;
                }
            }//end for loop
            if (mixer == null) {
                System.out.println("can't find a PulseAudio mixer");
            } else {
                Line.Info[] lines = mixer.getSourceLineInfo();
                if (lines.length >= 1) {
                    try {
                        dataLine = (SourceDataLine) AudioSystem.getLine(lines[0]);
                        System.out.println("Got a Pulse Audio source line");
                    } catch(Exception e) {
                    }
                } else {
                    System.out.println("no source lines for this mixer " +
                                                     mixer.toString());
                }
            }

And that's it! I can now write to this SourceDataLine and my sampled data is going straight to the Linux sound mixer, bypassing whatever the Java Sound Midi system is doing. Latency problem solved.

Now on to the next steps...

Streaming audio and Android

n my home audio setup, I have all the music files in OGG format on my web server with an M3U playlist for each CD. This works fine on all my PC browsers. But almost none of it works for Android 2.2.Here is the stuff I tried and eventually got working for a Kogan Android TV Portal.

REST, HTML, Ajax and PHP

What is REST?

REST is an architectural style for the Web. The term was coined by Roy Fielding in his PhD thesis. Roy was one of the architects of HTTP/1.1 so he knows what he was talking about. His thesis was rather abstract, so his ideas were ignored by the many who went chasing after SOAP, WSDL and UDDI in the hope that salvation lay that way. It didn't of course, and I have written about that in A Critique of Web Services.

REST as applied to the Web is based around the HTTP verbs of GET, PUT, DELETE and POST. Requests from a user-agent (such as a browser or web client application) signal what type of action they expect from a Web server by the choice of verb in the HTTP request. In a way, this is reminiscent of O/O programming where you get (GET) or set (PUT) a field of an object, delete (DELETE) an object or perform some arbitrary action (POST). I have explored this in An Overview of REST.

REST has become more popular as an approach to designing web applications, and has influenced many well-known systems such as Twitter and Google maps. Ruby on Rails is said to have adopted REST as a backend architecture. But the problem with an architecture is that it doesn't tell you how to do things, just what they should look like.

So in this blog, I'm going to look at building a simple web application from the front- to the back- end. It's a simple one: a database of people where each person has a name and ID, and you can query the database or add or delete from it.

From the REST viewpoint, a client will make these queries of the backend. The meanings of these are essentially given  by the HTTP specification:

HTTP method	URI	result
GET	/people	GETs a collection of people
POST	/people	POST: does something to the collection (using data in the POST request)
GET	/people/jan	GETs jan
PUT	/people/jan	PUTs a new version or an updated version of jan (using data in the PUT request)
DELETE	/people/jan	DELETEs jan
POST	/people/jan	POST: does something to "jan" (using data in the POST request)

The client will make these requests to an HTTP server. It will perform appropriate actions and return a result. The HTTP specification also specifies the result types:

HTTP method	Status
GET	200 - found resource
	404 - not found
POST	200 - resource returned
	204 - no resource to return
	201 - new resource created
PUT	201 - resource created
	200 or 204- resource modified
	4XX, etc - error occurred
DELETE	200 - deleted and resource returned
	204 - deleted but no resource returned
	202 - will be deleted
	4XX, etc - won't be deleted

Python client
 
First I will give Python code to send these requests and interpret the responses. It is just a simple command-line interface, but you could wrap it in a GUI such as PythonCard.

#!/usr/bin/python

import httplib

baseURL = "/boxhill/ict329/webservices/people/"

def connect(method, url, data):
    connection = httplib.HTTPConnection("localhost")
    connection.request(method, url, data)
    response = connection.getresponse()
    status = response.status
    body = response.read()
    return (status, body)


def get(name):
    (status, body) = connect("GET", 
                              baseURL + name, 
                             "")
    if (status == 404):
        print name + ": no such person"
    else:
        print body

def put(name, ID):
    (status, body) = connect("PUT",
                             baseURL + name,
                             ID)
    if (status == 201):
        print name + ": created"
    else:
        print name + ": modified"

def delete(name):
    (status, body) = connect("DELETE",
                             baseURL + name,
                             "")
    if (status == 404):
        print name + ": couldn't delete"
    else:
        print name + ": deleted"
        print body

def getAll():
    (status, body) = connect("GET",
            baseURL,
            "")
    if (status == 404):
        print "No people"
    else:
        print body
 

You can run this with requests such as

get("Peter")
put("Fred", 21)
delete("Paul")

The backend will act on these requests and return suitable responses.

HTML and REST

The Web isn't just HTTP of course. HTTP is there to carry content, and most of the content is HTML (I'm talking value of content here, not size of content as in PowerPoint slides :-). How does HTML co-operate with REST?

HTML allows you to create documents with links, using tags such as <a href="...">. In the case of http: links, all browsers use an HTTP GET, although I can't find this specified anywhere.

HTML also allows you to create forms where the content can be submitted to a server. The type of submission is controlled by the "action" attribute of the form tag, and can be either POST or GET. This is true of HTML 4 and also currently true of the HTML 5 draft. I do have a copy of an 2009 draft which did allow PUT and DELETE, but now in 2011 those options have disappeared.

So HTML offers no support for PUT or DELETE and so cannot be considered supportive of REST. Using POST to mean GET, PUT, DELETE as well is not support as I would consider it.

JavaScript and REST

With JavaScript you can load() a document. This uses HTTP GET. You can submit a form by form.submit(). This uses the form's action attribute, which is either POST or GET.

"Standard" JavaScript does not have support for REST.

Ajax and REST

You have to turn to Ajax to get proper support for REST. That means you have to make Ajax calls using XMLHttpRequest if you want a browser to make the correct REST calls, whether you like it or not.

 I'll use JQuery as that is a little easier and takes care of browser differences compared to straight JavaScript. There is an object $.ajax which takes a dictionary of attribute: action pairs. This can be used to specify the calling method (GET/PUT/...) as well as the URL, and the action to take when a result is returned. For our purposes here, the HTTP return codes are 2XX Okay codes or 4XX error codes. These can be handled by $.ajax success: and error: elements respectively.

A function to GET a value and display the results or an error in an Alert box is

      function doGet() {
        name =  document.getElementById("get_name").value;
        $.ajax({
           type: "GET",
           url: "people/" + name,
           success: function(data, status, jqXHR) {
                      alert(data);
                },
           error: function(jqXHR, textStatus, errorThrown) {
                      alert(name + ": no such person");
                },
           async: false,
        });
      };

which extracts the name from a textbox with id "get_name" in a form and makes a synchronous GET request, and shows the appropriate alert box when the call returns.

A similar function can be used for the DELETE call:

      function doDelete() {
        name =  document.getElementById("delete_name").value;
        $.ajax({
           type: "DELETE",
           url: "people/" + name,
           error: function() {
                         alert(name + ": couldn't delete");
                  },
           success: function() {
                         alert(name + ": deleted");
                  },                     
           async: false,
        });
      };

For the PUT method, we want to distinguish between a response of 201 (the resource was created) and 204 (the resource was modified). Both of these are success values, but done in a different way. From JQuery 1.5 onwards, we can examine the status code of the HTTP response:

      function doPut() {
        name =  document.getElementById("put_name").value;
        ID =  document.getElementById("put_id").value;
        $.ajax({
           type: "PUT",
           url: "people/" + name,
           data: ID,
           statusCode: {
                201: function(jqXHR, textStatus, errorThrown) {
                         alert(name + ": created ");
                     },
                204: function(jqXHR, textStatus, errorThrown)  {
                         alert(name + ": modified");
                     }
           },
           async: false,
        });
      };

Finally, we link these calls to forms:
<ul>
  <li>
    <p>
      Get info about one person
    </p>
    <p>
      <form>
        Name <input type="text" id="get_name"/>
        <br/>
        <input type="button" value="Submit this"
                    onClick="doGet()"/>
       </form>
    </p>
  </li>

  <li>
    <p>
      Create a person
    </p>
    <p>
      <form>
        Name <input type="text" id="put_name"/>
        <br/>
        ID <input type="text" id="put_id"/>
        <br/>
        <input type="button" value="Submit this"
                    onClick="doPut()"/>
      </form>
    </p>
  </li>

  <li>
    <p>
      Delete a person
    </p>
    <p>
      <form>
        Name <input type="text" id="delete_name"/>
        <br/>
        <input type="button" value="Submit this"
                    onClick="doDelete()"/>
      </form>
    </p>
  </li>
</ul>

Apache and REST

That's enough of the client side - now for the server! Most of the world uses Apache and I do too (except for Lighttpd on my hacked MyBook World server).

Besides the HTTP verbs, the other major component to REST is resources. We have already been using them as "http:people/jan", "http:people/fred" where the URL labels the resources "Jan" and "Fred". REST assumes that URLs are identifying resources, which may be documents, people, items on a shopping cart, etc.

So why didn't we use URLs such "http:people/get.php?jan" when we want to do a GET? Simple: REST URLs label resources not actions on these resources. get.php is an action on some parameter rather than a label. The value of using resources for URLs is that we can perform different actions on the same resource as in "GET people/jan", "DELETE people/jan" etc, and each time we refer to the same resource. This would not be clear if we had "GET people/get.php?jan" and "PUT people/put.php?name=jan&id=1".

In addition, just for maintenance reasons, we would want to separate implementation from data: suppose we decided to move away from PHP and use Ruby on Rails. We wouldn't want to revise all of our implementation-specific URLs - that should be an action behind the scenes in how to handle our resource URLs.

That said, how do we go about mapping resource URLs into actions on those resources? There are several methods. I've chosen Apache's Rewrite rules. These are specified in Apache configuration files, such as /etc/apache2/sites-enabled/000-default on an Ubuntu system. A rule consists of two components: a condition to be satisfied and then a rewrite of the URL. The rewrite rule uses regular expressions as in Perl, where "^(.*)$" means any characters between a begin and end of a string.

I have rules that apply only to the directory on my server box where I keep the "people" files. The Apache directive is

        <Directory /home/httpd/html/boxhill/ict329/webservices/people/>
                RewriteEngine on

                RewriteCond %{REQUEST_METHOD} =GET
                RewriteRule ^(.*)$ get_request.php?$1 [QSA]

                RewriteCond %{REQUEST_METHOD} =PUT
                RewriteRule ^(.*)$ put_request.php?$1 [QSA]

                RewriteCond %{REQUEST_METHOD} =DELETE
                RewriteRule ^(.*)$ delete_request.php?$1 [QSA]
        </Directory>

which calls get/put/delete_request.php with the person's name extracted from the URL and appended as a parameter to the PHP call. After you have made changes like this to the Apache configuration files don't forget to restart or refresh Apache (kill -HUP `cat /var/run/apache2.pid`).

My database

For this blog, I use a MySQL table with two fields, name (text) and id (integer).

PHP

The PHP code is fairly straightforward. It just extracts the parameter information from the $_REQUEST, accesses the database and returns the results using the appropriate HTTP return codes.

The get.php program is

<?php
$user="*****";
$password="*****";
$database="*****";
$localhost="localhost";
mysql_connect($localhost,$user,$password);
@mysql_select_db($database) or die( "Unable to select database");

$req=$_REQUEST;
$keys=array_keys($req);
$name=mysql_real_escape_string($keys[1]);
if ($name=="") {
   $query="SELECT * FROM people";
} else {
  $query="SELECT * FROM people WHERE name=\"$name\"";
}

$result=mysql_query($query);
mysql_close();

$num=mysql_num_rows($result);
if ($num == 0) {
   header("HTTP/1.1 404 Not Found");
   exit();
}

$n=0;
while ($n < $num) {
      $id=mysql_result($result,$n,"id");
      $name=mysql_result($result,$n,"name");

      echo "$name\n";    
      echo "$id\n";

      $n++;
}
?>

The delete_request.php is slightly more complex as we have to determine if the DELETE  has succeeded by counting if the number of rows affected is zero or not.

<?php
$user="*****";
$password="******";
$database="******";
$localhost="localhost";
mysql_connect($localhost,$user,$password);
@mysql_select_db($database) or die( "Unable to select database");

$req=$_REQUEST;
#print_r($req);
$keys=array_keys($req);
$name=mysql_real_escape_string($keys[1]);

$query="DELETE FROM people WHERE name=\"$name\"";

$result=mysql_query($query);
$num=mysql_affected_rows();
mysql_close();

if ($num == 0) {
   header("HTTP/1.1 404 Not Found");
} else {
  header("HTTP/1.1 202 OK");
}
?>

The put_request.php is also complicated by needing to tell if we are creating (INSERT) or modifying a row (exists, DELETE, INSERT). It is

<?php
$user="*****";
$password="******";
$database="*****";
$localhost="localhost";
mysql_connect($localhost,$user,$password);
@mysql_select_db($database) or die( "Unable to select database");

$req=$_REQUEST;
$keys=array_keys($req);
$name=mysql_real_escape_string($keys[1]);

$putdata = fopen("php://input", "r");
$ID = intval(fread($putdata, 1024));
fclose($putdata);

# are we updating?
$query="SELECT * FROM people WHERE name=\"$name\"";    
$result=mysql_query($query);

$num=mysql_num_rows($result);
if ($num == 0) {
   header("HTTP/1.1 201 Created"); # creating
} else {
   header("HTTP/1.1 204 OK"); # updating
   # delete current row first
   $query="DELETE FROM people WHERE name=\"$name\"";

   $result=mysql_query($query);
}

# now add new person
$query="INSERT people VALUES (\"$name\", $ID)";

$result=mysql_query($query);
mysql_close();

?>

Summary

This blog has discussed beginning to end of a web application using REST. Hope it is useful!

If you like this blog, please contribute using Flattr




or donate using PayPal

Ubuntu 11.04

I've got a laptop running Ubuntu 10.10, a netbook running 10.04 and as an alternative boot, the 10.10 netbook remix and also a rather odd system: I'm using an HP thin client as media centre in my lounge connected to a 58 inch Panasonic TV via DVI to HDMI. It streams from a MyBook World NAS that I hacked into to become a general server.

The TV screen is about 3 metres away from where I sit, so I want a "10 foot GUI". Surprisingly perhaps some of the netbook distros do very well at that as while they are designed for small screens they also do well for big screens a long way away. Currently I am using xPud which I have customised to my environment.

But I have to keep rebuilding my version of xPud to keep up, so I am always trying other distros with an easier management cycle. So of course Ubuntu 11.04 looks like a nice candidate.

It's worse than a disappointment, it doesn't display a usable screen at all. Bits of icons smeared across the screen and the left tool bar only showing when you click on the right-hand-side (!) of the screen.

It runs okay on my netbook, but I'm loathe to upgrade my main laptop while there are such problems with other systems. I guess I will wait till 11.10 for Unity to stabilise before doing a general upgrade to Ubuntu 11.

Shell scripts to handle filenames with spaces

Posix/Unix/Linux was not designed to handle filenames with spaces in them. However, Linux and Windows filesystems allow them and also many other "funny" characters. This has been brewing as a topic in Linux Journal recently, and Dave Taylor has just written an article on it in the February, 2011 issue. He spots files with spaces in them by the shell pattern "*\ *" and then mucks around changing spaces into other things. It's good stuff, but overkill for some cases.


For a long time now, I've been writing scripts that handle filenames both with and without spaces. You've got to know your shell and how Posix commands work! Commonly, I want to list files in a directory and do things to them whether or not they have spaces in them.

Shell patterns such as "*" break strings into "words" based on whitespace (spaces, tabs, newlines). This stuffs up a filename if its has spaces in it, since the name then gets split into separate words. But commands such as "ls" (when not directed to a terminal) list each filename on a separate line. So if you have something that distinguishes between spaces/tabs and newlines then you can get complete filenames with or without spaces.

The shell command "read" reads a line and breaks it into words. so

 read a b c

with input
 a line of text
will assign
 a="a"
 b="line"
 c="of text"
Just
  read line
will read all of the line into the variable. It stops reading on end-of-line so it has the distinction type I often need.

But how to use it? Well, the shell while loop is just a simple command, and as such can have its I/O redirected.  So I do this:
 ls |
 while read filename
 do
    #process filename e.g.
    cp "$filename" ~/backups
 done
This works for all files, with or without spaces. Just don't forget the quotes while processing the file! 

Of course, this doesn't work for all uses: note the find and xargs combination that Dave also commented on:
 find . -print0 | xargs -0 ...