6. Evaluation

For the execution performance measurement, we mainly used Solaris for J2SE environment; the Jabber server, which is C program, is built and runs on Solaris 5.7 under Sun Ultra-4 (4 CPUs, 1GB physical memory and 2GB swap space), and JIM-J2SE and infopipes including the Image Converter Infopipe were executed on Solaris 5.7 under Ultra-30 (1 CPU). For comparison, Windows NT under Pentium II 333MHz (256MB RAM) was also used. The Palm device is Palm IIIxe (Palm OS 3.5 8MB RAM), on which the KVM/Palm runs with 211KB heap for JIM-J2ME KVM/Palm.

Figure 10 illustrates the basic configuration of the IM system. Each IM client connects to the IM server. One may connect to it through an Infopipe in order to receive filtered information.

Figure 10: Basic Configuration of JIM for Performance Measurement

6.1 Messaging Round-trip

First of all, we measured the execution performance of messaging round-trip (Figure 11). A JIM client sends a message to another side; when it receives the message, it reply the same message to the sender immediately. In order to measure this, some lines of System.currentTimeMillis() were added to the sender side and a couple of lines to reply automatically were added to the receiver side. GUI at the receiver side might affect to the measured value.

Figure 11: Messaging Round-trip

The following table (Table 1) is the result of this evaluation:

Table 1: Messaging Round-trip

This result shows that the massaging with Palm is about 100 times slower than messaging without Palm; while, messaging only by Solaris and one with Win NT are relatively the same. (compared to the big difference from the messaging with Palm.)

6.2 Messaging Round-trip over Null-Infopipe

In order to measure the overhead of Infopipe, we built the Null-Infopipe whose middle method is empty; thereby, the overhead of IRT/JIM for the minimum case can be evaluated (Figure 12).

Figure 12: Messaging Round-trip over Null-Infopipe

Table 2: Messaging Round-trip over Null-Infopipe

The result shows that the overhead is about 16 msec. Please note that the overhead may be affected by the contents of messaging, even though the result of this experiment shows that the overhead for messaging "Hello!" (15.9 msec) is a little smaller than the other overhead for the null messaging (16.2 msec).

6.3 Image Attachment without Infopipe

Next experimental measurement shows massaging round-trip with an image. In this evaluation, 14,046 byte 320x240 JPEG image is attached by the sender, the null message (or the "Hello!" message) is sent with the image, and then the receiver gives only the message without the image back to the sender (Figure 13). Since the receiver client displays the image after it replies, though, that might affect to the evaluation result.

Figure 13: Image Attachment without Infopipe

Table 3: Image Attachment without Infopipe

The result shows that it took about 600 msec, which is much slower than sending just a short message. Also, sending an image with a short message, such as "Hello!", affected the performance. The big difference between messaging the null message with the image (603.8 msec) and the other messaging "Hello!" with the image (670.8 msec) is not analyzed yet.

6.4 Image Attachment over the Image Converter Infopipe

This evaluation shows the execution time of messaging round-trip with the image, which is the same one used in the previous experiment, over the Image Converter Infopipe (Figure 14). This may show the execution performance of the Infopipe Middle method, which contains to reduce both the size and number of colors and to save the converted bitmap image at the file system for the web server.

Figure 14: Image Attachment over the Image Converter Infopipe

Table 4: Image Attachment over the Image Converter Infopipe

The result shows the execution time of the Middle method of the Image Converter Infopipe is about 620 msec.

The execution time of the messaging with the image over the Infopipe between Solaris and Palm is pretty slow; since only the URL is sent from the Infopipe to Palm rather than the image is delivered, the execution time was supposed to be about 3,500 msec (= 2,276 (Table 1) + 600 (the image handling without the Infopipe) + 620 (the overhead of the Middle method)). The reason of the extra overhead is not analyzed yet.

6.5 Retrieving an Image for Palm over HTTP

In order for the Palm client to get an image, the client needs to access the web server where the converted bitmap image is stored (Figure 15).

Figure 15: Retrieving an Image for Palm over HTTP

It takes 4,290 msec in this evaluation.

Since the HTTP protocol has some overhead for client to establish the connection with server, we measured the execution time to receive the image over the native socket; it took 1,480 msec. It means potentially the user interface that is affected by such a performance can be improved.

6.6 Parsing XML Document by AElfred

Since we realized that the messaging with the Palm client was slow, the execution performance of the XML parser processing by AElfred, which is one of main parts of the client code, was evaluated for the varieties of platforms.

For this evaluation, both Solaris / Win NT and Palm use the same code; AElfred that is ported onto J2ME KVM/Palm was used also on Solaris and Win NT. To evaluate in the same (or similar as possible) condition, the XML documents were stored on memory before to be parsed.

In the Table 5, the time that is in () is the initial evaluation and the other is the average of 10 times evaluation.

Table 5: Parsing XML Document by AElfred

The result shows that cache seems to work efficiently in Solaris and Win NT while the Palm does not. In any size of target XML document, the Palm processing is about 100 times slower than the same processing in Solaris and Win NT (in the case of hot-start).

Please note that this result shows just a possible reason of the slowness of the Jabber messaging with the Palm client (JIM-J2ME KVM/Palm) that runs on Java under the Palm platform; the native application may have much better performance. In addition, the main bottleneck is not retrieved yet. Further investigation needs to be performed.

6.7 Summary of the Execution Performance Measurement

JIM-J2ME KVM/Palm is relatively slow and about 100 times slower than JIM-J2SE on Solaris and Win NT. From the viewpoint of the amount of the client code, XML processing by AElfred is the main parts of the procedures; the execution performance measurement of the XML processing shows the 100 times slowness. Further investigation may be required.

After the Image Converter Infopipe does store the converted image on the web server for the Palm or other clients to retrieve the image over HTTP, the JIM-J2ME KVM/Palm user's feel of messaging with images has become better drastically. This is because the initial implementation of the Image Converter Infopipe sends the base64 encoded image in XML to the client, and the Palm had to decode it; however, it took more than 30 msec to read the image from socket and to decode it. It means that it was obviously good idea to move the base64 decoding function away from the Palm that has limited processing power to the Middle of the information flow. At that time, ISG enables the author to concentrate on implementing the Middle method.