Tech Talk

My highlights of YOW! 2011 Melbourne

2011-12-05T06:34:00.035+11:00

I attended the YOW! 2011 Conference in Melbourne on 1-2 December 2011.

The talks

There were many great talks, and some not-so-great. For me, two talks in particular stood out as special. If I only got to listen to these two and nothing else, it would've been worth it.

Product Engineering :- Mike Lee

I first saw Mike Lee's talk on InfoQ several months ago and was really inspired by the passion of that guy, and was so happy to discover he was making an appearance at YOW.

He didn't impart any novel technical knowledge to us, nor did he tell us about the latest agile technique. In fact, everything he said in YOW was common-sense in hindsight, and virtually identical to the talks on InfoQ. And yet, it was pure pleasure to watch this man share his vision of what constituted a good product. I just did not get bored listening to this same talk for the 3rd time! Maybe it was his costume.

Two things that stood out from the talk:

The hook :- the thing that makes a product special
Appsterdam :- New IT hub for freedom loving developers

Temporally Quaquaversal Virtual Nanomachine Programming In Multiple Topologically Connected Quantum-Relativistic Parallel Timespaces...Made Easy! :- Damian Conway

Speaking of the hook, the title of this talk did the job. Ever since I watched Fred Simon's JavaPosse lightning talk on Positronic Variables a few months ago, I was curious about the Damian Conway's original work. As luck would have it, Damian came to tell us all about positronic variables.

Damian's Perl (and Rod logic) code was not very familiar to me, but his tongue-in-cheek introduction to quantum physics and the multi-verse was pure entertainment. Personally, I was not convinced that an Einstein-Rosenberg bridge could be harnessed as described for running computer programs due to excessive energy requirements. A computer program using positronic variables would probably be unstable and generate incoherent results due to the unpredictable superposition of multiple variations of positronic values from an indeterminate number of future multi-verses.

Other notable talks

Jim Webber gave his usual talk on RESTful systems. I'd listened to his similarly themed talks before, but it was still entertaining to watch Jim Webber's performance. The one lesson that stuck in my mind was: Use an ATOM feed to publish events so that different clients could independently maintain / reconstruct their own internal states.

Another interesting talk was Dan Ingalls telling us about Lively Kernel. Dan built a simple online load monitoring service simply using drag-on-drop from the Lively Kernel interface... cool.

The games

As usual, there were lots of fun and games between talks. Sensis developed a online (mobile) quiz game, CodeJuicer, that morphed into to be a fun hacking game instead. Too bad I did not have a laptop with me to hack on it all day, though I did manage to hack it to zero-time from home at the end of Day 1. Thoughtworks put up a quiz game and a movie title matching game, plus frozen nitrogen ice cream... all very popular. Atlassian let us shoot Nerf guns... hard to beat that.

The food

First day's food was not too good, but got better on the second day. The Beef Stroganoff was delicious.

The best parts were the free (real) coffee paid for by Aconex, and the desserts, especially the mango mousse.

Looking forward to YOW! 2012.

Personal recollections of the YOW Melbourne Conference

2010-12-06T22:53:00.062+11:00

I attended the YOW!2010 Australia Software Developer Conference in Melbourne last week, and here is some recollection of my favourite talks. Please note this is all from memory, so I might some parts wrong.

Main themes

From the sessions I attended, some of the main recurring technical themes in this conference were:

Consistency should be traded for Availability in typical distributed systems
Eventual consistency over ACID
Always expect network failures, i.e. systems must be able to tolerate partitioning
NoSQL

Most informative talks

The Rich Get Richer: Rails 3 :- Obie Fernadez

Obie gave a enlightening summary of the events leading to and the people involved in the merging of the Merb and Rails codebases, resulting in the newly released Rails 3.0. The main drivers behind the new release were decoupling, modularity and prevention of code bloat. Many components of Rails 2 had been abstracted out and managed as third-party plug-in projects. He described Rack, Bundler, Arel and the upgrade process from Rails 2.x to 3.0. I look forward to the day when I get the chance to put all these into practice.

Integrated Tests Are A Scam :- J. B. Rainsberger

Having written myriads of tests over the years, it was easy to lose sight of the different roles of tests in different parts of the system. JB gave an excellent refresher on two fundamentals of unit tests : interaction testing and contract testing.

Interaction tests were the "normal" unit tests that I've been writing, where the external dependencies were replaced with stubs/mocks . In the example, he described interaction tests for a client assessing an external supplier. These tests were concerned with the pattern in which the system-under-test (SUT) called out to the external system (mocked out), and how the SUT responded to various return values from the external system (stubbed).

Contract tests referred to tests that verified that an external system satisfied a certain API and expected behaviour, i.e. the contract. In a strongly-typed language like Java, this contract could be abstracted out from the actual implementation of the external system into a public interface, that our code could depend on. In fact, our code would only depend on this public interface, with no knowledge of the implementation. In a strongly-typed language, any implementation of this interface would then satisfy the minimum requirements of the contract, therefore, the real implementation could be replaced with a stub. As long as the stub passed the contract tests, it would in theory make no difference to our code. Hence, we could then write integrated tests for our SUT that used the stub instead of the real implementation, which would make such tests extremely fast.

Personally, I was not too convinced that interaction and contract tests could fully replace the need for integrated tests against real external systems. There would always be unexpected surprises lurking in system configuration, and some behaviour could not be adequately captured by the abstracted interface. Furthermore, in dynamically typed languages, it would be much harder to enforce that the stub fully implemented the public interface.

Designing and Implementing RESTful Application Protocols :- Ian Robinson

Ian walked us through an example from his new book, REST in Practice, to illustrate the interactions of a client program ordering coffee beans from a coffee supplier via a RESTful interface.

The client first asked the server for a list of allowed operations. After that, it requested a quote of available product prices. It then placed an order by resubmitting all the pricing data it had previously received back to the server. In this way, the client could maintain conversation state over a stateless protocol. A checksum of the data was used to prevent price tampering by the client. Finally, the server immediately returned a HTTP 202 response, before kicking off the long-running order fulfillment process asynchronously.

Ian explained several basic concepts such as the use of media types, link relations and XForms in driving the process flow.

I found Ian's talk to be clear and concise, therefore I'm putting his book on my wish-list.

Most interesting talks

Exploring NoSQL :- Erik Meijer

This talk started a bit a slowly with an (mostly unecessary) explanation of how an object graph was represented in memory using pointers. This type of representation was embodied by noSQL, whereas SQL embodied a fully normalized relational view of data.

Things became interesting when Erik started describing Category Theory. Erik reasoned that because the direction of parent-child relationships was reversed between SQL and noSQL, they were duals of each other according to Category Theory. In other words, noSQL was actually "coSQL", i.e. for any statement that was true of SQL, the dual-statement (opposite meaning) would hold true for noSQL. Some examples of this duality were:

SQL	noSQL
The identity is embodied in the row, i.e. primary key (extensional)	Identity of an object depended on other objects in the environment (intensional)
Closed-world view of data	Open-world view of data
Synchronous operations	Asynchronous operations

A very interesting concept indeed. Furthermore, this talk had the most obscure slide ever...

Computational Information Design :- Ben Fry

This talk was worth attending just for the visual candy itself. Ben talked about data visualization using his Processing programming language (IMO really a DSL over Java 2D/3D). Some of the examples shown were really amazing, such as:

Mapping the program flow in old cartridge-based video games
Tracking changes to six editions of Charles Darwin's work
Comparing the human genome with other mammals

Ben then introduced Processing.js that allowed code written in Processing to be run on any HTML5 compatible browser. People could simply go to sketch.processing.org to write and run Processing code all within a Web browser. What was even more exciting was a port of the Processing runtime to Android, which meant I could run Processing programs on my smartphone. Super cool!

Workshop

For me, the best part of the conference was the iPhone workshop. Having done some Android development, I was eager to see how things worked on the iPhone side. In this half-day workshop, I learned some basic syntax of Objective-C, basic usage of xcode IDE, some iPhone frameworks, and finally put together a simple iPhone application. My verdict of Objective-C :- dynamic typing and late binding were nice, but the extremely verbose syntax really sucked. I guess I'll stick with Android for now.

Tethering HTC Desire to Mandriva Linux 2010.1

2010-10-27T09:27:00.017+11:00

The other day, I found myself in a situation where I needed to go online but there was no Wifi around. So I plugged my HTC Desire (Android 2.1) smartphone into my laptop running Mandriva Linux 2010.1 Spring and got tethering working. Here are the steps.

Preparation

While connected to a wired or Wifi Internet, install the dhcpcd package. As root user (or using sudo):
$ urpmi dhcpcd

It was also useful to install usbview.
$ urpmi usbview

Connecting the phone

First, connect to Mobile Internet from the phone (this must be done before plugging into laptop). Then plug the phone into the laptop using the USB cable and select the Internet sharing mode.

On the laptop, start usbview and confirm that Android Phone was shown.

As root user, start the usb0 interface:
$ ifconfig usb0 up
Check that it had really started the usb0 interface:
$ ifconfig usb0

Getting an address

Finally, on the laptop, obtain an address from the phone using:
$ dhcpcd usb0

When this command had completed, confirm that an IP address had been assigned using:
$ ifconfig usb0

After this, the laptop can get online using the HTC Desire phone as a gateway.

Agile Australia 2010: Personal Recollections

2010-09-15T22:28:00.076+10:00

I attended the Agile Australia 2010 Conference in Crown Conference Centre, Melbourne, on September 15 and 16, 2010. Here are some of my personal recollections, remarks and photos.

DAY 1

First impression when I got to the venue :- lots of people. The expected Agile consulting/training firms were there with their promotional booths (some free iPads to give away), with Thoughtworks being prominently at the centre. Personally, I was more interested in training courses, and so grabbed some brochures from Renewtek and Agile Academy.

Promotional booths and the crowd

The conference was kicked off with a keynote talk by Jim Highsmith, who presented various pieces of data, statistics and graphs to tell everyone how to measure Agile successes.

Jim Highsmith's keynote talk

This was immediately followed by an excellent talk by Jeff Smith of Suncorp. His point that trust and respect were the crucial values necessary to foster effective teams resonated with my own recent experience with several teams in my own company. He also introduced the Agile Academy, which I didn't know of before.

Next, the conference was split up into three parallel streams. I attended the talks by Ben Hogan (Using Agile techniques to adopt Agile) and Neal Ford (Implementing Emergent Design).

Ben introduced us to John Kotter's 8 steps to successfully implement change, that he contended were not adaptive due to lack of any feedback mechanism. Therefore, he proposed using Scrum's feedback cycle to introduce adaptability to some of these steps. In order to introduced changes such as Agile adoption to people and organizations, he suggested using a longer iteration/cycle time (1 month) than in software development.

I was initially more interested in Neal's topic due to its developer-centric focus. He spent quite a lot of time explaining software design concepts such as:

inherent and accidental complexity
technical debt and negotiating payment
over-engineering for genericness

He presented enablers of emergent designs such as:

test-driven design
refactoring to harvest idiomatic patterns

He even went through some calculations of code complexity and class coupling metrics. However, given that Neal's talk was marked as for those thoroughly experienced with Agile, I was actually a bit disappointed to find it rather lacking in novel in-depth content. Surely, experienced agile practitioners would already know about complexity and technical debt. I doubt any experienced developer would have learned much from the refactoring examples. I did learn something useful though:- In Pacman, a ghost could not detect Pacman running right up next to it (his example of anti-object).

Neal Ford on idiomatic patterns and software design

By lunch time, I was eager to find out what delicacies might be served in the buffet. Alas, it was just beef rolls and sandwiches. Nothing to brag about. How I missed the comparatively luxurious offerings from the bygone Sun Developer Days conferences.

Sandwiches, rolls and some salad for lunch

After lunch, I watched the panel session, where five top agile practitioners sat together to discuss their preferred flavours of Agile. It was nice to be able to ask questions using Twitter. I was particularly interested in Kanban, and managed to tweet a Kanban-related question for Bruce Taylor from the comfort of my phone. By the way, my boss was also one of the panel members.

This was followed by the lightning talks, my favourite of which was Ben Arnott's Agile@Home. A light-hearted, entertaining contrast to the other serious topics of the day.

One of the lightning talks

The last full talk I attended on Day 1 was Craig Smith's Building An A-Team. The content itself was nothing new, mainly around how to identify and lead good teams. However it was concisely and brilliantly delivered with good humor.

Finally I decided to pop into the Agile Games session next door midway. This was what I saw :

Not too sure what was going on here

Groups of people throwing balls around. After 30 minutes and still failing to make sense of this, I switched to another room and caught the end of Jay Jenkins' talk on how ANZ did Agile.

Thus ended Day 1.

DAY 2

The second day started with an awesome talk by Martin Fowler. He reminded us of the reasons why we did Agile, the advantages of adapting to late changes, the importance of continuous integration and delivery, and finally the different types of technical debt. Of course, the best part was his impersonation of Uncle Bob Martin.

Martin Fowler giving the first talk of Day 2

Next, I attended a couple of more practical talks. First was Jason Yip and Marina Chiovetti on how to properly measure and assess Agile adoption and success in different organizations. They pointed out that external assessments were misleading because results could be faked. Only when an organization defined a vision ("true north"), be transparent and internally reflected on how it was progressing towards this vision could it have a true understanding of its level of Agile adoption.

The second talk was very relevant to me as a developer. Chris Mountford described how he solved monster builds problems in Atlassian. His approach involved the following:

Always measure first
Be selective about what to test
Parallelize tests
Canary tests with a good proven version to detect environment/external problems
Choosing the right tools

At this point, the #agileaus Twitter feed was flooded with reports of how fantastic talks in the other rooms were. All of the talks during this time slot looked interesting, but alas I could only be in one place at a time.

I felt lunch on Day 2 was a slight improvement over the day before. The salad, rolls and sandwiches were still there, but they tasted a bit better, or maybe I'd lowered my expectations.

More salad, rolls and sandwiches for lunch

In my opinion, the talk with the best analogy (Death Star) and slides would be Nigel Dalton's talk after lunch. Nigel described how various non-IT teams (lawyers, finance, publishers etc) in Lonely Planet adopted Agile. Photos of their various Kanban boards managed to convey how these teams track their work in a highly effective and visible way. It was reassuring to know that Agile was not just restricted to us IT folks. Having personally spent a few months working at the Lonely Planet office in Footscray a few years ago, I was happy to see such a drastic transformation throughout the company.

Nigel Dalton described the Death Star as the perfect Agile project : "Build the weapon first!"

Nigel's entertaining presentation was followed by a more serious, enterprise-level talk by John Sullivan from Sensis, on how best to do "Enterprise Architecture" in an Agile company. This was a topic that interested me as a developer. Some of John's main points were:

Traditional enterprise architectural groups were unnecessary barriers between developers and business
Architects produced elegant but unworkable solutions
Better to form a temporary group of senior developers to sit with business in a requirements gathering session to identify key business and technical requirements
Implement an initial version of the architecture in code, or sometimes spikes, to gain technical insights to be able to provide estimates
"Architect" as a temporary role on an as-needed basis, not a job title
Get rid of people who get in the way
Split the backlog into
- in-build : Only build this, but make the system extensible
- in-plan : Something to keep in mind, but expect this to change
- future-release : (I would guess don't worry about this)

John Sullivan declaring the end of the Enterprise Architect job title

Up to this point, DevOps people might have felt a bit left-out. Luckily, Jez Humble changed that with his informative presentation on DevOps and Agile Release Management. This was another area I had a special interest in. Jez stressed the importance of the definition of DONE, i.e. DONE when a feature got to production. The problems he presented were that DevOps people:

used to see changes as too risky
resisted new changes because they favoured stability
spent too much time firefighting rather than doing strategic work

. The solutions he presented were:

Culture : Have cross-functional delivery teams. Adopt Kanban to schedule work.
Automation : Environments could be procured using cloud computing and configuration managed by Puppet or Chef. He even mentioned cucumber-nagios which interested me a lot.
Measurement : Collate business and technical metrics. Perform root cause analysis.
Sharing : With big visible displays
Overcoming objections : By showing visibility, control, automated scripts and auditing
Deployment pipeline : Automatic deployment to various environments including production.

That was a lot of new tools for me to read up on.

Jez Humble talking about DevOps

Finally, I attended the Open Space session, first with the lightning talks group, then finally with a group discussing automated performance tests. The idea sounded good, but I felt there wasn't enough time to flesh out too much technical details.

One of the Open Space lightning talks

At 4:30 pm, snacks and drinks came out for the final networking session that concluded the conference.

CONCLUSION

Overall a worthwhile conference to gain an overview of the current state of Agile. I was glad to see the previously neglected area of DevOps getting some attention. As for the technically focussed talks, I would have preferred:

more technical depth
updates on the latest development, testing or monitoring tools
some discussion on the latest Agile practices when developing on mobile devices

Maybe next year...

Detect broken links on a Web site using wget

2010-08-06T16:38:00.005+10:00

The other day, I started thinking about writing a simple Web site validator that detects broken links on a Web site, similar to W3C Link Checker. After looking at various code samples in Java, Ruby, etc, I figured out that GNU wget 1.12 on my Linux machine could do the job just fine, with no programming required. It even detected broken resource links in CSS, not just broken <a> links.

Here is how to write a simple script to check the site. First, pretend to be a Mozilla-based browser and spider the site to the depth of one level:


wget --spider -r -l 1 --header='User-Agent: Mozilla/5.0' \
-o wget_errors.txt http://the_site_i_want_to_validate

Then, simply look at the return code to determine if there is any error. If the code is larger than zero, there is an error.


EXIT_CODE=$?
if [ $EXIT_CODE -gt 0 ]; then
    echo "ERROR: Found broken link(s)"
    exit 1
fi

To find out the actual links in question, just grep for 404 in the wget error log.


BROKEN_LINKS=`grep -B 2 '404' wget_errors.txt`

The -B 2 outputs the 2 lines above any matching line, which in this case contains the broken link in question.

Loading large reference database in Android

2010-08-03T16:13:00.066+10:00

Reference data normally refers to (mostly) read-only data that is used to validate or resolve other pieces of data. For example, a list of postcodes and suburbs, that can be used to provide auto-suggestions in a UI or validate a user's address.

I recently developed an Android (version 2.1 / Eclair) application that used a large set of reference data, stored in a SQLite database. The fact that this was targeted for Android smartphones introduced some constraints in the loading of this data that would not be present for a desktop or Web application, primarily processing time and storage space.

The initial design was based on my prior experience working on Web applications. It involved packaging a largish (~10MB) text file into the apk archive. The application performed a one-time initialization step that loaded this file, parsed it and executed SQL statements to insert rows into the database. The application then accessed the reference data via a subclass of SQLiteOpenHelper. It soon became apparent that this would not work in the Android world, due to the following reasons.

Loading the database, even if its a once-off, took too long. People are not likely to want to wait for many more minutes, especially after waiting for the 10MB download.

The database storage took up 10+MB, and when combined with the input text file, totalled 20+MB. Android 2.1 required the application to be installed onto the phone's limited internal memory, where 20MB was a relatively massive chunk.

The input text file in the apk archive could not be removed after initialization, so it just uselessly consumed precious phone memory.

After a few tries, I arrived at the following final design that seemed to solve these problems.

Preload database

The database loading code had to be removed from the Android code and rewritten as a separate program. This program read the text input file and inserted the rows into a SQLite database, with the database file stored on my desktop. Note that the primary key column of the reference table should be named _id and the following must be added to the database for it to be usable by Android:


CREATE TABLE android_metadata (locale TEXT);
INSERT INTO android_metadata VALUES('en_US');

This preloaded database file was then uploaded to a Web site so that it could be downloaded by the Android application.

Download database

The Android application didn't need to be packaged with the text input file any more, hence trimming the apk archive from 10+MB to a few hundred KB. The application's starting Activity would check and download the database file if it didn't already exist on the SD-Card. According to the Android Dev Guide on Data Storage, the proper place to save the database file would be:

for Android 2.2 and above :- the directory returned by getExternalFilesDir()

for Android 2.1 :- the directory starting with getExternalStorageDirectory(), then appended with /Android/data/<package_name>/<file_type>, which resolved to something like /sdcard/Android/data/com.mycompany.myapp/db

I had to use the latter option as I was targeting 2.1.

It was important to ensure that the UI was not frozen during the few minutes it took to download the 10+MB database file. That meant that the downloading code must be executed in a separate thread to the main UI thread. The best way to achieve this was to create a subclass of AsyncTask and put the downloading code in its doInBackground() method. This class also displayed a ProgressDialog to keep the user informed, and acquired a WakeLock (SCREEN_DIM_WAKE_LOCK) to prevent the phone from going to sleep. Another important thing to remember was to prevent the screen orientation from changing when the phone was flipped over while the ProgressDialog was showing, otherwise the application would crash when the dialog was dismissed. This was achieved by temporarily setting the requestedOrientation property of the activity to "no-sensor". Here is the skeleton code of the AsyncTask:


// FileDownloader is my own delegate class that performs the 
// actual downloading and is initialized with the source URL.
public class InitializeDatabaseTask extends
        AsyncTask<FileDownloader, Integer, Object> {
    private ProgressDialog progressDialog;
    private File dbFile;
    private PowerManager.WakeLock wakeLock;
    private Activity activity;
    private transient int originalRequestedOrientation;

    public InitializeDatabaseTask(Activity activity, File dbFile) {
        super();
        this.dbFile = dbFile;
        this.activity = activity;

        wakeLock = ...;  // Obtain a wakelock for SCREEN_DIM_WAKE_LOCK             
        progressDialog = ...;  // Create a ProgressDialog instance with title, message ,etc
    }

    @Override
    protected void onPreExecute() {
        super.onPreExecute();
        originalRequestedOrientation = activity.getRequestedOrientation();
        activity.setRequestedOrientation(ActivityInfo.SCREEN_ORIENTATION_NOSENSOR);
        wakeLock.acquire();
        progressDialog.show();
    }
    
    @Override
    protected Object doInBackground(FileDownloader... params) {
        FileDownloader downloader = params[0];
        try {
            downloader.downloadTo(dbFile);
        } catch (IOException e) {
            throw new AndroidRuntimeException(e);
        }
        
        return null;
    }
    
    @Override
    protected void onPostExecute(Object result) {
        super.onPostExecute(result);
        progressDialog.dismiss();
        wakeLock.release();
        activity.setRequestedOrientation(originalRequestedOrientation);
    }
}

The main Activity used InitializeDatabaseTask like so:


File dbFile = ...; // File pointing to /sdcard/Android/data/com.mycompany.myapp/db
new InitializeDatabaseTask(this, dbFile).execute(new FileDownloader(DOWNLOAD_DB_URL));

Mount SD-Card in emulator

I developed the application mainly using the Android SDK emulator in Eclipse. The emulator did not mount any SD-Card by default. In order to test the downloading code, the emulator must be set up with a SD-Card image file, like so (on my Linux system):


$ android-sdk/tools/mksdcard 64M ~/.android/avd/Android.2.1.avd/sdcard.img

This created a 64MB image file (more than enough for the data) in a special Android avd directory, so that it would be automatically mounted by the emulator.

Implement custom SQLite helper

Finally, I had to write my own custom version of the SQLiteOpenHelper to access the database file from the SD-Card, because the standard one would only read from the default phone internal memory. Given that the reference data would be read-only, this custom class was a lot simpler than SQLiteOpenHelper and only needed to open a read-only database connection. The code:


public abstract class ExternalStorageReadOnlyOpenHelper{
     private SQLiteDatabase database;
     private File dbFile;
     private SQLiteDatabase.CursorFactory factory;

     public ExternalStorageReadOnlyOpenHelper(
         String dbFileName, SQLiteDatabase.CursorFactory factory) {
        this.factory = factory;
        
        if (!Environment.getExternalStorageState().equals(Environment.MEDIA_MOUNTED)) {
            throw new AndroidRuntimeException(
                "External storage (SD-Card) not mounted");
        } 
        File appDbDir = new File(
            Environment.getExternalStorageDirectory(),            
            "Android/data/com.mycompany.myapp/db");
        if (!appDbDir.exists()) {
            appDbDir.mkdirs();
        }
        this.dbFile = new File(appDbDir, dbFileName);
     }
     
     public boolean databaseFileExists() {
        return dbFile.exists();
     }
     
     private void open() {
         if (dbFile.exists()) {
             database = SQLiteDatabase.openDatabase(
                 dbFile.getAbsolutePath(), 
                 factory, 
                 SQLiteDatabase.OPEN_READONLY);     
         }
     }
        
     public synchronized void close() {
         if (database != null ) {
            database.close();
            database = null;
        }
     }

     public synchronized SQLiteDatabase getReadableDatabase() {
         return getDatabase();
     }

     private SQLiteDatabase getDatabase() {
        if (database==null) {
            open();
        }
        return database;
     }         
}

A concrete subclass of ExternalStorageReadOnlyOpenHelper was created to query the reference data via the SQLiteDatabase object returned by the getReadableDatabase() method.

The databaseFileExists() method allowed the main Activity to check if the database file already existed to decide whether to initiate download.

Conclusion

This final design had greatly improved the application in the following ways :

The user only had to download a small apk file initially, greatly reducing the barrier to installation.

The separate download step for the database file presented an opportunity to tell the user what was happening and to use Wi-Fi if available.

Most of the data resided on the SD-Card, where space was much more abundant.

There was ever only one copy of the reference data on the phone, no redundant duplication.

The main application code could be updated without having to download the reference data.

Large User Story vs Human Psychology

2010-05-27T20:48:00.032+10:00

Much have been written about the problems of oversized user stories. They are commonly indicative of too much uncertainty, usually over business requirements, but sometimes over certain technical aspects. While large stories (or epics) are acceptable during a high-level planning stage, they should be either simplified or broken down by the time they are scheduled within an iteration. Otherwise, they usually end up greatly exceeding their (very inaccurate) estimates, or even becoming blocked.

As a software developer, I've been involved with a few such stories recently. In addition to the above-mentioned problems, I noticed another adverse side-effect. With each passing day on the same story, I became increasingly less motivated to observe good coding practices such as refactoring, cleaning up messy code and proper OO design. As the days dragged on, I became more tolerant of sloppy practices. I eventually finished the story and handed over a piece of spaghetti code that I wasn't proud of. In hindsight, I could have done a lot more to improve code quality.

Thinking back, I realized that this same behaviour manifested itself many times before in the past. Perhaps, I might just assume that I was a lazy bum, if not for the fact that I was practising pair-programming, and the same tendency was also apparent in my pairing partners (some more resistant than others). Maybe it was just the long hours and looming deadlines. However, my experience tells me that doing multiple smaller consecutive stories, while they may add up to the same volume of work as a giant story, does not lead to this behaviour.

Image from: weber.edu

This leads me to speculate that there is a psychological explanation. Research has shown that the human brain has two regions that compete to form a decision when presented with a choice between a small short-term reward and a longer-term goal (larger reward). The more emotionally driven region of the brain seeks the short-term reward while the rational region tells us to hold-out for the overall better outcome. Our brain battles itself to a decision. As professional software developers, we constantly have to make a similar decision :- should we take a short-cut to deliver a piece of work now, or refactor/write more tests/clean up so that the overall code quality improves in the long run?

It is likely that by breaking up a large body of work into many small discrete stories, the completion of each story provides an emotionally satisfying reward in a short period of time, even if extra effort/time has been added to ensure good quality. This extra effort/time is usually at least proportional to the original estimated size of the story, but may cost even more if the story gets so large that it touches many interdependent parts of a complex system.

In other words, a small story usually touches a small area of the code base, hence its parts are easier to visualize in one's head, and thus presents only a small mental barrier to keeping it clean. Conversely, a massive story that encroaches on a myriad of different moving parts becomes hard to visualize. Soon, problem areas stack up beyond our ability to keep track of them in our heads. On top of that, the uncertainties inherent in a large story may reveal themselves to be time-sinks. Before long, the extra effort required to deliver quality becomes a mountain of hurdles. Having spent most of our time and energy just getting to code to work, with no psychological reward from completing anything yet, it is no wonder that we become disinclined to continue.

In conclusion, large user stories work against human psychology for the desired long-term goal of good quality code. Small stories are more satisfying due to frequent emotional rewards, at the same time contributing towards the greater goal.

It is not always possible to foresee how big a story can become at the start, therefore, developers should be given the freedom to split up stories while they are being developed. Personally, I think a story should not take more than 2 to 3 days. If it looks bigger than this, split it. This I endeavor to do more of in the future, with the hope of always delivering code I can be proud of.

Testing jQuery with env.js + RhinoUnit

2009-11-02T06:48:00.011+11:00

The previous (Web application) project I was on started adopting jQuery to implement more client-side browser behaviour. We were already using RhinoUnit for testing some generic Javascript logic, but jQuery required access to the browser DOM. Hence, env.js was used to provided a DOM to RhinoUnit. This combination would be very quick due to RhinoUnit running headlessly and not relying on a browser, which was perfect for TDD. Getting these three pieces to work together properly took some work, as described below.

Versions

jQuery 1.3.2
env-js.1.0.rc7, using env.rhino.js
RhinoUnit 1.2.1

Replaced self
As mentioned in Adam Scott's blog, references to self in RhinoUnit needed to be replaced. So at the top of the rhinoUnitAnt.js, I added this:
var rhinoUnitSelf = self;
All references to self below this were replaced with references to rhinoUnitSelf.

Replaced print
Adam Scott's blog also mentioned the need to replace calls to print with self.log function. Specifically, reimplemented $env.log in env.rhino.js like so:


$env.log = function(msg, level){
    rhinoUnitSelf.log(' '+ (level?level:'LOG') + ':\t['+ new Date()+"] {ENVJS} "+msg);
};

Implemented load
For some strange reason, env.js relies on a load function that could not be found. The solution was just to implement it using RhinoUnit's loadFile. The following was added to the top of env.rhino.js:


load = function(x) {
 eval(loadFile(x));
};

Ignored global variables
jQuery and our unit tests created a lot of global variables that RhinoUnit didn't like. Instead of configuring RhinoUnit to ignore a massive list of global variables, I decided to simply remove the code that checked for this. Specifically, reimplemented ignoreGlobalVariableName in rhinoUnitAnt.js as such:


function ignoreGlobalVariableName(name) {
    return true;
}

Cleanup after tests
At this stage, our unit tests could run individually, but they failed when executed together (consecutively, from Ant). This was because the global variables that were created during the first test were not cleaned up and they corrupted the environment for all subsequent tests. In order to fix this, I modified the end of the runTest(file) function in rhinoUnitAnt.js so that global variables created in one testcase (test file) were cleaned up prior to running the next testcase.


/*****  ADDED to remember global variables that existed prior to each test file *****/
    var origGlobalVars = {};
    var varName;
    for (varName in this) {
        origGlobalVars[varName] = true;
    }
/************************************************************************************/

    eval(loadFile(file));

    var testCount = 0;
    var failingTests = 0;
    var erroringTests = 0;

    executeTestCases(test, test.setUp, test.tearDown);

    if (testCount === 0) {
        erroringTests += 1;
        testfailed = true;
        failingTestMessage(file, "No tests defined");
    }

    rhinoUnitSelf.log("Tests run: " + testCount + ", Failures: " + failingTests + ", Errors: " + erroringTests);
    rhinoUnitSelf.log("");

/******  ADDED to remove only those global variables that have been 
         added by the test file, but not those that existed before.  ******/
    for (varName in this) {
        if (origGlobalVars[varName] != true) {
            delete this[varName];
        }
    }
/*********************************************************************/

After these changes, our jQuery unit tests ran perfectly, allowing us to do TDD for our jQuery code.

Cacti : Graphing remote service WITHOUT using SNMP

2009-03-26T18:00:00.036+11:00

What is Cacti

Cacti is a popular Web-based tool for generating graphs of various data/statistics from remote servers. It provides default "templates" for generating graphs of CPU load, processes, memory, etc out-of-the-box, mainly retrieving these data from remote machines via SNMP (Simple Network Management Protocol). Cacti can be set up to graph custom data, again using SNMP by default. In fact, most of the instructions available online assume that Cacti talks SNMP to remote machines.

This blog is based on Cacti version 0.8.7d, running on a Linux machine.

The problem with SNMP

However, I find SNMP to be very complicated to set up for anything more complex than a single value data. Personally, I find SNMP to be overkill and far too difficult to manage, especially when it comes to setting up a custom SNMP table to group data in a way that can be traversed by Cacti.

Alternative to SNMP

This blog shows how Cacti can be used to retrieve data, via HTTP, from a URL that points to a "Web service" (not the heavy SOAP kind) that returns data in a simple text format. This can be implemented using JSP, PHP, a simple Java servlet or just any Web technology. All it needs to do is to return in its response the data in the following textual format:

fieldName1:fieldValue1 fieldName2:fieldValue2 fieldName3:fieldValue3 ...

Example

Lets assume I'm hosting a Java Web application, and I want to graph the following:
- The number of requests that generated successful responses.
- The number of requests that generated errors.

Two things are required on the Java application side:
- Some sort of filter/interceptor to capture these two types of request.
- A servlet to count the number of these requests and return a response in the desired format. This servlet will reset its counters after sending each response, so that each time it returns the count since the last response. The response content type is set to text/plain and the content looks something like:

successCount:1623 errorCount:23

This servlet is deployed on the same Web container as the main Web application, and mapped to the URL:
http://remoteserver:8080/myapp/statistics

Cacti Configuration

Poller Settings

Firstly, tell Cacti to simply use Ping to detect hosts for polling. Click on Settings on the left bar, then click on the Poller tab and change Downed Host Detection to just Ping, as shown below:

Device

After that, set up a device to point to the server that is hosting the Web application. Assume it is reachable by name remoteserver. Create a device of type None as shown below, meaning Cacti will use ping to detect the server.

Note: This version of Cacti has a bug (at least on my Firefox browser) where it tries to validate that the SNMP Version 3 passwords match even when SNMP is set to Not In Use. To work around this, first select SNMP Version 3, blank out both passwords, then select Not In Use.

Data Input Method

A Data Input Method is set up to tell Cacti to retrieve the data from the servlet (using wget), and map the field names in the response text to fields that Cacti understands. Refer to screenshot below:

Name this Data Input Method as RemoteServerInputMethod. Select Script/Command as the Input Type, and type the following in the Input String :
wget --quiet --no-cache -O - http://<hostname>:<data_port>/myapp/statistics

Create hostname and data_port input fields, that are used to pass in values to the <hostname> and <data_port> parameters in the script command.

Create successCount and errorCount output fields, that are mapped to the field names in the response text. Make sure these output field names exactly match those in the servlet response text.

Data Template

The Data Input Method is used to create a Data Template, as shown below:

Name this Data Template as RemoteServerStatisticsDataTemplate. Name the Data Source as RemoteServerStatisticsDataSource, this will automatically become the data source name (see later).

Create a Data Source Item to map to each of the data fields, i.e. successCount and errorCount. Most of the settings here can stick to the defaults. The important part is to make sure that the Output Field selectors are mapped to the correct fields.

Its better to tick the checkbox under the Port on server Custom Data to tell Cacti to assign the port from the data source, so that we don't have to hardcode 8080 in the Data Template.

Data Source

Create a Data Source that connects the Data Template with the Host Device that were configured earlier. Select appropriate options for these two drop-down selectors. Do not change the Data Source Path. Enter 8080 as the Port on server.

This screen creates a Data Source that automatically gets the name as specified in the Data Template screen before.

Graph Template

Set up a Graph Template to tell Cacti how to plot the data in the graph. Name it RemoteServerStatisticsGraphTemplate, and give the graph a descriptive Title. All other fields below the Title can be left in their default values. Create two Graph Template Items as shown in the screenshot below.

Note that creating a Graph Template Item will automatically create a corresponding Graph Item Input.

The screenshot below shows the screen for creating a Graph Template Item. Simply choose the matching Data Source for the field, and enter a descriptive Text Format.

Graph Management

Enter Graph Management to create an association between Graph Template, Host Device and the Data Sources. The screenshot below should be self-explanatory.

Graph Tree

Finally, enter Graph Tree to place the graph in the hierarchical tree. This is quite straightforward. Create a Tree Item of type Host, and point it to the Host.

The new Tree Item will appear like so in the Graph Tree:

DONE!

Finally, its all done. Just click on the Graph tab, drill down the tree and see the graph of successful and error request counts.

SQL*Plus connection to remote Oracle database

2009-02-26T11:58:00.001+11:00

SQL*Plus is a very handy command line tool when you want to quickly access an Oracle database or execute SQL statements from scripts, e.g. BASH scripts. However, getting it to connect to a remote Oracle database server in a script proved to be quite a challenge. This blog documents how to write the connection string in various cases.

Local database

Connecting to a local database is trivial, just use:

$ sqlplus dbUser/dbPassword@dbSid

Remote database from command line

Here's the nasty syntax for connecting to a remote database using its SID:

$ sqlplus dbUser/dbPassword@'(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=remoteServer)(PORT=1521)))(CONNECT_DATA=(SID=dbSid)))'

You can also connect using a SERVICE_NAME instead of SID.
Note that the single-quotes are needed to preserve the brackets.

Remote database from a BASH script

I needed to write a script that called SQL*Plus several times. In order to make it maintainable, the username, password and the part after the @ were specified as environment variables. Unfortunately, there did not seem to be any obvious way of setting the value of the environment variable so that the single-quotes were properly sent to the sqlplus command at run time. I had to explicitly escape each bracket with a backslash, like so:

DB_CONNECTION=\(DESCRIPTION=\(ADDRESS_LIST=\(ADDRESS=\(PROTOCOL=TCP\)\(HOST=remoteServer\)\(PORT=1521\)\)\)\(CONNECT_DATA=\(SID=dbSid\)\)\)

Because this script was part of a Java project, I subsequently moved this value to a Java properties file. This means every backslash had to be escaped with another backslash to prevent Java from interpreting it. The entry in the Java properties file:

db.connection = \\(DESCRIPTION=\\(ADDRESS_LIST=\\(ADDRESS=\\(PROTOCOL=TCP\\)\\(HOST=remoteServer\\)\\(PORT=1521\\)\\)\\)\\(CONNECT_DATA=\\(SID=dbSid\\)\\)\\)

(I know I could have just used JDBC, but there was a good reason why I had to use SQL*Plus for this Java project.)

Define a local service name to a remote server

An alternative to using the big ugly connection string is to define a local service name in the local tnsnames.ora that references the remote instance. Add the following:


DB_LOCAL_NAME =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL= TCP)(Host= remoteServer)(Port= 1521))
    (CONNECT_DATA = (SID = dbSid))
  )

to

${ORACLE_HOME}/server/network/admin/tnsnames.ora

Connecting to the remote database is simply:

$ sqlplus dbUser/dbPassword@DB_LOCAL_NAME

Of course, this solution has a few drawbacks (enough to dissuade me from using it).
- Need write permission to the tnsnames.ora.
- The application relies of custom entries in tnsnames.ora, which must be maintained together with the application code.
- Every environment (e.g. development, staging, UAT, production) in which the application is deployed to will need to have its tnsnames.ora modified and maintained.

Save the Net

2009-02-20T11:30:00.000+11:00

Date sensitive integration testing of database code with Oracle

2009-01-09T17:22:00.000+11:00

Note:

Commands starting with $ are to be entered in a shell, e.g. BASH.

Commands starting with SQL> are to be entered in a database client, e.g. SQLPlus or JDBC code

Scenario

Lately, I've been working with a lot of database code, mostly packaged as stored procedures, running on Oracle 10g XE. These stored procedures are tested using a set of automated integration tests written in Java (JUnit + DBUnit). Everything was fine until one day, I needed to implement a logic that filters out data based on whether a date field stores a date within a certain period of time. The SQL for this was quite trivial, for example:

SELECT ... WHERE table1.the_date_field > ADD_MONTHS(SYSDATE, -6)

This example would filter out all results with the_date_field storing a date older than 6 months ago.

The Problem

The problem arose when I needed to write automated tests for this. The tests loaded up a set of hardcoded test data fixture (using DBUnit). The test data would have hardcoded date values. Obviously, these tests would give different results over time because SYSDATE would return the current system time by default.

The Solution(s)

One possible solution would be to dynamically inject the dates into the test data when the tests were executed. However, this would require a lot of work because of the quantity and spread of the dates. I would need to convert existing hardcoded dates into offsets from SYSDATE, then write some code to perform the injection at runtime.

Given this, I chose the reverse approach :- make SYSDATE return a fixed value.

Stopping time

Luckily, Oracle provided a way to let me do just this.
Firstly, I had to login to the database as a SYSDBA and grant my Oracle user the ALTER SYSTEM system privilege.
Now, I could invoke the following SQL in the set-up code of the JUnit tests:

SQL> alter system set fixed_date = '01-APR-2005'

This would make SYSDATE always return April 1st, 2005. All the test data could now be measured relative to this date, and the tests would not break no matter when they were run.

The tear-down code of the JUnit tests would invoke:

SQL> alter system set fixed_date = 'none'

to revert SYSDATE back to its default behaviour.

PFile and SPFile

However, when I tried to restart Oracle server, it failed to start up. After a few hours, I found out why. Oracle by default stored parameter settings in a binary file format called SPFile. Any changes made with ALTER SYSTEM were always persisted to this file, so that when Oracle was restarted, it tried to load them again. For some reason (a bug IMHO), Oracle could not load the fixed_date value of 'none', it simply died.

The workaround was to change to a different file format called PFile. PFile is a normal text file that could be editted, and unlike SPFile, Oracle would not persist any parameter changes to it. In my opinion, PFile is a better choice for a development or automated testing database, where changes made during tests should not be persisted or interfere each other. To convert from SPFile to PFile, first login as a SYSDBA:

$ $ORACLE_HOME/bin/sqlplus sys as sysdba

then execute:

SQL> create pfile from spfile;

Then shut down Oracle server, and remove the SPFile. The SPFile name has the following format: spfileSID.ora, where SID is the database SID. I was developing on a Ubuntu machine and my SID was XE. Hence, I ran this:

$ rm $ORACLE_HOME/dbs/spfileXE.ora

The PFile name has the format initSID.ora, so it was initXE.ora on my machine. Edit this file with a text editor and delete the line:

*.fixed_date='NONE'

Finally, restart Oracle.

Limitations

I found that when SYSDATE was fixed to a hardcoded value, jobs could not be scheduled using DBMS_JOB functions. DBMS_JOB.SUBMIT did not seem to work at all. Therefore, I can't yet think of a way of both freezing time and testing any code that rely on scheduled jobs.

Mandriva 2009 on Toshiba Portege M800

2008-12-12T23:49:00.002+11:00

(Screenshot of KDE 4 desktop with Dolphin file manager, photo and desktop widgets)

I've just installed the newly released Mandriva 2009.0 Powerpack on a company supplied Toshiba Portege M800. Almost everything worked out of the box... almost.

The following required some tweaking:

Video Playback
I did not like DragonPlayer that was configured by the installer. so the first thing I did was install MPlayer (my favourite video player that plays everything) from PLF, together with all the popular codecs. Initially, my video did not play. After reading the Mandriva Errata page, it turned out that because I had enabled desktop effects using Compiz, I needed to also enable the Compiz Video Playback plugin, using ccsm. Once that was done, MPlayer worked fine. However, other video players like Kaffeine and Totem did not seem to work, but I didn't care much for those.

Wifi
After trawling the Internet, I realized that I needed to reinstall the iwlwifi-4965-ucode package, because the installer had removed some firmware files required by the built-in Intel 4965AGN wireless adaptor. After that, wifi seemed to connect fine to my home wifi network that used WPA authentication.

MandrivaUpdate over proxy
At work, MandrivaUpdate must be configured to go through an authenticated proxy before it could go out to grab updates from the Internet. Unfortunately, there seemed to be an inconsistency between how the proxy login and password were specified in the urpmi configuration and Network Settings. As a result, it was not possible to perform an update via MandrivaUpdate. The workaround was just to do updates from the command line, using:


urpmi --proxy server:port --proxy-user login:password ...

MandrivaUpdate worked fine at home where it did not need to go through a proxy.

Things I did not quite like include:

Auto-partitioning
The Portege M800 came with a 130GB harddisk. I chose to let the installer manage the partitioning this time and told it to copy the contents of the installation DVD onto disk. I had expected it to give several tens of GB for the "/" partition, a separate partition just to hold the DVD contents, and everything else for "/home". I was disappointed to see that the installer allocated a mere 8GB for "/" that also contained the DVD contents (~4GB). That left only 4GB for the entire OS plus any other programs I wanted to install. No where near enough because I needed some large applications like an Oracle database. To fix this, I moved the DVD contents to a new directory under "/home", and made a symlink to it from the original location. That freed up 4GB that should hopefully be enough until my next upgrade to Mandriva 2009.1. Next time, I'll choose custom partitioning.

Missing applications

Missing altogether were KDE 3.x applications that had not been ported over to KDE 4.x, notably K3B. Hopefully, this situation is remedied in the near future.

Now, the good bits...

Overall, I was impressed with this release. KDE 4 looked a lot better than KDE 3 in 2008.1. The desktop widgets were a pleasure to use, desktop effects were nicely integrated, performance was acceptable, and there was an abundance of applications from the PLF and contribs repositories. The"Start Application" menu had a new tabbed-style that I found more user-friendly on a laptop. Even the SD card reader worked properly.

Sun Developer Day 2008, Melbourne, Australia

2008-11-26T21:29:00.000+11:00

Note: Click on images to view in full size.

I attended the Sun Developer Day in Melbourne, on November 26, 2008. It was hosted in the very fabulously furnished Grand Hyatt hotel. Upon arrival at the hall, I was greeted by Sun logos on brightly lit wall displays. I got there just in time for the Keynote speech. Below is a summary of the talks I attended, and what I thought about them.

Keynote, by Reginald Hutcherson

Reginald is undoubtably the best speaker of the day, in terms of presentation skills and the consistent pace of delivery. All the subsequent speakers seemed hurried. He stressed the importance of Open Source technologies for Sun, then briefly mentioned a host of technologies like virtualization, MySQL, Glassfish, Rich Internet Applications (RIA), JavaFX, but the most interesting one for me was zembly. Unfortunately he only had a single slide on zembly, but I later gathered that its a Web site that allows users to create social applications that work on Facebook, iPhone, etc. At this point in time, it is free to use while still in beta. Interesting... worth a second look once its released.

Demo shootout, by all three subsequent presenters

Two of the three demos were on JavaFX. They were simple applications that mostly served as eye-candy to keep the audience entertained, as well as introducing Netbeans. However, I've seen similar JavaFX demos in previous Sun talks, so there wasn't much in them for me. I wished they showed different aspects of development, for example writing unit and integration tests for JavaFX applications.

The remaining demo was on OpenSolaris. As expected, it started with GUI eye-candy effects like wobbly windows (nothing more than Ubuntu), followed by a sample MySQL driven application. One thing that caught my interest was VirtualBox that could seamlessly run applications from different hosted systems on the host desktop, as if they were running natively, only with different window decorations.

JavaNext - The future of Java SE, by Chuk Munn Lee

For me, this was the highlight of the whole seminar. I was looking forward to hearing about upcoming features in Java, and this talk did not disappoint. New features that particularly interested me were:

Annotations on Java types (JSR 308) to do stuff like enforcing invariants.
Modules to control access to Java packages. This involved adding a new module keyword and a module-info.java file to specify access rules. Modules would be packaged into JAM archives, that could also be pulled from OSGI repositories.
Closures - woo hoo!
New APIs like Swing Application Framework (JSR 296) and Beans Binding.

There was mention of a new soft real-time garbage collector called GarbageFirst for those having to do time sensitive applications, but this was not so relevant to me.

I wish Chuk told us what version of Java all these goodies would be coming in.

Connection the World with REST, by Chuk Munn Lee

The problem with this talk was too much introductory material on REST. I felt that REST is so widely known by most Web developers nowadays that basic knowledge such as verbs, resources, etc should have been assumed instead of wasting precious presentation time. Towards the end of the talk, Jax-RS (JSR 311) and the Jersey implementation were mentioned. Chuk briefly talked about how when Jax-RS is integrated into Java EE6, it will be able to work with/on EE resources like EJBs, Web Services, security, etc. Chuk should have spent much more time on these new upcoming support for REST.

JavaFX - SDK and Compiler, by Angela Caicedo

This talk turned out to be more interesting and informative than I expected, probably due to me being new to JavaFX. Angela introduced the newest syntax for variables, functions, binding, effects, animation and events. Pages and pages of code snippets... just the way I liked them. In addition, the following were discussed:

Architecture of the JavaFX framework.
Pull parser that supported JSON and XML.
Ability to access RESTful resources.
New browser plugin architecture.

SOA with OpenESB and Java CAPS 6.0, by Chuk Munn Lee

There was one word to describe developing for SOA :- tedious, and this demo-talk did little to persuade me otherwise. A demo loan approval application was implemented as a document-centric Web Service, using OpenESB running on Glassfish. Of course, Netbeans 6.5 was the IDE. Starting from some existing XSD data schemas, the basic steps were to create a WSDL, define a BPEL, bind with a Stateless Session EJB, run some integration tests against the service, then finally generating a Java client. The presenter had to go back and forth through various IDE panes and dialogs, to define WSDL parameters, access various URI and port settings, etc, ... I really hated these sort of complex manual steps.

MySQL - A Developer Perspective, by Peter Karlsson

This was an interesting talk about a database I've used a lot. Peter compared several storage engines in MySQL 5.1 and usage scenarios for them. This was followed by a discussion on how to optimize database queries, which were pretty basic stuff not specific to MySQL. There was nothing much that was new here, nevertheless a useful reminder of the things to consider when designing a database schema.

Netbeans 6.5, by Angela Caicedo

I guessed this talk was meant to show off as many features of Netbeans 6.5 as possible in the time available. Unfortunately, there was so much in this IDE to fit into a single talk. Quite a lot of time was wasted showing off standard features like refactoring, perhaps the speaker should have concentrated on features that were not found in Eclipse.

The compile-on-save and live-update-refresh features seemed handy. I wished there was some mention of Ruby/Rails support and Unit Testing of new technologies like JavaFX.

Skipped talks

I skipped the last couple of talks, on DTrace and virtualization, as they were both Solaris specific and I don't use Solaris.

LUNCH

Lunch was surprisingly good considering this was a free seminar. There were abundant sandwiches, desserts, salads, chicken/beef satay skewers, and pizzas. I especially enjoyed the salmon pizza.

All photos were taken using my Nokia N95-8GB.

Images and videos for Sansa E250

2007-06-20T17:16:00.000+10:00

After several days of researching, I finally found a way of encoding photos as 16-bpp bitmap images for my Sandisk Sansa E250 Media player on my Mandriva Linux 2008.1 desktop.

Gimp 2.4 (I’m using 2.4.5) supports saving of 16 bpp bitmap which works on the E250. First crop and scale image to 176×224 pixels. Then flip the image vertically, so that top becomes bottom and vice versa. Finally, save as bitmap. A dialog box will appear that provides Advanced Options. Expand it and select “16 bits R5 G6 B5″.

I can now view my photos using my media player. :)
Grant, from http://www.visi.com/~grante/sansa, provided a script to encode videos for the E250, but only in black-and-white. I modified it to encode audio to standard pcm codec, instead of Grant’s specially patched pcms. Then I run it though ffmpeg to convert the audio to big-endian format.

Here’s how my script looks like (it writes to a temporary file) :

mencoder -ofps 15 -vf rotate=1,scale=160:208 -ffourcc mjpb -ovc lavc -lavcopts vcodec=mjpeg:vbitrate=150:vhq:gray -oac pcm -srate 11025 -of lavf ${1} -o /tmp/video.mov

ffmpeg -i /tmp/video.mov -vcodec copy -acodec pcm_s16be ${2}

rm -f /tmp/video.mov

Note that older versions of mencoder, prior to 2008 I think, required the following option.

-lavfopts i_certify_that_my_video_stream_does_not_use_b_frames

in order to produce a video that works on the E250. This is no longer the case with the current version.

SanDisk E250 + Linux

2007-06-12T18:46:00.000+10:00

I purchased a SanDisk E250 Media Player with 2GB storage, mainly to play MP3s, but it also views images and plays video clips. It was on sale in JB Hi Fi, so I paid A$168 for it. I’m using it with my Mandriva Linux desktop PC at home.

I’ve only played with it a few hours. Despite good Linux connectivity support, there are some annoying shortcomings that dragged this product down.

Here are the pros and cons (in my opinion):

Pros:

Very easy USB connectivity to my Linux PC. Just set it to use Mass Storage mode and it automatically mounted when plugged into the PC. This is a VERY BIG PLUS for me, since Linux is all I use.
Clear screen, good quality sound.
Intuitive and easy-to-get-started interface.
Supports Chinese title text (but only after changing the language to Chinese).

Cons:

Does not seem to be able to group songs into directories. All songs from all sub-directories are listed. There is however a play-list that one can add songs to, but only one play-list. It does grouping based on ID3 v2 tags, so I work around this limitation by setting the Album tag using a little program called id3v2. Its still a hassle to have to manually set this tag for all the MP3s that I upload to the player.
Very limit support for image ( 16 bpp BMP) and video (MJPEG Quicktime) formats. The player comes with a Windoze-only media converter software that’s useless to me of course. Gimp can now be used to save images as 16 bpp bitmap. Video encoding is also very tricky as most other blogs on the Web are also complaining of the difficulty of encoding to the very peculiar color format. Hopefully someone out there knows the required mencoder options for this. I have no idea why SanDisk chose such weird formats for an otherwise very capable player. Very frustrating!
Built-in batteries that can only be charged when plugged into a computer’s USB port, and the computer must be left on. There is an option to use a AC adaptor, but that’s an extra accessory.

Overall, I can only give it a barely passing mark.