Large User Story vs Human Psychology

Much have been written about the problems of oversized user stories. They are commonly indicative of too much uncertainty, usually over business requirements, but sometimes over certain technical aspects. While large stories (or epics) are acceptable during a high-level planning stage, they should be either simplified or broken down by the time they are scheduled within an iteration. Otherwise, they usually end up greatly exceeding their (very inaccurate) estimates, or even becoming blocked.

As a software developer, I've been involved with a few such stories recently. In addition to the above-mentioned problems, I noticed another adverse side-effect. With each passing day on the same story, I became increasingly less motivated to observe good coding practices such as refactoring, cleaning up messy code and proper OO design. As the days dragged on, I became more tolerant of sloppy practices. I eventually finished the story and handed over a piece of spaghetti code that I wasn't proud of. In hindsight, I could have done a lot more to improve code quality.

Thinking back, I realized that this same behaviour manifested itself many times before in the past. Perhaps, I might just assume that I was a lazy bum, if not for the fact that I was practising pair-programming, and the same tendency was also apparent in my pairing partners (some more resistant than others). Maybe it was just the long hours and looming deadlines. However, my experience tells me that doing multiple smaller consecutive stories, while they may add up to the same volume of work as a giant story, does not lead to this behaviour.

Image from: weber.edu

This leads me to speculate that there is a psychological explanation. Research has shown that the human brain has two regions that compete to form a decision when presented with a choice between a small short-term reward and a longer-term goal (larger reward). The more emotionally driven region of the brain seeks the short-term reward while the rational region tells us to hold-out for the overall better outcome. Our brain battles itself to a decision. As professional software developers, we constantly have to make a similar decision :- should we take a short-cut to deliver a piece of work now, or refactor/write more tests/clean up so that the overall code quality improves in the long run?

It is likely that by breaking up a large body of work into many small discrete stories, the completion of each story provides an emotionally satisfying reward in a short period of time, even if extra effort/time has been added to ensure good quality. This extra effort/time is usually at least proportional to the original estimated size of the story, but may cost even more if the story gets so large that it touches many interdependent parts of a complex system.

In other words, a small story usually touches a small area of the code base, hence its parts are easier to visualize in one's head, and thus presents only a small mental barrier to keeping it clean. Conversely, a massive story that encroaches on a myriad of different moving parts becomes hard to visualize. Soon, problem areas stack up beyond our ability to keep track of them in our heads. On top of that, the uncertainties inherent in a large story may reveal themselves to be time-sinks. Before long, the extra effort required to deliver quality becomes a mountain of hurdles. Having spent most of our time and energy just getting to code to work, with no psychological reward from completing anything yet, it is no wonder that we become disinclined to continue.

In conclusion, large user stories work against human psychology for the desired long-term goal of good quality code. Small stories are more satisfying due to frequent emotional rewards, at the same time contributing towards the greater goal.

It is not always possible to foresee how big a story can become at the start, therefore, developers should be given the freedom to split up stories while they are being developed. Personally, I think a story should not take more than 2 to 3 days. If it looks bigger than this, split it. This I endeavor to do more of in the future, with the hope of always delivering code I can be proud of.

Testing jQuery with env.js + RhinoUnit

The previous (Web application) project I was on started adopting jQuery to implement more client-side browser behaviour. We were already using RhinoUnit for testing some generic Javascript logic, but jQuery required access to the browser DOM. Hence, env.js was used to provided a DOM to RhinoUnit. This combination would be very quick due to RhinoUnit running headlessly and not relying on a browser, which was perfect for TDD. Getting these three pieces to work together properly took some work, as described below.

Versions

• jQuery 1.3.2
• env-js.1.0.rc7, using env.rhino.js
• RhinoUnit 1.2.1

Replaced self
As mentioned in Adam Scott's blog, references to self in RhinoUnit needed to be replaced. So at the top of the rhinoUnitAnt.js, I added this:
var rhinoUnitSelf = self;
All references to self below this were replaced with references to rhinoUnitSelf.

Replaced print
Adam Scott's blog also mentioned the need to replace calls to print with self.log function. Specifically, reimplemented $env.log in env.rhino.js like so:

$env.log = function(msg, level){
    rhinoUnitSelf.log(' '+ (level?level:'LOG') + ':\t['+ new Date()+"] {ENVJS} "+msg);
};

Implemented load
For some strange reason, env.js relies on a load function that could not be found. The solution was just to implement it using RhinoUnit's loadFile. The following was added to the top of env.rhino.js:

load = function(x) {
 eval(loadFile(x));
};

Ignored global variables
jQuery and our unit tests created a lot of global variables that RhinoUnit didn't like. Instead of configuring RhinoUnit to ignore a massive list of global variables, I decided to simply remove the code that checked for this. Specifically, reimplemented ignoreGlobalVariableName in rhinoUnitAnt.js as such:

function ignoreGlobalVariableName(name) {
    return true;
}

Cleanup after tests
At this stage, our unit tests could run individually, but they failed when executed together (consecutively, from Ant). This was because the global variables that were created during the first test were not cleaned up and they corrupted the environment for all subsequent tests. In order to fix this, I modified the end of the runTest(file) function in rhinoUnitAnt.js so that global variables created in one testcase (test file) were cleaned up prior to running the next testcase.

/*****  ADDED to remember global variables that existed prior to each test file *****/
    var origGlobalVars = {};
    var varName;
    for (varName in this) {
        origGlobalVars[varName] = true;
    }
/************************************************************************************/

    eval(loadFile(file));

    var testCount = 0;
    var failingTests = 0;
    var erroringTests = 0;

    executeTestCases(test, test.setUp, test.tearDown);

    if (testCount === 0) {
        erroringTests += 1;
        testfailed = true;
        failingTestMessage(file, "No tests defined");
    }

    rhinoUnitSelf.log("Tests run: " + testCount + ", Failures: " + failingTests + ", Errors: " + erroringTests);
    rhinoUnitSelf.log("");

/******  ADDED to remove only those global variables that have been 
         added by the test file, but not those that existed before.  ******/
    for (varName in this) {
        if (origGlobalVars[varName] != true) {
            delete this[varName];
        }
    }
/*********************************************************************/

After these changes, our jQuery unit tests ran perfectly, allowing us to do TDD for our jQuery code.

Cacti : Graphing remote service WITHOUT using SNMP

What is Cacti

Cacti is a popular Web-based tool for generating graphs of various data/statistics from remote servers. It provides default "templates" for generating graphs of CPU load, processes, memory, etc out-of-the-box, mainly retrieving these data from remote machines via SNMP (Simple Network Management Protocol). Cacti can be set up to graph custom data, again using SNMP by default. In fact, most of the instructions available online assume that Cacti talks SNMP to remote machines.

This blog is based on Cacti version 0.8.7d, running on a Linux machine.

The problem with SNMP

However, I find SNMP to be very complicated to set up for anything more complex than a single value data. Personally, I find SNMP to be overkill and far too difficult to manage, especially when it comes to setting up a custom SNMP table to group data in a way that can be traversed by Cacti.

Alternative to SNMP

This blog shows how Cacti can be used to retrieve data, via HTTP, from a URL that points to a "Web service" (not the heavy SOAP kind) that returns data in a simple text format. This can be implemented using JSP, PHP, a simple Java servlet or just any Web technology. All it needs to do is to return in its response the data in the following textual format:

fieldName1:fieldValue1 fieldName2:fieldValue2 fieldName3:fieldValue3 ...

Example

Lets assume I'm hosting a Java Web application, and I want to graph the following:
- The number of requests that generated successful responses.
- The number of requests that generated errors.

Two things are required on the Java application side:
- Some sort of filter/interceptor to capture these two types of request.
- A servlet to count the number of these requests and return a response in the desired format. This servlet will reset its counters after sending each response, so that each time it returns the count since the last response. The response content type is set to text/plain and the content looks something like:

successCount:1623 errorCount:23

This servlet is deployed on the same Web container as the main Web application, and mapped to the URL:
http://remoteserver:8080/myapp/statistics

Cacti Configuration

Poller Settings

Firstly, tell Cacti to simply use Ping to detect hosts for polling. Click on Settings on the left bar, then click on the Poller tab and change Downed Host Detection to just Ping, as shown below:

Device

After that, set up a device to point to the server that is hosting the Web application. Assume it is reachable by name remoteserver. Create a device of type None as shown below, meaning Cacti will use ping to detect the server.



Note: This version of Cacti has a bug (at least on my Firefox browser) where it tries to validate that the SNMP Version 3 passwords match even when SNMP is set to Not In Use. To work around this, first select SNMP Version 3, blank out both passwords, then select Not In Use.

Data Input Method

A Data Input Method is set up to tell Cacti to retrieve the data from the servlet (using wget), and map the field names in the response text to fields that Cacti understands. Refer to screenshot below:



Name this Data Input Method as RemoteServerInputMethod. Select Script/Command as the Input Type, and type the following in the Input String :
wget --quiet --no-cache -O - http://<hostname>:<data_port>/myapp/statistics

Create hostname and data_port input fields, that are used to pass in values to the <hostname> and <data_port> parameters in the script command.

Create successCount and errorCount output fields, that are mapped to the field names in the response text. Make sure these output field names exactly match those in the servlet response text.

Data Template

The Data Input Method is used to create a Data Template, as shown below:



Name this Data Template as RemoteServerStatisticsDataTemplate. Name the Data Source as RemoteServerStatisticsDataSource, this will automatically become the data source name (see later).

Create a Data Source Item to map to each of the data fields, i.e. successCount and errorCount. Most of the settings here can stick to the defaults. The important part is to make sure that the Output Field selectors are mapped to the correct fields.

Its better to tick the checkbox under the Port on server Custom Data to tell Cacti to assign the port from the data source, so that we don't have to hardcode 8080 in the Data Template.

Data Source

Create a Data Source that connects the Data Template with the Host Device that were configured earlier. Select appropriate options for these two drop-down selectors. Do not change the Data Source Path. Enter 8080 as the Port on server.



This screen creates a Data Source that automatically gets the name as specified in the Data Template screen before.

Graph Template

Set up a Graph Template to tell Cacti how to plot the data in the graph. Name it RemoteServerStatisticsGraphTemplate, and give the graph a descriptive Title. All other fields below the Title can be left in their default values. Create two Graph Template Items as shown in the screenshot below.



Note that creating a Graph Template Item will automatically create a corresponding Graph Item Input.

The screenshot below shows the screen for creating a Graph Template Item. Simply choose the matching Data Source for the field, and enter a descriptive Text Format.

Graph Management

Enter Graph Management to create an association between Graph Template, Host Device and the Data Sources. The screenshot below should be self-explanatory.

Graph Tree

Finally, enter Graph Tree to place the graph in the hierarchical tree. This is quite straightforward. Create a Tree Item of type Host, and point it to the Host.




The new Tree Item will appear like so in the Graph Tree:

DONE!

Finally, its all done. Just click on the Graph tab, drill down the tree and see the graph of successful and error request counts.

---

SQL*Plus connection to remote Oracle database

SQL*Plus is a very handy command line tool when you want to quickly access an Oracle database or execute SQL statements from scripts, e.g. BASH scripts. However, getting it to connect to a remote Oracle database server in a script proved to be quite a challenge. This blog documents how to write the connection string in various cases.

Local database

Connecting to a local database is trivial, just use:

$ sqlplus dbUser/dbPassword@dbSid

Remote database from command line

Here's the nasty syntax for connecting to a remote database using its SID:

$ sqlplus dbUser/dbPassword@'(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=remoteServer)(PORT=1521)))(CONNECT_DATA=(SID=dbSid)))'

You can also connect using a SERVICE_NAME instead of SID.
Note that the single-quotes are needed to preserve the brackets.

Remote database from a BASH script

I needed to write a script that called SQL*Plus several times. In order to make it maintainable, the username, password and the part after the @ were specified as environment variables. Unfortunately, there did not seem to be any obvious way of setting the value of the environment variable so that the single-quotes were properly sent to the sqlplus command at run time. I had to explicitly escape each bracket with a backslash, like so:

DB_CONNECTION=\(DESCRIPTION=\(ADDRESS_LIST=\(ADDRESS=\(PROTOCOL=TCP\)\(HOST=remoteServer\)\(PORT=1521\)\)\)\(CONNECT_DATA=\(SID=dbSid\)\)\)

Because this script was part of a Java project, I subsequently moved this value to a Java properties file. This means every backslash had to be escaped with another backslash to prevent Java from interpreting it. The entry in the Java properties file:

db.connection = \\(DESCRIPTION=\\(ADDRESS_LIST=\\(ADDRESS=\\(PROTOCOL=TCP\\)\\(HOST=remoteServer\\)\\(PORT=1521\\)\\)\\)\\(CONNECT_DATA=\\(SID=dbSid\\)\\)\\)

(I know I could have just used JDBC, but there was a good reason why I had to use SQL*Plus for this Java project.)

Define a local service name to a remote server

An alternative to using the big ugly connection string is to define a local service name in the local tnsnames.ora that references the remote instance. Add the following:

DB_LOCAL_NAME =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL= TCP)(Host= remoteServer)(Port= 1521))
    (CONNECT_DATA = (SID = dbSid))
  )

to

${ORACLE_HOME}/server/network/admin/tnsnames.ora

Connecting to the remote database is simply:

$ sqlplus dbUser/dbPassword@DB_LOCAL_NAME

Of course, this solution has a few drawbacks (enough to dissuade me from using it).
- Need write permission to the tnsnames.ora.
- The application relies of custom entries in tnsnames.ora, which must be maintained together with the application code.
- Every environment (e.g. development, staging, UAT, production) in which the application is deployed to will need to have its tnsnames.ora modified and maintained.

---

Save the Net

---