Thursday, October 25, 2012

The hidden cost of bad implementation

We have two systems integrated: one processes the internet form and outputs a XML (the forms system), the other receives and stores the output and processes further user input (the frontend system).

For economical reasons the output is stored as text (CLOB) in database, violating the principles of rational relational database usage.

The alternatives were:

  1. a dedicated table (entity) that would need to be changed everytime the other system changed 
  2. table with attributes: form_id | field_name | field_value 
  3. dynamic class with .toXML_CLOB() and .readFromXML_CLOB() methods to parse the XML

The first idea was rejected because it was unpractical - changes to XML output would break the integration. The changes to forms system are unpredictable and the solution was rightly rejected.

The second idea was used in some systems previously in our company and served us well.  I've maintained such systems for 4 years. I liked possibility of mass update of fields in case of :
  • application needing different format of value
  • app ceasing to accept some values/value ranges

The third idea is riddiculous in static java world.

The rationale

The reason simple text storage was choosen was that very little bugs or updates were expected. This turned out to be true, because of enormous experience of team making this decission. Bear in mind hovewer, that it might have turned otherwise.
The type of decission is beeing called 'engeeneering 95% decission', which means that is is solves 95% of requirements/problems.

The misscalculation

But was it truely the effective decission or unwise bow to the budget? Please notice, that the implementation cost of field_name | field_value table is not very much higher than CLOB. I'd say 2 or 3 times more work (max 40 hours more). And we're talking about two year project in team of 15 people. And we're talking about integration.

Integration is something that can't be easly changed once it starts being used.

The hidden cost

What was not taken under consideration are the lost opportunities of good software. The value and stability of well written app is stressed across many books and lectures in the field. Yet it gets forgotten so often.

Let's explore the opportunities of good, simple implementation field_name | field_value. Those are things easly done with this implementation and very hard to be done upon XML CLOB:

  • mass update of date format
  • when a field may no longer be empty, a mass update of default value is easly done
  • analysys of values in specified field is easly deliverable
  • one may search for a form with specified e-mail
  • easy duplicate values (e-mails to be precise) detection and rising alerts
  • additional business-critical validation of user-entered values is possible. Some data is unavailable for forms system but is available in frontend system. Some critical assumptions could be tested (and re-tested) after form has been submitted.
  • any request to alter or analyse forms data would be reasonably priced. Reasonable pricing of simple operations is good. Prohibitive prices for simple things are very, very bad for business relations.
  • alternate ways to deliver the forms data to the system would be easly possible - the frontend system would have a possibility to become some kind of center of processing data. Good for business, right?


  1. Good implementation delivers higher value to customer who paid for the system.
  2. Good implementation allows better business-to-business relations
  3. Good implementation allows system to grow and become a importand bond between business partners

The hidden cost are those lost opportunities.

Tuesday, September 11, 2012

Fix on four branches

Commit this to following four branches

Yesterday one of developers in my project got a message to commit her changes to four branches in central repository. I said "you must be kidding me" and immediately went to investigate.

We develop new features on separate branches and cherry pick them to release candidate branch that is put to testing and possibly deployed on production. There is also a "master" branch for non-client requested changes and a branch for immediate fixes on production.

As the fix was needed by two different changes (branches), two commits seemed reasonable. What about the master and "fixex" branch?

It turned out that the developer in charge of changes process requested immediate merge to those two branches in fear of someone taking over him (less experienced programmer) might have trouble applying the patch without errors upon master and/or fixex branch also needining this fix.

So some part of new feature was developed on branch, needed the fix, but the fix was also to be applied to other branches is case the changes were to be added to those branches.

This is bad. This is anticipating a problem that did has not yet happened. This potentially breaks the stable branches (not requested, surprise change). This a real problem of trust and leadership.

Don't do it, please. Trust your co-workers to do the good job. Anticipate problems by documending prodedure and quirks. Advice and help verbally. But do not spoil the code.

Wednesday, May 30, 2012

Simple statistics alghoritm that beats A/B methodology hands down

The alghoritm shows [Buy me!] button in 3 different colors. If user clicks it, "click through ration" for color is risen. It user does not, it's lowered. After 100 clicks (and thousands of visits) one gets very good estimate of both click througrh ratio and which button works best.


Friday, May 25, 2012

Friday, February 3, 2012

Memory efficiency in java

I've encountered great resource about memory efficiency of huge java applications:$FILE/oopsla08%20memory-efficient%20java%20slides.pdf

Main points of the presentation are:

  • Representation overhead is sometimes huge
  • Representation overhead not always diminshes with data size (!)
  • Caches should have bounded size
  • Many tiny strings consts you more than you think

Upon reading this paper you will

  • Understand why your simple, low scale java app takes 2GB RAM
  • How to use memory more efficiently
  • Why and how to use good old mmap in java
  • .. have a few good ideas for refactoring

This has been a very good and interesting paper for me, despite the fact I don't like java much. There is hacking spirit and great amout of war-field experience in it. Go read it if you are java pro.