Droidcon Bangalore 2011

Had great fun attending the first Droidcon in India at namma bengaluru. Here's my somewhat longish take (in schedule order):

The Good:

• Diogo's keynote on Cyanogenmod's growth and potential. Interesting to know that they support 80 devices. My cheapo LG Optimus P 500 isnt one of them, however :(.
• James Hugman's talk on making X-platform apps suck less. Bonus: the Kirin framework. Interesting idea of writing business logic in javascript, but using the builtin webview-based javascript engine as the runtime. Definitely worth checking out. 
• Hard Earned Android experiences by Khashif. Lots of sane advice here. Some interesting face-offs with his own (seemingly very young) CEO made me feel like I was at a tennis match but in all a nice talk. Most controversial point: optimizing database writes. 
• New features in ICS: Good round up of the new features. Had to ask the question on the new Google expectation to move physical buttons to soft ones even on phones. More on this below.
• App demos: Selected output from the Hacknight event that HasGeek held as a leader for this event. droid2chrome was interesting, as was the bookcompare (i forget the actual name). The former was interesting because most of the team members didn't want to do android and yet managed to do something; while the latter was interesting cos they actually screen scraped book prices from indian vendors!
• Anand Virani's end-of-day talk on the mobile growth wave: I caught the end of this talk but clearly this is a man who can talk and present well. Nice projections on how the mobile world is shaping up. Some sharp questions on open sourcing Qualcomm tech which he quite adroitly side stepped without pissing people off :) 
• Tibaut's keynote on i'm not sure exactly what (it had strands of droidcon history,android growth and political incorrectness), but it was by far the most entertaining and engaging talk i've seen in a while :). He made an interesting connection between Minitel - the French govt sponsored draconian internet offering that was in blue, green and while colors - and the recent fragmentation of the mobile market between blue(nokia), green(android) and white(apple). I'm not really sure what his final slide asking us to challenge the status quo of mobile really meant, but he DID get a few hundred indian geeks to actually talk to each other! That's an achievement for sure.
• IPR aka Who let the lawyer in the room: Again, I caught the last 5 mins of this talk, and immediately felt I should have taken it in whole. Malavika looks like she could be the india PJ and she likes open source. I asked her about groklaw's impact on relations between devs and lawyers, to which her quip was: "I think its great because usually developers never read the terms and lawyers never see the apps"! Nice.
• Indoor tagging engine: Was intrigued before the session and liked what I saw. Tagin is a project that allows identify indoor locations based on the surrounding wifi beacons. If this thing is accurate 50% of the time, it has great personal potential. The only downside that I see is that you have to have wifi on all the time. Barring that, however, I can think of some simple applications that extend the GPS's accuracy - especially within buildings that have lots of wifi routers - like apartments. Example: "Ding! you're near the mailbox, do you want to drop your bills off?"
• Creating apps that work at all sizes: Lots of good information here for UI devs. I couldn't help but feel that the raster vs vector wars were being fought all over again on mobile, however.
• Android and Arduino: Very enthusiastic presenter who knew his stuff and presented it in an honest, no-frills way. Learnt how easy it was to interface with arduino more than anything else; and that all this stuff (and knock-off cheaper boards) were available locally. Now all I have to do is magically make my family disappear so I could just hole up playing with these toys :)

Ok, could be better:

• Tech challenges of apps in limited connectivity: This is a topic that could have been a lot of things - especially given that mobile is becoming cheap and offline apps and syncing them up is something (IMO) a lot of developers should be worrying about. While the initial "here's the problem" part of the talk was good, the rest felt like reading a Craig Larman architecture book - dry, high on theory and boring as hell. If you read between the bullet points, there were gobbles of information just waiting to be let out. They stated the problem, stated the things you have to consider and then stopped. A better approach would have been: "here's the problem, here's the things you have to worry about, and here's what we did. Now lets discuss". My other takeway: offline and sync should be a framework problem, not an app developer one. More on this below.

Meh:

• Android Service Patterns: Was over my head because I'm an android tourist dev. However, I'm surprised someone built a new IDL to define service contracts. Doesn't the world have enough of those already without android creating one more? God forbid some (very possible) future where EAI vendors have to support AIDL! Oh look, the horrors of integration combined with those of android fragmentation!
• Honeycomb codelab: There were glitches in the setup - primarily because most people did not have the ICS Dev setup on their machines. It might have helped to have a note on the schedule asking people to prep for this one. Also, there was a whole lot of "how" and no "why" at all in the content - but maybe that's because I don't know the nuances between honeycomb and older versions. And: it WAS called a codelab, so maybe I shouldn't expect theory or reasons. Regardless, it would have been nice to have a brief this is why what we're going to do is important/fun/great. It lacked that oomph, IMO. Personal peeve: Seemed like the presenter was a bit miffed that "in this day and age" we dont have connectivity. The whole tutorial was online, with no way of even getting offline as it was on appspot. We ended up curl-ing the pages so that our unconnected buddies could follow along.

• Note to organizers: Tell people to setup machines prior or tell them about the thumb drives at registration
• Note to speaker: Do not expect a perfect environment; have fallbacks. Most of us were trying very hard to manage to follow you, but a little understanding would go a long way.
• Note to self: Read up on the content to know the why before attending a codelab instead of bitching about feeling listless at it :)
• Android app: A deconstruction: I was distracted; hence the meh. Seemed like a pretty ok breakdown of building an app for AAA. However, I've attended a similar talk earlier, and the guy then made me want to have the app; so I was interested in how it was made. This was a lot of the hows; and a lot of it was pretty standard stuff when looked at through the lens of web development. My bias, I know.

Did not attend, would have liked to:

• Android Multimedia Internals
• Pricing models for apps
• Continuous delivery for android apps
• Cloud to device messaging.
• Android Memory optimization
• Enterprise App Development, admin api.
• Close to metal programming with the NDK.
• OpenGL on Android: My colleague attended it and liked it.
• Humanizing Android.
• Extending android with new devices
• From stock to CM: Ericsson.
• Smartphone Platform Frameworks.
• Android porting for dummies.

Just plain Did not attend:

• Android UI secrets: Heard from another guy that this was not as secretive gyaan as expected, but that's second hand info.
• Sensors on UI: My colleagues attended; they were not the right audience, however. Must have been interesting to the the right people.
• Demystifying mobile advertising
• Infusing Android with social.
• ICS Camera and connectivity.
• The Phonegap session.
• Android products with TI tech.

Sitting through some of the sessions, I couldn't help but feel that Google should:

• Think of phones as phones, not as small tablets. Some of us actually use these things when we're not looking at them, and some of us actually (gasp) LIKE  "Call" and  "hang up" buttons. Not all things are meant to be used with two hands.
• Create offline and sync apis: I'm not sure if there is an api for these in the android platform; but if there isnt, there should be one. I spoke to at least 2 app developers building their own solutions for these problems, and clearly, these are not application-level problems. "In this day and age". 

Surprises:

• Robosoft: A company based in Udupi, doing Android! Go figure. I might just retire there - if they'll have my sorrily backed-up-in-tech-skills behind :)
• Copperspiral; Fronted by a very informed and polite Vineeth, this RFID comapny was the first that showed up in India when I googled for local vendors; and I had quite a few informative chats with him. I'm not an RFID expert, but he's the guy to talk to if you are a hobbyist, methinks.
• Github: They sponsored the party (which I couldnt attend owing to stupid excuses such as having a family) was conspicuous by they absence. Sure, the cut outs were there. Maybe there was more of them at the live music show? Thanks anyway, github. Love your software, love your support.
• The 2 guys volunteering to set cyanogenmod for random strangers: Completely unplanned, apparently, and completely cool. My LG still didnt get love, sadly.
• Dextera: I learnt that this company from Cochin built a Siri-clone called Iris while at their CEO's session. Have to definitely check it out.
• The venue: Did not expect this good a venue, actually. Except for the small room, a top-notch place; and still a little away from the madding crowd. I particularly liked the steps that also doubled as an access ramp on the way to the courtyard. I'm by no means an architecture expert, but I have not seen that before.

The experience:
Hats off to HasGeek for the organization. I'm one of the entitled bunch that gave them grief about the lack of net connectivity, but Kiran's saga of their attempts to get wifi for a month won me over. Kiran, if you're reading this, I'm the guy who was willing to pay Rs.500 more. You have my apologies.That said, here are my suggestions:

• A panel discussion would be nice. I know you guys had plans for it, but it didnt work out. Now you know the people will come, so have one for Droidcon 2012.
• An app for the event. Methinks this could be community-sourced. I met some guys from Cochin who were in mobile tech training. Seems like they can whip this thing out :)
• Prior notice for things like the codelab. Update: I just read your blog. You did tell them in advance.
• Content: Android scripting, esp JRuby on Android aka Ruboto.

In all, a fun two days.

Jack and Turing completeness

From the previous posts on Jack's primitive capabilities it should be pretty obvious that it is turing complete. It supports sequences and selection naturally; and allows iteration of any node's children.

The intent, however, is to NOT use Jack as a primary programming language. I do not see a hello world in Jack's future - at least not in the conventional sense.

Jack's Turing completeness is intended to provide a full complement of computing power so that program logic may be effectively TRANSFORMED. The more appropriate set of hello world would programs may be:

• A program that edited another program and represented the editing as a sequence of Jack steps
• A program that reasoned about another program using Jack
• A program that expressed the history of a program and exposed its "why" using Jack
• A program that expressed the future roadmap of a program using Jack

Jack: the whys and wherefores

As I pen down these thoughts that I have about Jack, I realized that there's a little bit of the "losing the forest for the trees" going on - while the initial posts were  what-oriented, the latter ones have been decidedly how-oriented. The "why" is kinda sprinkled around. This post attempts to correct that gap.

Jack is the culmination of a series of vaguely connected concepts that have been swimming below the surface in my consciousness. Some of the individual strands are:

• Code today suffers from want of better abstraction for itself. The simplest manifestation of this is the conflation of its persistent, modeling and editing formats into one single form - text. I posit that treating code in its true form - an AST - directly will reap higher benefits.
• There are sub-strands to this where concepts from Subtext (and the whole language workbench movement, for example) finds resonance with my thinking and therefore has been subsumed into it.
• True code structure is latent and the structure made apparent by its organization into files and folders is misleading. I posit that all code will look like graphs (a la social networks) once we represent (and visualize) the actual relations between code components. At that point, graph CRUD techniques can be used to manage the code and graph analysis techniques can be applied on it to reduce the comprehension overhead.
• Today's applications require connecting together at least 4-5 different languages. There is no conceptual glue to hold these together, splintering comprehension and adding to the accidental complexity. I posit that treating apps as a graph containing code components will allow for easier comprehension and maintainability.
• Today's development practice is woefully snapshot based. There is no way to represent the code's past (so that you may learn from it or make corrections easily) nor its future (so that you may plan for it or code in a certain way)
• The average developer follows Reality Driven Development, no matter what methodology they say they follow. 
• All of this has direct bearing on EXISTING code. Most approaches to bettering software development talks about tools and techniques that you can use on your NEXT project. I'd like to come up with tools and techniques that you can use on the CURRENT or PREVIOUS project, or even that ANCIENT one from the '80s - the one that's already in production and making money for the business and will never be replace (at least not completely) that you have to either maintain or talk to. This is also the hard and unglamorous side of the development cycle - the maintenance side. However this is also the place where code spends most of its time and yet is mistreated much. 

Jack is the code name I use to represent my solution to these issues. It started out addressing the format and representation issue and has burgeoned into including the rest of the concerns. It therefore has gained the larger qualities that any solution to the problems above should have:

• It should not favor any particular language or framework - at least not design and definitely not in the ultimate implementation.
• It should not introduce another language either; instead use existing language runtimes as host environments and introduce ways to work with the code by adding metadata and the ability to programmatically refactor it.
• It should introduce a runtime environment that is live in the same way a LISP repl or Smalltalk IDE is. Compiled environments should get as close as possible to this ideal. This is fluent.
• It should allow for models of the code to be built at varying levels of abstraction from the code itself. These models would be used to understand code and build it. It should therefore subsume disparate technologies used to build working apps or make it easy to add new ones into the modeling platform.
• It should allow incremental adoption of itself. No big bang, lose-your-religion type steps because we're dealing with maintenance code and developers following RDD.
• By definition, therefore, all tools and techniques within should have models for incompleteness: incomplete code, design, architecture, relations, etc. This should be used to help comprehension of the code in "comfortable bite sizes" and to allow description of future code whose specifics are not yet known.
• It should have a native access to the history of the code so that comprehension can follow. This is the reason the previous post has the scm FFI.

So there you have it: the WHY and WHAT of Jack in one post.

Jack: some new thoughts, a few iterations

Some strands of thoughts have come together of late as I re-read my posts about Jack, watched Subtext and talked about CodeViz to people. Things congealed into place as I revisited OMeta and decided to give it a shot as a prototyping platform.

Iteration 0

As a precursor to trying OMeta out I listed out the features that Jack was supposed to have based on my posts. Here's what I came up with:

Transforming this to OMeta syntax, here's what I came up with:
As you can see, some compression has happened from my BNF-ish syntax to the OMeta one.

Iteration 1
Pumped by this quick success, I immediately started counting my unhatched chicken. How would people easily share Jack code, I wondered? I'd already thought of YAML as a storage format so, I quickly wrote up the YAML version of the function from above:

I also came up with the whole slew of pipelines that would bring Jack (and some part of Fluent) to life:


text syntax   --JackTxtParser-------> JackAST
JackAST       --JackUIRenderer------> JackViewTree
JackViewOp    --JackUIModifier------> modifiedJackViewTree
JackViewTree  --JackInterpreter-----> JS Evaluation
JackViewTree  --JackCompiler--------> JS Source
JacViewTree   --JackTxtGenerator----> text syntax output

Iteration 2
Then reality set in. What are the primitives I'd need to build for this to take off? A reverse read of all jack posts revealed the following set of "things to be built":

• A primitive conditional
• The ability to make a module out of anything and vice versa
• make module (code list)
• devolve module into codelist(module)
• A way to denote a function as optimized via a Foreign Function Interface (FFI). This could be in the fact language, but this means the fact language and the compiler should talk.This is probably required anyway; the interpreter should be able to query the analyzer for facts' veracity.Note: This facility is like annotations, but doesn't entail arbitrary side effects as the annotation processor allows through arbitrary code running.Each FFI, however, should expose a way to call the underlying optimization, with ways to map values to and forth.
• The ability to refer to any single piece of code: built into the structure already. url tbd
• The ability to refer to any set of code pieces: define a continuous code range, define an arbitrary list of code pieces (is this required)
• The ability to comment on such a set of code pieces: ie attach a comment to a code set. this is somewhat similar to the modularize reqt above
• An FFI to scm tools with the basic functions supported
• check in/out
• commit
• branch
• merge
• snapshot
• A functional version expression
• The ability to represent WIP code
• Most importantly, an interpreter that supports all this

Iteration 3

Then I took a step back and looked at the complexity growing. Could there be some abstraction done? Here's the outcome of some furious (re)thinking:

• There are nodes
• Nodes have attributes
• Standard attributes are:
• [id] is used to uniquely identify a node. it can be system-generated or user-provided
• [comment] is used to provide a comment about the node.
• [fact] is used to state a fact about the node using FOPL; and is typically used to derive some "higher order knowledge" about the node
• [name] is used to provide a referencible alias for the node.
• [kind] is used to identify the type of node. this may be dynamically assigned to implement duck-typing.
• [context] is used to "run" or execute the node.
• Other attributes can be added at will and used in execution
• A collection of nodes is also a node, and therefore can have the same attributes and be executed similarly.
• Standard nodes are:
• base:
• block : a continuous sequence of nodes to be executed one after the other
• if : a node that conditionally executes one of its child nodes
• define : a node that adds a name to a node
• function : a named block
• module : a collection of functions
• app : a collection of modules
• meta:
• group : a node that groups other nodes (and optionally names)
• split : a node that splits an existing group
• insert : a node that inserts a node at a given point in a collection
• delete : a node that deletes a node from a collection
• versioning:
• checkout : a node that checksout a version of its input node ref from scm
• commit : a node that commits a version of its input node ref to scm
• branch : a node that branches a version of its input node into a new branch
• merge : a node that merges two node refs
• standin : a node that can stand in for any other node. to be used for nodes that dont exist yet 

Thoughts

I reached this far and realized something larger abstractions are possible:

• Jack code could be stored in any data format that can handle trees - backward compatible data format!
• Jack could use any language as host - backward compatible code! Jack is "just another scripting language on top of $my_fav_lang

So, it sounds like:

• Jack's true role will be to consolidate sequences to specific hosts, create and break interfaces and manage the change that happens - a sort of super shell.
• Ultimately Jack should enable code comprehension and legacy support.

Maybe Jack should be renamed Glue.

Intent to sms contact

Write an android intent to allow sms-ing a contact, cos the default phone app doesnt have one.

Update: This has been Implemented. Woohoo!

Physics of Software: a review

I stumbled upon an interesting series of blog posts that attempts to bring in a "physics" for software. This attempt in and of itself seems to me a worthy cause as IMO Computer Science could do with some foundations like these. There's too much on the engineering side and less on the science one.

The other reason I liked the series (which I'm still reading) is my own latent ideas on using FEM-ish analysis on the structure of software.

Halfway through the series, however, it seems to me that the analysis is overly reliant on real world physical concepts. Software having a center seems fine, as does mass and inertia; but extrapolating that to speed and acceleration seems contrived.

My own take is that software DOES have a physics of its own. However, concepts from Natural Physics may be applicable as analogies, but not directly to express this phsyics. My suspicion is that the physics of software would be multi-dimentional (which the author of this series concedes as well), graph-like (which he also alludes to with his attraction-repulsion diagrams), fractal (also stated, but not evolved fully) and much in need for a set of operators that define the effect of change (which I've not found yet in these papers).

Still, a very worthy exercise indeed; and one that introduced me to Coplien's forays into the area from the late 90s. Bonus: Reinteroduced me to Christopher Alexander's Design pattern concepts.

PS: While the posts dont talk about change, I was led to them via link from Micheal Feather's attempt to quantify design change by measuring correlation between methods/classes changed in a single commit.

On self containment

I like systems that are self contained. No unnecessary dependencies.

I woke up today and decided that lispdown needs an implementation. After all, the rules are simple; should be a shell script, right?

True; but I couldn't help thinking: shouldn't I write lispdown in Lisp? Whoever needs it (lispdownville: pop 1) obviously will have lisp - and that's all they'll need. Sure they most probably will have bash, but why use it at all?

This is the same kind of thought process that led me to build cuke4jas. I found joy in the fact that the application being built and the tools being used to build it are both from the same base platform/language.

I know everyone has to be a grasshopper first, but...

... could I do it matrix "I know Kung-fu" style , please?

I bull-in-a-china-shopped my way through SICP so that I can get to the metacircular evaluator bit as soon as possible, but now I see even more clearly that each chapter firmly builds on the previous one in this book.

As I read the implementations of eval and apply I realize I have to go back and read the foundational material before I can expect to just grok them. My kingdom for the knowledge packs from the Matrix :)! 

On a personal note: Remembering 9/11

It's been 10 years.

Like many in our generation, I remember where I was and what I was doing that day. I know Latur probably had greater deaths, as probably did Bhopal; but this is the one that I was closest to.

I lived in Parsippany, NJ then and remember heading out to my car to go to work only to be told that the towers were bombed. I remember watching in disbelief as people jumped from one certain death to another. And then the videos of the plans actually crashing. And then the short messages people sent their dear ones from the planes. I remember a nation coming together immediately.

I remember walking into our regular diner the weekend after 9/11 and being asked: "You're indian, right?" by the guy who'd seated us every time before. If I were him, I'd ask too. I remember the Indians touting American flags on their cars for the next few months.

I remember feeling sad that this tragedy of epic proportions was becoming a media circus, yet being so part of it that I couldn't look away. I remember going to Ground Zero a month or so after 9/11 and feeling viscerally affected - the scene left enough clues as to how it happened: from the huge gaping hole with twisted steel and crumbled slabs of concrete to the charred buildings nearby I could imagine the myriad personal horrors that the people who were there on that day went through. I remember being annoyed by the street hawkers who had already started selling touristy swag nearby, but was numbed enough by the event and the place themselves to ignore such pettiness.

I remember going to Ground Zero last year and still feeling the same as before. I remember being a tad ticked off that the scene didn't look much different from before - maybe it had, but where was the monument to all the people who died that day? Where was the imposing answer that showed that we as humans will not bow down to terrorism?

I remember the firemen being nice to my kids and posing for pictures (they asked to sit on the fire engine), but couldn't ignore the thought that maybe some of them had friends who were no more. And I still couldn't understand the picknicky people swarming the site nor the hawkers with their "I've been to Ground Zero" T-shirts and mugs.

Fiercely apolitical, I have actively shunned news media in all its forms since my college days, but I couldn't get enough of the coverage of the Bin Laden capture. Real or not, it gave me closure to those events from a decade ago - and I didn't even lose any dear ones.

I cannot remember  a date to save my life, but I remember going to the World Trade Center on July 4th 2001.  I was badly hung over from the previous night and the meal from the restaurant at the top of the tower set me right so I could enjoy the view. I remember there being barbed wire surrounding the deck to prevent jumpers - seems kinda pointless now.

I kept the ticket stub to the towers and joke to my friends that I'm keeping it for its antique value, but whenever I happen on it while sorting my keepsakes, I'm taken back to this day, ten years ago.

A personal exploration on Java's Enum declaration

What:

A personal exploration on why Enum is declared as Enum<E extends Enum<E>>

Why

I stumbled upon this interesting question a while ago when a co-worker showed it to me. Googling it provied enough links to satisfy my initial curiosity, but recently the question surfaced again; so I tried to implement my own version of the explanation provided online to "see for myself".

What follows is largely based on Angelika Langer's FAQ and can be thought of as my worked-out version of that page. My enum class is called Enum1 to avoid clash with the SDK's version; and has been tested with JDK 1.7.0_07.

The exploration

Say you're the Java team and you want to support syntax like:

    enum Color(RED, GREEN,BLUE);

... so that it could be used like so:

• As constants : Color.RED ... which means that public static constants must be available
• In variables : Color color=Color.BLUE ... which means that it must of type Color
• In comparison : color==Color.BLUE ... which means it must implement compareTo()

Further, say you intend to add this feature with as little syntactic sugar as possible; so you want to convert the basic enum syntax alone via a special operation in the compiler and generate everything else using standard Java language constructs.

The generated code would look something like so:

    public class Color implements Comparable<E>{
        private static final Color RED;
        private static final Color BLUE;
        private static final Color GREEN;
        private static final Color $VALUES[];
        static{
          RED = new Color("RED",0);
          BLUE = new Color("BLUE",1);
          GREEN = new Color("GREEN",2);
          $VALUES = (new Color[]{ RED, BLUE, GREEN });
        }

        private final String name;
        public final int ordinal;

        //private since nobody outside needs to use this
        private Color(String name, int ordinal){
            this.name = name;
            this.ordinal = ordinal;
        }

        public final int compareTo(Color c){
            return this.ordinal-c.ordinal;
        }
    }

If you wrote more than one of these, you'd have quite a bit of logic that's common and could be extracted out:

• Each item in the enumeration has a name and an ordinal.
• Each item is built privately and the instances built statically are made available publicly as constants and via an array
• The constants have comparable values (optionally internal)

So you could write a base class like so:

    public class Enum1<E> implements Comparable<E>{ //line1
        protected static Enum1 $VALUES[]={};

        private final String name;
        public final int ordinal;

        protected Enum1(String name, int ordinal){
            this.name = name; 
            this.ordinal = ordinal;
        }

        public final int compareTo(E e){
            return this.ordinal-e.ordinal;  //line 2
        }
    }

... where Enum1 needs to be defined with a generic type E because any implementation of Comparable needs to be generic. Doing that, however will cause the compiler to throw a warning that line 2 above may be using unsafe operations. This is because Enum1<E> is equivalent to Enum1<E extends Object>; so we could potentially send in any subclass of Object into compareTo().

To fix that, we'll need to constrain the type of E to Enum1s or its subclasses. This can be done by changing the type parameter in line 1 to : Enum1<E extends Enum1>. The modified code thus becomes:

    public class Enum1<E extends Enum1> implements Comparable<E>{   //line1
        protected static Enum1 $VALUES[]={};

        private final String name;
        public final int ordinal;

        protected Enum1(String name, int ordinal){ 
            this.name = name;
            this.ordinal = ordinal;
        }

        public final int compareTo(E e){
            return this.ordinal-e.ordinal;  //line 2
        }
    }

Now Color can be written as:

    public class Color extends Enum1<Color>{
        public static final Color RED;
        public static final Color BLUE;
        public static final Color GREEN;

        static{
          RED = new Color("RED",0);
          BLUE = new Color("BLUE",1);
          GREEN = new Color("GREEN",2);
          $VALUES = (new Color[]{ RED, BLUE, GREEN });
        }

        protected Color(String name, int ordinal) 
            { super(name,ordinal); }
    }

...and test code can be written like so:

    public class TestEnum1{

        public static void main(String[] args){
            Color c1=Color.RED;
            Color c2=Color.RED;
            Color c3=Color.BLUE;
            Color c4=Color.GREEN;

            System.out.println((c1==c2));       // 0
            System.out.println(c1.compareTo(c3));   //-1
            System.out.println(c4.compareTo(c3));   // 1
        }
    }

We can even write another enum like so:

    public class Vehicle extends Enum1<Vehicle>{
        public static final Vehicle BIKE;
        public static final Vehicle CAR;
        public static final Vehicle BUS;

        static{
          BIKE = new Vehicle("BIKE",0);
          CAR = new Vehicle("CAR",1);
          BUS = new Vehicle("BUS",2);
          $VALUES = (new Vehicle[]{ BIKE, CAR, BUS });
        }

        protected Vehicle(String name, int ordinal) 
            { super(name,ordinal); }
    }

... and enhance the test code to check out things like the ordinals of two Enum1-derived enumerations not matching even though the numerical values are same:

    public class TestEnum1{
        public static void main(String[] args){
            Color c1=Color.RED;
            // ...

            Vehicle v1=Vehicle.BIKE;
            // fails with error: incomparable types
            System.out.println((c1==v1));
        }
    }

Pending Question

So I've still not figured out what the final <E> in:

    public class Enum1<E extends Enum1<E>>

... is for, considering the code above works without it.

Jack: Allow WIP pseudo code

You know how you sometimes mock up code with some keywords to represent the structure of the logic but then put in hand-wavy text for rest of it because you've not fleshed it out yet?

Jack should support that directly. Something like:
if --calamity happens--
    --dont panic--
    --call 911--
end

Just a hint of an idea at this time. I'm not sure if -- is the right sigil/delimiter to use. "<<" seems like a good one from personal experience, but it has bad web mojo :). 

Lispdown

As I read SICP, an idea arose in my mind to create a ()-less lisp. Its been done before, of course, but I was not really trying to redesign the language as I was attempting to remove visual clutter. To that end, I present:

 Lispdown. Its lisp, and its like markdown, get it?

This is still WIP, so if by some bizzare chance this gets into HN or reddit, bear with me. Its not done yet. In fact, its vaporware at this point. Just a concept. And a strawman at that :)

Alright, here we go:

• Lispdown is to lisp as markdown is to html. Therefore, valid lisp is always valid lispdown. If you cannot use lispdown syntax, just use lisp's.
•  Known forms are like begin-parens; add an end to finish them (Thanks ruby!)
• A list is formed by any set of lines that has some sort of list begin marker(each line is an element), or a line separated by commas, or a part of a line in () or [] or {} separated by spaces or commas
• A pair is formed using a colon to separate the two bits. this is only for readability as a space will do just fine.
• I've yet to figure out how complex trees of function bodies can be reliably converted to lispdown without introducing python-style whitespace interpretation.

Update: There is now a Lispdown implementation

SICP: The first pass

I just finished my first pass through SICP. Well, "finished" might be a bit presumptuous because I skimmed through the latter chapters, but that was intentional: I couldn't wait to figure out what the book was "all about" -  especially the metacircular evaluator bits - that I HAD to skip ahead to find out if "the butler did it".

And boy did he do it! I havent read the book in detail at all, but already I see some core concepts that I had about programming being presented inside-out. For somebody who didn't grok polymorphism (for example) unless it was mentally transformed into vtable lookups, this feels like looking at a sculpture from the inside - while its being made. If that makes sense at all.

Let me try to explain. And before I go further, let me add: its not that its all functional programming (although the functional bits that blew my mind I've already blogged about) nor the lispy geekiness of it. Its more fundamental than that.

The method of imparting knowledge on programming that I've encountered to date has been additive: you're introduced to some basic concepts like assignment, looping etc; then taught how to string them together in expressions and statements; then string these into functions and procedures; and so forth. SICP seems subtractive in comparison: it states the problem and set out to decompose its solution into smaller bits till the basic statements are arrived upon, at which point some primitive "forms" (such as cond() are introduced for use). This would merely be bottom-up vs top-down if not for SICP showing along the way that programs and data are equivalent by creating a data store mechanism in pure functions, demonstrating data-driven programming (command pattern implemented using config file+ interface + n implementations for you java guys) using table lookups, shown how symbolic programming could be done with examples in calculus and circuitry, and finally demonstrating simulating a real programming system with stack frames, VM et al.

No doubt the internal reason is that functional programming works that way, but this is a powerful way of presenting programming for two reasons:

1. It IS the right way to solve problems - especially large ones
2. If you present it as the only way to do things - as the book does - new programmers wont know any worse :)

There is also a marked "peeling the onion" feel to the book - each chapter builds on the previous one. Here's the "storyline" that I've figured out to date:

• Write programs by breaking them down into their constituent functions
• Data can similarly be broken down using abstraction
• Programs = data and vice versa
• Programs may require state. Introducing state has its advantages and disadvantages
• State => environment, and here's how you go about building one.
• Now that you know how to build an environment, here's how you use it - to build a language evaluator.
• Finally, here's how you'd do it a real world machine (I could be wrong on this one - I just read the heading "register based machine").

A emacs+clojure guy that I met over the weekend felt that the drawback with SICP was that there was no clear direction in the book as to how this knowledge can be applied. Maybe so, I'm not sure at this point.

When I bought the book I had a couple of goals in mind:

• See what the book is all about given that it has all this fame
• Learn Lisp in the process

The first one of these has been achieved, I think; and now that I've "got" it, I plan to go back and read the book in earnest, trying out the exercises along the way. The second goal, however, seems a bit misplaced. The book certainly uses lisp, but is not about lisp. I think I will still learn the kernel of lisp that's pure and universal to all its dialects - but that's about it. Now that I think about it, that's all the lisp I probably need :)

Having understood what the book is about, though, I think a reframing of goals is in order:

• Understand program decomposition and data abstraction as espoused in the book. Try to use it in a non-trivial application
• Try out exercises using Racket and post work to github (A la vru3dd)
• Understand the language evaluator chapter. Try to use it on a toy language if that's possible
• List how the concepts in this book are applicable in daily programming.

On Software Testing and the case for the Common Test Scenario Database

TL;DR

As a participant (and sometimes contributor of ideas) to Test Automation efforts (Strategy, tool selection and implementation), it has been my long-standing opinion that the true gains from automation can be achieved only when Test Scenarios (enshrined as exemplar test data) are a heavily reused resource across the organization. This page is an attempt to "arrive" at that opinion by progressively optimizing the software testing process.

I take the example of testing the hotel search feature and progressively optimize it through 4 stages to illustrate the value of each such optimization; and the value of shared test scenarios in particular.

What is Software testing?

Software Testing is the art and science of validating that a particular piece of software behaves as expected. Since most words in the previous sentence are loaded with meaning, I'll dissect it:

• Art and science: Art because some of it is based on the skill of the tester and science because most bits are tractable into well-defined, repeatable procedures with predictable outcomes
• Validating: This might involve the atomic step of input-process-output, or a sequence of many such atomic steps (typically known as "following a script"). Each such step (or steps) entails comparing an expected value (or more generally outcome) with a known one
• Expected: This implies that there are some requirements that the software is supposed to meet which is an accumulation of the most important (or all) of the comparisons mentioned above.

A Concrete Example - Testing Hotel Search functionality

Any travel website has a hotel search functionality. You enter a destination, start and end dates and number of guests; and the website returns a list of hotels. Each hotel card has some details (name, location, address, picture, reviews, rating, description etc); and the list itself has some details (count, sort order, #pages, etc).

Notice also that there are some implicit inputs (eg: choice of POS, choice of currency) and some implicit outputs (eg: sort order).

To truly test this feature, you need to validate that:

• The functionality is right:
• The list is as expected (has the expected number of hotels which are in the expected order, paginated at the right place, etc)
• Each hotel card is as expected (Has the name as expected, image shows up, description is correct, etc)
• The display is right:
• Visual characteristics for each data item is correct
• No visual elements overlap
• And so on
• Adding the feature didn't break anything else:
• tests on features that already exist pass
• tests on features that are in-flight at the same time and targeted to the same release pass
• The performance is within acceptable limits

Focusing on functional testing alone for a moment, we see that:

• Each combination of values that we input produces a unique output; and
• Each such output has details that need to be validated against expected values.

The universe of all things that need to be tested therefore is of the order:

             Number of input combinations  X  Size of output for each input combination

Now, for the hotel search scenario,

            Number of input combinations = Product of (Size of the set that an input belongs to) for all inputs.

And

            Size of each output = Size of result set details + Size of details for all hotel cards

Note: inputs include both implicit and explicit ones

Using the details mentioned above and ignoring the etc's for now,

Size of inputs for hotel search

Number of inputs = 2 implicit + 3 explicit = 5 Product of sizes = Size of the set of destinations X Size of the set of Possible Start dates (and invalid ones) X Size of the set of Possible End dates (and invalid ones) X Size of set of number of guests (subset of Integers) X Size of set of POSs allowed (subset of integers) X Size of set of Currencies allowed   = 50000 (assumed) X 365 (assuming a year of allowed booking) X 365 (assumed similarly) X 4 (assumed to be max allowed guests per booking) X 1 (assuming USPOS) X 1 (assuming USD)

Size of each output

= 3 result set details + N x (7 hotel card details)   Where N = size of result set itself

If N = 100, the Test Universe = 50000 x 365 x 365 x 4 x 1 x 1 x (3 + 100 x 7) = 1E13 Tests

Onto this set of 1e13 Tests, we'll have to add the Regression, UI Validation and Performance Tests as well.

Sidebar for the mathematically inclined

 All testing can be considered validation of functions. Given an abstract function y = f(x), testing can be considered validation that for all x in X (the input set), there exists the expected y in Y (the output set)

The domain of the function represents the size of the input and the range of the function represents the size of the output. The Cartesian product is, therefore, the set of tests to be conducted to validate the function. 

Obviously, this is a lot of testing to do; and if we're actually able to do all of it, that would be Exhaustive Testing. Also obviously, anything larger than the simplest feature would quickly be intractable due to the combinatorial explosion of tests required; so we apply the "art and science" part and try to pare the problem down a bit. I'll focus on the functional testing here, but most of the concepts apply to UI validation and Performance as well.

Before we start, however, I'll define some factors to evaluate the efficacy of our optimization with:

• The number of tests required to be run (or reduced as a result of the optimization)
• The cost of running the tests (in terms of time, resources, $, etc)
• The overall quality of the product is still within acceptable limits

Optimizing testing - Round 1: Scenario Based Testing

This optimization concept is very simple - reduce each set mentioned above to a representative subset. Each element of this subset would "stand for" or represent an entire class of values within the original set. A success or failure of the chosen value is deemed a success or failure of the entire class.

To continue the Hotel Search example, the inputs could be reduced to the following scenarios (for example):

Destinations	Top 10 destinations 1 non-US destination 3 destinations with "known issues" or special edge cases
Booking Dates	Last minute bookings (1 date in the next 3 days) Planned bookings (4 dates in the next 2 quarters) Peak date bookings (2 National holidays) Weekend bookings (2 weekends in the next 2 months)
Guests	4 is small enough a number to test all combos, but we could still pick a subset, say 1 and 4 only

With this, the input size drops to 13 x 9 x 2 = 234

And the test universe becomes 234 x (3 + 100 x 7 ) = 164,502

That's still a huge number, but 8 orders of magnitude less already! We could optimize this further by reducing the validation on the output values if we want to. Realistically, we can probably get away with checking 4 of the 7 details for the first 25 hotels; and checking the result set details just once throughout. So the test universe reduces to:

234 x ( 25 x 4) + 3 = 23,403

How has this impacted our evaluation factors?

• The number of tests required to be run has obviously come down.
• The cost of running each of the tests still remains the same; we haven't optimized that yet.
• The resultant overall quality depends on the scenarios chosen. If we've chosen well; it should be within acceptable limits.

Note that there are more algorithmic ways of arriving at a subset of tests; Orthogonal Array testing to name one. I'll not elaborate on this further as the optimization is the same in principle - that of reducing the total number of tests required in validating a feature.

Similarly, on the regression testing side of the house, scenario-based reduction of tests can be done by carefully analyzing the areas that changed code is likely to impact; aka Impact Analysis.

Optimizing testing - Round 2: Automation

When you have many features similar to the one depicted above, scale effects come to bear:

• The total number of tests to be run is still a large overall number
• The cost of regression is a constant despite reducing the regression test suite using impact analysis.

The optimization to counter this is conceptually simple - relegate repeatable tests to a machine so that human cycles can be spent on the unique ones. This is easier said than done in practice; and the quickest way to get started is - surprisingly similar to BDD precepts - Outside In. That is, start at the outermost layer and automate tests at that layer. Work gradually inwards; or even not at all. Automating regression alone can have significant benefits.

One of the biggest issues with automation, however, is that you miss out on the human ingenuity bit. Scripts WILL break if data changes over time, so environments have to be stable; something that the human tester would easily sidestep by validating "outside the box" that the changed data is indeed still valid.

To continue with the Hotel Search example, assuming both the human and the machine take the same time for a test, the gain in human cycles due to various levels of automation are:

Feature	Manual Tests	Human cycles saved with 10% automation	25%	50%	75%
Hotel Search	23403	23404 * .1 = 2340.4	23404 * .25 = 5851	23404 * .5 = 11702	23404 * .75 = 17553.0

Reviewing our factors again, we see that with this additional optimization,

• The number of tests required to be run by humans has come down again and the tests run by machines can be run repeatably so.
• The cost of running each of the tests has reduced, but the cost of maintaining test environments went up as both manual and automated environments have to be kept running.
• The resultant overall quality still depends on the scenarios chosen and our trust in our automated tests. If we've chosen well and trust our scripts; it should still be within the same acceptable limits.

Optimizing testing - Round 3: Component based testing

The cost of maintaining test environments mentioned above is typically the tip of the iceberg. All testing espoused to this point has been strictly end-to-end, ie, the environment has been a live one from the UI all the way to the database (or back-end). There is a non-trivial cost associated with maintaining these environments; and a collateral cost of maintaining scripts (or known data for use by the scripts) as those environments change. Additionally, some kinds of testing may not be possible in live environments. Booking scenarios are typically such tests - contractual obligations or the cost of test bookings may deter such tests from being done in a large scale or at all.

In addition, end-to-end testing forces the entire organization into a train wreck of dependencies. Since all testing is done from the UI, all code must be built and integrated before ANY testing can start. This not only delays testing, it also puts pressure on changes to the inner layers of the application - that code has to be completed WAY IN ADVANCE of the UI code, but cannot validate their output until the UI is done.

Component testing attempts to fix these issues by testing each component

at ITS interface, not at the final user interface. That way, the owners of that component know for sure that they produce valid output for given input; a large live environment need not be maintained; and the validation of a test scenario is spread across multiple component tests which together comprise the larger validation.

Component testing almost always predicates the mocking of dependent components because the cost gains are not realized otherwise. That is, if A -> B -> C is a string of components involved in a particular test scenario, C must be mocked out to test B and B must be mocked out to test A; otherwise we've taken on the additional job of maintaining a separate instances of A,B and C solely for component testing purposes, thereby increasing cost of maintaining environments more than without it.

Component testing also typically requires some means of creating mock data - manual means will not suffice; especially if the request-response payloads are huge object graphs.

The choice, adoption and usage of an organization-wide mocking framework is therefore a non-trivial task and I will not go into the details of how to achieve this. I will, however, analyze the impact of adopting such component testing on the evaluation factors mentioned above.

To continue the Hotel Search example, a hotel search typically involves a string of internal components:

                                                               |------------> GDS

UI -> Business  -> Business  -> Business  ->    Back End-------|------------> CRS1

Dispatcher        Facade     Logic Executor  Abstraction Layer

                                                               |------------> CRS2

(Some details may be missing; sorry. I'm trying to make a larger point here).

Let's take the following Test Scenario:

Input	Expected Output
Top Destination (LAS), Weekend (start: next Friday, ret: next sun), 1 guest	1st page has 25 hotels 1st hotel is Caeasar's Palace @ $100 2nd hotel is MGM @ $125 …and so on

…and break it into component tests:

Component	Given a test script that provides this input	..should provide this output	..using this mock data
UI Layer	LAX, Next fri, Next Sun, 1 guest	1st page has 25 hotels 1st hotel is Caeasar's Palace @ $100 2nd hotel is MGM @ $125 …and so on	Business Dispatcher response containing the 25 hotels
Business Dispatcher/ Facade	LAX, mm/dd/yyyy,mm/dd/yyyy,1 + other Dispatcher required params	Arraylist of Business Layer objects	Executor response containing the 25 hotels
Business Logic Executor	LAX, mm/dd/yyyy,mm/dd/yyyy,1 + other Executor required params	Arraylist of Executor-specific objects	LAX-to-internal-location-object response Back end Abstraction Layer responses
Back end Abstraction Layer	LAX, mm/dd/yyyy,mm/dd/yyyy,1 + other Back end required params	Arraylist of Back end Abstraction Layer objects	Back end-specific responses (1 per link)

We'd have to do a similar exercise for each such scenario identified before as required, but if we did, the impact on the factors would be:

• The number of tests required to be run by humans has come down again and the tests run by machines can be run with lesser resources.
• The cost of running each of the tests has reduced; so has the cost of maintaining test environments as live environments no longer need be maintained.
• The resultant overall quality now depends largely on the fidelity with which the end-to-end scenarios have been mapped to component tests. If there is a high correlation between the two, overall quality should still remain within the original acceptable limits.

This last is not easy to do for various reasons:

• There needs to be a coherent effort to ensure scenarios matchup between components; else component owners could define their sphere of influence differently from the scenarios deemed organizationally important.
• The larger the size of mock data to be created, the more difficult it is to create it with high fidelity. Shortcuts such as record-replay mechanisms might help, but only if they've been sanitized to remove volatile data and then made generic to match the expected scenarios.
• Ownership is shared; so division of labor should be clearly defined via policy. For example, each component owner should be responsible for her front end, ie, provide mock data and a set of regression test scripts to his clients; she can therefore rely on her dependents to do the same. Without such a policy, key pieces of the puzzle will be missing and the overall quality will suffer.

Optimizing testing - Round 4: Common Test Scenario Database

These issues with implementation of component testing may even lead to a regression back to manual testing. The crux of the problem is that the cohesive force of the end-to-end test is lost in component testing very easily.

The central idea with the common test scenario database is retain the benefits of component testing while bringing back that cohesion via data: we need to ensure that test data that is distributed across the various component test scripts still have the same tie-in to the original scenario. That way, every component owner in a particular scenario refers to the same scenario using the same language. While we're at it, it would also be beneficial to change the mock data in two ways:

• Replace significant pieces of live data with values that stand for the class of test data that we will use in the specific scenario. E.g., the destination data item when used in a scenario where it represents a top destination could be given the canonical name "Top Dest1". Alternatively - assuming this is clearly understood company-wide - a real value can be used as a canonical one ; eg, in this case "Las Vegas" could stand for top destination; but then it shouldn't be used in any other scenario.
• Clear out any recorded values from the mock data so only known values remain. This eliminates side effects from remnant data but requires a higher degree of discipline.

The larger change would be to introduce full-fledged exemplar data sets for application domain concepts that cannot be confused with live data, but clearly denote the exact scenario in which they can be used; and use policy to drive adoption of these exemplar data sets as the mock data backbone.

To continue on the hotel search example, the first step would be to define the following exemplar data:

Concept	Exemplar Data	Comment
Top Destination	LAS
Regular Destination	USDEST1
Special Destination	Acme Cozumel	Added "Acme" to Destination to call out that this is a test value
Next Week Friday	(Computed Value)	Mock data framework should be able to generate such values and respond appropriately
Hotel Chain	Acme Hotels LLC
Hotel	Grand Acme Ritz Chicago
Top Hotel @ Top Destination	Acme Hotel and Casino

The component test from before can then be rewritten like so:

Component	Given a test script that provides this input	..should provide this output	..using this mock data
Web-wl	LAS, Next fri, Next Sun, 1 guest	1st page has 25 hotels 1st hotel is Acme Hotel & Casino @ $100 2nd hotel is Acme MGM @ $125 …and so on	TBS response containing the 25 hotels (Note: other hosts required to bring wl up ignored for now)
TBS/Plugin	LAS, mm/dd/yyyy,mm/dd/yyyy,1 + other TBS required params	Arraylist of BookProduct objects	HSE response containing the 25 hotels
HSE	LAS, mm/dd/yyyy,mm/dd/yyyy,1 + other HSE required params	Arraylist of objects	Market/Markup response Supplier Link responses
SL Host(s)	LAS, mm/dd/yyyy,mm/dd/yyyy,1 + other Supplier Link required params	Arraylist of objects	SL-specific responses (1 per link)

More importantly, when a second scenario has to be mapped to component tests, the exemplar data table above should be checked to see if the concepts in that scenario already exist, and if so they should be reused.

So, to convert the following scenario:

Input	Expected Output
Top Destination , Peak Weekend, 4 guest	1st page has 25 hotels 1st hotel is Acme Hotel & Casino @ $100 2nd hotel is Acme MGM @ $125 …and so on

...into component tests, the following data items will have to be reused:

• Top Destination (LAS)
• Top Hotels (Acme Hotel & Casino, Acme MGM)

…and some new data items will have to be added:

• Peak Weekend (Thanksgiving dates, for eg)

…which will further be reused when automating the scenario:

Input	Expected Output
Top Packaging Destination , Peak Weekend, n guests Acme Mexico Dest1 Labor Day Weekend dates 2 guests	1st page has 25 hotels 1st hotel is Acme Cozumel Resort1 @ $100 2nd hotel is Acme Cozumel Resort2 @ $125 …and so on

.. Which will further require new data items to be created, and so on.

When a new feature is added, say separate pricing for children or prepaid hotel rooms, that's the time for a completely new set of hotel chains and hotels to be created.

Over time, this practice of reusing test scenarios results in the creation of the Test Scenario Database which becomes the lingua franca across the organization when talking about Quality issues.

Let's see how our measuring factors score with this optimization:

• The number of tests required to be run by humans hasn't changed since the last round.
• The cost of running each of the tests remains as before. If there were any collateral increase in costs of using live environments due to inability to trust component tests, that is removed however.
• The resultant overall quality still depends largely on the fidelity with which the end-to-end scenarios have been mapped to component tests; but there's a direct correlation possible because the organization now has the "common language" of the scenarios enshrined in the test data. This is the killer quality of the common scenario database.

Notes on implementation

• Just as with automation, creating a Test Scenario Database is easier said than done. Policy from "up top" will certainly help; but a grassroots approach is also possible because the database can be built one scenario at a time. It does require a few enthusiastic converts, but some key component owners being convinced will create the kernel of good, reusable (and reused) test scenarios which can then be supported via policy.Once their use is common, network effects will take care of their popularity.
• The Quality Organization should own the Test Scenario database and gate-keep use and reuse. Developers may create test scenarios, but should get approval from Q.
• Component owners should be responsible for their component and their front end interface; and expect their dependents to do the same. That way they have the responsibility towards their clients and expect the same from their dependent servers.

Todo

• Create a Test Scenario database tie-in for the Robot Framework