Why literate programming has not been adopted

I've tried literate programming multiple times and not really succeeded in adopting it. The problem lies in the fact that programming is a constant switch between thinking about design and implementing it, and the latter provides more useful feedback since its at the business end of things. This is why programmers prefer documenting code rather than embedding it inside documentation.

What you really want is a solution that allows you to document your design to the extent that it clarifies your thought about implementation, then write some code which gives you results, then correct your design assumptions, and repeat while keeping the code and documentation in sync.

Assuming code and docs coexist in the same source repo, the keeping in sync part is the tough piece. What if the editing surface was based on content addressable objects, like a git repo? Names would be pointers to specific versions of the content, allowing for doc content and code content to be linked with each other's content hashes.

Pwd Rules

This happens to me all the time: I have to create a password for a site that I dont use everyday, but its an important site, like a bank or insurance site. The site usually has some wierd set of rules that I obeyed when I created the account, but have forgotten now.

So when I go in after a while and have to get in, I have some inkling of my password, but cannot get past the auth page because of some arcane rule that is not obvious to me then. Of course, I go through the reset password procedure and THEN they list out all the requirements for the new password.

If I could see the rules one page ahead, though, I'd probably be able to remember my original one!

So here's the idea: A website called PasswordRules that stores the password rules by site. You can query by site to get the rules for that site and use it to login. Simple!

Better yet: Browser plugin that pops the rules up when you visit the site.
More: API for contributors and website owners to publish the rules programatically.

Design:

• Create a website that has:
• add new site and its rules
• edit site rules
• search for rules by site
• Create an API that allows:
• Search for rules by site
• [optional] add/edit site rules
• Create a plugin for Firefox, Chrome, etc that:
• Uses the API to search for the current site and load up its rules in a popup

Lingaa->Rajni->Balleilakka->Choirs : How mass movies lead to high music

So I watched Lingaa yesterday.

Because of which I was googling and youtubing for rajni's impact outside india; and was largely expecting to find videos from japan - which I did.

But what was really interesting is that I found at least 5 to 6 videos of one song from Rajni's last movie - Sivaji - being sung by school choirs across US.

The song's called Balleilakka, and here's the original:
https://www.youtube.com/watch?v=Tma8zoLEPJU

Here's a sampler of schools choirs singing it:

Best that I found: https://www.youtube.com/watch?v=XQWLqcNr9B8
Really fast: https://www.youtube.com/watch?v=Rvo1_vLGAGo
Pretty good, but smattering of indians :) : https://www.youtube.com/watch?v=dUhMQYW-xmk
Pretty good: https://www.youtube.com/watch?v=09C8s1QhgqE
Shitty accoustics, but points for the effort: https://www.youtube.com/watch?v=jSJYVAB8XQU


.. and I began to wonder: why this song alone? None of them actually mentioned the source of the song; they usually called it an indian song in tamil. I think one mentioned that its AR Rehman's song.

So they were actually using it because it was a complex song musically - it was a tongue twister even for natural speakers of the language and what was really obvious from listening to it when sung by a chorus is that it had many layers and voices to it. I dont obviously know the right musical terms here, but in programming terms, it was a mutlithreaded song - multiple concurrent voices had to be active to make it happen, and each voice was sure to get knotted up in the tongue twister lyrics; not to mention that the beat was really fast! No wonder the choruses are taking it up as a challenge.

Its amazing that I live in a time when composers like Rehman have enough commercial clout to put complex music into a movie meant for the masses. Even better: not only do the intended audiences love the song, so do people who are serious about music!

Idea: A screenreader that's a market place, not a queue

Had this idea while in a meeting at work today where ADA was being talked about; and how making a page ADA compliant required reordering the markup - sometimes entirely.

What if the screen reader didn't read sequentially - like a queue of people who talk to a teller - and instead read everything that's on a page at the same time - almost like a marketplace where everyone's talking at the same time?

I'm not blind, but I have to imagine that this is how blind people perceive the real world anyway - as a cacophony of sounds that they have to filter out noise from. This is not that different from fully able people sensing the world and filtering what's important.

If we had such a screen reader, the document markup could be the cue for how "loud" each element would be, child elements could be "muted" in favor of the parent and so forth.

Todo: check if screen readers already do this.

Fixed order graphs with Graphviz

Graphviz continues to amaze me. I did not know until recently that they had an awk-like language for graphs within the graphviz package, but even closer home - there's a tool to control the order of nodes!

This is the one thing I've battled with out-of-the-box Graphviz for a long time - any new edge is likely to throw your neat-looking graph off-kilter *just because* that's the most optimal graph layout.

Well, not if you use osage, packmode and sortv.

I dont exactly know why its called osage, but this layouting engine actually obeys the order of nodes that you can specify with the sortv attribute.

You have to set the appropriate packmode setting, of course, and the correct incantation is:

packmode=array_uX

where X=1 if you want one column of nodes, 2 if two and so forth.

Here's a sample graph for a taste of what it looks like in practice (from the man page for osage, see example section):

digraph test123 {
    pack=8
    subgraph cluster0 {
        packmode=array_u1
        x[sortv=5]
        y[sortv=4]
        x0[sortv=3]
        y0[sortv=2]
        x1[sortv=1]
       
        subgraph cluster1 {
            mn
        }
    }
    b[shape=box];
    c[label="hello\nworld",color=blue,fontsize=24,fontname="Palatino−Italic",fontcolor=red,style=filled];

    a->z
    x->z
    a->b->c;
    a->{x y};

    edge [style=dashed,color=red];
    b->x;
}


Once you do that, you issue something like this:

osage testosage.dot -Tpng -o testosage.png

... and you should see a column of components with x at the bottom and x1 at the top.

Also, use

pack=16

to space the nodes out a bit. Default is 8

I fixed some hardware!

This is a personal first. I took my wife's laptop apart and repaired it!

My wife has a Lenovo 670 that we bought in India. Its a low-end laptop that I chose for her because we were in the market due to my son spilling water over the good Toshiba that she originally had. "You use it mostly for browsing", I had reasoned, adding: "Besides, its a Lenovo, which used to be a Thinkpad, so these things are rugged; and more importantly it has the chicklet keyboard so we're covered on the water spillage scenario from now on".

While it indeed was rugged, it didnt endear itself to my wife because it was slow as hell. Still, she was not a fan of the Mac and was happy that it was the one Windows machine in the household.

Fast forward 2.5 years later and across the globe where we are now, the poor laptop had fallen for the umpteenth time from the side table (I think it was the son again - this time a well-placed kick in sleep) and the wife found that she could no longer charge the laptop. The connector could still be seen inside, but it was not in its slot and couldn't be paired with the charger.

"Husband, can you see if you can fix this?" she asked. I quickly googled the nearest Lenovo service center and left it open in a tab for her to see.

A few weeks later, she got around to calling the service center only to be told that they didn't service any models sold outside the US of A, thank you very much.

We considered shipping it off true desi style with some hapless acquaintance to be delivered to a service center in India that DID service such models, but thought it might be not worth all that hassle.

So my wife repeated her original question and went one step further and found a web page with instructions on how to do it.

Two or three weeks later, I got around to looking at it. The initial steps were all pretty standard - unscrew 13 screws, turn over, unscrew 8 other screws and so forth. The rest were decidedly not unless you've been assembling and disassembling laptops (or other electronics) all your life.

Here's the thing about electronics these days: They're giant 3-D puzzles that have been put together in a space the size of a football field and then squished into the final size that you get it in. There is apparently one and only one way to take the thing apart and almost everything requires you to take the whole thing apart. Must be nice to be in a market where your product has no serviceable parts or the act of servicing it will render it non-serviceable.

So I got around to removing the battery (of course), back cover, the hard disk, the connection to the wireless lan and the optic drive. I had skipped the memory hoping it would not be germane to my fixing the power supply connection. Going through the next steps. however, it looked like the only way I could get to the power supply was by removing the fan, the cpu heat sink and the cpu itself.

Now this is not new territory to me - I HAVE disassembled a laptop before - solely because of my cheapness. Back in the early 2000s, they sold an abomination of a laptop called eMachines - they had great specs and were cheap; except they killed themselves whenever you ran more than 3 applications because the cpu heated up past its allowed range due to the cheap heat sink they put in. One way to solve this was to open it up and clean out the sink yourself.

Anyhoo, because of that experience, I knew that taking the CPU out implies putting in new compound - something that I didnt have then nor had now (side note: I had taken a chance with the eMachines and put the hardened compound back as-is and it did work for a while :) ). So I did what everyone would do - tried taking either the bottom half of the laptop or the bevel surrounding the keyboard without touching the CPU or its sink.

After about 5 hours, I gave up. All natural light had gone by then and I really couldnt work with the incandescent lights and the Walmart tools that I had. So I gave up for the next few weeks.

Finally, I picked it back up today. Started at about 11 am and by 6 I had the bevel around the keyboard removed. Here's the kicker: Remember I told you that these things are 3D puzzles? Well, not only are they puzzles, they're also traps: manufacturers put in blue paint on screws and stickers on specific components that will tell them if we've actually looked inside. The thing that held me up for a good 4 hours? One screw that was hidden under a Lenovo sticker. After I found it at 3 hours and 57 mins counting, I realized that special gum those kindly folks at Lenovo used would not allow me to peel the sticker off. It had to be scratched out - leaving no doubt that whatever warranty this laptop could have been under has now been voided.

Once the bevel was removed, I could look at the connector. Sure enough everything was fine and all it needed was to be put back in place. To their credit, this Lenovo had taken all kinds of weird angles of the charger and worked well until recently. I dont know if there was a bit that did hold it in place, but it looked like all they had going to keep it there was a snug fit. So I made that a lil bit snugger with some quickfix glue and closed the whole thing up.

And by "closed the whole thing up" I mean "Spent the next 2 hours painstakingly following the instructions in reverse order, of which half an hour was spent looking - unsucessfully - for the one washer that got away and was now hidden in the beige savannah called my carpet, then reaching the last step only to realize that step 8 was reinstalling the keyboard (a link off to another page) that I'd missed so going back a few steps to take gajillion screws out again and popping the keyboard out and finally getting it all together with 2 screws outside"

As if on cue, the wife showed up and asked nonchalantly: "How's it going?". This must be how CEOs and VPs feel.

We fired it up and it hasn't exploded - yet.

My wife, of course, noted dryly that it seems somewhat slower - like my hardware machinations could have magically slowed down the electronic pathways inside Windows 7.

I was just happy that it worked :). We hi-fived. She said she's proud she married someone who knows how to fix things.

A better portable environment

For a long time I've tried to create a portable environment for myself, without much success.

A portable environment is a collection of all software that I need and use for development available as a set of files on a USB stick. I should be able to put it into any machine (Win, OSX or Linux) and do everything that I want to do.

PortableApps comes pretty close. Portable msysgit gives a uniform bash shell everywhere and as long as i'm miserly with the languages I want to work with, I should be OK. Maybe cheat a bit with an install of Sublime Text (zip install used via Wine on Linux) and I'm golden.

At least that's what I've settled for now.

But sometimes its not enough. You want a full environment. You want to not have to copy files or git rebase. So I tried VirtualBox. The idea was that the only install I'd need was Virtual Box on each machine that I want to use the environment on, and the VDI file would be on the stick. That way everything remains in one place.

Except the VDI files dont work right with OSX. Or at least didnt on Snow Leopard. I've not tried on Mavericks.

But in the meantime, I had an idea: How about a node-webkit based environment? As long as the stick has executables for all OSs, the app itself can be run. There are enough and more terminal emulators, editors and so forth in JS these days. What is still not in JS (versioning, file system, etc) could probably be filled in using emscripten.js maybe?

Its a thought for 2015 :)

Idea: Stable visualization of a codebase

I have been thinking lately about visualizations of a codebase - spurred on because of recently rediscovering Software Catography and its successor - Codemap. Coincidentally, I also wanted to create my personal website as a "visual representation of the web of connections that it is", which essentially boiled down to a stable visualization.

When I looked at the tools that are currently available to do this, it seems like they are overly complex. The closest was Gource, but it focuses on the people who worked on the code and doesnt generate a stable visualization.

So here's my idea:

• The visualization will be created from the commit history of the codebase.
• Once created, the visualization is not a snapshot, but can be enhanced over time to show changes. So the output format should contain the history of changes.
• The visualization is essentially a Treemap-ish diagram with time along the X-axis and size along the Y.
• Each object(file or directory) is drawn as it comes to life in the commit log and is represented as a rectangle.
• Position: The first object  that is created gets the position x=0 within its parent, the second gets x=1 and so forth. Once assigned, these positions are permanent even after the object is moved or removed.
• Dimensions: The width remains the same for all: files have a width of 1 unit and directories have a width equal to the sum of the widths of its contents. The height is equal to the size of the file.
• When an object is changed, its old size shows up as a faded outline within the newly sized rectangle - somewhat akin to the age rings of trees. Size reductions may show age rings outside the current rectangle.
• When an object is moved, its old position shows a faded outline and objects after it do not move to take up the position.
• Similarly when an object is deleted, its old position shows a faded outline.
• Keeping the visualization contained: This is where the Treemap concepts are helpful. The complete visualization's size will be calculated inside-out: the size of the deepest directory will control the % contribution of its parent and therefore transitively its grandparent, and so forth. This way, the visualization can be contained in a finite space. At its smallest size, each "rectangle" will be reduced to a line: the position still remains as described above, the width is reduced to 1 pixel and the length is still the size of the file. No rings are possible at this level of compaction.
• Controls: The visualization will have:
• Play: A way to see the evolution of the codebase a la Gource
• Zoom in and out
• Time Filter: A way to filter out older rectangles. This will essentially show the current state of the codebase, but since all positions are fixed, it will give an idea of how far the current state is from the original.
• Object Highlight: this will highlight a particular file or directory to "show where it is in the map"
• Object Trace: This will high light the path of the object throughout its evolution in the codebase.
• Commit Highlight: Highlight all files in a commit

The advantages I see with such a visualization is that it combines a stable spatial representation of the code along with its evolution over time. Using a treemap representation essentially keeps it bounded so that the view could be injected into current developer environments without taking up too much screen space.

Implementation notes:

• A quick way to implement this might be using html divs.

Last post of the year

Its been almost 9 months since I've blogged!

I realized it only yesterday.

A lot has happened in that gap  - I moved halfway across the globe, was in "transit housing" before and after, getting adjusted to the new place and so forth.

However, there's another reason for the dry up in blog content: I started building more and - more importantly - started journaling my work within each project.

The "building more" piece came from an an increased focus on finishing. Not that I'm good at it, but I'm moving in that direction.

The "journaliing" piece came from switching projects too much and having to spend a lot of time catching up with where I was when I left off last. However, it quickly became a "pensieve" of sorts - a place where I store my thoughts and half-baked ideas on how to move forward with that project. Since most (all?) my projects are single-person ones, its easy for me to plop in a journal.md and just start writing.

So, in some sense, the ideas that I come up with used to be put up here as blog posts now are likely ending up as another project folder on my laptop. It also is probable that the bandwidth that I had for ideas in general is now being channelled into specific ideas for projects that I'm already working on.

Anyhoo, that's by way of apology to the teeming millions visiting this blog :)

Since this is the end of the year, I usually end up thinking about what I've achieved and do some half hearted attempt at a resolution list. Not that I actually execute on that list, but its good to at least think about it once a year :)

So here's what I'd like to focus on in 2014: Actually finish up (even a v1.0) of these projects:

1. my website
2. GUTS
3. plnext
4. ed

This should be quite enough considering I'm staring a part-time masters soon. Hopefully I wont get derailed with weekend projects that take 2 months to reach v1.0, but hey - that's the fun of personal projects, right?

On the bright side, that project was something that I always wanted to do, so there's always a bright side :)

Eternal digressions of the engineer mind

I had an idea for a stream-of-consciousness editor. So I started creating a prototype using node-webkit, codemirror and my non-existent ui skills.

Since I did all this on the PC and wanted to sync up with the mac, I started to setup node-webkit on it, only to realize that there are no binaries for osx 10.6.x.

So I downloaded node-webkit (using up 35gigs of bandwidth by the 10th of the month - kids and wife will not see their videos) and got it running.

Then I realized that the code for the editor itself could do with some modularization - modularization of the kind that my betterscript language promises. That project, which was last worked on last year on one day - a hacknight - now got revived.

Of course, betterscript uses ometa-js, which I'd copied directly onto the betterscript project (along with its dependency - rhino) and duct-taped together a script that worked only for that version of betterscript.

Today's me, however, was not happy with this state of affairs. First off, ometa-js should be an external dependency. Also, even if it were an external dep, the original author had not really organized the code much. He'd just done the important bits - ie the really interesting peg parser in JS that parsed both text and objects alike - and left the rest of the "productionization" to us vogons. This wouldnt do, of course. I'd like some organization in that code, for cryin' out loud.

This required looking up the original github project and its 400 some-odd forks to see which one met my needs. There were a few that did re-org the code, but not in the "just right" way: they'd either stop short of what I wanted, or go way beyond the call and rewrite the code.

So of course, I forked the original, reorg-ed the code my way and tried to use it as a dependency on betterscript. Which, of course, meant that now I had to rewrite the original ometa-rhino.js with whatever changes I'd made in the file with the same name within betterscript. This lead to the realization after one sleepless night (doh) that outputing an object to stdout will reduce it to a string that cannot be read back as an object. Off to Crockford's for a bit of json2.js.

As of today:

• my fork of ometa-js works when run using my lanucher with rhino. Other modes not working yet.
• betterscript works using my fork of ometa-js on simple samples. Still a long way to go before I send that post out to jashkenas :)
• soced is encouraging concept-wise, still to reach its usability nirvana!

Final note: I thought it might a good idea to represent all these dependencies as a Graphviz graph. That turned into a journey down the "how do I get graphviz to show the nodes in the order I want " rabbit hole.

Sigh.

The phrase "Eternal Digressions" is somewhat inspired by the phrase "Infinite Digressions", which is copyrighted by Praveen. Alluded to here with no permission whatsoever :) 

Built node-webkit...finally

So I'm still on Snow Leopard, which means node-webkit's pre-built binaries are of no help - they start at 10.7. The pithy response to the very pertinent question of "How the hell do I run this awesome piece of software on my apple-fanboy-version-of-win2k?" has been "Oh, just build it on 10.6.x yourself. I hear it works fine then".

In a rush induced by the success of my "Code as you think" prototype working really nicely on the PC, I decided that I must have it running on the Mac too. I did consider upgrading to Mountain Lion, but decided that building node-webkit like they'd blithely suggested might be smaller a peak to scale than having to deal with a whole new version of OSX.

Downloading the code took about a day and a half. Of course, I was not surprised, considering this was  the Chromium codebase, plus node-webkit's own. Actually, the final download took about 3-4 hours, but it failed (ie 'gclient sync' got stuck and had to be ctrl-c'd) about 4-5 times. Here're the things that I learnt/thought up during these failures:

• How to setup the mac so it doesnt sleep - using screensavers and energy settings. Ended up using neither: was going away for the night, didnt want to leave machine unmanned.
• How to piss spouse off with blank stares and disconnected answers while download process failed yet again.
• How it would be nice if code sync processes had pause and resume options. wget has it, why not gclient/git?

Anyhow, all the code finally came down at around 1:40 am. Of course, I had to build it right then, so after some more consulting of multiple documents, I fired off the build commands. A full 1.5 hours later, I was the proud owner of a self-built version of node-webkit. The process completely maxed out all 4 cpus on my mac for the majority of the time, which I spent browsing the ninja build tool manual. Of course, it was almost finished by the time I'd reached the part on how to control the number of cpu's to allow for ninja's use :). The mac, however, performed admirably under stress, I must say. The rest of the running processes were still very usable and the terminal itself was still responsive. Not the same experience on a Windows or linux box, in my experience.

In all, time well spent; and catharsis for disappointments from the past. I might just run the build commands for Chromium next.

And once I've patched up with the spouse, I might even try getting it to run on Windows with VS Express!

Code as you think

I was in the middle of deciding how to proceed with my current side project when I realized that I'd not yet created an architectural description for it. Since I already have an architecture description tool that I wrote, I have no excuse for not writing one out. So I stopped to think why I didn't create an architecture definition first. Ans: Because it would stop me from doing what I was doing right then, which was to decide:

• how to separate out the current spike of the language from its sample code and actions.
• how this will be deployed in and of itself, ie how to dogfood it?
• how to move from the spike folder to the actual folder
• how to move from bitbucket(where I'm keeping all this stuff till its public-worthy) to github or a public bitbucket repo (where I expect this to be shared with the world)
• ...all in the middle of adding commit messages for changes
• while juggling code between the PC and the mac
• ... and so on.

As I'm writing this, more thoughts are streaming through, and they're all jumbled together: code ideas, architecture ideas, documentation, behind-the-scenes reasons, journal entries - all in one single stream of consciousness. Stopping now to create an architecture definition would not just slow me down, it would take me down a totally different path. And I don't want to go down that path because the IDEAS ARE JUST FLOWING!

Another reason I didn't want to context switch into "architecture definition mode" was the high cost of that switch : create a new document, decide where to save it, type in the actual content, optionally see how it looks once the markdown becomes html and so forth. IMO, this is also why documentation is out-of-date with code, especially expository documentation like architecture and design docs. The comment-above-code kind of documentation may still be in touch with the reality of the code, but higher level documentation like READMEs and user guides quickly become disconnected from the code they're talking about.

That's when it hit me: What we need are tools and languages that understand the stream-of-consiousness way of building software: code snippets flying into the system from the human mind along with documentation and random thoughts and todos and reorganizations of such thoughts - all of which the system will record in sequence and generate meaningful views out of. 

Imagine an editor that:

• Allows creation of content without the need to pick a name or a location for it.
• Knows enough about the content to provide appropriate editing affordances - syntax highlighting, preview, whatever.
• ... and autosaves it for you - no separate save step required
• ... then allows tagging it as being "of a particular type"... whatever that means to you
• ... then allows you to seamlessly retag it differently
• ... including being able to arrange it in hierarchies (if that made sense) or graphs (ie nodes and links - if that made better sense)
• Presents all existing content and its associated "tag graph"
• Allows linking content with each other
• ... or even parts of content with each other.
• Shows the links visibly
• Allows tagging parts of a document as being of a different tag, but still housed in the source document. Content tagged thus will be displayed in both tagged content, but the second display will be a reference to the original.
• Tracks all changes to the content graph in a stream-of-consciousness journal that's visible and accessible at all times so that you can see what you did before in the order that you did it.
• Allows you to add your own manual entries to the journal to track your thoughts at that point in time.

Such an editor could be used to make the context switches mentioned above as seamless as possible. The only bit of automation it promises is the decidedly "big brother"-like journal, but in my opinion that's something the computer should do for us - track what we're doing automatically so we can look back and "see the trace of our thoughts from before in our actions". Features like seamless switching allows easy creation of design, code, comments  and code in back-n-forth style, for example; while the ability to link content allows for todos to be linked to the point in code where change is required, but still can be maintained as a separate list of todos for easy review.

To allow easy adoption, such an editor should add the following:

• Treat an existing directory structure as the basis for its tags: folders become tags, folder structures become hierarchical tags, files become fully described hierarchical tags. If starting from a blank folder, however, allow for the initial tagged content to remain as files until the user discovers the "natural order of things" for himself.
• Seamlessly convert files to folders when the user renames them.
• Save the changes to the underlying folder as soon as possible so that there's no "last minute unsaved content change"
• Allow for some scripting/automation triggered by content changes.

UI Design Notes:

• Start the editor from a particular dir, which we'll call <dir> in these notes. The startup sequence allows choosing the default language to use optionally, else the default is text.
• The editor starts with two windows: a default one titled "<dir>" and the journal titled "stream". The former is editable, the latter is read-only. 
• The default window is "in" the default language (if chosen), ie is syntax highlighted for that language. Typing text into it and hitting enter causes text to show up like any editor, but it also causes the same text to show up in the stream window with a date/time stamp. Internally, the text in the stream window is a reference to the line in the original window, not a copy of its contents. In the background, all the text changes are saved back to <dir>.
• Typing F2 allows you to rename the window's title, which is also its tag. The editor allows creation of hierarchical tags like "tag1/tag2/tag3". It also allows setting the language of the content by naming the tag with the format "tag.type" - similar to how files are named today.
• Typing an esc puts you into "enter a tag" mode. a small edit area (or vim/emacs-style command area) shows up which allows you to type in the name of a tag such as "todo" or "comment" or whatever. hit enter when done and two things happen: the edit area disappears and a new window titled with the tag just entered shows up and focus moves to that window. Type text and hit enter to enter content in this mode, while it also appears the stream window like before. if the tag is already present, focus shifts to the existing window for that tag and text input proceeds from there.
• Selecting text and then pressing esc to select a tag will leave the text in the original window, but add that text to the window corresponding to the chosen tag (via reference again), as well as denote that the tagging event occured in the stream window. Presssing ctrl-esc instead of esc will cause the text to be moved to the tag chosen.
• Ctrl-tab will cycle through all open windows and allow you to make any window the main one. The same esc and enter behavior holds for all windows.
• Everything is save-as-you-type. No ctrl-s required. The tag is the name of the content.

More features:

• Windows are tiled, not free. this allows pinning of windows to specific locations so as to create familiar screen layouts.
• A third kind of window is a modal window, which can be in input or output mode. this will allow creation of repls. issue a command when its in input mode and view results when its in output mode, which is read-only. The "stream" window can also be recast as a modal window, allowing journal entries in input mode.

Ok, that was the easy part. What about a s-o-c language?

Such a language would:

• Allow for programs to be expressed one chunk at a time. The language runtime could be invoked at any time during the expression, at which event the runtime would evaluate the current set of chunks and determine if it can do anything with it. If the runtime is a compiler, it would decide if object code can be generated or not; and if its an interpreter, it would interpret the chunks available.
• Not have comments. Instead it would allow a more generic interpretation of commented content as "chunks from another language embedded within itself, which could be "understood" or "translated to itself" given an appropriate cross-interpreter/compiler.
• Allow chunks to be linked to each other and handle link lifecycle events. Eg: chunk A is related to chunk B could imply that if A changes, B should also change; thus flag such changes on A as requiring changes in B as well. Or, if chunk A is deleted, chunk B should be deleted as well because it doesnt make sense to keep it around anymore. Or, if chunk A is created, a corresponding chunk B MUST be created.

More thoughts on the language TBD.

Implementation Notes

Editor:

• Use node-webkit, codemirror and sharejs to allow evented entry of text
• Make a repl using a bot that's sitting at the other end of a sharejs connection.

WIP notes

Started work on a prototype using node-webkit. A basic version of the UI Design has been implemented, and the updated version of this post reflects some experiences from this implementation.

bash tip: setting terminal tab title programatically

This one has been annoying me for quite a while - looking at the string of tabs in the mac's Terminal App, all of them titled "bash" except for the one trying to download webkit which is usefully titled "git".

So I finally decided to do something about it, namely google the solution. Here're the results:

As a bonus, the cd() function adds the ability to change the title to whatever directory you currently are on - which works just fine when you're in your projects directory :)

I think this should work on Linux as well; confirmation TBD.

The Ultimate Personal Task Manager

I've been prototyping a GDoc-based 43 folders for a few weeks now. The biggest two problems I have to date are:

• the ease with which I miss my daily review of things to do
• "Too much trees, too less forest". I need a dashboard that's higher than the daily, ground level view

In a fit of frustration about the second point (cos there's no point fretting about the first:) ), I set about designing the ideal personal task management system - once again. Here's the result:

The Ultimate Personal Task Manager needs:

1. The open task list: which contains all tasks that are to be started or WIP. Routine/repetitive tasks are represented as new open tasks for the next instance when the current one is completed.
2. The review/decide process: which reviews current status and decides what to do next.
3. The todo list: which contains tasks that have been decided upon, and will be acted on in a particular time period - day/week/month/whatever. <==> 43 folders. no sequencing, just "these tasks mush happen within this period or before this milestone"
4. The plan: sequenced list of tasks. <==> day planner. contains time slots when things will be done
5. The dashboard: "dont break the chain" style display of only top level projects. ideally this should be the top 5 things you care about in life shown as a red/yellow/green style view
6. The nagger: Most important piece for procrastinators. the physical reminder to actually use this system. could be a phone reminder system or a spouse/friend. should know what your frequency to run the decision process is and nag you to do it.

Key point about this system: It could be all in your head or on paper/pencil or software

Yes, this is pretty close to GTD. The additions are the dashboard and the nagger. 

If implemented as software it also needs to:

1. Work offline
2. Works across OSs and systems
3. Allow versioning

Software implementation notes:

- Something that works offline and universally is required. simplest i've seen to date is gina trapani's todo.txt. Using it, the functional requirements:

- #1 & 3 are built in.  Use 'repeat' addon for sub point under 1

- #2 and 6 are human tasks anyway.

- #4 and #5 will need additional dev. In particular #5 will require (at least for me) enhancing the project field to a tree of projects. Some simple testing with bash showed that a format like "proj1/sub1/sub11" will work, as will "proj1:sub1:sub11"

  The non functionals have direct answers:

  - it works offline and across OSs - being text and a shell script

  - putting it in a git repo will handle versioning

- Idea: Create an Outlook plugin to auto convert mail into tasks

- how to run cygwin bash from windows? ans: Cygwin's run command

- how to create an outlook plugin? TBD

Groovy DSL Voodoo... or why I think imperative programming makes more evolutionary sense

I recently had to consider the Groovy space for a project proposal as the client had decided on Groovy/Grails as the platform of choice. While evaluating the platform for the proposal, I naturally gravitated towards Groovy's DSL capabilities and started thinking about using it for my personal projects, obviously. Abu seems like a natural fit (why didnt I think of Groovy before, I wonder? I'd hit a wall with JRuby due to generics anyway); as did Jack.

Anyhoo, with all this background, I hit slideshare and found tons of decks from the language leads on the specific topic of DSLs. One example piqued my interest in particular: slide 106 of Paul King's deck, which I've replicated here as a gist:

I wanted to both understand how this worked and to replicate it by myself. Here's all the flailing I went about in trying to achieve those two goals:

As you can see, I failed miserably. My last trial had a near-working version, but left me completely disillusioned about Groovy's DSL capabilities - especially considering I came out of the process feeling that all the work was done by the single 'of' closure with the rest playing dummies to the Groovy language's optional syntax rules - especially because in my Englishy version, you could switch the order of the closures passed into 'of' and still have it work fine.

However, all of this didn't explain the "extra code" in the original, which left me with the nagging feeling that no language author would be trying to pull this much wool over people's eyes. So I went back and expanded the original expressions like so:

Now it made much better sense.

Mind you, I still haven't figured out how you go from a problem statement like "I want to express the ability to compute square roots in an English-y fashion" to the solution presented above. This is as far as I can get:

1. I'll obviously need a way to call Math.sqrt and then print it. This needs to be a function name that's english-y
2. An englishy description could be something like "Show the square root of 100". Using Groovy's existing rules, that makes "of" the candidate for the function name.
3. How now do we make the rest of the words work? Well, "show" and "square root" are exactly the two actions to be taken. So as long as I can define those as functions and compose them, I'm good.
4. How do I pass in the functions without naming them?

Obviously, its all the baggage of years of imperative programming knowledge holding me back. Somebody with more skill in functional programming might find this the intuitive enough to roll out.

But I have to wonder: There's too much magic going on here. The imperative approach to the stated requirement would be to define a grammar, write a parser and allow actual interpretation of such a language. Painstaking and error prone, sure; but explicit and clearly out of the programmer's head and into a language that is so dumb that there's no doubt as to what's happening.

This sounds like I'm propounding Worse Is Better in other words,  but there's something more: I realized that the cognitive overload required to understand the Groovy version is higher. More importantly, the cognitive overload to retain that understanding over time is even more so - for "normal" developers who have not walked the functional way for very long. That's the majority in today's world, at least.

More insidious, however, is the implied promise that this is real; that the Groovy DSL is actually English. Could I, for example, expect the slightly less polite "show the square_root of 100" to work? Or the even curt "square_root of 100"?  As an English speaker, why would I not?

As a programmer, I see why 'please' and 'the' are the required glue to make the english-y sentence work within Groovy's pretend-English world. Its extensible in that you could replace square_root with cube_root, for example; but not so that you could change the grammar of the sentence itself. That would require a different set of closures, like the ones you'd find in slide 105 of the same deck, for example. Note that in this version its 'the()' that is the active closure. But I fail to see why this should prevent me from expressing myself naturally as an English speaker.

This then, IMO, is the larger problem with DSLs. When you make something like its real-world counterpart, the human mind immediately taps into the huge body of latent real-world knowledge and tries to use it with that thing: although this DSL doesnt promise it, I immediately wanted to use English structure on it. It's all fine that your programming language has risen to talk your user's language, but has it captured enough of the user's world to go beyond "training wheel use"? To paraphrase Carl Sagan,: To make an apple pie DSL, you must make a universe that's up to scratch. 

How will the DSL author handle this? In general, not very well, I think. I realize I'm picking on a toy example here, but on extrapolation I cannot imagine a domain where all possible concepts and all of their possible combinations are easily enumerated; and then retain their meaning over time and across minds.

I begin to wonder then, at attempts like the VPRI's FONC, where one of the overarching ideas seems to be to create a multitude of DSLs to produce an OS in under ~20 KLOC; and at feeble attempts like my vaporware designs of Jack and Fluent: would they build a better world or is the imperative code bloat of Windows and Linux the more sustainable way?

JMeter as a webapp

I've long felt that it might be a good idea to create a web-based controller for JMeter. JMeter has been and still is a good tool for performance testing. Its Swing-based UI, however, has always been a bit of an eyesore and IIRC a memory hog. The one interesting behavior it has, however, is "save on tab", ie any data you type is automatically saved when you tab out of the control. Of course, the save happens to memory, so you still have to Ctrl-s to actually save. And then there's the "feature" of saving any chosen node as a new file. Why, I'll never know.

Anyhoo, my idea is to create a webapp version of the JMeter UI. The possibilities are interesting:

• Even a straight copy of the existing design should right away make a JMeter instance accessible via the intra/internet. Add features to actually kick off performance runs and you have a saucelabs equivalent for performance.
• If instead we tweaked the UX a bit and actually model the Performance testing workflow (which, btw, the JMeter XML supports inherently), the result could be vastly better. Create the test run workbench separate from actual runs, show reports as a separate view instead of as attached nodes, and so forth.

Explicit Language constructs for architectural concepts

You know how we have modules, layers and environments for code but none of these concepts are actually IN the language?

Modules are typically self contained chunks of code that expose an interface to some useful logic. It is not important how it does what it does; but it IS important that we be able to find the module and call it. In most OO languages, we use functions or object/class methods to hold such logic, but then have to actually build the concept of module using user-visible mechanisms.

Layers are another conceptual tool for organizing code. This typically reduces to packages or namespaces, but neither of these concepts actually enforce a hierarchy or "can/cant call" rules. Similarly, it would make great sense for dependencies themselves to be layered this way, but no language that supports imports also supports defining the layer for those dependencies.

Environments - which I mean the container or context in which code runs - are similarly underspecified.   Most languages define a main entry point and are call it done. Why not have a generic way of stating an entry point that includes a declaration of the environment (or even environments - plural)? The declaration of environment allows for the actual standup of said environment and trigger of the main code to be separated from the code itself; while the ability to declare that "this code runs in both gui and cli environments", for example, allows explicit declaration of such purity in the code and allows determining if this is true statically.

Well, this idea is to explicitly add those concepts to the language syntax itself.

The idea is that the implementation explicitly states the architecture assumed. Two advantages:

• The intended architecture is explicitly available to future developers
• Tools can be brought to bear on code directly, maybe even in the language

Of course, this does mean that the language knows only about specific types of organizing code. Within that limitation, however, things are much more clear, IMO.

Excel as a UI Mockup tool

I just finished using Excel as a UI Mockup tool and am amazed at just how usable it was.

More (with pictures, tips and techniques) when my left index finger is out of the bandage.

"Whose Turn?" App

This idea is from a recent outing of 3 couples with kids under six:

A Phone app that tells whose turn it is to change diaper/ deal with the lil monsters / feed them / whatever.

You choose the activity, it keeps score and tells you whose turn it is.
Once you're done you say you did it, and it updates the score.

Bonus points for doing things out of turn is a configurable setting :)

An agent-based parser for a structural IDE

I had been thinking about the difference between text editing and structured editing and it dawned upon me that the latter will never win simply because it is not "natural" to edit structurally and that such editing is not forgiving of "mistakes". Sadly, text editing is good by default; and structured editing is not so by design.

We have a grammar, so we use it. But do grammars have to be policemen all the time? After all, we humans created grammar; and I have to think that we created it by convention to make sense of the otherwise chaotic barrage of communicative sounds we invented. So if natural language grammar is intended to be "guidelines and lighthouses to the islands of information in the oceans of noise", why shouldn't computer language grammar be the same?

The problem, of course, is that such a grammar would be non-deterministic. How would you specify a terminal symbol with any kind of certainty when any possible other symbol could intervene and has to be considered noise in a very context-sensitive way?

I wonder if there's an other way; and the rest of this post will record my thoughts to date on this other way.

Imagine each symbol of the language's alphabet as a cell waiting for user input. Imagine each non-terminal in the language's grammar also as a similar cell, only it waits for symbols to activate.The moment a symbol appears in the input (this could be the read of the next symbol from a file or the next key pressed by the user) the symbol grabs it and announces that its now "active". This triggers "interested" non-terminals to claim that symbol as part of themselves, although the ownership is shared - until all symbols of a particular non-terminal are satisfied and it claims all symbols to itself; and announces that it is active. This triggers the next level of non-terminals to wake up, and so forth.

This sounds pretty close to how Agent-based systems work, hence the term in the title. If I were Microsoft, I'd probably call it ActiveParsingTM.

The key advantage with such parsing would be that ambiguous input could be handled by the system. Also, input that doesn't match any production rule in the grammar would still be allowed into the system; which means that erroneous user input is tolerated, not berated.

In addition, such a system would be incremental from the ground up: although it is backed by a grammar, input is not constrained to "start at the top and end at the bottom of the grammar". It is also inherently event-driven (see design notes below) and therefore should be able to react to editing actions better. This ties in well with "natural" structural editing.

Finally, such a system would allow for hybrid languages that embed other languages within themselves, eg HTML+CSS+JS.

I cannot imagine that this hasn't been thought of before, as always. My Goog-fu not being upto snuff, however, I was not able to ferret up anything concrete. Of course, I did find a lot of references in the NLP literature to agent based systems and automata based parsers and such like, but they usually devolved into stats or deep math to prove the value which I couldn't comprehend.

Design Notes

• The system works on heartbeats called ticks. Everything happens once every one tick.
• There are cells. Cells wait for external input. All cells are guaranteed to get a new datum input into the system within the same tick. Cells activate (or not) during that same tick for a particular datum. Cell activate recognizes a single symbol or character.
• There are cells that wait on other cells for input. These cells  activate when all the cells they need are activated. Their activation recognizes a token or non-terminal.
• The input could be control input (such as the delete key) as well. This would result in previously activated cells becoming deactivated, changing the state of the system. The impact would be felt in the system over the next few ticks, but also means that it would be local to only those parts of the grammar that are indeed affected by the change, not universal as is typical in standard parsing systems.
• At all levels, activation triggers an event that can be handled outside the system. In particular, recognition of a non-terminal could be handled as highlighting it as a known keyword or identifier, etc. This is how syntax highlighting would happen
• At a sufficiently high level, activation of a non-terminal means something larger than syntax highlighting, eg, validation, interpretation/compilation or execution can happen.
• Finally, all cells can be supplied with a prerequisite activation signal which would HAVE to be "on" for them to become on. This would allow controlling the "viewport" of visible cells easily enough to constrain the calculation to that part of the text that is visible.

Update

Found it! An undated paper about Object Oriented Parallel Parsing for Context-free Grammars. Although I couldnt find a date on it, references to Symbolic Lisp machines dates it to at least the 90s if not the 80s. Google search terms "parallel parsing computer language" 

Idea: Email + Document Server in a box

I dont yet have a good name for this concept, but here's the idea anyway:

You know a lot of good ideas are better put in an email instead of a separate document because people will not open that attachment?

What if you could actually then easily extract out such content and make it available as a document? Further, what if the original recipients of your email continue to see it as an email, but it has a separate life as a document?

This is what I mean by the Email server being a document server as well. Each email could contain multiple such documents; or each email could itself be a document. The inline vs attachment line is blurred, but at the same time, individual bits within emails become addressable entities outside of the email thread.

Right now, we save such emails, re-format them as documents and stick them in a wiki or sharepoint. What if the Email server did all this for us?

JS++

Sometimes you dont think of the obviousness of an idea until someone hits you on the head with it.

Typescript was released today and it finally dawned on me (despite the very visible existence of Coffeescript in that exact same mold for ages) that creating a pre-processor for Javascript that cleaned up all the messy bits for you was a viable option.

After all, that's how C++ came to be, isnt it?

But if I were to do it, I'd do it a bit differently:

• I'd not have the TS-style classes, interfaces and the like; instead I'd just have a simple object creation syntax that's better than the default JS one or the JSON-style one
• Modules would map to CommonJS or AMD modules, but they would behave (from a deployment perspective) more like Fantom pods than the module-within-module craziness that TS seems to expound. It sucks in Java and I'm convinced the Fantom model is better simply because "One conceptual container == one deployment container" is such a reduction in cognitive overload.
• Embrace hoisting for "top-down style" writing of code, but introduce true scopes to retain sanity.
• Some implementation-neutral way of embedding the ultimate javascript into its host environment through the preprocessor. This is something tools like yeoman and bower miss, IMO.

Those are some of my main themes, I'd say; there's more that another reading of the Crockford bible should produce.

Design notes: 

1. Since this should be a simple "if you see this jspp code, spit out that js code" type thing, I'm thinking something as simple as StringTemplate might suffice, although it might be too early to speculate that far. If it did though, I'd build the whole preprocessor as a series of calls to a function called becomes(srcstr,deststr)
2. The implementation will have to support output of multiple files from the processing of a single input file.
3. If this turns out to be small enough, we could finally have a JS language that runs transparently on Rhino. CS currently faces the issue that its main code is larger than the size that Rhino can handle.

Postscript: I'm tentatively calling this idea JS++ simply because that's how C++ came to be; but I can see how this might not make good marketing sense. If so, here are some alternative names:

• Better Javascript : .bs :)
• Easier JS: .ejs

Idea: Use a map framework to depict code

Today's XKCD comic and its interpretation as a zoomable view using Leaflet had me thinking of the possibilities this presents:

Software cartography already demonstrated how code could be converted into a map. It even has the interesting property that it attempts to map the mental model of the code instead of its specific implementation - which IMO is way better than something like Code City simply because the city (or country) looks the same even if a few buildings disappear - if you know what I mean.

The only missing piece is scale - how do you scale this up to larger and larger codebases? Well, using a map engine is one way, IMO.

The problems of scale have already been solved there, as is that of display form factor: most of the map frameworks are already mobile-ready. The UI metaphors are familiar with most people too.

The only possible thing that detracts from my grandiose view of an n-dimensional version of CodeBubbles  to depict the true complexity of code is that map engines are decidedly 2 dimensional. But even that is a weak argument - layers provide sufficient degrees of freedom to annotate the display appropriately.

Why new bottles with the same old wine works... everytime!

I attended the second BangaloreJS meet today. The sessions were good and so was the food; and I met some really interesting people so in all a fun time was had.

Before the sessions started, however, Jon - the meetup organizer - showed a quick demo of Yeoman: a new tool released by Paul Irish and some of his friends from Twitter that eases building client-side JS projects.

I had a senior moment when I finally realized what it was.

This was Rake in client-side JS clothing, which was ant in Ruby clothing, which was make in Java clothing, which was ... I dont know what came before make because that was waaay before I was born, but I know there was something else that all the cool kids left for make.

Yes, I realize this does way more than a simple build tool; much like rake does way more than just build code. And it did it in a way that seemed magical when it came out first, too. You could do all that with ant tasks/plugins but it was not the same thing. But that's besides the point. It was a build tool in its core.

Why do we as developers keep.reinventing.these.common.tools?

Answer: For the same reason tool vendors have different versions of the same tool by language.

See, I happened to be browsing JetBrains' website the other day and saw that they claim that their non-Java IDEs share the "same features as other Jetbrains IDEs" or something to that effect. It struck me that they surely have a generic base version of the IDE that is skinned and customized for each "vertical" i.e., language-specific IDE.

Why then do they have one tool per language?

Because there are very few polyglot programmers in the world. Majority of us stick to one language, maybe a platform with a few related ones. That's IT. So we want something that "speaks to us, knows our problems", not something that can boil the PL ocean. And why would we need all those generic, all-language features anyway, right?

So the vendors sell us the shiny new bottles. And we drink it up.

And that's the way it should be. Why should client-side JS devs (who are in an unprecedented hype cycle after a long winter) be grudged a "robust and opinionated client-side stack, comprised of tools and frameworks that can help developers quickly build beautiful web applications"?

Note to self: When you finally finish creating Fluent, remember to brand it by language. FluentC, FluentRuby, FluentJava all sound very usable :)

The "Save Download to Category" Browser Add-on

I download things all the time, and all of them to go to the same place always - my downloads folder. Its nice to have a single place to look at all the things I downloaded, but most of the time, I'm moving the download to someplace else once its downloaded. This two step process is annoying.

So the idea is this:
Have a browser add-on that allows you to pick where to put the downloaded file(s) with one click. Preferably with category tags that map to actual folders.

Todo: Check if something already exists. It most probably does.

Tools are the bane of the Beginner

"Huh? Isn't that counter-intuitive?", you ask?
I realizing something particular as a result of some experiences and wrote that line down, but to me even it sounds counter-intuitive; so let me hasten to explain.

A beginner, for the purposes of this discussion, is somebody who's beginning something. This could be a novice starting to learn a skill; but it could also be an expert who's beginning a new project within his area of expertise.

A a beginner (thus defined) is hindered by the presence of tools and frameworks for two reasons: 

• They obscure the what and the how of the problem at hand by hiding it within ( usually for novice beginners).
• They prevent easy exploration of the why of the problem and its solution space through ceremony and preventing access(usually for experienced beginners).

Allow me to present the experiences that led to this realization. It all began one mundane work day in the not-too-distant past....

Story#1

My team had just got a bunch of freshers. They were recent engineering graduates (presumably with some exposure to programming) who have passed through the company's training program (which again presumably imparts further such exposure). We, however, found that they couldn't do some simple tasks like write code outside Eclipse. They didn't know how to deploy a web application except through the Eclipse plugin; had never debugged an application via logs and in fact didn't know  about webapps as an idea independent from "Tomcat". Their OO concepts were shaky at best, but they had implemented small-but-complete web applications using Tomcat, Struts and Hibernate and passed an EJB exam. When asked to build their study app the same from scratch using a command line, however, they were lost. When asked to build a different application (than the one they'd done) *using Eclipse*, they were similarly lost.

While a large portion of the blame should rightly lie in the teaching methodology (or lack of one), the tools and frameworks too, IMO, should bear some of it. "Deploying" for them meant clicking on the Tomcat icon in Eclipse, so they had no use of knowing what "web.xml" did nor did they know that it was no longer required. The same wizards and menu options that make the life of a practitioner easy actually obscure the underlying process (and why its required) to a beginner.

Story#2

The same group of freshers were slowly getting on track with (re)learning the basics of programming, when I thought it might be a good idea to instill in them at this "early age" the values of Test Driven Development. I immediately checked myself, however, because they'd have to learn JUnit and how to use it. On second thoughts, however, I realized that they didn't HAVE to use Junit or any such framework to do TDD. All they had to do was write a test before writing the actual code, have it fail, write the code and have it pass the test. The test could be code in the same function or in main() or as a series of calls to the program stored as a shell script or as JUnits. All of them are equally valid as "tests". We generally, however, recognize only the last of these as tests. The concept of TDD has been usurped by the concepts of the tools that implement it - to the extent that TDD doesn't seem have a life outside of those tools.

Tools, therefore, seem to be actively scuttling the consumption and adoption of Concepts, even when they were created explicitly for the reason of automating the repeated application of known concepts.

I personally have been struggling with this - the guilt of not doing TDD vs the allure of just seeing working code - especially when I'm beginning something and still feeling the problem and solution spaces out. In some cases, the solution space doesnt have readily available tools (BDD on the browser, anyone?) and in others there are tools but I'm still not ready to commit to them because I dont know what my solution is yet (should I build the parser first to figure out the syntax or the AST interpreter to see how it would run?). My liberation came when I declared that a test will be whatever I call a test for the situation at hand, not what some framework determines to be one. Since then the test-red-code-green-repeat cycle is a much more doable one.

Full Circle

Back to the initial "Huh?" moment from the beginning. Why then do we generally consider tools to be useful - especially for beginners? Tools are generally time-savers. They do one thing and they do it well; and that is their value. They do, however, have an "operating range" in which they're most useful. Below that, they're overkill and above, they're obstructive - as depicted in this highly accurate graph on problem size vs tool effectiveness:

So when we usually talk about tools being useful, we're talking about the useful operating range. Specifically for beginners, tools are solution accelerators at that range. The stories presented here represent the two ends of the spectrum, however, where tools are sub-optimal.

Note: I've glossed over frameworks in this discussion, but the concept is the same; or applicable even more so for frameworks. Frameworks by definition are a standard solution to a common problem, with room for customization so that application specifics can still be implemented. The framework is one because it has a known world view and exposes an interface that allows operations on that world view. The concept of operating range is well-ingrained, therefore; as are those of the limits on either side.So please read "tools/frameworks" wherever you see "tools" below.

So...

Armed with this framework for evaluating tools, we can start asking some interesting questions.

1. What is a good tool?
2. When are tools not required?
3. When are tools required?
4. How do we determine the operating range of a tool, then? 
5. What can we do to use tools more effectively?
6. What can tool builders do to make effective tools

Attempts at answers to these questions in part 2 of this article.

What if there IS no source code?

...yeah, kinda linkbait-y heading, I know.

What I mean is: What if there is no single source of the code?

Let me explain.

Typically we have source code. Its written by someone, stuck in a source control system somewhere, changed by others and so on.

The projectional editing school of thought modifies that picture somewhat by suggesting that we could have different views of the same code - a functional/domain-specific one, a folded one, a UML(ish) one, a running trace one and so forth. The relationship between the source and the multiple views, however, remains decidedly one-to-many.

What if instead of this master-slave relation, the different views themselves were the source? That is, the "whole picture" is distributed across the views - like a peer network?

Assuming the views are consistent with each other, modifying the source in one view should retain the true intent. But is that possible even? Views are, by definition, projections of the code; meaning some parts are included in the view and some aren't. So how would we maintain consistence across multiple views?

Two paths lead from here:
One: We cannot. This is why we need the "one true source" that's the parent of all views.
Two: Maybe we don't need consistency all the time. Maybe we can do with the "eventual consistency" that big data/nosql guys are raving about?

#2 seems like an interesting rabbit hole to explore :)

Naked Objects for Mobile

Why not, I ask?

The Naked Objects Pattern has been an idea that I've always found interesting and useful for quick-n-dirty apps.

The Apache Isis way of NO seems a little bit overwrought (as most of Apache projects are), but I've been following JMatter for quite a while now and have been impressed despite it being a Swing-only UI.

For the simple one-user app, the NO pattern seems sufficient: the thing gets built quickly, there's lowered expectation of customized UI from the user and probably even platform portability.

Snipped from the web: an idea related to GUTS

HN > Techcrunch story > Google Search for paper mentioned > Ch1 of paper, which had this nugget:

"I noticed that, often, the mere presence of a feature or capability incurs some cost, even when not being used. For example, as the number of services and options multiply, extra code is required to select from among them, and to check for possible interference between them. This code does no useful work processing the application's data, yet it adds overhead to each and every call."

I like this idea of the presence of code causing things to change around it - seems like something that can be quantified usefully by GUTS. 

I am Jack's total lack of unlayered design

A nebulous idea grows within me partly from GUTSE's insight that all programming is mere translation from one  language to another: All Von-Neumann machines are merely SSI (Sequence, Selection, Iteration) executors that essentially facilitate these translations. Why not enshrine these facts directly, then? The design of Jack, therefore, stands refined to:

• A simple SSI executor at the base with no idea of what data is beyond the generic map that it reads in as input.
• A data layer (built using the SSI + its environment) that actually does know about the bits of data stored in the map. This layer essentially provides the box/unbox, serialize/deserialize, parse/print dual functions as required
• The data transformation layer (built using the layers below) that enable and reason out the transformations declaratively so that they can be composed, collapsed etc.

Coin, Lambda, Jigsaw and EJB Home Automation: My take on the JavaOne 2012 Hyderabad

So nearly 13 years after my first java program, I finally attended a JavaOne. I remember sitting in Chennai years ago wishing I were in California when I first heard of them (or read about talks from them months later); so when I heard about a JavaOne in Hyderabad, I figured it was time. Of course, its not the same thing without Sun hosting it, but I was also interested to see how Oracle rolls one out. Few days of finagling with the powers-that-be later, I was set to go.

I've been to Hyderabad before, but not by air. So when I landed and found an airport that could be mistaken for any mid-size one in Europe or the US, I had to concede that Bangalore with its sneeze-n-you're-either-out-or-on-the-runway airport has been severely outclassed. Add to that the faithful reproduction of a US expressway (with exit numbers, no less) that led from the airport to the city and a die-hard bangalorean like me had to just weep and die a little bit inside. Damn you, you f-in politicians and whoever else who's responsible for our rinky-dink airport and the tragedy that is NH-7. You know you've had the revenue in spades to do a better and quicker job.

And then there was the convention center itself. Pretty much world class, IMO. Easily configurable main hall (attendees know what I'm talking about), a control console in each room, each room equipped with at least 2 screens and 2 LCDs facing the speaker - one to show the slides and one with a timer, ample wait staff and F&B counters, attached hotel - the works. Of course, the event management itself had some typical desi-ness to it that I'll mention below, but in all a very nice place. It had the feel of somebody with systemic thinking being in charge.

Anyhoo onto the sessions themselves. There were 5-7 parallel talks spread across Oracle-specific and Java themes so I'll talk about the sessions I attended:

• Day 1:
•  Keynote: Pretty OK state-of-mobile talk by Mr. Rego. Nice demos of Nano Ganesh (controlling irrigation pumps using Nokia phones) and of controlling cars via phone. Nice message about "looking for problems to solve and the money will follow". Also liked Nokia's message that they're not about creating new things themselves, but enabling and partnering with people who are creating those new things. I liked this message because I've always like Nokia phones as sturdy workhorses of communication and IMO with all the smartphone craziness this is a good middle path to take. Hope it works out for them.
• Aside: The Nano Ganesh project uses the phone's voice connection to control pumps. I spoke to Santosh Ostwal - the creator - and he confirmed there's no API to this - its direct control of the hardware. Sad. It would have been great to write this kind of phone app - the kind that doesnt need a separate data connection.
• Head and tail of Project Coin: Good "Directors cut" review of what its about by Dalibor Topic. Much better than just the changes.
• Data Parallelism with Fork/Join: Blew my mind when they got to Spliterable. Very well explained by David Holmes. He didn't like my fanboy idea of reviving the old java idea of sending code via RMI to another JVM so that we could federate fork/join over multiple JVMs, however. Probably with good reason :). 
• Oracle SOA suite: Caught the last of this because I was talking to David Holmes, so can't say much about it.
• Tech Keynote 1: Nice overviews of Coin, Jigsaw and Fork/Join. Particularly liked Dalibor's animated slides that converted boilerplate java into its functional equivalent.
• Improving MSQL performance with Hadoop: Lost patience. Speaker gave very naive definitions of MySQL and Hadoop and showed an agenda that didn't actually have the title of the talk listed.
• Project Lambda: Excellent build up of the material and exposition of the JSR as it stands.
• Java ME and LWUIT: Attended the end of this lab session. Seemed pretty standard UI stuff.
• OTN Night: Not easy to get a few thousand indian geeks to "get jiggy", but to their credit Oracle tried. First there was a stand-up comic who apparently was a developer at some time, so was billed as a comic in 4 languages: English, Hindi, Java and C++. Sadly his tech jokes stopped at calling himself a Null Pointer. Aside from asking a whole crowd of south indians if "there were any south indians" and making the cardinal sin of assuming all southies are the same when there was a whole wealth of jokes with just mallus alone, he was OK. That was followed by Vasundara Das trying really hard to get 5000 engineers to confirm that the "party was on the dance floor" to her chartbuster "Where's the party tonite?". When I left about 100 hands were up in the air somewhat in chorus with the slightly-off background troupe. The food was good, though :)
• Day 2
• Oracle Develop Keynote: Some clueless event management guy goaded me into this with a "Developer keynote starting, please go, please go", so I went in. 5 seconds in I was thinking "How the hell is this a developer keynote, I'm walking out right now". I remembered, however, that I should probably listen to some Oracle speak for use "at the office" and settled down. It turned out to be pretty Ok after all, with Oracle's Cloud strategy taking center stage. David Pease presented a nice (possibly standard, but nice anyway) cloud adoption strategy and spoke about their public and private cloud offerings. Afterwards, him and another Oracle Manager did a pretty good job of answering my two questions:  How the cloud affects their main revenue stream - big iron database servers (answer: complimentary, not cannibalistic), and what their official response was to NoSQL(ans: berkleyDB. Not convincing, IMO) and Big Data (Ans: Will be treated as extension of existing solutions, not a separate one).
• Interfacing with the interface: Impressive demo of JavaFX, Wiimote, Kinect and Hotspot. It was one of those "It CAN be done with Java too" kinda things. Cool, but dont see the point. Wouldnt you just use the native technologies for Wiimote and Kinect? Still a very balanced and nice talk. Take away: Heard about the JMonkey Engine. Also heard that its pretty buggy :)
• Agility and Scalability @ eBay: Good, solid talk; but nothing new. Walked away into the "interfacing.." talk after a while, returned towards the end and found it taking a predictable course. No fault of the speaker, though. It was me, not him; entirely like seeing George Carlin for the 5000th time and not laughing. The rest of the crowd seemed to like it just fine.
• Jigsaw: Another solid talk by David Holmes. I started reading about the project after the talk and immediately wanted those slides that presented the problem, its implications and solution(s) so well. Reaction to the project itself: Cant wait for it, cant wait to use it. Although I dont get the need for the "default" keyword for default methods. If you're writing a method body within an interface definition, shouldnt that be sufficient indication that its a default method? Holmes' take was to join the mailing lists and either understand or speak up. 
• Nashorn - Javascript on JDK: Short, well presented talk on the new JS interpreter on the JVM. Sunderrajan was awesome at spinning out quick examples to questions right on spot. He was evasive about performance due to safe harbor rules, but was forthcoming enough to say its way better than Rhino. Killer announcement: Node is coming to the JVM, and they already have it working internally. Cant wait.
• Aside: I mentioned that I was excited about Node-on-JVM to Steve Chin and Kevin Nilson and their response was: "why would anyone want to do that?". My answer: admins love the JVM. Put anything inside it, and they'll be OK with it. The JVM is the ultimate production grade sneakware platform :)
• OTN Room: Monitoring Cloud Apps: Spent 10 mins here. People in the audience kept interrupting the speaker with more knowledgable information than he had. Good to get good information, not so good to have the talk's flow being interrupted thus. IMO Ranganath should have just set the rules of engagement better than being polite like he was, or the session should have been a BOF one.
• Java beyond the IDE: Nice if boring-in-the-middle talk by Jay Suri on things less talked about in the Java scene, namely, issues in production. After a long list of issues the Java Platform Group has heard from customers (and their solutions), Suri introduced two tools that were coming to the JDK via JRockit - Mission Control and Flight Recorder. This latter tool is what everyone wants in production - instrumentation for a running JVM - and you can now have it! Expressly to be run in production! From the source! Of course, you'll need an Oracle license for production use, but depending on the price, I can easily see this being put on at least "Test in Production" servers for those hard-to-debug-in-preprod problems. Awesome.
• Having fun with JavEE and Home Automation: Who would have thought of using an EJB server to do home automation? A crazy, funny, energetic brazilian and his wife, that's who.  Vinicius and Yara Seneger were genuinely happy to be in India and gave what was possibly the most lively presentation that I saw. They demoed JHome - a home automation software platform that uses Glassfish and Arduino to create home automation such as lamps that turn on via http, LED lamps that turn to any RGB colour, controlling appliances  via sound. The final demo was a heartbeat monitor that Vinicius had hooked up to some EJB code (Yes! EJB code!) so as to light up a lamp when the heart rate goes over 110 BPS. And of course, he had to jog to get it working. Awesome and endearing. Who cares if they used a whole herd of elephants to swat a fly? The energy of the guy was great. Couldn't have asked for a better last session :).
• Speaking of: why wasn't there a one last hurrah Oracle? I'd have liked one last closing session on the center stage. And: some of us like to see the people behind the show, you know? Get 'em up the stage and let's take a gander at them. 

Tech Reactions/ Opinions:

• UI Frameworks: Java has AWT(hopefully dead now), Swing, JFX and LWUIT (which I didn't know existed till this conference). Exactly how many of them does one language need? I spoke a bit about this to Steve, but he seemed unconvinced that it might be a good idea to consolidate. Maybe the way I put it was off, but is it too much to ask for one language to have one (preferably declarative) way to define a UI across all its deployment profiles?
• Mobile: 
• The elephant in the room was walked all the way around pretty carefully by Oracle. Nokia had one A-bomb, while Oracle billed the "mobile platform that must not be named" quaintly as "other linux tablets". The stoopid questioners mentioned below, however asked pointed questions on integrating with it, however. Was kinda funny to watch the Oraclers tippy toe around these ones.
• The heartening news, however, was that the release schedules for J2ME was merging with the J2SE one and that Java 8 will have one of the ME profiles built in. Should bring solace to J2ME users, i guess. Do you know any? I dont.

General Opinions:

• Speakers almost always took up all the time, leaving no "public Q&A" time. They were of course, super available in the hallway after the talk, but that meant fighting the hordes to get your one question answered.
• There were some really stoopid questioners, which the speakers were gracious enough to deign to answer. 
• Sample: To the nashorn team member: "Will it support eval?". Ans: Of course, its a language requirement. "Will it support closures, lambdas and anonymous functions?" Ans: Of course, its a language requirement.". "Ok, I understand it will support lambdas because of Project Lambda, but what about closures and anonymous functions?" Ans: "Look, we have to, cos its part of the language. And this has nothing to do with Lambdas in Java". This went on for a few more rounds.
• It was even worse in the Oracle Dev sessions where people generally were summarizing the talks in the Q&A sessions.
• Event management was a bit weird in places:
• No wifi. Had to be said. In fact, their "suggestion" was to not get any laptops either. I get that the majority attendees are local and probably have their own data connections, but what about us out-of-towners who might want to tweet you some kudos? Surely the world's largest database company should be able to afford that?
• Long queues to get into a session because Oracle wanted head count. Couldn't you have set the Reader at the door and made it self-service? Many people kept leaving sessions and walking into others anyway thereby skewing your count; and this made for an annoying delay at each session.
• Food service had some desi quirks: 
• I had to literally fight for a second plate at lunch (I discarded my first one after my first course) because they dont give out a plate unless you have a token. Explaining that I'd given mine already and all I wanted was a fresh plate didn't go too well. The second day I learnt my lesson.
• Tea was served at these stations with really nice looking tea cups-n-saucers stacked up. But they didn't serve us tea in them. Instead we got it in paper cups. Yeah, no clue why, or even why they had them on display if they didn't intend to give them out. Once you got your tea and took a sip as you walked away was when you realized there's no sugar. Walk back, and you're pointed to these tables with bowls containing sugar sachets, but no stirrers. Those precious bits of cutlery obviously cannot be left unguarded, so they're in the safe custody of the F&B guy - who could have told you that the first time, but of course he didn't. So walk back, get stirrer, walk back, get sugar. Then drink. I saw this happen over and over again in those two days. Its funny when you know what's gonna happen and you watch some poor sap go over this drill.
• The Swag embargo: At the end of the conference I saw a line snaking towards the registration booth and some people emerging with a t-shirt and mug. The mug I didn't care about, but I'm a t-shirt whore. So I stood in line, not really taking note of the people in line frantically filling out the feedback form. Apparently, you HAVE to fill the feedback form, else you dont get the swag. What if I took the things and came back with the form filled out, I asked? No sir, its the rules, they said. I'll wait right where you can see me, I countered. No sir, those are the rules, they said; helpless against the all-seeing all-dancing mother directive. So I filled it out as Doofenschmirtz, Evil Genius. They gave me the stuff without once looking at the paper. 

Thus went my first JavaOne. Oracle, if you're reading: Now you have my real feedback. Looking forward to more of the good stuff and less of the silly stuff.

Update: Not finding the slides from the talks in some official place, I scoured the internets for them so I could share with my team. Here're what I've found to date:

• Tech Keynote, alternate
• Another version, but with Fork/Join and Spliterable stuff included.
• Fork/Join Alone
• Project Lambda
• JDK 7 and Beyond
• Nashorn: The sort-of-official link from Dalibor's blog post doesnt work. I couldnt find any other place that had the "Future of Javascript on JDK" slides. Comment below if you find it.
• Project Coin
• Project Jigsaw
• The JavaOne India site seems to be geared for uploading of slides at some future date, going by what I found on the site from the one that happened last year in South America. Look there a while later to see if they've uploaded any. Right now (Mid May 2012), the list of talks itself is incomplete.