Eternal digressions of the engineer mind

I had an idea for a stream-of-consciousness editor. So I started creating a prototype using node-webkit, codemirror and my non-existent ui skills.

Since I did all this on the PC and wanted to sync up with the mac, I started to setup node-webkit on it, only to realize that there are no binaries for osx 10.6.x.

So I downloaded node-webkit (using up 35gigs of bandwidth by the 10th of the month - kids and wife will not see their videos) and got it running.

Then I realized that the code for the editor itself could do with some modularization - modularization of the kind that my betterscript language promises. That project, which was last worked on last year on one day - a hacknight - now got revived.

Of course, betterscript uses ometa-js, which I'd copied directly onto the betterscript project (along with its dependency - rhino) and duct-taped together a script that worked only for that version of betterscript.

Today's me, however, was not happy with this state of affairs. First off, ometa-js should be an external dependency. Also, even if it were an external dep, the original author had not really organized the code much. He'd just done the important bits - ie the really interesting peg parser in JS that parsed both text and objects alike - and left the rest of the "productionization" to us vogons. This wouldnt do, of course. I'd like some organization in that code, for cryin' out loud.

This required looking up the original github project and its 400 some-odd forks to see which one met my needs. There were a few that did re-org the code, but not in the "just right" way: they'd either stop short of what I wanted, or go way beyond the call and rewrite the code.

So of course, I forked the original, reorg-ed the code my way and tried to use it as a dependency on betterscript. Which, of course, meant that now I had to rewrite the original ometa-rhino.js with whatever changes I'd made in the file with the same name within betterscript. This lead to the realization after one sleepless night (doh) that outputing an object to stdout will reduce it to a string that cannot be read back as an object. Off to Crockford's for a bit of json2.js.

As of today:

• my fork of ometa-js works when run using my lanucher with rhino. Other modes not working yet.
• betterscript works using my fork of ometa-js on simple samples. Still a long way to go before I send that post out to jashkenas :)
• soced is encouraging concept-wise, still to reach its usability nirvana!

Final note: I thought it might a good idea to represent all these dependencies as a Graphviz graph. That turned into a journey down the "how do I get graphviz to show the nodes in the order I want " rabbit hole.

Sigh.

The phrase "Eternal Digressions" is somewhat inspired by the phrase "Infinite Digressions", which is copyrighted by Praveen. Alluded to here with no permission whatsoever :) 

Built node-webkit...finally

So I'm still on Snow Leopard, which means node-webkit's pre-built binaries are of no help - they start at 10.7. The pithy response to the very pertinent question of "How the hell do I run this awesome piece of software on my apple-fanboy-version-of-win2k?" has been "Oh, just build it on 10.6.x yourself. I hear it works fine then".

In a rush induced by the success of my "Code as you think" prototype working really nicely on the PC, I decided that I must have it running on the Mac too. I did consider upgrading to Mountain Lion, but decided that building node-webkit like they'd blithely suggested might be smaller a peak to scale than having to deal with a whole new version of OSX.

Downloading the code took about a day and a half. Of course, I was not surprised, considering this was  the Chromium codebase, plus node-webkit's own. Actually, the final download took about 3-4 hours, but it failed (ie 'gclient sync' got stuck and had to be ctrl-c'd) about 4-5 times. Here're the things that I learnt/thought up during these failures:

• How to setup the mac so it doesnt sleep - using screensavers and energy settings. Ended up using neither: was going away for the night, didnt want to leave machine unmanned.
• How to piss spouse off with blank stares and disconnected answers while download process failed yet again.
• How it would be nice if code sync processes had pause and resume options. wget has it, why not gclient/git?

Anyhow, all the code finally came down at around 1:40 am. Of course, I had to build it right then, so after some more consulting of multiple documents, I fired off the build commands. A full 1.5 hours later, I was the proud owner of a self-built version of node-webkit. The process completely maxed out all 4 cpus on my mac for the majority of the time, which I spent browsing the ninja build tool manual. Of course, it was almost finished by the time I'd reached the part on how to control the number of cpu's to allow for ninja's use :). The mac, however, performed admirably under stress, I must say. The rest of the running processes were still very usable and the terminal itself was still responsive. Not the same experience on a Windows or linux box, in my experience.

In all, time well spent; and catharsis for disappointments from the past. I might just run the build commands for Chromium next.

And once I've patched up with the spouse, I might even try getting it to run on Windows with VS Express!

Code as you think

I was in the middle of deciding how to proceed with my current side project when I realized that I'd not yet created an architectural description for it. Since I already have an architecture description tool that I wrote, I have no excuse for not writing one out. So I stopped to think why I didn't create an architecture definition first. Ans: Because it would stop me from doing what I was doing right then, which was to decide:

• how to separate out the current spike of the language from its sample code and actions.
• how this will be deployed in and of itself, ie how to dogfood it?
• how to move from the spike folder to the actual folder
• how to move from bitbucket(where I'm keeping all this stuff till its public-worthy) to github or a public bitbucket repo (where I expect this to be shared with the world)
• ...all in the middle of adding commit messages for changes
• while juggling code between the PC and the mac
• ... and so on.

As I'm writing this, more thoughts are streaming through, and they're all jumbled together: code ideas, architecture ideas, documentation, behind-the-scenes reasons, journal entries - all in one single stream of consciousness. Stopping now to create an architecture definition would not just slow me down, it would take me down a totally different path. And I don't want to go down that path because the IDEAS ARE JUST FLOWING!

Another reason I didn't want to context switch into "architecture definition mode" was the high cost of that switch : create a new document, decide where to save it, type in the actual content, optionally see how it looks once the markdown becomes html and so forth. IMO, this is also why documentation is out-of-date with code, especially expository documentation like architecture and design docs. The comment-above-code kind of documentation may still be in touch with the reality of the code, but higher level documentation like READMEs and user guides quickly become disconnected from the code they're talking about.

That's when it hit me: What we need are tools and languages that understand the stream-of-consiousness way of building software: code snippets flying into the system from the human mind along with documentation and random thoughts and todos and reorganizations of such thoughts - all of which the system will record in sequence and generate meaningful views out of. 

Imagine an editor that:

• Allows creation of content without the need to pick a name or a location for it.
• Knows enough about the content to provide appropriate editing affordances - syntax highlighting, preview, whatever.
• ... and autosaves it for you - no separate save step required
• ... then allows tagging it as being "of a particular type"... whatever that means to you
• ... then allows you to seamlessly retag it differently
• ... including being able to arrange it in hierarchies (if that made sense) or graphs (ie nodes and links - if that made better sense)
• Presents all existing content and its associated "tag graph"
• Allows linking content with each other
• ... or even parts of content with each other.
• Shows the links visibly
• Allows tagging parts of a document as being of a different tag, but still housed in the source document. Content tagged thus will be displayed in both tagged content, but the second display will be a reference to the original.
• Tracks all changes to the content graph in a stream-of-consciousness journal that's visible and accessible at all times so that you can see what you did before in the order that you did it.
• Allows you to add your own manual entries to the journal to track your thoughts at that point in time.

Such an editor could be used to make the context switches mentioned above as seamless as possible. The only bit of automation it promises is the decidedly "big brother"-like journal, but in my opinion that's something the computer should do for us - track what we're doing automatically so we can look back and "see the trace of our thoughts from before in our actions". Features like seamless switching allows easy creation of design, code, comments  and code in back-n-forth style, for example; while the ability to link content allows for todos to be linked to the point in code where change is required, but still can be maintained as a separate list of todos for easy review.

To allow easy adoption, such an editor should add the following:

• Treat an existing directory structure as the basis for its tags: folders become tags, folder structures become hierarchical tags, files become fully described hierarchical tags. If starting from a blank folder, however, allow for the initial tagged content to remain as files until the user discovers the "natural order of things" for himself.
• Seamlessly convert files to folders when the user renames them.
• Save the changes to the underlying folder as soon as possible so that there's no "last minute unsaved content change"
• Allow for some scripting/automation triggered by content changes.

UI Design Notes:

• Start the editor from a particular dir, which we'll call <dir> in these notes. The startup sequence allows choosing the default language to use optionally, else the default is text.
• The editor starts with two windows: a default one titled "<dir>" and the journal titled "stream". The former is editable, the latter is read-only. 
• The default window is "in" the default language (if chosen), ie is syntax highlighted for that language. Typing text into it and hitting enter causes text to show up like any editor, but it also causes the same text to show up in the stream window with a date/time stamp. Internally, the text in the stream window is a reference to the line in the original window, not a copy of its contents. In the background, all the text changes are saved back to <dir>.
• Typing F2 allows you to rename the window's title, which is also its tag. The editor allows creation of hierarchical tags like "tag1/tag2/tag3". It also allows setting the language of the content by naming the tag with the format "tag.type" - similar to how files are named today.
• Typing an esc puts you into "enter a tag" mode. a small edit area (or vim/emacs-style command area) shows up which allows you to type in the name of a tag such as "todo" or "comment" or whatever. hit enter when done and two things happen: the edit area disappears and a new window titled with the tag just entered shows up and focus moves to that window. Type text and hit enter to enter content in this mode, while it also appears the stream window like before. if the tag is already present, focus shifts to the existing window for that tag and text input proceeds from there.
• Selecting text and then pressing esc to select a tag will leave the text in the original window, but add that text to the window corresponding to the chosen tag (via reference again), as well as denote that the tagging event occured in the stream window. Presssing ctrl-esc instead of esc will cause the text to be moved to the tag chosen.
• Ctrl-tab will cycle through all open windows and allow you to make any window the main one. The same esc and enter behavior holds for all windows.
• Everything is save-as-you-type. No ctrl-s required. The tag is the name of the content.

More features:

• Windows are tiled, not free. this allows pinning of windows to specific locations so as to create familiar screen layouts.
• A third kind of window is a modal window, which can be in input or output mode. this will allow creation of repls. issue a command when its in input mode and view results when its in output mode, which is read-only. The "stream" window can also be recast as a modal window, allowing journal entries in input mode.

Ok, that was the easy part. What about a s-o-c language?

Such a language would:

• Allow for programs to be expressed one chunk at a time. The language runtime could be invoked at any time during the expression, at which event the runtime would evaluate the current set of chunks and determine if it can do anything with it. If the runtime is a compiler, it would decide if object code can be generated or not; and if its an interpreter, it would interpret the chunks available.
• Not have comments. Instead it would allow a more generic interpretation of commented content as "chunks from another language embedded within itself, which could be "understood" or "translated to itself" given an appropriate cross-interpreter/compiler.
• Allow chunks to be linked to each other and handle link lifecycle events. Eg: chunk A is related to chunk B could imply that if A changes, B should also change; thus flag such changes on A as requiring changes in B as well. Or, if chunk A is deleted, chunk B should be deleted as well because it doesnt make sense to keep it around anymore. Or, if chunk A is created, a corresponding chunk B MUST be created.

More thoughts on the language TBD.

Implementation Notes

Editor:

• Use node-webkit, codemirror and sharejs to allow evented entry of text
• Make a repl using a bot that's sitting at the other end of a sharejs connection.

WIP notes

Started work on a prototype using node-webkit. A basic version of the UI Design has been implemented, and the updated version of this post reflects some experiences from this implementation.

bash tip: setting terminal tab title programatically

This one has been annoying me for quite a while - looking at the string of tabs in the mac's Terminal App, all of them titled "bash" except for the one trying to download webkit which is usefully titled "git".

So I finally decided to do something about it, namely google the solution. Here're the results:

As a bonus, the cd() function adds the ability to change the title to whatever directory you currently are on - which works just fine when you're in your projects directory :)

I think this should work on Linux as well; confirmation TBD.

The Ultimate Personal Task Manager

I've been prototyping a GDoc-based 43 folders for a few weeks now. The biggest two problems I have to date are:

• the ease with which I miss my daily review of things to do
• "Too much trees, too less forest". I need a dashboard that's higher than the daily, ground level view

In a fit of frustration about the second point (cos there's no point fretting about the first:) ), I set about designing the ideal personal task management system - once again. Here's the result:

The Ultimate Personal Task Manager needs:

1. The open task list: which contains all tasks that are to be started or WIP. Routine/repetitive tasks are represented as new open tasks for the next instance when the current one is completed.
2. The review/decide process: which reviews current status and decides what to do next.
3. The todo list: which contains tasks that have been decided upon, and will be acted on in a particular time period - day/week/month/whatever. <==> 43 folders. no sequencing, just "these tasks mush happen within this period or before this milestone"
4. The plan: sequenced list of tasks. <==> day planner. contains time slots when things will be done
5. The dashboard: "dont break the chain" style display of only top level projects. ideally this should be the top 5 things you care about in life shown as a red/yellow/green style view
6. The nagger: Most important piece for procrastinators. the physical reminder to actually use this system. could be a phone reminder system or a spouse/friend. should know what your frequency to run the decision process is and nag you to do it.

Key point about this system: It could be all in your head or on paper/pencil or software

Yes, this is pretty close to GTD. The additions are the dashboard and the nagger. 

If implemented as software it also needs to:

1. Work offline
2. Works across OSs and systems
3. Allow versioning

Software implementation notes:

- Something that works offline and universally is required. simplest i've seen to date is gina trapani's todo.txt. Using it, the functional requirements:

- #1 & 3 are built in.  Use 'repeat' addon for sub point under 1

- #2 and 6 are human tasks anyway.

- #4 and #5 will need additional dev. In particular #5 will require (at least for me) enhancing the project field to a tree of projects. Some simple testing with bash showed that a format like "proj1/sub1/sub11" will work, as will "proj1:sub1:sub11"

  The non functionals have direct answers:

  - it works offline and across OSs - being text and a shell script

  - putting it in a git repo will handle versioning

- Idea: Create an Outlook plugin to auto convert mail into tasks

- how to run cygwin bash from windows? ans: Cygwin's run command

- how to create an outlook plugin? TBD

Groovy DSL Voodoo... or why I think imperative programming makes more evolutionary sense

I recently had to consider the Groovy space for a project proposal as the client had decided on Groovy/Grails as the platform of choice. While evaluating the platform for the proposal, I naturally gravitated towards Groovy's DSL capabilities and started thinking about using it for my personal projects, obviously. Abu seems like a natural fit (why didnt I think of Groovy before, I wonder? I'd hit a wall with JRuby due to generics anyway); as did Jack.

Anyhoo, with all this background, I hit slideshare and found tons of decks from the language leads on the specific topic of DSLs. One example piqued my interest in particular: slide 106 of Paul King's deck, which I've replicated here as a gist:

I wanted to both understand how this worked and to replicate it by myself. Here's all the flailing I went about in trying to achieve those two goals:

As you can see, I failed miserably. My last trial had a near-working version, but left me completely disillusioned about Groovy's DSL capabilities - especially considering I came out of the process feeling that all the work was done by the single 'of' closure with the rest playing dummies to the Groovy language's optional syntax rules - especially because in my Englishy version, you could switch the order of the closures passed into 'of' and still have it work fine.

However, all of this didn't explain the "extra code" in the original, which left me with the nagging feeling that no language author would be trying to pull this much wool over people's eyes. So I went back and expanded the original expressions like so:

Now it made much better sense.

Mind you, I still haven't figured out how you go from a problem statement like "I want to express the ability to compute square roots in an English-y fashion" to the solution presented above. This is as far as I can get:

1. I'll obviously need a way to call Math.sqrt and then print it. This needs to be a function name that's english-y
2. An englishy description could be something like "Show the square root of 100". Using Groovy's existing rules, that makes "of" the candidate for the function name.
3. How now do we make the rest of the words work? Well, "show" and "square root" are exactly the two actions to be taken. So as long as I can define those as functions and compose them, I'm good.
4. How do I pass in the functions without naming them?

Obviously, its all the baggage of years of imperative programming knowledge holding me back. Somebody with more skill in functional programming might find this the intuitive enough to roll out.

But I have to wonder: There's too much magic going on here. The imperative approach to the stated requirement would be to define a grammar, write a parser and allow actual interpretation of such a language. Painstaking and error prone, sure; but explicit and clearly out of the programmer's head and into a language that is so dumb that there's no doubt as to what's happening.

This sounds like I'm propounding Worse Is Better in other words,  but there's something more: I realized that the cognitive overload required to understand the Groovy version is higher. More importantly, the cognitive overload to retain that understanding over time is even more so - for "normal" developers who have not walked the functional way for very long. That's the majority in today's world, at least.

More insidious, however, is the implied promise that this is real; that the Groovy DSL is actually English. Could I, for example, expect the slightly less polite "show the square_root of 100" to work? Or the even curt "square_root of 100"?  As an English speaker, why would I not?

As a programmer, I see why 'please' and 'the' are the required glue to make the english-y sentence work within Groovy's pretend-English world. Its extensible in that you could replace square_root with cube_root, for example; but not so that you could change the grammar of the sentence itself. That would require a different set of closures, like the ones you'd find in slide 105 of the same deck, for example. Note that in this version its 'the()' that is the active closure. But I fail to see why this should prevent me from expressing myself naturally as an English speaker.

This then, IMO, is the larger problem with DSLs. When you make something like its real-world counterpart, the human mind immediately taps into the huge body of latent real-world knowledge and tries to use it with that thing: although this DSL doesnt promise it, I immediately wanted to use English structure on it. It's all fine that your programming language has risen to talk your user's language, but has it captured enough of the user's world to go beyond "training wheel use"? To paraphrase Carl Sagan,: To make an apple pie DSL, you must make a universe that's up to scratch. 

How will the DSL author handle this? In general, not very well, I think. I realize I'm picking on a toy example here, but on extrapolation I cannot imagine a domain where all possible concepts and all of their possible combinations are easily enumerated; and then retain their meaning over time and across minds.

I begin to wonder then, at attempts like the VPRI's FONC, where one of the overarching ideas seems to be to create a multitude of DSLs to produce an OS in under ~20 KLOC; and at feeble attempts like my vaporware designs of Jack and Fluent: would they build a better world or is the imperative code bloat of Windows and Linux the more sustainable way?