Concept: Treat requirements as external forces, especially non-functional requirements

Along the lines of my idea to use FEM-like analysis on code, I realized that it should be possible to model the things that cause the architecture and design of the software over time as physical forces.

Case in point: My team is currently working on adding multi-lingual and multi-currency support to a fairly large app. Adding this predictably requires changes across the board, and led me to visualize all the places we're pushing and pulling at the current design/architecture to make this happen.

Could this instead be considered a single external force acting on the elements of the system? Can we study the "material properties" of the code under such forces and "sum up" the forces?

If we could do that, it would certainly move us out of the current dark ages where cost of change is still measured as a list of things to change per a specific change plan.

Maintainability of code

Most of my career has involved maintaining code built by others. I'd wager that most developers' career has been the same, even if they didn't want to admit it. Sometimes maintenance goes under the name of updating your own code, so people delude themselves into thinking they're not maintaining code, they're just creating the next version. From a maintainability POV, any code that's built is legacy.

And yet most (all?) methodologies of software development either address completely start-from-scratch greenfield development, or propose ideal practices that mature, passionate developers who are in the field for the love of programming.

The reality, however, is that most development involves enhancing or maintaining code - usually somebody else's, and mostly crufty; and in all likelihood the crufty code is from somebody who's presence in the industry is a happenstance rather than a planned event. If you're lucky it merely represents somebody's state of maturity at the time of writing, and that person has now improved.

A comprehensive methodology for maintainable software, therefore, must:

• To address the legacy code angle:
• Provide mechanisms to create "good enough" comprehension of the code
• But avoid attempts at large scale or totalitarian model-driven comprehension. Such attempts will always fail in the real world simply because there will always be something outside the model
• That is, allow for leaky abstractions
• That is, allow for manual steps. There will always be manual steps. The methodology should allow for that by exclusion or explicit statement.The former is easier.
• Help identify what changes and what doesn't. Easier: help identify what has changed and not in the past and let the humans extrapolate to the future.
• Provide a means to migrate from what is to what should be that's incremental.
• To address the maturity concern:
• Allow for different levels of maturity
• Allow for the ability to define "good enough" architecture, design and code; and the ability to easily enforce it
• Allow quick enough comprehension of these definitions
• Allow for gradual adoption, and a means to measure progress

The usual solution is to relegate this to Software Engineering, which typically spirals into talks of CMM and suchlike - process heavy, people agnostic.

The reality, however, is that software development is largely a human effort, and precisely because it lacks the usual shackles of other human endeavors. A mechanical, electrical or electronics engineer will always hit upon some natural limit. Not so the software engineer. His limits are the limits of the mind. If you can think it, you can make it.

And therein lies the problem. If you can think it, so can a multitude of other software engineers; and each such mind can think of at least one variation to the same problem using the same essential solution. This is why I believe we will not see Software ICs any time soon. Most process oriented methodologies tend to gravitate towards this concept, or to the equivalent one of "resources tasked to do X will repeatably produce output of increasing metric Y".

Meantime, in the real world, human programmers are finding ways to game that system. 

As software practitioners, what are we doing to better this? There seem to be promising starts with the BDD and (to a smaller extent) TDD movements, and (in a less focused, but generic scale) with the general move towards declarative programming. There're some incremental gains from tooling in general, but those gains are largely in the area of reducing the steps that the interface requires you to go through to achieve any particular task. There's also some progress in architecture analysis, validation and enforcing that constructs such as DSM and its ilk provide - if practiced regularly.

However, all of these lean toward the mature, self-aware programmer. By the time it reaches Joe Developer, its an organization mandate, not something he WANTS to do. So the gaming cycle begins. This time, however, its insidious because it projects the impression that code quality has improved. We therefore need tools and methods that "work in the trenches". I don't have a good enough answer, but here're are some interim steps that I can think of:

• Allow for easy documentation of what exists, and in such a way that the document follows changes. Working diagrams are a good way of documenting the subset of the code that's understood. 
• Use tagging as a means of collation of documentation created thus. Its easy and "good enough".
• Don't loose what you gain. Developers gain a lot of system knowledge during debugging. There's no easy way of saving that for posterity, so the knowledge is lost. Do not expect developers to write a wiki page - they don't have the time. This area is ripe for tools such as 
• bookmarking and code traversal trails being saved. 
• A "tweet my XApp-fu" app is what's required here.
• A way to share all of this that's searchable
• Make creating tests a 1-click affair. Any run of the software should be convertible into a test, especially a manual  during the normal write-compile-test cycle that the developer engages in.
• Allow cheating on setting up the context. In most production systems, its not easy to inject mocks or test doubles. Test automation should allow for as little of the system being mocked out as possible.
• Mindset changes:Somebody on the team needs to evangelize these:
• "Works on integration box" is less useful than "PROVABLY works on any box". This implies CI without requiring buying into the CI religion.
• Exemplar test data is better than just test data. Exemplars are test data sets that "stand for" a scenario, not a particular test run. 

Webster/Fluent/Jack: Create a text-based syntax as well

I realized that while I'm trying to usher in better programmer tools, most programmers might already be comfortable with their existing ones. So to foster better adoption, the simpler idea might be to allow a text-based syntax as well.

Hopefully, they wont be able to escape the wily wares of better tools for too long!

Idea: Queues: a visualization of work queues

I have been thinking about the concepts mentioned in Product Development Flow, and visualizing a dashboard based on the Queues mentioned in that book. An agile team could be represented minimally by a queue - the list of tasks in its backlog, and how the team processes that list. If we had a visualization of this, it would be easy to represent how well/efficiently the team is working

Extending this downwards, the team's performance could be measured in different ways: as a summation of each team member's work queue (which would allow per team member productivity and efficiency estimation), or as a summation of work completed, or (even interesting) as a summation of the different workstreams it's currently undertaking.

Extending this upwards, a set of teams or the entire organization could be measured in terms of the queues for each team, and the attributes of each queue chosen to be measured.

Speaking in software terms, I'm thinking of creating:

• A system that allows the creation of a queue, and a visualization of it.
• The ability to create parent and child queues
• The ability to "sum up" based on arbitrary attributes of each queue

An easy launcher for redcar on Windows

I just (re)discovered Redcar, and have been playing with it. I love it so far (especially the seamless git integration, and the tree-document-view concepts) but I find unnecessary windows irritating, so I obviously didn't like that Redcar needed a command window open in addition to its own SWT one. Here's how I made launching Redcar a 1-click affair.

1. Install redcar and make sure it works on its own. ie, typing redcar on any command window should start it up.
2. Convert the redcar icon from png to ico using imagemagick. You obviously need a unix box for this, or have imagemagick installed on windows. Online tools are also available, but this seems the most programmable and quick - especially if you have a unix box around :)
   

    [user@box]$ convert -resize x16 -gravity center -crop 16x16+0+0 redcar-icon-beta.png -transparent white -colors 256 redcar-icon-beta.ico

3. Create a windows shell script (redcar.vbs) that loads up redcar without a command window
   

       Set WshShell = CreateObject("WScript.Shell")
       WshShell.Run chr(34) & "redcar" & Chr(34), 0
       Set WshShell = Nothing

4. Open up a launch folder of your choice. Mine is the Quick Launch bar.
5. Rt click on the folder, New -> shortcut, and in the wizard that shows up, enter the following:
1. Location: drive:\path\to\redcar.vbs
2. Name: redcar
6. Rt click on the shortcut -> Properties. Then change the following:
1. Change Icon -> Look for icons in file: drive:\path\to\redcar.ico
7. In the folder (or in my case, on the Quick Launch bar), Click on the Redcar icon. Redcar should now load up straight to the splash screen.

Notes:

• The one drawback with this is that you no longer see the debug information that the command window provides. I could not have know that Redcar looks for git during its launch without that output. So there's still reason to launch redcar from the command line. This method doesn't prevent such launches.
• I noticed that there's a github project creating a windows launcher for redcar, but it seems to me like the steps above could be easily automated and the artifacts added to the redcar core distribution for a simpler launcher.Maybe I'll fork redcar and add them :)
• If you're using Portable Git on Windows like me, the key to having redcar find git outside MinGW is to add the following to your path:

drive:\path\to\PortableGit-xxx\cmd

Jack: Javascript should be the base language

It just struck me today morning as I was getting ready that the base language for Jack - my idea of the next gen programming language - should be Javascript. The good parts, of course, but definitely javascript.

This ensures that it has the widest spread of runtime environments, has the functional bits built in, and - as long as we keep the base concepts to the good parts -  none of the ugliness

Idea: A textmate rewrite in Fantom

Textmate seems ripe for a rewrite to a platform outside of Macs, and what better language than Fantom?

Sigh. Another project idea that'll have to wait.

Idea: More Effective Workspaces

Workspaces are invaluable in keeping focus on the job at hand. I've been used to Workspaces on Linux Desktops for a long time, and recently discovered a Windows powertoy - MSDVM - that does the same on windows. This idea is a couple that I've had that can make them more effective:

• A per-workspace todo list: So that when you form a plan of what to do in that WS, the plan is visible always
• A "Send todos to workspace X" feature: I frequently find myself remembering to do something official while I'm on my personal workspace. It would be nice to be able to send a todo to that todolist without losing context.
• An "I'm stepping out" button with tickler for when you return : Click on the I'm stepping away shoudl allow you to enter quick text on what you're currently doing, so you have a high visibility "This is what I was doing before I stepped away" note on the workspace when you return. I constantly find myself doing this using a new notepad instance even though i have other todo lists - just so its front and center when I return. It would be nice if the workspace did it for me.

Of course, all of this can be done with current software and some manual setup, but it would be nice if the todo lists were workspace-aware, so this works out of the box.

A goal for Fluent

Fluent, pka Webster is the Web-based DSL metalanguage+editor I'm working on currently. It seems like the language workbench challenge would be a good fit for a goal to work towards.

True BDD for Webapps

Current BDD implementations have a "core" language they support. How about a bdd framework for ANY webapp?

"Outside in", the core BDD concept implies that you start with the outer layer of the app and implement inwards. Well, why not apply that starting with the html, going inwards to js, refactor js to make it mvc etc.

So the true BDD implementation for Web Apps should load in a browser, and allow any browser action to be validated naturally.

Implementation Ideas:

• a html based runner of the spec a al jscoverage (which should be rewritten in js, btw) is required, so your spec can be loaded in html, and runs the target app in html as well
• spec will be in a cucumber/robot syntax - very english-y, which will be converted into functions in javascript. Nothing new here, this is what all existing BDD implementations do anyway. We'll just leverage the best Javascript one.
• building html must use zen coding for example so its easy to do so.
• Adding behavior will involve writing a spec for it, and adding the basic js code for it, then refactoring it to use an appropriate framework (mvc, etc) and so forth.
• As that progresses, the main spec becomes more detailed, and contains nested specs for specific layers. Eg, the javascript layer could be tested directly using rhino, etc

Simple Admin Tool

Most applications have configuration defined in files. This idea is to automatically wrap a simple admin tool around the configuration file instead of building one from scratch.

So, for eg, a java properties file when input to this tool would automatically generate a website that allows  values in the properties file to be created, display, updated and deleted.

Not in scope: updating in-memory variables with the new values. Although a simple reloadProperties() call can be supported.

Wireless Projection

You go to a meeting room, and typically connect up your laptop to the projector or plasma screen in it.
But do you really need to? Why cant you connect wirelessly to the display?
That then, is the core of this idea.

If the projector/plasma is connected to one machine, others can connect to it wirelessly and display the content of their screens on it.

Alternately, the one connected attendee can then easily enable other attendees to use his machine to remotely display their screen contents! This has the added advantage of allowing anybody to use the display, not just the one person connected to it - without the hassle of disconnect/connect/key combo to show etc; not to mention the chance of allowing group participation through easy passing of the display baton.

Implementations possible:

1. Xserver on a box that's physically connected to a projector/plasma.Wrapper server & client programs that allow Webex/vnc-style connect-ability to project local display onto that display.
2. VNC client/server setup with client in listen mode
3. Custom wrap of xserver or vnc that automates all this.

Metalysis

Metalysis: The analysis paralysis that results from thinking about meta and meta-meta programming. A prime sample
language = domain model + syntax (template + css)
the model till now describes:
    - the elements of the language
    - how to display/edit it
but the editor's interaction model is fixed based on the meta model   

so true language editor is
language editor = domain model + display model ie syntax (template + css) + interaction model (javascript behavior)

now this behavior could be modeled as a domain language itself, but for it to be useful, it would need something that can execute the domain, not just understand and allow its editing

so true language = domain model + editing model (display model + interaction model) + executor (interpreter/compiler)
:
So I should just create the meta model and its interpreter, and running it will spit out my language editor, ready to run!

This is when I came back to the ground and started writing an event loop and handler. Nothing like normal GUI programming to make you realize you're nothing special!

Webster: switch modes instead of making it hybrid

I just had a brainwave: I have been mulling about ways to bypass the creation of a hybrid editor, when it seems like there's an easier way: allow fluid change to text mode of the whole document.

When you dont want to treat the code as code, just switch to text, and use text semantics. Simple :)

Of course, change "whole document" to "part of document" and you get a hybrid editor for free.

Which of course, goes back to the "islands of one language floating in a sea of another" idea that I've been mulling about. What if the "document" was actually a disjoint data structure held together by multiple instances of small editors? Each such editor would handle its own language fragment.

More thought required, obviously :)

Idea: Photo to Graphviz diagram: an android app

Take a picture of a whiteboard diagram with your android phone.
This app converts it into a graphviz diagram.

A delicious alternative

Delicious is the one intenet resource I use, and that is now on the verge of being sold or shutdown. Here's my idea of an alternative. As always, I like them to be self-sustained, and preferable controllable by my self.

All we need is a web server, and hits to a particular url in which the link to be saved is contained. The url doesnt even need to be active; it just needs to log to the web server's logs. An offline job runs as frequently as needed, trawls the logs, extracts the links and stores it.

Entry and viewing of the link database is mediated through a browser plugin as now, because that's the most common usage anyway.

Simple, and as long as you have a web server (which you should anyway), you're good. The only custom component is the offline job.

Idea: Photo to Graphviz dot file for android

Finally posting this idea from last year. The idea is that you'd take a picture of your latest-n-greatest app architecture that you just scrawled on the whiteboard (as you usually do nowadays at the fag end of that big architecture meeting), and this app would automagically convert it into a graphviz dot file.

Simple, huh :)

Things that I'm really interested in

This is one of those big picture posts. Here're the "hard problems" I'd really like to solve:

• Code Comprehension: How to understand a new codebase easily. If i wrote an oreilly book it would be called "Understanding Code"
• Code creation and maintenance: A lot of this is manual, repetitive, lossy and tedious. Programmers should use the tools (that they make for users) for their own use. And: less snapshot-based development, more fluid one.
• Applying AI techniques to code itself.

Slots: It's like freemind converted to twitter :)

Pardon the cheeky title.


The idea is to have an app that will allow you to take a piece of text, and define "slots" in it. Essentially something like the "Its like x for y" sentence above. It then stores the template sentence, names for each slot, and the values. Once these are stored, Slots then allows for them to be queried using the template sentences. My previous idea of the app to remember where things were could then be just an application of slots.

Further development of the idea:

• It need not have any specific interface; it should embed itself into other apps or work on clipboard text.Input is therefore any text. This means that it could be an android app, or browser extension and still work the same.
• Storage can be local our cloud.
• Once a match is made, action can be taken in any/many ways to actually do something
• 
  

OT + Grammar = New way of defining languages?

Operational transformation is a way of defining documents via the series of steps required to create the document - among many other things.
Language grammars are usually defined by rules on how to recognize a document once its created.

What if the latter was done in the former's style?

Need to think this through but if the rule set is not prohibitively large this would be a nice way to achieve incremental parsing, not to mention the ability to have "islands of known information" mixed with WIP unknown fragments, methinks

Idea: A language that has DI built in

... in fact how about a language that:

• Has only mixins, no pure interfaces (this is probably there in at least a dozen languages already, albeit without the no pure interfaces restriction)
• Allows objects to interact only through dependency injection?

As always, I'm pretty sure a language expert would poke some serious holes in these ideas :)

Layered model for software components

Another idle thought; obviously having your laptop around when you're half asleep at 5am is good :)

How about  a OSI Layer model for application level software components? Lower layers wouldnt have guaranteed delivery or security, which would be provided by the higher layers.

Concept: Dynamic analysis of code statically

Some idle browsing of the architects' dzone led me to think about how programs can not be completely analyzed statically; which led me to think of how humans understand code by just looking at the source.

We run it in our heads.

If we can do it, can't computers? And I don't mean it in a Godel-limit-breaking way, but surely there should be small enough chunks of code that a computer can "run in its head"?

Related thought bits:

• Since the godel limit is that representation != understanding, how about we settle for metadata calculus = very weak for of understanding? Semantic web for code, that is.
• Would knowing the type and range of inputs to a particular piece of code help us understand the code better - a la Agitar creating unit tests on the fly?
• Often missed out is the "world" around the code, and its impact on the running of the code. It might be useful to model the world as well, and probably develop a taxonomy of such environments. Eg: when function x() is run in java it will behave in one way vs when run as javascript. This is a loaded statement 

Need to think more about this.

Slashchrome: an idea whose time is long overdue

Its been a while since I considered taking this project up, and decided against it because it seemed like too simple a project. The idea is to enable Google Chrome to be able to search using the forward slash key. Nothing new, spectacular, or revolutionary.

I had, in fact, gone through the bug trackers of both the Chrome and Chromium(issue 150) projects a while ago, but a recent re-reading got me back to the old fervor: Here was a dead simple feature that the core developers refuse to implement, and the only reason that I can see is that some webapps (read GMail) capture the slash for its own purposes and the browser doesnt want to get in the way.

The developers insistence in ignoring users who have been begging for the feature since its inception is reminiscent of the Pidgin vs Carrier saga. Its time to fork Chromium and include slash search, methinks.

Since so many people are asking for it, I thought somebody would have already forked it, but a google search did not turn up any such projects.

I also looked briefly at an extension that was written to overcome this limitation, but comments on the thread seem to indicate that a core implementation is still desired. Hence this idea. However, there are some really nice features in the extension that I'd borrow when implementing this.

Roadmap:

• V0.1: fork Chromium and change the keyboard mapping of search from ctrl-f to /
• V0.2: Make the switch configurable via options so that people can still switch back to ctrl-f behavior if needed
• V0.3: Blacklist support for sites that use slash for their own purposes (this from the typ-ahead extension)

Thoughts:

• I looked at the source briefly assuming that finding the keyboard map should be simple. I've not yet been able to.
• The build setup also seems pretty time consuming. I should probably setup on windows (being the first implementation and all, and probably the most users), but that entails setting up on VS Express - which seems like an excursion down the proverbial rabbit hole in itself.

Idea: A browser plugin that answers the question: "How did I get here"

Sometimes I reach a page through a long and circuitous traversal of the interwebs, and I wonder how I reached there. It would be nice if there was a browser plugin that showed me that path.

And no, the back button is not exactly the answer because of new tabs.

Better: We need a browser that shows the part of the web that has just been browsed as a graph, so other traversals can be considered openly instead of through our memory of where we have been!

Fantom: a new language to learn

Today I discovered Fantom - a surprisingly interesting language that seems to get so many things right. My first thought was to re-implement Joey in it. We'll see.

The language is definitely interesting - especially considering 2 brothers built it in the (mostly) spare time.

Things to learn from/use:

• Fan(tom) is self hosting, ie the compiler is written in fan itself. So building it from source involves using an older release of fan. This is pretty standard in compiler land, I guess, but still novel to me; and probably novel in that the although the final executable is a native binary, it goes through intermediate non-binary output levels
• It compiles to its own intermediate code called fcode, primarily because the original intent was to be able to generate code that worked the same on Java and dotnet.
• Javascript compile - sweet.
• The FWT toolkit - havent started using it yet, but any library that allows declarative UI is good in my book. Especially since the same fan source can be used to generate  web ui a la extJS.
• It also seems to have a fluent api to building html - another nice thing
• Units (a la Frink) and a datetime api with durations built in.

PS: When I saw the link in LtU I half expected it to be inspired by the Phantom, but apparently not. I have decided, however, that if I do develop in fantom, the projects will be Phantom themed :)

Some structural editor links

Saving here as there's no delicious bookmaklet on the iPad.

an interesting study of actual developer usage of text editors, and implications for structural editors

Linked discussion at LtU http://lambda-the-ultimate.org/node/3567

FrankenEditor: an unholy marriage of emacs and vi

As is obvious from my previous post, I have just re-discovered these editors; and obviously think there should be a best-of-both-worlds editor. See the bottom of said post for feature list.

Another thought: recast Webster into a vim/emacs mode. That will go a long way in adoption.

Doh! Vi and emacs ARE structured editors

In hindsight this was to be expected. I have been reading up the vi and emacs documentation and finally realize that they are both structural editors - not like Alice and Concrete and Webster (when/if its made) is, but definitely they're smart editors, and in some parts, they're structural editors too.

Vim got it right with "command mode is normal, you drop into edit briefly", while emacs got it right with its major modes.

Thoughts in no particular order:

• They seem to have a fixed concept of the "constructs" that comprise structured text, although that could well be my superficial reading of the docs. ESP considering the file explorer mode, for eg. That seems to be a full-fledged structural editor
• emacs already has the concept of skip words ie structural elements that should be skipped over in navigation which is exactly what Webster/concrete does. Dunno if vi does.
• vi's idea of normal mode seems to be more efficient than emacs's "type normally, use ctrl/escape sequences to insert commands" as a user interaction strategy. Should use this for webster.
• this explanation of why the command sequences need to be as weird as they are is useful in understanding emacs, as was Tim ottinger's explanation of the vi command syntax
• they both separate function from it's keystroke - an important detail.
• The good things, therefore are:
• structured text
• structured navigation
• undo/redo/buffers
• command syntax, ESP of vi
• normal mode is navigation, not text entry
• emacs mnemonic navigation seems better than vi's home row based one, although using modifiers for the levels of objects (char,wore,line,para) doesn't.

It seems like there's opportunity to take a modern stab at an editor that's in the spirit of these two behemoths, and yet goes beyond their limitations.I would not be the first to think this :)

I propose, others implement...And a vim hater recants

Predictably, some of the ideas ive been writing about are being implemented by others, notably webster.
I recently found concrete on github, and it's a pretty close approximation of Webster. It doesn't have all the niceties I meant to implement ESP cos it's viewpoint is MDA, but it's there for the most part, and does the modeling in a neat and lightweight way by reusing html, CSS and standard dom - something I intended to do, but didn't succeed with.

The real reason for this post tho? I just realized that vim could just be half way down the syntax editing route!

Coming from someone who doesn't think much of vim's weird modes and keystrokes that's as about face as it could get. Tim ottinger's vim tutorial was what made me change my mind. This is by far the most approachable vim tutorial iv'e read and finally had me grok the vim way. Now the need for command and edit modes seem almost required, especially considering all syntax driven editing is in such a mode! Vim could be thought of as a syntax driven editor that supports the "text language"!

Maybe I'll implement Webster as a vim mode someday!

Doc Dep: A tool to handle the dependency of code and its corresponding documentation

As I work on Abu, I realize that documenting a project is a separate side project in and of itself. I started with a simple Idea.txt, and that has burgeoned into the following:

• A readme.md (for markdown) for github's main page
• A contents.md showing the contents of the codebase
• A design.md for the design "discussion"
• A user guide - WIP
• A developer guide - to be created
• A roadmap.md
• A log.txt for my personal use - as a tickler for when i stop working and have to pick back up later.
• A news.md (yet to be created) letting people know of recent changes

And what's interesting is the interplay between changes I make to the code (or design) and these documents. Each change in code/design obviously impacts one or more of these documents, but there's no easy way for me to keep track. Eg, change in design impacts the user and developer guides, the news file, and sometimes the readme and roadmap files. Any new top level directory added  changes the contents file and so forth.

Now, what if I had a way to say "this code file impacts these document files", so that when the code file changes, I have to change the corresponding document? That then, is the idea - DocDep - which will have:

• A means to list all the documents in a codebase - some kind of config file should do, or probably a /doc directory could be assumed if we want to go CoC.
• A means to represent the dependency between a file and these documents - easily done by creating a tag - embedded within a comment - in the files
• A means to alert the developer when a change has occurred on a file, which therefore needs a change in the document file linked to it.
• This could be as simple as generating a todo list by commit, Eg: todo: change guide.md to handle change in commit# xxx; Or...
• We could go anal and prevent checkin of the code changes without the documentation.

Codeviz: Some thoughts

This post to be fleshed out better later, but here's an insight I had today.
The format that codeviz will use must contain the following types of information:

• graphs
• nodes
• edges

... and these pieces of information should be easily intermingled with each other, and must be line oriented, ie the data that a node has about its parent should be expressed in that line itself, not presorted by parent to include this child. This enables easier sequential processing.

Quantum Commuter

This is an idea that I've had for quite a while, and thought I'd blogged about already, but apparently I hadn't.

You ever had the choice of two routes to the same destination and wanted to find out if the "road not taken" was the better one? Well this idea is one that'll help you find out.

Chicago's public transport system - CTA - provides an api to find out bus times and a whole slew of transit related information. The idea is to create an app that allows you to select two (or more) buses/trains that ply between points A and B; and pit them against each other using the API. The display would show a live display of the progress of the buses/trains, and who won.

I've always wanted to "clone myself" quantum mechanics-style to do something like this IRL, so I call it Quantum Commuter. Nice play on words, even if I say so myself :)

Btw, this could be an easter egg in an otherwise normally useful bus tracker app :)

Idea: A URL/URI spec for code

This is one of those early morning ideas that I have to put down before I forget it: How about a URL spec for code? Something like:
code://somecompany.com/someapp/package/structure/goes/here/AClass.java#methodName#if3@rev234

The idea is to be able to refer to any part of the code externally, easily, and transparently. guarantees on the posterity of the resource returned similar to general URLs, so they're more URLs than URIs.

Elements that should be in the url:
- language used (better notation that file extension tbd). alternative the protocol could be the language name instead of "code"
- ownership of the app by company/individual identifier. i think the java package concept of inverting the domain name works fine here
- package structure
- source object reference. could be file name, but doesnt have to be so in this abstract space
- intra source object references to methods and specific lines, eg the 3rd if in a method
- version information in the form of a scm version specifier

Code Canvas

I just discovered a Microsoft research project from 2008 that foreshadows quite a bit of my ideas with Webster. Its called Code Canvas (demo video).

In line with the N+1 concept, the few things I can think of (if I were to think of Webster as a product) are:

• Treats everything as a graph - down to the individual tokens. This is a game changer in the way wave was :(
• This also means using graph operations to edit code.
• Any language, not just CLR
• Any mix of languages eg, one web app uses shell scripts, build scripts, assemblies/jars, html+javascript+css, ui frameworks (jsp/asp/etc), compiled code (c#, java, et al), ORM scripts, so forth

I do like the semantic zoom, and the Quad tree algorithm to enable it.

Idea: The "Switch around the And" plugin

I was watching a colleague editing a document, and was struck by the the number of steps he need to change a phase like "cat and dog" and convert it into "dog and cat". I myself have had this problem quite a few times. How difficult would it be to make a plug in (in whatever editor) to automate this simple task?

So, this then, is my next idea: Create the "Swith around the And" plugin. For specificity let's call it a Word Plugin. But no reason to stop there - how about for most other editors?

In fact, when I think about, this could become like one of those projects that write "hello world" in every possible language. The "And switch" plugin for everything from vim to word to what have you :)

And of course, this is just the tip of the proverbial iceberg. Why stop at just "and" as the matching pattern? Why not add this as a feature to any code editor so swapping arguments of a binary operator is easy and automated?

Todo: Somebody must have done this already. Google it.

Idea: Boxview - a new kind of html editor

This is an idea for a better html editor:

• Expose the box model to the html programmer by visually showing the boxes that elements are contained within
• allow changing of padding, margin and border properties directly instead of through property sheets
• automatically calculate the minimal set of properties required to specify the box model built by the user. Eg, if a child and its parent element have the same property, bubble the property up an dinherit it.
•  Allow abilty to specify "global" overflow and containment of elements directly instead of through interacting css properties.
• Allow views of the model with no content, some content and  a lot of content directly, so that the model can be stress tested.
• 
  

Visual != Graphical

It just struck me that visual programming doesnt necessarily have to mean a graphical programming environment.  It should be possible to denote the visual aspects of the program in text mode as with graphical mode.

It does mean some environment, however. A text editor is not one. And that might be the deal breaker for most visual programming environments.

Now what if we had a visual programming IDE in text mode?

Syntactic Sugar IS UI for programming languages

I'm reading Paul Graham's On Lisp, and this section from the chapter on macros struck me as interesting:

Backquote is usually used for making lists.1 Any list generated by backquote can also be generated by using list and regular quotes. The advantage of backquote is just that it makes expressions easier to read, because a backquoted expression resembles the expression it will produce.

...

The longer the macro definition, however, the more important it is to use backquote

 This may not be original, but I just realized that syntactic sugar is the UI for programmers. It just happens to be stuck in the limitation of text. I couldnt help but think that a Structured Editor would obviate the need for backquotes, or the usual "lisp is full of parens" complaint.

All they were trying to do was to make a UI for the programmer

Idea: Where's my stuff

An app that remembers where you keep your stuff. It could be as simple as a wunderbar-style interface. You fill in the blanks for the following sentence:
"I keep/ have kept in/at "
Input could be text or images.

Nothing revolutionary, but should be really useful, methinks. esp as a phone app
maybe this'll be my android app.

webster: avoiding the editing problem

- "everything is a graph": the problem (as is obvious from various discussions on the web on the topic) with structured editors is to allow seamless editing of code. the standard example being converting a for loop to a while loop. My idea is to treat everything as being part of a graph, with infinite breakdown. The default level will be at the "syntax" level, but if you want to break that further, press a hotkey, and the syntax-based placeholders and user inputs get reduced to text nodes.

Superficially, this is similar to the "hybrid editors" that have a mode to switch to text input which is then parsed back. But the key difference is that the conceptual graph is retained.

Combine this with the leaf based parsing should help keep the experience smooth.

Some more thought:
operations allowed by a text editor

• navigate text
• create, modify and delete text
• copy and paste arbitrary text, either in line mode or block mode

to emulate this, how about a structured editor has a "text grid" view

• at every text grid event, the structure model is notified, and it changes the model accordingly
• if a placeholder value is changed, no model changes are required, just the slot value has to change
• if a keyword is changed, a triage construct consisting of the following is constructed:

the rest of the structure is retained in the anticiaption of rollback,

a new structure is anticipated and candidates are stored, which through multiple events may get realized into an actual new structure

• if a new structure is realized, the old one is discarded
• the engine will aim for the quickest new structure that can be realized, given the language syntax

More thoughts on webster

I've been browsing on the general concept of structured editors hoping to find precedents, and finally hit paydirt recently - they were apparently a rage in the 80s and 90s and quite a bit of research was done on them. Of course, they were called syntax directed editors then.

Google returned a lot of hits on this new term, and one paper in particular caught my attention: http://arxiv.org/PS_cache/arxiv/pdf/0710/0710.2358v1.pdf

It talks about a syntax directed editor called Absynthe which allowed editing of code via the grammar of its language - the editor would provide templates for entry of valid language constructs, complete with placeholders for the free text pieces. Of course, they use the right terminology - syntax elements and terminal symbols, respectively.

The paper also went into some detail as to why such editors have not found much favor:

A first problem within a pure template model is that only terminal symbols can be entered as free text. This characteristic is absolutely intolerable in practice. Suppose you want to enter an arithmetic expression containing three or four operators. Then you should select the appropriate template by means of commands or menus for each operator and argument - which results in many additional commands being triggered.

This is why a pure template model has never been implemented in practice. Both the Cornell Program Synthesizer ([Reps 84], [Teitelbaum]) and Mentor ([Donzeau]) have used an hybrid model of top-down replacement of place holders with template and bottom-up construction of phrases from free text. We have also adopted such a model in Absynte. In a hybrid model, a focus is used to delimit an area of free text editing. In this area the text may be temporary incorrect. In the systems where the internal representations are ASTs, a first drawback is the administration required for parsing and unparsing the new representation.

If a free area is really useful to enter arithmetic expressions it does not solve the most important defect of the template model: how to handle modifications within a tree structure ? The replacement of a while statement by an until or if statement is more difficult than the corresponding change in a traditional text editor (the later is quite simple). In general, the replacement of one construct by another causes all information belonging to the old construct to be deleted. If the user wants to save some pieces, he can sometimes do it in special buffers or windows, but he must perform these operations separately (for each syntactic category and, of course, before the complete removal) because every piece of program to be saved must be inserted in its future template separately.

I don't have any better answers than the one suggested in the paper itself - that of storing deleted node content in a special buffer for use later - but here're some of my thoughts:

• Navigation = change of mode. That is, if you navigate out of one of the free text/terminal symbol areas, you're automatically in "edit node mode" - transparently.
• Allow pre-order creation of child nodes. That is, instead of requiring that the parent be created before the children can be, allow the tree to be created from all possible left (or right, if the language is RTL) terminal symbols, and keep pruning the tree as you go.
• More to come as I think of it
  

Webster: the web-based structural editor

This is an update on my previous post of a web-based MPS editor.

I've actually found some time to work on this, and I have a basic version working - well in 2 pieces.

1. A renderer that takes a language spec and an instance of the language, and spews out an editable html version of it - with keywords non editable etc.
2. An editor that takes this rendered version, and adds behavior to it a la MPS - allows navigation, ctrl-space like popups etc.

I still need to combine these together to realize the real goal, ie that of a web based AST editor, which will effectively be Language Workbench meets bespin with an MPS flavor.

But wait, there's more!

My real intent is to go much farther than that:

• Generalize the standard editor commands so that they are the true graph edit operations that are required to transform one version of code to the next. Eg, to add a new import statement, today we insert a line, type "import java.blah.foo;". This should be transformed into: "insert a new import node, which produces an editor node that allows us to choose java from the ctrl-space drop down, then blah, then foo. And so on and so forth.
• Generalize it further such that editing the structure of an application is no different from editing the structure of a class/program - these are graphs too, inherently. So I'm thinking of a zoom in/out feature. Zoom far enough in, and you see the standard AST editor. Zoom far enough out, you see something similar, but it now shows your packages, applications, deployment graph, what have you. Consider using SVG for this.
  
• Webster meets Subtext: Create capabilities similar to the ones demonstrated by Jon Edwards' Subtext. This might however be part of a language that uses Webster more than
  Webster itself.
• Implement Jack in Webster
• Implement Robot framework in Webster

FEM - like analysis on code

This idea is partly from a section I remember from the "Inmates are running the Asylum". Alan Cooper compares code to a huge stack of cards one on top of the other, with just that single flick threatening to bring the whole thing crashing down.

How about treating code as a structure, and using finite element method-like analysis on it? Of course, the math would be discrete, not continuous. But the possiblities are interesting - we already create software in layers, with upper ones depending on the lower ones. Surely some definitions of structural stabilty can be got from those? Similarly the dynamic nature of programs could possibly be modeled as forces...

food for thought..but that's as far as I've got with this idea.

A Web-based MPS Editor

I have been playing around with IntelliJ's Meta Programming System for a couple of days now, and I find it a very interesting and promising tool - to teach myself its constructs, I'v started writing an IDE for Robot scripts. I've got a basic editor and generator done and it looks pretty good, especially for completely declarative code.

I was wondering, however, at the applicability of the concept to the web - as in how easy it would be to port the concept of a declarative structural editor to the web. The MPS editor language is surprisingly similar to divs and spans - all one would need to move the editor to the web would be to slap on a generator that creates D/HTML/5 widget wizardry!

Another project to start....

Implementation thoughts:
This might not need a full fledged editor like EditArea - since its structural, it might actually be cumbersome if it were. It might be easier to have a simple div/span structure or even a table with variable row and col spans, with a single in-place editor component for the current element being edited. Of course, keypress handlers would have to be written to make it usable for keyboarders (like me), and we'd have to find a way to show a cursor always.

Interesting ideas on continuation-style parsers for javascript editors: http://marijn.haverbeke.nl/codemirror/manual.html

Further implementation idea: use the MPS Editor language to define it, and create a generator for the editor from it that outputs GWT! This way the editor is web-enabled!

Thoughts on Joey

• It will mimic Chandler to a great extent - many things they just got right. Eg, all items show up in one flat space.
• It will not have folders at all - only tags a la Gmail.
• Everything will first show up in the main dashboard which is effectively the inbox. There will be no explicitly named folder called inbox
• similarly there will be no Sent items folder. They will automatically tagged as "reference" or "info" or show up in the dashboard with a sent tag. The idea behind this is that there is really no other use for sent items than as reference material unless we're expecting a response and want to watch it. So the user has the option to tag it as watching before its sent, in which case it will automatically trigger a tickler when the user needs to be reminded if a response came, or a follow up needs to be sent.
  
• Projects and subprojects will be handled via hierarchical tagging, instead of the much more complicated route of have tasks and sub tasks. This comes from the insight I had that a) you cannot execute a project, only its tasks, and b) at the execution level, it need but just a flat todo-list. So taking the hierarchical aspect out of the task list helps, and we still retain that structure in the hierarchical tagging. So my strawman design for hiearchical tags is:

simple tag: home, fun, work
hierarchical tag: Projects. Project A. task1, Projects. Project B. Sub Project B1. task 3

• Bring ubiquity/awesome bar concept to PIMs. This is already there in Chandler, but enhance it to make that a primary use vector.
• Make it generic enough that the PIM can be made behave either like a GTD PIM, or like Outlook, if so required.
• Nice to have: Tasks show up in Calendar view if no date is provided, so that allocation can happen right there.
  

Design thoughts

• The core of Joey will know only about items and display them. The user can then setup a profile that has rule on what to do with the items.
• Items can be automatically got from an Exchange/IMAP/POP server or created by hand; and Joey will not do anything special with them like putting them in a special folder called "Inbox". Instead the rules of what should go where will be created by the user.
• Similarly the actions that can be done on an item would be configurable in the profile, and they're effectively two kinds:
  
• tag it
• change its type
• "act" on it. which depends on the item's type. For a mail its to respond. For an event its to attend and dismiss it, etc.
  
• The profile, therefore, will say the following:
• what to do with an item when its rendered on the dashboard - tag it to a particular type or not
  
• what action to take on it when the user "acts" on it.
  

New ideas

I've had a bunch of ideas lately that are sitting on a post-it on my desk, so thought this would be the better place to dump them:

• Twitterbot to show traveler updates: The idea is that you'd add the bot as a friend, and send it a tweet asking it to track your trip, and it would pull the itinerary and send tweets to your account with the progress of your trip. This way your followers (which i'm hoping are family) can keep up with where you are, and if you reached on time.
• Ruby Route for Tax Software: This is the idea that I use Route to create an open source tax prep software that mimics the manual forms
• AST Editor: already expanded on in previous posts. Look at IntelliJ's MPS as a potential solution that already exists. But not quite - I'm interested in maintainability, they're interested in creating a DSL ecosystem.
• Map reduce for code comprehension and static analysis: This comes from my efforts to get XRadar to work for Novo codebase, which it balked on. So the idea is to use map-reduce to break the problem down into manageable size, and still be able to do static analysis of large codebases.
• PIM based on Chandler: This is the new incarnation of my long-standing PIM idea: I recently went back to see how Chandler is getting along, and it seems like they have some great ideas, which seem even greater in the light of my recently drinking the GTD koolaid. But its slllllooooowww as hell - not good for a productivity tool, and it still doesnt do mail. So my idea is to create a PIM tool that does mail, and has all of Chandler's goodness. Chandler itself is not an option because its slow. So I looked for existing open source java mail clients that I can convert, and found Columba which seems to be really mature, but the source seems really impenetrable. Plus it still didnt do Outlook, which is on the top of my list of required integrations. Why have a GTD PIM, if you still have to import/export from the most used email client in the world? So now I'm investigating writing the thing in Ruby. I'm tentatively calling it Joey, as in we already have Chandler, so why not Joey :)?
  

Visual Programming: how it should really look

A meandering walk through the net that started at LtU and ended in Source code in Database led me to thinking (again) about how the real visual programming environment should be. And here's what I came up with:

• It should reflect the fact that code is essentially a graph. It should therefore be basically a graph editor.
  
• At the single program level, however, this graph is essentially left to right, top to bottom, so a standard representation (as shown by current editors) will be natural
• Folds and hyperlinks are still useful to have around, though; and this represents the 2.5D solution to what is essentially an n-dimensional graph
• It should have the ability to move in at least 3 dimensions - zoom in and out being the 3rd axis. This should behave according to usability principles - zooming way out from a single program should show its package within the other packages' context, but highlighted; while zooming just a little should show more of the program, but not fold or summarize it in anyway (font zooming goes away completely- we are looking at code, not text)
• Multiple views should always be available - eg, package view must be vieweable as a tree, or a treemap, or a graph, or a list of packages
• editing must be keyboard based to allow for a completely console based implementation if required. The focus is on exposing the true structure of the program, not to sugar coat its display
• Constructing programs must be a structural editing task, not a text editing task.
  
• language constructs such as keywords, classes, packages etc must act as single "characters" and be deletable atomically.
  
• since the AST is being represented visually, the editor must provide auto-complete and hints as to expected parts of the current statement being constructed. Eg, typing an "if" or choosing a command to insert an if statement should allow autocomplete to finish choosing if, and provide VB/eclipse style popup hints as to the other components expected.

Ted Neward's Languages talk

Ted Neward gave a talk at my office, and spoke about Languages, and how we're seeing a renaissance of languages currently. I spoke to him afterwards about visual programming, and his comment was:

"You know, every time I do this talk, someone comes to me afterwards and asks about visual programming. The thing is: text is currently the most succinct way to represent code"

I had to agree with him. But I thought some more about it later, and felt that while that that's true for presentation of code, it need not be for its persistence. Why should text also be the storage mechanism?

define: roomba football

Roomba Football (room-bah fut-bol): The practice roomba owners have of kicking crud towards the path of a running roomba so that it gets a second (or third, or fourth...) chance at picking it up.

Its quite an interesting sport, especially because the device is prone to make crazy changes in course, as any player of the sport will attest.

Tax season and opensource

For various reasons, I cannot use e-filing. So I have to fill out the IRS forms manually. And while I'm doing them, I cant help but think that those forms are a workflow implementation waiting to happen.

The forms themselves are pretty easy to follow, so it shouldnt be hard to automate them. But in between there are deviations you've to take to determine if you're eligible to take an earned credit, or (in my case) if you're a resident or non-resident. And a lot of these deviations are presented as worksheets embedded in the instructions!

Wouldnt it be better to have it in the form itself? And while the pdfs that allow data to be input (and saved, which the federal forms do but some state forms dont - thank you New York), it would be nice if they were in spreadsheets. Or better still - as a workflow implementation. So here's my idea for the week:

An opensource workflow implementation for creating your tax return.

Nothing original, of course, as tons of websites make this idea (sans the opensource requirement) their means of existence. But I thought it would be challenging as an exercise. Here're my high level requirements:

• Workflow based: Each step in the process of filing a return is expressed as a workflow step, and execution proceeds as the user works through each step
• Modular: Deviations and side steps are expressed as sub-processes, which allow them to be optional or ignored if required
• Useful: Allows user to verify results, or actually generate manual pdf output that can be sent to IRS
• Extensible: The basic implementation should cover the common case, but it should allow addition of special cases, or conversion of values to calculations that are output from sub-processes and so forth; so that eventually it covers the span of the IRS forms.

Implementation Ideas:

• Use ruby and Route
• Use Route's web implementation as the basic UI

Minor nice to haves

1. A File Explorer that allows you to put directories as favorites similar to a browser - saves on all the dir tree navigation. I'm sure some file explorer already has this. But it would be nice if windows explorer did. It has something close - a favorites side bar. But I'd like a toolbar.

2. And then of course, an app suite that uses this favorites list. You know how the office suite's File Open dialog has the side bar with My Docs, Recent Docs, etc? Why cant one of the options be My favorites? Or is this already in Vista, and I'm still barking up the XP tree?

3. Another brain-dead feature of the office suite's file open (or most windows app for that matter) is that they will not do what Textpad has been doing for ages - open up the file open dialog to the same directory as the currently open document. Its highly likely that you'll open up one more document from the same place, or somewhere thereabouts. Why should I go back to My Docs all the time? If I was in My Docs to start with, this behavior would still end up effecting the same result. I dont get it.

Anyway, it'd be nice to see these tiny fixes into windows.

A true code format, and its editor

This is post is influenced by my recent preoccupation with code representation (see post on coding in thought, for instance) and inspirations from Subtext and ediola.

Imagine a code editor that allowed you to edit the actual AST of the language instead of text that will be parsed into the AST of the language. It would obviously make it transparent to you that you're editing a tree instead of free form text, except for taking out the inefficiencies. For instance it would allow you to use the normal navigation keys to scroll up and down through the code, but would skip keywords and language constructs automatically because they're are actually tokens and don't need to be treated as English words. When you navigate to a user entered value such as a variable, it would drop into an in-place editor that allowed you to change it, and it would then automatically change all occurrences of that variable because it knows that its a variable.

For the same reason, it would be easily able to provide auto suggestions /completions as it knows the language's syntax. An add-on feature would collect data about imported and declared classes, methods and variables as you edit, and make those available in the auto suggest as well.

The best part of course would be it would have none of the limitations of text - no need to worry anymore about whitespace as it would become exactly what it should be - white space that is merely a visual thing, not something that's stored in the code.

Sidebar: I realized after I wrote this that such an editor could be at a loss with a language like Python, which essentially mandates whitespace. However, I do think the reason Python went the way it did is reconciling to text being the universal code format, and if so, making the best of it, namely to avoid all visual clutter caused by begin/end tokens, and instead rely on whitespace to denote it for both humans and compiler alike. That being the case, there's no reason this editor can mimic the indentation, and provide the output that the python interpreter expects.

How would such code be stored? Why, as the AST itself. The actual storage format (as long its not back to text) is not important, as any number of those are applicable. What is important, though, is that it is in the exact format that the language allows for human thought to be enshrined as code - nothing more (specifically, text formatting warts), nothing less.

Once we have this, we can think of more improvements. How about treating comments as annotations on code, and extracting them out completely? This way, all the issues related to comment obsolescence and non-locality are resolved. We could just tie a comment to a particular piece of code, and wherever the code goes, the comment follows. If the code chunk is removed, broken or altered, it would be trivial to mark the comment as outdated and to be treated with a pinch of salt.

There's another gem hiding in the previous paragraph - that of tying a comment to a particular piece of code. Well what exactly is "a particular piece of code" unless we're talking about some named chunk such as a method? Nothing defined, yet. However, consider this: what if we had a way of identifying every piece of code uniquely - an id for each atomic piece of code? Notice that I've specifically avoided considering the line number as the id because is not just another of those text formatting appendages, but also an unreliable anchor to any particular piece of code as it can easily change with an unrelated edit of the file. I'm thinking more of :

• a unique id for each statement (or its component) (eg, the 5th statement of the main method - which could well be on the first "line" if it were text)
  
• expressions for groups of such statements (eg, first 5 statements of method foo),
• reference via scope/namespace (eg: 3rd if in Class a's method foo)

- an XPath for code, if you will. Once we have this, refactoring becomes analytical, even expressible as code. And of course, we'll be able to attach all kinds of metadata to chunks of referencible code, including comments.

Update: This from my wife: "3rd 'if' in method a.foo" is as arbitrary as line number. The better idea is to sequence them in order of creation, ie, the first 'if' created in a.foo gets id #1, the second #2 and so forth - for eternity. That way edits will keep the piece of code still identifiable. Ids could be periodically cleaned up to remove unused ones.

Ok, all of the above has been about editing code, not necessarily the format laid down by a language itself. But the more I think about it, formats have been more a limitation laid down by the tools that surround a language rather than the languages themselves. Compilers require classes to be contained in single files. Compilers require directory paths for package name spaces and so forth. There is really no reason a compiler cannot read a pre-parsed version of code as its primary source. By the same token, there is no reason source code cannot be stored as the AST - in version control or outside. The newer source code tools shouldnt have problems dealing with whatever binary format is chosen to store the AST, and diff against them.

And you know the best(est) part? No more tabs vs spaces wars!

Further Ideas for a new language

Some more random additions to the new language idea:

A name: Jack
This is mostly tongue in cheek but I thought if I ever got around to creating this language, I'd call it Jack. You know, like in Jack of all trades, cos its trying to be one - with its polyglot leanings. I did consider calling it Poly, but the options for source file name extensions would be .pl or .py or .ply. First two are taken, and the third doesn't seem that interesting. OTOH, .jak comes out nice.

Aside, I did also consider calling it V after me, but vetoed it cos it didnt give any indication about the language itself, just self promotion.

Anyway, Jack it is. And I've not yet checked if there's already a language with that name - hope not. But think of the possibilities with such a name:

- You could tell others off by saying "You dont know Jack"
- Like Java Jars and Ruby Gems, there could be Jack (in-the) boxes :)
- I know I'm gonna call the hello world in Jack "Jackrabbit slims" :))

Enough buffoonery, now for some serious stuff:

It has optimisations
You know how you have to "drop down a few layers" to optimize things? Like for eg, you can do 3d graphics using the Java3d APIs or you can use OpenGL or you could just drop into the video card's API? Well, Jack will have the same capability. If a thing can be done in more than one way, and one way is the composition of multiple functions, while the other is a single function that does effectively the same, you will be able to specify that. Instant reuse. Want to use the MFC classes, or OS sys calls to do something? Go right ahead. And supported in the language too. Sandbox to be specified, but definitely much easier than JNI et al - remember I'm still dreaming :).

The key thing I want to point out is the "supported in the language" bit. We as programmers routinely cut across language/API boundaries. However, each language is in its own sandbox held together by scripts, duct-tape and magic. Why not be explicit about it? Obviously there will be need for sandboxes, but making it explicit enables maintainability. It should be clear how different parts of the application ecosystem actually connect.

Ideas for a new language

A number of nebulous ideas have been swirling in my mind of late on what's lacking with today's languages. I woke up today morning with the remnants of a dream where I was coding in a language that magically had closed all those gaps, and even improved things a bit. Here's what my conscious self remembers from that dream.

It was based on conditionals
I have been thinking a lot about conditionals lately, partly due to being impressed by Subtext. But more closer to work, I've been thinking of a large legacy application and how to refactor it so that chunks of logic that are in the wrong layers can be relocated easily, and how other chunks of logic that are interwined with each other even though they logically belong to different paths can be extracted, and then combined via configuration using (for lack of a better term) a micro ESB.

And it struck me that the conditional (represented by the plain vanilla if, or an inheritance hierarchy) is the cleave point of "paths". All conditionals that are based on a particular check (eg: is the object of type foo, or is the value equal to x) are essentially in the same path. Therefore code should be organised (or auto-organised) such that conditionals are matched by their paths, and the largest conditional branch naturally gravitates to the top. This is, of course, exactly what Jonathan Edwards' Subtext does, with the additional advantage that it also ensures that there are no gaps in the conditionals - all possible values(or value ranges) have to be filled in. This would be a great thing for a language to have.

So the language from my dream had this feature. It worked using conditionals, and like subtext it auto arranged them - but all in text. That's the dream part, i guess.

Now when I think about it though, I can envision this conditional cleaving happening incrementally - a module can be conditionally complete within itself, but not so wrt other modules in the application.

It had modules/services/components as a first class language construct
Meaning, the "chunks of code" concept that I mentioned above were directly supported by the language. Think COM/XPCOM, or SOA/Web Services - with more accent on the fact that there's a published API backed by an implementation; and less on how they would be discovered, remoted etc. Code by contract, basically.

I don't remember if it had explicit support for constructs such as classes and interfaces that combine to form the components. It wouldn't matter if it did or not, but the key thing was that the component WAS defined, and was the cornerstone of composition.

It was extractably modular
And by this, I mean that chunks of code could be extracted out into modules by the language itself.Not a preprocessor or optimizer - the language itself had operators to extract code, define the api based on the contents of the code, and place it elsewhere. It therefore had meta programming capabilities, and more importantly, code elements were addressable in a platform- and source file-neutral way. Every unit of execution from the statement to the module was addressable to enable the modularisation operators to work.

It had DRY testing
Defining a module or class implied testing it. The language statically checked for bounds and ranges based on the conditionals. In addition, every run with values can be recorded as a test, adding to the test set.

It supported legacy apps
It had the ability to read old app code; and then show the gaps in logic or discover (if not extract) the true modules. It might have had a compatibility mode or it allowed incomplete (from its perspective) code to run.

It was functional at its core
and used monads for everything - especially the micro esb.
... but it had a lot of dsl-ish sugar to make it seem more general purpose.

And finally, it was (j)vm based.

Aftermath
After I wrote this far, I got to thinking about what goals a feature set such as this would arise from? Here's what I got:

• backwards compatibility
• support multiple programming styles
• direct support for common use cases in development and deployment

Idea: Working Diagrams

This is an idea to represent working systems. Working applications are manifested by the files that they are comprised of. So the idea is to rely on expressing the nature of the application using those very files. This is pretty similar to what anybody usually does when whiteboarding out an application - draw boxes representing pertinent pieces of the application. The boxes could be the whole application, or a part of the application or a line in a file of a part of an application. We then draw lines to represent relations that we're interested in. Specs like UML try to formalize this, but for most purposes, we dont need it to be true to the spec - just understandable enough. And there are representations that UML still doesnt (and probably wont) have in it that we'd like to represent - eg the call chain of our build xml's for example, or how two bash scripts relate to each other.

So the idea is to create an app that given the following input, spits out a graphviz (or eclipse emf/visio/jpg/whatever) diagram of the relationships represented by the input.
* one or more dir paths which are considered the nodes of the graph
* each directory and file is a node
* each directory is a hierarchical node and can be folded/unfolded
* a list of plugins that help to identify relationships expressed in each file that can be parsed using, for eg, regexes. The idea of using regex is to to avoid parsing the whole file. Though there's nothing precluding actually parsing the whole file. Eg, a java file plugin would contain regexes to identify method calls and create relations to external classes used as arguments.

Version 2:
Make an interactive version of the output that can be folded/unfolded at will.

Uses:
* Point the tool at your deployed app and get your whole deployment stack displayed as a diagram.

Note: The notion of a file can be generalized to anything whose representation can be queried for interesting relations. Consider, for example a process tree visualization by traversing the /proc directory in unix, and now think of a tool that translates a windows process list into something similar for processing by this tool

I call it Working Diagrams. Its informal and works for purposes of conversation and discussion.

I wonder if there's antisocial networking, and webapps for that

I'm a reluctant participant in the whole social networking thing. Must be my INTJ personality type. But as I get connected into facebook and its ilk, I wonder: how about antisocial networking?

How about a bad mouthing network? This was an idea that i got separately as well - a venting wall where anybody can enter some person's name, and rant on how that person ruined their life, and how they hope that person went to hell. I'd call it cheethoo.com, which should be obvious to anybody from bangalore.

but as i think about it, the possiilities are endless. How about an anti-facebook? When you login, it'll tell you how many new people are not your friends, and therefore you can safely ignore. It'll have an app called fortress of solitude, where nobody can get you (except maybe lex luthor).

Tongue in cheek post, this; but shows my frustration with the whole social networking thing. Do we really need to know everthing everyone is doing right now?

Update: I googled anti social networking after I wrote this up, and apparently this is a solved problem.

Auto indexer

Over the course of my adult life, I've had 1 desktop, 3 personal laptops, 2 official laptops, and a blackberry. Of these, at least 1 personal laptop, 1 official laptop and 1 pda of some sort are actively used depending on which one i'm closest to at the moment. My data, notes and files, therefore, are splintered across all these repositories.

The idea is to have an auto indexer that allows me to retrieve the location of a piece of information regardless of where it is. Essentially, the moment I create a document, it must be indexed with the central repository so that whenever I'm interested in that document, the repository should be able to point me to the location of the document, and possibly allow me to (pre)view it instantly on whatever machine I happen to have handy.

OS Ideas: Tagging FS

I'm sure this has been thought of, and probably implemented, but here goes anyway:
The idea is for each "file" in a filesystem to be tagged instead of put into a hierarchy, and the tags themselves are used as the organizational structure. Tags will be just one of the attributes allowed to each file, so adding meta data will be easy, and even encouraged.

Add to this the ability to organize tags into hierarchies or graphs and you have all possible organizational models available to the user.

Finally, search will be the default mode of finding a file - either by its contents, or its attributes, with tags getting special status.

Other applications: version could be another special tag understood by the file system framework, allowing files to be versioned intrinsically.

All of this of course, will be horrendously slow; but wonderfully flexible as a result :)

Painless Timesheets

This is an idea I got while listening to a few colleagues griping about filling out their timesheets. Nobody I know likes timesheets. Yet everyone has to fill them out. So is it possible to make them at least tolerable?

I think maybe.

There are a few programs out there that monitor you while work. PTM is one - it looks exactly like the windows task manager, but it tracks what you're doing automatically, and you can then map it to a set hierarchy of tasks that you set up and it will generate summarized and detailed reports based on the hierarchy.

What if we used a program similar to this to capture how time is spent, use inputs both from live monitoring and scheduled information. Further we make the mapping simpler by providing the ability to tag any timeslice with multiple keywords, and a means to map keys to a hierarchy or graph - whatever makes sense to the official timesheet system. Finally we provide a browser plugin or expect/mechanize type mechanism that auto fills the fields so that the user doesnt have to ever enter time. To top it off, we make the whole system pluggable so multiple timesheet systems can be supported - and extended by anybody.

Advantages:

• User entry is automated, and never requires remembering what was done in the past
• We use the best evidence in hand - calendar info, and actual tracked time on pc
• Its still not big brother as there's no reporting to anyone but the individual
• Data entry is also automated.

Learning Haskell -1

I started learning Haskell as part of "learning a new programming language every year". Here's the frist post:
Am using Real World Haskell, first ed, as that's whats available on Safari books.

Notes:

• Installed Hugs, then ghci cos the book uses that, and there're sufficient differences in the output of the interpreters for me to not bother about it.
• Have started reading about types.
• They are strong, static and inferred.
• Type names are Capitalized
  
• ints are signed, fixed width integers dependent on the machine word size.
• Integers are signed integers of unbound size
• Add :: TypeName after a declaration to set the type. Mostly not required
• function args are written after the name without ()'s or ,'s. Eg fn arg1 arg2
• lists are written as [item,item,..]
• head and tail work similar to lisp car cdr.
  
• Strings are character lists. So tail "abcd" returns "bcd"
• Tuple is a collection of items of different types. Witten as (itemofType1, itemofType2, ..)
• () is the type and value of a tuple of zero elements, called unit.
• take, drop work on lists
• fst, snd work on tuples. due to strict typing, we cannot define a fst fn for all kinds of tuples, only pairs.
  
• Side effect explained: If a function is dependent on a global variable, each run of the function will differ based on the current value of that variable, which alters the way it executes. And therefore, even if it doesnt ever change a value, it has a side effect.
• Pure functions dont have side effects. That is, they're idempotent - every execution of such functions return the same value. Kinda like GET.
• Impure functions have side effects. They are identified by the Monad tag before their type. Eg:Prelude> :type readFile
  readFile :: FilePath -> IO String

Next week:

2.9. Haskell Source Files, and Writing Simple Functions

A pc my 2 year old can use

My kid already can use a pc - minimally. She knows how to start a pc. With the right settings she manages to get it to hibernate. She figured out all by herself that hitting the space bar will pause and resume video on youtube and windows media player.

The key to her ability IMO is following simple rules: She presses the biggest buttons, and recognizes simple patterns. I was wondering if it were possible to create a controlled environment where everything is consistent with what she knows, and therefore she's able to use it better. Some of my ideas:

Direct manipulation: these are based on my observation that she sometimes doesnt make the connection between hitting a key/clicking a mouse and the action it triggers on screen. No fault of hers, imo, its akin to Pickard talking into a mouse.

• Use concepts from Johnny Lee's wiimote project to create a direct manipulation display for her
• Combine this with creating a projection tv using projector panel + OHP to make a viewing + playing + learning environment for her

Sandboxed input: These are based on the earlier mentioned observation - she tends to hit the larger keys, or the ones at the edges. So the idea is to put a fixture around the keyboard that allows access to only those keys. Or put on a template such that all the usable keys are highlighted in red, and the others aren't. And the usable keys would have simple one-to-one mappings to actions -eg, start, shutdown, run video, pause/resume, play (gcompris) game, nav keys. Similarly, the mouse too would be in controlled mode - no right click, click=double click, etc.

The good part about sandboxing is that it allows her to learn, and as her understanding expands, I should be able to scale up towards full keyboard and mouse functionality and this enables her to be productive with existing hardware sooner.Need to research laptop templates for this.

Easy Interface: The OLPC's Sugar UI comes to mind. But methinks an even simpler UI is in order to get her to understand how to use things - with a lot of association thrown in. Eg, really large buttons on the screen, that are the same shape, color and location as the keyboard. Icons that are the same on the keyboard as well.

Scheduled Programs: This is an idea I got from noticing that my kid seems to gravitate towards one of the screens we have at home (no tv, but 4 laptops and 1 dvd player) automatically, and turns it on - invariably to her favorite dvd or cartoon. What if the laptop came up, and the only option allowed at that time was to play an educational game? Or maybe if she's already done one ed game, it shows a cartoon as a reward?

Subtext and monads

Over the xmas week holidays, I got to thinking about Subtext again. Since my conversation with the author had dropped off at the point where he said he's skeptical about getting subtext to work with normal programming languages as they're not functional and he doesnt want to bridge that gap, I set about thinking what would it take to bridge that gap.

And this led me to reading up on a couple of functional programming languages, and of course to Haskell. The thing that caught my attention was Monads, and the assertion that Monads was Haskell's answer to imperative programming - how it allows you to keep things pure and functional, and yet provide the "do this first, then this other thing" style, side effects and all.

This was, of course, exactly the opposite of converting imperative code to functional, but I thought the insight might be helpful.

Two days of confusion and a screaming wife later, I realized there's a reason monads have so many tutorials - they're pretty confusing concepts to start with. I'm still on shaky ground with them, especially having not passed that rite of passage of writing one all by myself, but the a-ha moment came from a post by Oliver Steele about how a monad is a rule that defines how to get from one statement in a program to the next, and further from this comment that even in lazy pure functional languages, f(g(h x) implies a sequence in with h is evaluated before g and g before f.

IMO, a tutorial is required that actually maps a monad's working to that of a von-neumann machine. Then, maybe, we imperative programmers will get it :)

Where does this leave subtext and my original thread of thought? I'm not sure. Would a conversion like this work?
Imperative code --> Monad equivalent --> functional code

How would the first conversion even happen? Need to think about this.

Idea: Automatic unit tests

Although the TDD folks have been pushing the concept of "test code first, actual code next", I've seen that doing that actually requires some maturity and a larger amount of discipline. But what struck me was that all developers actually do test their code before putting it out; they just dont code the unit tests always. How do they do this? Well, by running the code and looking at the output - be it in a debugger or with SOP prints or logger output, or GUI display. Everything that a junit requires is there, except the assertion which is done visually by the developer.

So how about creating a Record mode for the program run - probably as an Eclipse plugin. Once you click record, it would record the state of the program run - every method call and its inputs and outputs. This by itself is a solved problem - debuggers do it all the time. Then when the program's finished running, you present the developer with the execution trace, and ask him to pick the ones he'd like to convert into tests. When he picks particular method calls, the plugin then allows him to choose what preconditions of that method call need to go into the setup() (and corresponding postconditions into the tearDown()) and also allows him to inspect the output and create assertions. Once these are chosen, a test is created with these values automatically, and added to that class' test suite.

Idea: GTD version of Lightning

Been using Thunderbird with Lightning, and its a bit tough as I'm trying to use it GTD style. So this idea is to tweak lightning so its Task features become more GTD friendly.

• First off, tasks should have sub tasks
• Also, they must be allowed true categories, ie one task can be in multiple categories. Right now only one category is allowed.
• Search Tasks feature should be there
• Group by multiple attributes would be useful. Eg group by category (Next Action, Someday/Maybe) and then by context (@Home, @PC, etc - which I use the location field for)
• Link tasks to the messages or events that they have been converted into.

.. and some more that i cannot remember now.

Can I code in Thought?

"I think in pictures, not English"

Way back when I was in probation in the initial stages of my career, I had a colleague who was completely out of his depth with the training assignment given to us. The assignment itself was'nt too difficult - create a client server library management system - the default assignment. At the risk of instantly dating myself as a dinosaur, I should add that this was in the then-state-of-the-art PowerBuilder IDE. He, however, was completely out of his depth as his background was mainframes. He had no frame of reference to start with on GUIs. Being a smart human being, however, he had an idea that stayed with me:

"I wish there was a 'Give up' button", he said, "which would generate all the code that I'm supposed to create once I press it!"

My brain latched onto that moment - maybe due to the novelty of the idea, or its absurdity in terms of actually implementing something like that. And then it linked that moment to another one from my college days:
I had quite innocently asked a classmate what language she thought in and she responded with what I've put as the heading to this para:
"I think in pictures"

This actually caught me unawares and dumbfounded as I'd had a whole brag in anticipation of her expected answer. But my brain latched onto that moment too probably due to the novelty of the idea, or its presumable absurdity from where I stood then.

Well, what if we did actually think in pictures?

With the recent interest in polyglot programming, I was taken back to these two moments in time as it set me thinking about the real-world machinery that actually allows us to convert thought to running code - programming languages.

I was reminded of my evolution along the computer languages scale starting from imperative/procedural languages (x86 assembly, GWBasic, Pascal), scripting (DOS scripting - I know this might sound blasphemous, but when I was growing up access to Unix was not easy :) and linux was still "under development", much like access to the internet - so DOS it was) to object oriented ones (Object Pascal, C/C++, Java). I call this an evolution for obvious reasons - each language was progressively more abstract, and presumably, a better model of the way humans think when solving problems.

These were the languages I could actually run a program in. Then there were others that I could imagine how they worked, but didnt have access to a compiler or runtime, and my only contact with them was through used books that I bought off the street about them - and these really intrigued me. Prolog (via logic programming texts from the 70s) was one, and so was SmallTalk (via old Computer Magazines) and to some extent Lisp and Forth (dont recall the actual sources).

The best of them, however, was APL of which I was able to get an old, but surprisingly well weathered textbook - and what an exposition of the language it was. It blew my mind that there existed a language that not only did not use ASCII, but used normal arithemetic and set theory operators naturally, and even extended the semantics of some of the vector calculus operators into an actual working programming language, and did it such a way as to put Perl to shame. If you dont believe me, go ahead and click that link above, and then come back.

More recently, I "discovered" Tcl/Tk, which seemed to do a similar houdini act - but with language extension; and in such a way that the translation agent (compiler, interpreter) was not too knowledgeable of the extension. How it implements this is in itself intriguing and specific to Tcl, but the feature is (by now) in other languages such as Scala. (I might be wrong in crowning Tcl as the first language to do this, but its definitely one of the first)

So both these languages APL and Tcl had the feature that they allow us to mirror what we want to achieve in a manner that is less natural in other languages - nothing new here, thats what DSLs do. But beyond that, these 2 languages in my experience broke some of the tacit assumptions that I had about how to write code, to which I could (now, in retrospect) add Lisps eval() and macros.

All of this led me to think: what if we "supersized" these breaks from tradition? Would that be closer to the way we think, and therefore let us write better code?

...and a few days of day-dreaming on the CTA buses later...

Here are some of the tangents that thought train led to, and as the list grew I grouped them as:

• Evolutionary changes:
  
  • What if we were able to extend ascii at will?
  • Why are we still fighting the whitespace wars?
  • Do we really need escaping?
  • Syntactic Sugar: why do we still call it that? Its there for a reason, isnt it?
  • How about programming languages "support" a language feature instead of just "enabling" it?
  • What if we used a codebase instead of text files in directories a la Visual Age?
  
  
  
• Revolutionary Changes:
  • What if we could code in something other than ascii or even text?
  • Related:What really requires us to code left-to-right, top-to-bottom other than a legacy of print media?
  • How about non-modal or multi-modal languages? AKA Polyglot programming on steroids.
    
  
  

forNow=UNTIL_NEXT_POST; giveUpButton.click(forNow);

Intrigued? Read on in future posts for a drill down on each question. Fair warning: there are no specific answers. Just some more thoughts..