Fantom: a new language to learn

Today I discovered Fantom - a surprisingly interesting language that seems to get so many things right. My first thought was to re-implement Joey in it. We'll see.

The language is definitely interesting - especially considering 2 brothers built it in the (mostly) spare time.

Things to learn from/use:

• Fan(tom) is self hosting, ie the compiler is written in fan itself. So building it from source involves using an older release of fan. This is pretty standard in compiler land, I guess, but still novel to me; and probably novel in that the although the final executable is a native binary, it goes through intermediate non-binary output levels
• It compiles to its own intermediate code called fcode, primarily because the original intent was to be able to generate code that worked the same on Java and dotnet.
• Javascript compile - sweet.
• The FWT toolkit - havent started using it yet, but any library that allows declarative UI is good in my book. Especially since the same fan source can be used to generate  web ui a la extJS.
• It also seems to have a fluent api to building html - another nice thing
• Units (a la Frink) and a datetime api with durations built in.

PS: When I saw the link in LtU I half expected it to be inspired by the Phantom, but apparently not. I have decided, however, that if I do develop in fantom, the projects will be Phantom themed :)

Some structural editor links

Saving here as there's no delicious bookmaklet on the iPad.

an interesting study of actual developer usage of text editors, and implications for structural editors

Linked discussion at LtU http://lambda-the-ultimate.org/node/3567

FrankenEditor: an unholy marriage of emacs and vi

As is obvious from my previous post, I have just re-discovered these editors; and obviously think there should be a best-of-both-worlds editor. See the bottom of said post for feature list.

Another thought: recast Webster into a vim/emacs mode. That will go a long way in adoption.

Doh! Vi and emacs ARE structured editors

In hindsight this was to be expected. I have been reading up the vi and emacs documentation and finally realize that they are both structural editors - not like Alice and Concrete and Webster (when/if its made) is, but definitely they're smart editors, and in some parts, they're structural editors too.

Vim got it right with "command mode is normal, you drop into edit briefly", while emacs got it right with its major modes.

Thoughts in no particular order:

• They seem to have a fixed concept of the "constructs" that comprise structured text, although that could well be my superficial reading of the docs. ESP considering the file explorer mode, for eg. That seems to be a full-fledged structural editor
• emacs already has the concept of skip words ie structural elements that should be skipped over in navigation which is exactly what Webster/concrete does. Dunno if vi does.
• vi's idea of normal mode seems to be more efficient than emacs's "type normally, use ctrl/escape sequences to insert commands" as a user interaction strategy. Should use this for webster.
• this explanation of why the command sequences need to be as weird as they are is useful in understanding emacs, as was Tim ottinger's explanation of the vi command syntax
• they both separate function from it's keystroke - an important detail.
• The good things, therefore are:
• structured text
• structured navigation
• undo/redo/buffers
• command syntax, ESP of vi
• normal mode is navigation, not text entry
• emacs mnemonic navigation seems better than vi's home row based one, although using modifiers for the levels of objects (char,wore,line,para) doesn't.

It seems like there's opportunity to take a modern stab at an editor that's in the spirit of these two behemoths, and yet goes beyond their limitations.I would not be the first to think this :)

I propose, others implement...And a vim hater recants

Predictably, some of the ideas ive been writing about are being implemented by others, notably webster.
I recently found concrete on github, and it's a pretty close approximation of Webster. It doesn't have all the niceties I meant to implement ESP cos it's viewpoint is MDA, but it's there for the most part, and does the modeling in a neat and lightweight way by reusing html, CSS and standard dom - something I intended to do, but didn't succeed with.

The real reason for this post tho? I just realized that vim could just be half way down the syntax editing route!

Coming from someone who doesn't think much of vim's weird modes and keystrokes that's as about face as it could get. Tim ottinger's vim tutorial was what made me change my mind. This is by far the most approachable vim tutorial iv'e read and finally had me grok the vim way. Now the need for command and edit modes seem almost required, especially considering all syntax driven editing is in such a mode! Vim could be thought of as a syntax driven editor that supports the "text language"!

Maybe I'll implement Webster as a vim mode someday!

Doc Dep: A tool to handle the dependency of code and its corresponding documentation

As I work on Abu, I realize that documenting a project is a separate side project in and of itself. I started with a simple Idea.txt, and that has burgeoned into the following:

• A readme.md (for markdown) for github's main page
• A contents.md showing the contents of the codebase
• A design.md for the design "discussion"
• A user guide - WIP
• A developer guide - to be created
• A roadmap.md
• A log.txt for my personal use - as a tickler for when i stop working and have to pick back up later.
• A news.md (yet to be created) letting people know of recent changes

And what's interesting is the interplay between changes I make to the code (or design) and these documents. Each change in code/design obviously impacts one or more of these documents, but there's no easy way for me to keep track. Eg, change in design impacts the user and developer guides, the news file, and sometimes the readme and roadmap files. Any new top level directory added  changes the contents file and so forth.

Now, what if I had a way to say "this code file impacts these document files", so that when the code file changes, I have to change the corresponding document? That then, is the idea - DocDep - which will have:

• A means to list all the documents in a codebase - some kind of config file should do, or probably a /doc directory could be assumed if we want to go CoC.
• A means to represent the dependency between a file and these documents - easily done by creating a tag - embedded within a comment - in the files
• A means to alert the developer when a change has occurred on a file, which therefore needs a change in the document file linked to it.
• This could be as simple as generating a todo list by commit, Eg: todo: change guide.md to handle change in commit# xxx; Or...
• We could go anal and prevent checkin of the code changes without the documentation.

Codeviz: Some thoughts

This post to be fleshed out better later, but here's an insight I had today.
The format that codeviz will use must contain the following types of information:

• graphs
• nodes
• edges

... and these pieces of information should be easily intermingled with each other, and must be line oriented, ie the data that a node has about its parent should be expressed in that line itself, not presorted by parent to include this child. This enables easier sequential processing.

Quantum Commuter

This is an idea that I've had for quite a while, and thought I'd blogged about already, but apparently I hadn't.

You ever had the choice of two routes to the same destination and wanted to find out if the "road not taken" was the better one? Well this idea is one that'll help you find out.

Chicago's public transport system - CTA - provides an api to find out bus times and a whole slew of transit related information. The idea is to create an app that allows you to select two (or more) buses/trains that ply between points A and B; and pit them against each other using the API. The display would show a live display of the progress of the buses/trains, and who won.

I've always wanted to "clone myself" quantum mechanics-style to do something like this IRL, so I call it Quantum Commuter. Nice play on words, even if I say so myself :)

Btw, this could be an easter egg in an otherwise normally useful bus tracker app :)

Idea: A URL/URI spec for code

This is one of those early morning ideas that I have to put down before I forget it: How about a URL spec for code? Something like:
code://somecompany.com/someapp/package/structure/goes/here/AClass.java#methodName#if3@rev234

The idea is to be able to refer to any part of the code externally, easily, and transparently. guarantees on the posterity of the resource returned similar to general URLs, so they're more URLs than URIs.

Elements that should be in the url:
- language used (better notation that file extension tbd). alternative the protocol could be the language name instead of "code"
- ownership of the app by company/individual identifier. i think the java package concept of inverting the domain name works fine here
- package structure
- source object reference. could be file name, but doesnt have to be so in this abstract space
- intra source object references to methods and specific lines, eg the 3rd if in a method
- version information in the form of a scm version specifier

Code Canvas

I just discovered a Microsoft research project from 2008 that foreshadows quite a bit of my ideas with Webster. Its called Code Canvas (demo video).

In line with the N+1 concept, the few things I can think of (if I were to think of Webster as a product) are:

• Treats everything as a graph - down to the individual tokens. This is a game changer in the way wave was :(
• This also means using graph operations to edit code.
• Any language, not just CLR
• Any mix of languages eg, one web app uses shell scripts, build scripts, assemblies/jars, html+javascript+css, ui frameworks (jsp/asp/etc), compiled code (c#, java, et al), ORM scripts, so forth

I do like the semantic zoom, and the Quad tree algorithm to enable it.

Idea: The "Switch around the And" plugin

I was watching a colleague editing a document, and was struck by the the number of steps he need to change a phase like "cat and dog" and convert it into "dog and cat". I myself have had this problem quite a few times. How difficult would it be to make a plug in (in whatever editor) to automate this simple task?

So, this then, is my next idea: Create the "Swith around the And" plugin. For specificity let's call it a Word Plugin. But no reason to stop there - how about for most other editors?

In fact, when I think about, this could become like one of those projects that write "hello world" in every possible language. The "And switch" plugin for everything from vim to word to what have you :)

And of course, this is just the tip of the proverbial iceberg. Why stop at just "and" as the matching pattern? Why not add this as a feature to any code editor so swapping arguments of a binary operator is easy and automated?

Todo: Somebody must have done this already. Google it.

Idea: Boxview - a new kind of html editor

This is an idea for a better html editor:

• Expose the box model to the html programmer by visually showing the boxes that elements are contained within
• allow changing of padding, margin and border properties directly instead of through property sheets
• automatically calculate the minimal set of properties required to specify the box model built by the user. Eg, if a child and its parent element have the same property, bubble the property up an dinherit it.
•  Allow abilty to specify "global" overflow and containment of elements directly instead of through interacting css properties.
• Allow views of the model with no content, some content and  a lot of content directly, so that the model can be stress tested.
• 
  

Visual != Graphical

It just struck me that visual programming doesnt necessarily have to mean a graphical programming environment.  It should be possible to denote the visual aspects of the program in text mode as with graphical mode.

It does mean some environment, however. A text editor is not one. And that might be the deal breaker for most visual programming environments.

Now what if we had a visual programming IDE in text mode?

Syntactic Sugar IS UI for programming languages

I'm reading Paul Graham's On Lisp, and this section from the chapter on macros struck me as interesting:

Backquote is usually used for making lists.1 Any list generated by backquote can also be generated by using list and regular quotes. The advantage of backquote is just that it makes expressions easier to read, because a backquoted expression resembles the expression it will produce.

...

The longer the macro definition, however, the more important it is to use backquote

 This may not be original, but I just realized that syntactic sugar is the UI for programmers. It just happens to be stuck in the limitation of text. I couldnt help but think that a Structured Editor would obviate the need for backquotes, or the usual "lisp is full of parens" complaint.

All they were trying to do was to make a UI for the programmer

Idea: Where's my stuff

An app that remembers where you keep your stuff. It could be as simple as a wunderbar-style interface. You fill in the blanks for the following sentence:
"I keep/ have kept in/at "
Input could be text or images.

Nothing revolutionary, but should be really useful, methinks. esp as a phone app
maybe this'll be my android app.

webster: avoiding the editing problem

- "everything is a graph": the problem (as is obvious from various discussions on the web on the topic) with structured editors is to allow seamless editing of code. the standard example being converting a for loop to a while loop. My idea is to treat everything as being part of a graph, with infinite breakdown. The default level will be at the "syntax" level, but if you want to break that further, press a hotkey, and the syntax-based placeholders and user inputs get reduced to text nodes.

Superficially, this is similar to the "hybrid editors" that have a mode to switch to text input which is then parsed back. But the key difference is that the conceptual graph is retained.

Combine this with the leaf based parsing should help keep the experience smooth.

Some more thought:
operations allowed by a text editor

• navigate text
• create, modify and delete text
• copy and paste arbitrary text, either in line mode or block mode

to emulate this, how about a structured editor has a "text grid" view

• at every text grid event, the structure model is notified, and it changes the model accordingly
• if a placeholder value is changed, no model changes are required, just the slot value has to change
• if a keyword is changed, a triage construct consisting of the following is constructed:

the rest of the structure is retained in the anticiaption of rollback,

a new structure is anticipated and candidates are stored, which through multiple events may get realized into an actual new structure

• if a new structure is realized, the old one is discarded
• the engine will aim for the quickest new structure that can be realized, given the language syntax

More thoughts on webster

I've been browsing on the general concept of structured editors hoping to find precedents, and finally hit paydirt recently - they were apparently a rage in the 80s and 90s and quite a bit of research was done on them. Of course, they were called syntax directed editors then.

Google returned a lot of hits on this new term, and one paper in particular caught my attention: http://arxiv.org/PS_cache/arxiv/pdf/0710/0710.2358v1.pdf

It talks about a syntax directed editor called Absynthe which allowed editing of code via the grammar of its language - the editor would provide templates for entry of valid language constructs, complete with placeholders for the free text pieces. Of course, they use the right terminology - syntax elements and terminal symbols, respectively.

The paper also went into some detail as to why such editors have not found much favor:

A first problem within a pure template model is that only terminal symbols can be entered as free text. This characteristic is absolutely intolerable in practice. Suppose you want to enter an arithmetic expression containing three or four operators. Then you should select the appropriate template by means of commands or menus for each operator and argument - which results in many additional commands being triggered.

This is why a pure template model has never been implemented in practice. Both the Cornell Program Synthesizer ([Reps 84], [Teitelbaum]) and Mentor ([Donzeau]) have used an hybrid model of top-down replacement of place holders with template and bottom-up construction of phrases from free text. We have also adopted such a model in Absynte. In a hybrid model, a focus is used to delimit an area of free text editing. In this area the text may be temporary incorrect. In the systems where the internal representations are ASTs, a first drawback is the administration required for parsing and unparsing the new representation.

If a free area is really useful to enter arithmetic expressions it does not solve the most important defect of the template model: how to handle modifications within a tree structure ? The replacement of a while statement by an until or if statement is more difficult than the corresponding change in a traditional text editor (the later is quite simple). In general, the replacement of one construct by another causes all information belonging to the old construct to be deleted. If the user wants to save some pieces, he can sometimes do it in special buffers or windows, but he must perform these operations separately (for each syntactic category and, of course, before the complete removal) because every piece of program to be saved must be inserted in its future template separately.

I don't have any better answers than the one suggested in the paper itself - that of storing deleted node content in a special buffer for use later - but here're some of my thoughts:

• Navigation = change of mode. That is, if you navigate out of one of the free text/terminal symbol areas, you're automatically in "edit node mode" - transparently.
• Allow pre-order creation of child nodes. That is, instead of requiring that the parent be created before the children can be, allow the tree to be created from all possible left (or right, if the language is RTL) terminal symbols, and keep pruning the tree as you go.
• More to come as I think of it
  

Webster: the web-based structural editor

This is an update on my previous post of a web-based MPS editor.

I've actually found some time to work on this, and I have a basic version working - well in 2 pieces.

1. A renderer that takes a language spec and an instance of the language, and spews out an editable html version of it - with keywords non editable etc.
2. An editor that takes this rendered version, and adds behavior to it a la MPS - allows navigation, ctrl-space like popups etc.

I still need to combine these together to realize the real goal, ie that of a web based AST editor, which will effectively be Language Workbench meets bespin with an MPS flavor.

But wait, there's more!

My real intent is to go much farther than that:

• Generalize the standard editor commands so that they are the true graph edit operations that are required to transform one version of code to the next. Eg, to add a new import statement, today we insert a line, type "import java.blah.foo;". This should be transformed into: "insert a new import node, which produces an editor node that allows us to choose java from the ctrl-space drop down, then blah, then foo. And so on and so forth.
• Generalize it further such that editing the structure of an application is no different from editing the structure of a class/program - these are graphs too, inherently. So I'm thinking of a zoom in/out feature. Zoom far enough in, and you see the standard AST editor. Zoom far enough out, you see something similar, but it now shows your packages, applications, deployment graph, what have you. Consider using SVG for this.
  
• Webster meets Subtext: Create capabilities similar to the ones demonstrated by Jon Edwards' Subtext. This might however be part of a language that uses Webster more than
  Webster itself.
• Implement Jack in Webster
• Implement Robot framework in Webster

FEM - like analysis on code

This idea is partly from a section I remember from the "Inmates are running the Asylum". Alan Cooper compares code to a huge stack of cards one on top of the other, with just that single flick threatening to bring the whole thing crashing down.

How about treating code as a structure, and using finite element method-like analysis on it? Of course, the math would be discrete, not continuous. But the possiblities are interesting - we already create software in layers, with upper ones depending on the lower ones. Surely some definitions of structural stabilty can be got from those? Similarly the dynamic nature of programs could possibly be modeled as forces...

food for thought..but that's as far as I've got with this idea.

A Web-based MPS Editor

I have been playing around with IntelliJ's Meta Programming System for a couple of days now, and I find it a very interesting and promising tool - to teach myself its constructs, I'v started writing an IDE for Robot scripts. I've got a basic editor and generator done and it looks pretty good, especially for completely declarative code.

I was wondering, however, at the applicability of the concept to the web - as in how easy it would be to port the concept of a declarative structural editor to the web. The MPS editor language is surprisingly similar to divs and spans - all one would need to move the editor to the web would be to slap on a generator that creates D/HTML/5 widget wizardry!

Another project to start....

Implementation thoughts:
This might not need a full fledged editor like EditArea - since its structural, it might actually be cumbersome if it were. It might be easier to have a simple div/span structure or even a table with variable row and col spans, with a single in-place editor component for the current element being edited. Of course, keypress handlers would have to be written to make it usable for keyboarders (like me), and we'd have to find a way to show a cursor always.

Interesting ideas on continuation-style parsers for javascript editors: http://marijn.haverbeke.nl/codemirror/manual.html

Further implementation idea: use the MPS Editor language to define it, and create a generator for the editor from it that outputs GWT! This way the editor is web-enabled!