Groovy DSL Voodoo... or why I think imperative programming makes more evolutionary sense

I recently had to consider the Groovy space for a project proposal as the client had decided on Groovy/Grails as the platform of choice. While evaluating the platform for the proposal, I naturally gravitated towards Groovy's DSL capabilities and started thinking about using it for my personal projects, obviously. Abu seems like a natural fit (why didnt I think of Groovy before, I wonder? I'd hit a wall with JRuby due to generics anyway); as did Jack.

Anyhoo, with all this background, I hit slideshare and found tons of decks from the language leads on the specific topic of DSLs. One example piqued my interest in particular: slide 106 of Paul King's deck, which I've replicated here as a gist:

I wanted to both understand how this worked and to replicate it by myself. Here's all the flailing I went about in trying to achieve those two goals:

As you can see, I failed miserably. My last trial had a near-working version, but left me completely disillusioned about Groovy's DSL capabilities - especially considering I came out of the process feeling that all the work was done by the single 'of' closure with the rest playing dummies to the Groovy language's optional syntax rules - especially because in my Englishy version, you could switch the order of the closures passed into 'of' and still have it work fine.

However, all of this didn't explain the "extra code" in the original, which left me with the nagging feeling that no language author would be trying to pull this much wool over people's eyes. So I went back and expanded the original expressions like so:

Now it made much better sense.

Mind you, I still haven't figured out how you go from a problem statement like "I want to express the ability to compute square roots in an English-y fashion" to the solution presented above. This is as far as I can get:

1. I'll obviously need a way to call Math.sqrt and then print it. This needs to be a function name that's english-y
2. An englishy description could be something like "Show the square root of 100". Using Groovy's existing rules, that makes "of" the candidate for the function name.
3. How now do we make the rest of the words work? Well, "show" and "square root" are exactly the two actions to be taken. So as long as I can define those as functions and compose them, I'm good.
4. How do I pass in the functions without naming them?

Obviously, its all the baggage of years of imperative programming knowledge holding me back. Somebody with more skill in functional programming might find this the intuitive enough to roll out.

But I have to wonder: There's too much magic going on here. The imperative approach to the stated requirement would be to define a grammar, write a parser and allow actual interpretation of such a language. Painstaking and error prone, sure; but explicit and clearly out of the programmer's head and into a language that is so dumb that there's no doubt as to what's happening.

This sounds like I'm propounding Worse Is Better in other words,  but there's something more: I realized that the cognitive overload required to understand the Groovy version is higher. More importantly, the cognitive overload to retain that understanding over time is even more so - for "normal" developers who have not walked the functional way for very long. That's the majority in today's world, at least.

More insidious, however, is the implied promise that this is real; that the Groovy DSL is actually English. Could I, for example, expect the slightly less polite "show the square_root of 100" to work? Or the even curt "square_root of 100"?  As an English speaker, why would I not?

As a programmer, I see why 'please' and 'the' are the required glue to make the english-y sentence work within Groovy's pretend-English world. Its extensible in that you could replace square_root with cube_root, for example; but not so that you could change the grammar of the sentence itself. That would require a different set of closures, like the ones you'd find in slide 105 of the same deck, for example. Note that in this version its 'the()' that is the active closure. But I fail to see why this should prevent me from expressing myself naturally as an English speaker.

This then, IMO, is the larger problem with DSLs. When you make something like its real-world counterpart, the human mind immediately taps into the huge body of latent real-world knowledge and tries to use it with that thing: although this DSL doesnt promise it, I immediately wanted to use English structure on it. It's all fine that your programming language has risen to talk your user's language, but has it captured enough of the user's world to go beyond "training wheel use"? To paraphrase Carl Sagan,: To make an apple pie DSL, you must make a universe that's up to scratch. 

How will the DSL author handle this? In general, not very well, I think. I realize I'm picking on a toy example here, but on extrapolation I cannot imagine a domain where all possible concepts and all of their possible combinations are easily enumerated; and then retain their meaning over time and across minds.

I begin to wonder then, at attempts like the VPRI's FONC, where one of the overarching ideas seems to be to create a multitude of DSLs to produce an OS in under ~20 KLOC; and at feeble attempts like my vaporware designs of Jack and Fluent: would they build a better world or is the imperative code bloat of Windows and Linux the more sustainable way?

Explicit Language constructs for architectural concepts

You know how we have modules, layers and environments for code but none of these concepts are actually IN the language?

Modules are typically self contained chunks of code that expose an interface to some useful logic. It is not important how it does what it does; but it IS important that we be able to find the module and call it. In most OO languages, we use functions or object/class methods to hold such logic, but then have to actually build the concept of module using user-visible mechanisms.

Layers are another conceptual tool for organizing code. This typically reduces to packages or namespaces, but neither of these concepts actually enforce a hierarchy or "can/cant call" rules. Similarly, it would make great sense for dependencies themselves to be layered this way, but no language that supports imports also supports defining the layer for those dependencies.

Environments - which I mean the container or context in which code runs - are similarly underspecified.   Most languages define a main entry point and are call it done. Why not have a generic way of stating an entry point that includes a declaration of the environment (or even environments - plural)? The declaration of environment allows for the actual standup of said environment and trigger of the main code to be separated from the code itself; while the ability to declare that "this code runs in both gui and cli environments", for example, allows explicit declaration of such purity in the code and allows determining if this is true statically.

Well, this idea is to explicitly add those concepts to the language syntax itself.

The idea is that the implementation explicitly states the architecture assumed. Two advantages:

• The intended architecture is explicitly available to future developers
• Tools can be brought to bear on code directly, maybe even in the language

Of course, this does mean that the language knows only about specific types of organizing code. Within that limitation, however, things are much more clear, IMO.

What if there IS no source code?

...yeah, kinda linkbait-y heading, I know.

What I mean is: What if there is no single source of the code?

Let me explain.

Typically we have source code. Its written by someone, stuck in a source control system somewhere, changed by others and so on.

The projectional editing school of thought modifies that picture somewhat by suggesting that we could have different views of the same code - a functional/domain-specific one, a folded one, a UML(ish) one, a running trace one and so forth. The relationship between the source and the multiple views, however, remains decidedly one-to-many.

What if instead of this master-slave relation, the different views themselves were the source? That is, the "whole picture" is distributed across the views - like a peer network?

Assuming the views are consistent with each other, modifying the source in one view should retain the true intent. But is that possible even? Views are, by definition, projections of the code; meaning some parts are included in the view and some aren't. So how would we maintain consistence across multiple views?

Two paths lead from here:
One: We cannot. This is why we need the "one true source" that's the parent of all views.
Two: Maybe we don't need consistency all the time. Maybe we can do with the "eventual consistency" that big data/nosql guys are raving about?

#2 seems like an interesting rabbit hole to explore :)

I am Jack's total lack of unlayered design

A nebulous idea grows within me partly from GUTSE's insight that all programming is mere translation from one  language to another: All Von-Neumann machines are merely SSI (Sequence, Selection, Iteration) executors that essentially facilitate these translations. Why not enshrine these facts directly, then? The design of Jack, therefore, stands refined to:

• A simple SSI executor at the base with no idea of what data is beyond the generic map that it reads in as input.
• A data layer (built using the SSI + its environment) that actually does know about the bits of data stored in the map. This layer essentially provides the box/unbox, serialize/deserialize, parse/print dual functions as required
• The data transformation layer (built using the layers below) that enable and reason out the transformations declaratively so that they can be composed, collapsed etc.

Fluent in the real world: Light Table

Today's HN story on Light Table only tells me that I won't have to build Fluent myself - someone else will, soon enough!

In the discussion that ensued, there was however, this interesting comment:

Yes, but our code was entirely in these utterly unusable changeset files that couldn't work nicely with the version control that everyone else in the entire world was using; his version still uses files under the hood. There's a team that's trying to back Monticello with Git, I believe by saving each method into its own file in part of a Git source tree; that looks promising as a compromise.

That's a good lesson to learn from, methinks. Just as "text as code and not just its serialization format" has stood the test of time, so has version control using textual diffs. Anything else will face the migration gap.

I can't be bothered to find the link, but a while ago there was a post by somebody big in the blogosphere (Spolsky or Cringely) that spoke about a bill pay start-up that solved the migration problem by allowing users to forward their paper bills directly to them to have the bills digitized and paid. The idea was that once the bills were digitized they could be paid electronically thence, but it was the first time setup that was the hump that users didn't want to go over. Take that pain away, and you got yourself converts by the droves.

Tripit does something similar with Trip Management by parsing booking confirmation emails from travel websites.

I'm not so sure such an approach will apply to developers and their tools for a couple of reasons:

• Text is actually not a format that you want to get away from (unless you're in the structured camp)
• Providing the gateway solution involves fixing all the tools to work with the non-text format you come up with. And you must fix ALL of them - the IDE, the debugger, the built tool, you name it.
• Alternatively, you could change the environment to something where text input is actually less efficient. Like the Tablet. Then you have a chance.

Its a long, tough road ahead :)

Early thoughts on Jack

I found this writeup from 2009 in one of my umpteen storage media and thought it was evocative enough to publish. My later ramblings on the topic of Jack have been attempts to concretize the ideas rather than gaze starry-eyed at the wonder that is that ephemeral topic of a versionable language. This one is more of the latter kind than the others. There are 2 parts to the writeup: a exposition on top and notes at the bottom. The notes are actually the outline of the exposition, which is essentially unfinished. Except for some cosmetic formatting changes, I've reproduced the piece as is:

Think of an aspect. An aspect usually takes a chunk of code, and in the most general case (called the around advice), wraps around that chunk an envelope. This envelope acts as a guardian to any entry into the chunk and is not only capable of altering expected behavior of the chunk by altering its inputs, but is also capable of deciding if the chunk should execute at all.

Think of a monad. It effectively does the same, except it does it to separate non-functional bits from functional ones.

Think of an ESB. It too, does the same; except that it does it at a much higher level of granuality - at that of a service. Taking the generally understood definition that services are suitably large components that house some specific business functionality, an ESB orchestrates the sequence of operations between these components to achieve the affect of an application.

Now think of how we modify code. We do the same thing - alter the expected behavior, decide if the code should execute at all, (re)orchestrate the sequence of operations to achieve the effect we're expecting.

We are the aspect, the monad, the ESB. 

However there are major differences between us and these constructs that act to the disadvantage of humans while maintaining code:

• each of these constructs have the feature that it describes the change it effects on the component being changed. The humans changes are available only as diffs on an external tool - the version control system

• the human's change is at a textual, character level; not a statement/method/package/module/unit of execution level.So the change is not percieved as a change of language elements to effect it, but numerous characters being shuffled around

While this might seem like a tirade against text-based programming and possibly making a case for structural editors, there's more than that. The key problem is that the unit of execution is not identifiable. Statements in modern languages are anonymous, except by the line number of the source file. If they were identifiable, we could express something like "i had to remove the 5th if statement to after the assignment of var foo" instead of "cut line 150-234, paste at line 450". The former is what we usually do when we talk about it, but there's no direct way of enshrining that in a machine readable way.

Whats the use of such a feature you ask? Well, imagine a language that allowed us to identify the statements, and then express addition, deletion and modification of statements within itself. Something like:

insertStmt module3.class4.method1.if#5, AFTER, module3.class4.method1.letvarfoo

 The fact that the statements of code are addressable allows us to refer to them in a logical manner and the fact that the operations carried out to cause the code to change are operators allows us to maintain the change itself as code.Similar to insertStmt there'd be addStmt, deleteStmt and modifyStmt operators; and obviously we can extend the concept of a function to these operators too, so that the complete conversion from one version to another is expressed as a single operation - a changeset in code, if you will.Producing the next version of your app is no longer a snapshot activity - it can become incremental. Further, multiple changesets can be "compiled" into a fix pack of changes to produce any version at will. And all changes are expressed as logical changes, not textual deltas.

More importanly, think of what the language (or its units of execution - the statements) would have to support for this to work. They would have to become self-contained modules. Self-contained micro services, if you will, which can then be "orchestrated" my moving them to the location in the code which will cause whichever version we desire to be effected via these transformation functions. Code therefore becomes easier to change.

Now lets take it to the next level, and define these operators at all levels of abstraction/modularization that the language supports. So we'd have addMethod,addMethodArg, addModule, etc.

Notes:

• Identity
• modularity
• esb-style orchestration
• not just run time, but also statically, we can express the change occuring. which makes is amenable to machine learning.
• automatic modularization/aggregation is the key to useful versioning.
• forget versioning. what i'm really trying to do is to discover the steps in the computation being carried out that can be abstracted out such that an esb can act on it.
• deriving the higher order sequence from base description
• there are only 3 basic constructs in programming - the assignment, the goto (including the implied goto by the instruction pointer), and the conditional ie if. if is usually followed by the goto, and all instructions of the JZ, JNZ variety are combinations of the if/goto. so the real inflection point for the sequence of operations is the if. we can consider any block of code before an if as a single block with appropriate inputs, outputs and context, and similarly any block of code after an if. each if clause represents a micro service. if therefore is the micro esb. 
• so partitioning of code cana be done based on ifs. now if we take all the ifs at the same peer level - within a method, class or package/module, (or even app) and find the same conditions being checked, those paths can be collapsed, or refactored - this is similar to the ideas in subtext. find the right partitioning of code so that it can be expressed easily. 

Named ranges - a potentially useful concept for Jack (or any normal language)?

Consider the following code that calculates the maximum product of 3 integers possible from a given array (Java used for exposition; could be any language):

The code in the loop could probably be extracted into 2 subroutines but that would involve major interaction between the variables in this one and those subroutines. What if we were able to define the code between the recalc...end recalc comments as a named range that did not introduce a scope? Something like so:

IDEs could fold easily on this and this would only help comprehension.

Finally: Why a language construct and not a comment? Because it explicitly aids comprehsnsion. 

Nice idea from a HN thread

http://news.ycombinator.com/item?id=3650360

...storing here aa bookmark.

Information Density and textual vs visual

I was reading the Wikipedia page on Information Theory as part of my read the wiki project, when it struck me that there's possibly an objective way of measuring the effectiveness of text vs visual programming languages using Information Theory.

The central concepts (whose math I'm admittedly unable to fathom) are those of information, information rate, entropy and SNR. One of the age-old cases for text-based programming ( and therefore against non-textual programming languages) has been that it has very low SNR and the "information density" is high for given screen real estate.

Is that really true, though? How much "noise" does syntax add? On the other side of the spectrum, I've seen infographics that assuredly deliver more "understanding" in the given screen space than the equivalent textual description. Is it possible to design an "infographic-style" programming language that packs more power per square inch than ascii?

It would be interesting to do some analysis on this area. 

Jack and Turing completeness

From the previous posts on Jack's primitive capabilities it should be pretty obvious that it is turing complete. It supports sequences and selection naturally; and allows iteration of any node's children.

The intent, however, is to NOT use Jack as a primary programming language. I do not see a hello world in Jack's future - at least not in the conventional sense.

Jack's Turing completeness is intended to provide a full complement of computing power so that program logic may be effectively TRANSFORMED. The more appropriate set of hello world would programs may be:

• A program that edited another program and represented the editing as a sequence of Jack steps
• A program that reasoned about another program using Jack
• A program that expressed the history of a program and exposed its "why" using Jack
• A program that expressed the future roadmap of a program using Jack

Jack: the whys and wherefores

As I pen down these thoughts that I have about Jack, I realized that there's a little bit of the "losing the forest for the trees" going on - while the initial posts were  what-oriented, the latter ones have been decidedly how-oriented. The "why" is kinda sprinkled around. This post attempts to correct that gap.

Jack is the culmination of a series of vaguely connected concepts that have been swimming below the surface in my consciousness. Some of the individual strands are:

• Code today suffers from want of better abstraction for itself. The simplest manifestation of this is the conflation of its persistent, modeling and editing formats into one single form - text. I posit that treating code in its true form - an AST - directly will reap higher benefits.
• There are sub-strands to this where concepts from Subtext (and the whole language workbench movement, for example) finds resonance with my thinking and therefore has been subsumed into it.
• True code structure is latent and the structure made apparent by its organization into files and folders is misleading. I posit that all code will look like graphs (a la social networks) once we represent (and visualize) the actual relations between code components. At that point, graph CRUD techniques can be used to manage the code and graph analysis techniques can be applied on it to reduce the comprehension overhead.
• Today's applications require connecting together at least 4-5 different languages. There is no conceptual glue to hold these together, splintering comprehension and adding to the accidental complexity. I posit that treating apps as a graph containing code components will allow for easier comprehension and maintainability.
• Today's development practice is woefully snapshot based. There is no way to represent the code's past (so that you may learn from it or make corrections easily) nor its future (so that you may plan for it or code in a certain way)
• The average developer follows Reality Driven Development, no matter what methodology they say they follow. 
• All of this has direct bearing on EXISTING code. Most approaches to bettering software development talks about tools and techniques that you can use on your NEXT project. I'd like to come up with tools and techniques that you can use on the CURRENT or PREVIOUS project, or even that ANCIENT one from the '80s - the one that's already in production and making money for the business and will never be replace (at least not completely) that you have to either maintain or talk to. This is also the hard and unglamorous side of the development cycle - the maintenance side. However this is also the place where code spends most of its time and yet is mistreated much. 

Jack is the code name I use to represent my solution to these issues. It started out addressing the format and representation issue and has burgeoned into including the rest of the concerns. It therefore has gained the larger qualities that any solution to the problems above should have:

• It should not favor any particular language or framework - at least not design and definitely not in the ultimate implementation.
• It should not introduce another language either; instead use existing language runtimes as host environments and introduce ways to work with the code by adding metadata and the ability to programmatically refactor it.
• It should introduce a runtime environment that is live in the same way a LISP repl or Smalltalk IDE is. Compiled environments should get as close as possible to this ideal. This is fluent.
• It should allow for models of the code to be built at varying levels of abstraction from the code itself. These models would be used to understand code and build it. It should therefore subsume disparate technologies used to build working apps or make it easy to add new ones into the modeling platform.
• It should allow incremental adoption of itself. No big bang, lose-your-religion type steps because we're dealing with maintenance code and developers following RDD.
• By definition, therefore, all tools and techniques within should have models for incompleteness: incomplete code, design, architecture, relations, etc. This should be used to help comprehension of the code in "comfortable bite sizes" and to allow description of future code whose specifics are not yet known.
• It should have a native access to the history of the code so that comprehension can follow. This is the reason the previous post has the scm FFI.

So there you have it: the WHY and WHAT of Jack in one post.

Jack: some new thoughts, a few iterations

Some strands of thoughts have come together of late as I re-read my posts about Jack, watched Subtext and talked about CodeViz to people. Things congealed into place as I revisited OMeta and decided to give it a shot as a prototyping platform.

Iteration 0

As a precursor to trying OMeta out I listed out the features that Jack was supposed to have based on my posts. Here's what I came up with:

Transforming this to OMeta syntax, here's what I came up with:
As you can see, some compression has happened from my BNF-ish syntax to the OMeta one.

Iteration 1
Pumped by this quick success, I immediately started counting my unhatched chicken. How would people easily share Jack code, I wondered? I'd already thought of YAML as a storage format so, I quickly wrote up the YAML version of the function from above:

I also came up with the whole slew of pipelines that would bring Jack (and some part of Fluent) to life:


text syntax   --JackTxtParser-------> JackAST
JackAST       --JackUIRenderer------> JackViewTree
JackViewOp    --JackUIModifier------> modifiedJackViewTree
JackViewTree  --JackInterpreter-----> JS Evaluation
JackViewTree  --JackCompiler--------> JS Source
JacViewTree   --JackTxtGenerator----> text syntax output

Iteration 2
Then reality set in. What are the primitives I'd need to build for this to take off? A reverse read of all jack posts revealed the following set of "things to be built":

• A primitive conditional
• The ability to make a module out of anything and vice versa
• make module (code list)
• devolve module into codelist(module)
• A way to denote a function as optimized via a Foreign Function Interface (FFI). This could be in the fact language, but this means the fact language and the compiler should talk.This is probably required anyway; the interpreter should be able to query the analyzer for facts' veracity.Note: This facility is like annotations, but doesn't entail arbitrary side effects as the annotation processor allows through arbitrary code running.Each FFI, however, should expose a way to call the underlying optimization, with ways to map values to and forth.
• The ability to refer to any single piece of code: built into the structure already. url tbd
• The ability to refer to any set of code pieces: define a continuous code range, define an arbitrary list of code pieces (is this required)
• The ability to comment on such a set of code pieces: ie attach a comment to a code set. this is somewhat similar to the modularize reqt above
• An FFI to scm tools with the basic functions supported
• check in/out
• commit
• branch
• merge
• snapshot
• A functional version expression
• The ability to represent WIP code
• Most importantly, an interpreter that supports all this

Iteration 3

Then I took a step back and looked at the complexity growing. Could there be some abstraction done? Here's the outcome of some furious (re)thinking:

• There are nodes
• Nodes have attributes
• Standard attributes are:
• [id] is used to uniquely identify a node. it can be system-generated or user-provided
• [comment] is used to provide a comment about the node.
• [fact] is used to state a fact about the node using FOPL; and is typically used to derive some "higher order knowledge" about the node
• [name] is used to provide a referencible alias for the node.
• [kind] is used to identify the type of node. this may be dynamically assigned to implement duck-typing.
• [context] is used to "run" or execute the node.
• Other attributes can be added at will and used in execution
• A collection of nodes is also a node, and therefore can have the same attributes and be executed similarly.
• Standard nodes are:
• base:
• block : a continuous sequence of nodes to be executed one after the other
• if : a node that conditionally executes one of its child nodes
• define : a node that adds a name to a node
• function : a named block
• module : a collection of functions
• app : a collection of modules
• meta:
• group : a node that groups other nodes (and optionally names)
• split : a node that splits an existing group
• insert : a node that inserts a node at a given point in a collection
• delete : a node that deletes a node from a collection
• versioning:
• checkout : a node that checksout a version of its input node ref from scm
• commit : a node that commits a version of its input node ref to scm
• branch : a node that branches a version of its input node into a new branch
• merge : a node that merges two node refs
• standin : a node that can stand in for any other node. to be used for nodes that dont exist yet 

Thoughts

I reached this far and realized something larger abstractions are possible:

• Jack code could be stored in any data format that can handle trees - backward compatible data format!
• Jack could use any language as host - backward compatible code! Jack is "just another scripting language on top of $my_fav_lang

So, it sounds like:

• Jack's true role will be to consolidate sequences to specific hosts, create and break interfaces and manage the change that happens - a sort of super shell.
• Ultimately Jack should enable code comprehension and legacy support.

Maybe Jack should be renamed Glue.

Jack: Allow WIP pseudo code

You know how you sometimes mock up code with some keywords to represent the structure of the logic but then put in hand-wavy text for rest of it because you've not fleshed it out yet?

Jack should support that directly. Something like:
if --calamity happens--
    --dont panic--
    --call 911--
end

Just a hint of an idea at this time. I'm not sure if -- is the right sigil/delimiter to use. "<<" seems like a good one from personal experience, but it has bad web mojo :). 

Jack/Webster/Fluent: Use YAML as the text format

YAML seems like a nice fit for a text-based format for Jack or Webster/fluent. The things that attracted me to YAML are:

• Strings dont need to be quoted unless absolutely required. This is a huge advance over json
• YAML has references.

it still might not be the best way to represent code as data, but its close.

Jack: Javascript should be the base language

It just struck me today morning as I was getting ready that the base language for Jack - my idea of the next gen programming language - should be Javascript. The good parts, of course, but definitely javascript.

This ensures that it has the widest spread of runtime environments, has the functional bits built in, and - as long as we keep the base concepts to the good parts -  none of the ugliness

Idea: A URL/URI spec for code

This is one of those early morning ideas that I have to put down before I forget it: How about a URL spec for code? Something like:
code://somecompany.com/someapp/package/structure/goes/here/AClass.java#methodName#if3@rev234

The idea is to be able to refer to any part of the code externally, easily, and transparently. guarantees on the posterity of the resource returned similar to general URLs, so they're more URLs than URIs.

Elements that should be in the url:
- language used (better notation that file extension tbd). alternative the protocol could be the language name instead of "code"
- ownership of the app by company/individual identifier. i think the java package concept of inverting the domain name works fine here
- package structure
- source object reference. could be file name, but doesnt have to be so in this abstract space
- intra source object references to methods and specific lines, eg the 3rd if in a method
- version information in the form of a scm version specifier

FEM - like analysis on code

This idea is partly from a section I remember from the "Inmates are running the Asylum". Alan Cooper compares code to a huge stack of cards one on top of the other, with just that single flick threatening to bring the whole thing crashing down.

How about treating code as a structure, and using finite element method-like analysis on it? Of course, the math would be discrete, not continuous. But the possiblities are interesting - we already create software in layers, with upper ones depending on the lower ones. Surely some definitions of structural stabilty can be got from those? Similarly the dynamic nature of programs could possibly be modeled as forces...

food for thought..but that's as far as I've got with this idea.

A true code format, and its editor

This is post is influenced by my recent preoccupation with code representation (see post on coding in thought, for instance) and inspirations from Subtext and ediola.

Imagine a code editor that allowed you to edit the actual AST of the language instead of text that will be parsed into the AST of the language. It would obviously make it transparent to you that you're editing a tree instead of free form text, except for taking out the inefficiencies. For instance it would allow you to use the normal navigation keys to scroll up and down through the code, but would skip keywords and language constructs automatically because they're are actually tokens and don't need to be treated as English words. When you navigate to a user entered value such as a variable, it would drop into an in-place editor that allowed you to change it, and it would then automatically change all occurrences of that variable because it knows that its a variable.

For the same reason, it would be easily able to provide auto suggestions /completions as it knows the language's syntax. An add-on feature would collect data about imported and declared classes, methods and variables as you edit, and make those available in the auto suggest as well.

The best part of course would be it would have none of the limitations of text - no need to worry anymore about whitespace as it would become exactly what it should be - white space that is merely a visual thing, not something that's stored in the code.

Sidebar: I realized after I wrote this that such an editor could be at a loss with a language like Python, which essentially mandates whitespace. However, I do think the reason Python went the way it did is reconciling to text being the universal code format, and if so, making the best of it, namely to avoid all visual clutter caused by begin/end tokens, and instead rely on whitespace to denote it for both humans and compiler alike. That being the case, there's no reason this editor can mimic the indentation, and provide the output that the python interpreter expects.

How would such code be stored? Why, as the AST itself. The actual storage format (as long its not back to text) is not important, as any number of those are applicable. What is important, though, is that it is in the exact format that the language allows for human thought to be enshrined as code - nothing more (specifically, text formatting warts), nothing less.

Once we have this, we can think of more improvements. How about treating comments as annotations on code, and extracting them out completely? This way, all the issues related to comment obsolescence and non-locality are resolved. We could just tie a comment to a particular piece of code, and wherever the code goes, the comment follows. If the code chunk is removed, broken or altered, it would be trivial to mark the comment as outdated and to be treated with a pinch of salt.

There's another gem hiding in the previous paragraph - that of tying a comment to a particular piece of code. Well what exactly is "a particular piece of code" unless we're talking about some named chunk such as a method? Nothing defined, yet. However, consider this: what if we had a way of identifying every piece of code uniquely - an id for each atomic piece of code? Notice that I've specifically avoided considering the line number as the id because is not just another of those text formatting appendages, but also an unreliable anchor to any particular piece of code as it can easily change with an unrelated edit of the file. I'm thinking more of :

• a unique id for each statement (or its component) (eg, the 5th statement of the main method - which could well be on the first "line" if it were text)
  
• expressions for groups of such statements (eg, first 5 statements of method foo),
• reference via scope/namespace (eg: 3rd if in Class a's method foo)

- an XPath for code, if you will. Once we have this, refactoring becomes analytical, even expressible as code. And of course, we'll be able to attach all kinds of metadata to chunks of referencible code, including comments.

Update: This from my wife: "3rd 'if' in method a.foo" is as arbitrary as line number. The better idea is to sequence them in order of creation, ie, the first 'if' created in a.foo gets id #1, the second #2 and so forth - for eternity. That way edits will keep the piece of code still identifiable. Ids could be periodically cleaned up to remove unused ones.

Ok, all of the above has been about editing code, not necessarily the format laid down by a language itself. But the more I think about it, formats have been more a limitation laid down by the tools that surround a language rather than the languages themselves. Compilers require classes to be contained in single files. Compilers require directory paths for package name spaces and so forth. There is really no reason a compiler cannot read a pre-parsed version of code as its primary source. By the same token, there is no reason source code cannot be stored as the AST - in version control or outside. The newer source code tools shouldnt have problems dealing with whatever binary format is chosen to store the AST, and diff against them.

And you know the best(est) part? No more tabs vs spaces wars!

Further Ideas for a new language

Some more random additions to the new language idea:

A name: Jack
This is mostly tongue in cheek but I thought if I ever got around to creating this language, I'd call it Jack. You know, like in Jack of all trades, cos its trying to be one - with its polyglot leanings. I did consider calling it Poly, but the options for source file name extensions would be .pl or .py or .ply. First two are taken, and the third doesn't seem that interesting. OTOH, .jak comes out nice.

Aside, I did also consider calling it V after me, but vetoed it cos it didnt give any indication about the language itself, just self promotion.

Anyway, Jack it is. And I've not yet checked if there's already a language with that name - hope not. But think of the possibilities with such a name:

- You could tell others off by saying "You dont know Jack"
- Like Java Jars and Ruby Gems, there could be Jack (in-the) boxes :)
- I know I'm gonna call the hello world in Jack "Jackrabbit slims" :))

Enough buffoonery, now for some serious stuff:

It has optimisations
You know how you have to "drop down a few layers" to optimize things? Like for eg, you can do 3d graphics using the Java3d APIs or you can use OpenGL or you could just drop into the video card's API? Well, Jack will have the same capability. If a thing can be done in more than one way, and one way is the composition of multiple functions, while the other is a single function that does effectively the same, you will be able to specify that. Instant reuse. Want to use the MFC classes, or OS sys calls to do something? Go right ahead. And supported in the language too. Sandbox to be specified, but definitely much easier than JNI et al - remember I'm still dreaming :).

The key thing I want to point out is the "supported in the language" bit. We as programmers routinely cut across language/API boundaries. However, each language is in its own sandbox held together by scripts, duct-tape and magic. Why not be explicit about it? Obviously there will be need for sandboxes, but making it explicit enables maintainability. It should be clear how different parts of the application ecosystem actually connect.

Ideas for a new language

A number of nebulous ideas have been swirling in my mind of late on what's lacking with today's languages. I woke up today morning with the remnants of a dream where I was coding in a language that magically had closed all those gaps, and even improved things a bit. Here's what my conscious self remembers from that dream.

It was based on conditionals
I have been thinking a lot about conditionals lately, partly due to being impressed by Subtext. But more closer to work, I've been thinking of a large legacy application and how to refactor it so that chunks of logic that are in the wrong layers can be relocated easily, and how other chunks of logic that are interwined with each other even though they logically belong to different paths can be extracted, and then combined via configuration using (for lack of a better term) a micro ESB.

And it struck me that the conditional (represented by the plain vanilla if, or an inheritance hierarchy) is the cleave point of "paths". All conditionals that are based on a particular check (eg: is the object of type foo, or is the value equal to x) are essentially in the same path. Therefore code should be organised (or auto-organised) such that conditionals are matched by their paths, and the largest conditional branch naturally gravitates to the top. This is, of course, exactly what Jonathan Edwards' Subtext does, with the additional advantage that it also ensures that there are no gaps in the conditionals - all possible values(or value ranges) have to be filled in. This would be a great thing for a language to have.

So the language from my dream had this feature. It worked using conditionals, and like subtext it auto arranged them - but all in text. That's the dream part, i guess.

Now when I think about it though, I can envision this conditional cleaving happening incrementally - a module can be conditionally complete within itself, but not so wrt other modules in the application.

It had modules/services/components as a first class language construct
Meaning, the "chunks of code" concept that I mentioned above were directly supported by the language. Think COM/XPCOM, or SOA/Web Services - with more accent on the fact that there's a published API backed by an implementation; and less on how they would be discovered, remoted etc. Code by contract, basically.

I don't remember if it had explicit support for constructs such as classes and interfaces that combine to form the components. It wouldn't matter if it did or not, but the key thing was that the component WAS defined, and was the cornerstone of composition.

It was extractably modular
And by this, I mean that chunks of code could be extracted out into modules by the language itself.Not a preprocessor or optimizer - the language itself had operators to extract code, define the api based on the contents of the code, and place it elsewhere. It therefore had meta programming capabilities, and more importantly, code elements were addressable in a platform- and source file-neutral way. Every unit of execution from the statement to the module was addressable to enable the modularisation operators to work.

It had DRY testing
Defining a module or class implied testing it. The language statically checked for bounds and ranges based on the conditionals. In addition, every run with values can be recorded as a test, adding to the test set.

It supported legacy apps
It had the ability to read old app code; and then show the gaps in logic or discover (if not extract) the true modules. It might have had a compatibility mode or it allowed incomplete (from its perspective) code to run.

It was functional at its core
and used monads for everything - especially the micro esb.
... but it had a lot of dsl-ish sugar to make it seem more general purpose.

And finally, it was (j)vm based.

Aftermath
After I wrote this far, I got to thinking about what goals a feature set such as this would arise from? Here's what I got:

• backwards compatibility
• support multiple programming styles
• direct support for common use cases in development and deployment