Jack: the whys and wherefores

As I pen down these thoughts that I have about Jack, I realized that there's a little bit of the "losing the forest for the trees" going on - while the initial posts were  what-oriented, the latter ones have been decidedly how-oriented. The "why" is kinda sprinkled around. This post attempts to correct that gap.

Jack is the culmination of a series of vaguely connected concepts that have been swimming below the surface in my consciousness. Some of the individual strands are:

• Code today suffers from want of better abstraction for itself. The simplest manifestation of this is the conflation of its persistent, modeling and editing formats into one single form - text. I posit that treating code in its true form - an AST - directly will reap higher benefits.
• There are sub-strands to this where concepts from Subtext (and the whole language workbench movement, for example) finds resonance with my thinking and therefore has been subsumed into it.
• True code structure is latent and the structure made apparent by its organization into files and folders is misleading. I posit that all code will look like graphs (a la social networks) once we represent (and visualize) the actual relations between code components. At that point, graph CRUD techniques can be used to manage the code and graph analysis techniques can be applied on it to reduce the comprehension overhead.
• Today's applications require connecting together at least 4-5 different languages. There is no conceptual glue to hold these together, splintering comprehension and adding to the accidental complexity. I posit that treating apps as a graph containing code components will allow for easier comprehension and maintainability.
• Today's development practice is woefully snapshot based. There is no way to represent the code's past (so that you may learn from it or make corrections easily) nor its future (so that you may plan for it or code in a certain way)
• The average developer follows Reality Driven Development, no matter what methodology they say they follow. 
• All of this has direct bearing on EXISTING code. Most approaches to bettering software development talks about tools and techniques that you can use on your NEXT project. I'd like to come up with tools and techniques that you can use on the CURRENT or PREVIOUS project, or even that ANCIENT one from the '80s - the one that's already in production and making money for the business and will never be replace (at least not completely) that you have to either maintain or talk to. This is also the hard and unglamorous side of the development cycle - the maintenance side. However this is also the place where code spends most of its time and yet is mistreated much. 

Jack is the code name I use to represent my solution to these issues. It started out addressing the format and representation issue and has burgeoned into including the rest of the concerns. It therefore has gained the larger qualities that any solution to the problems above should have:

• It should not favor any particular language or framework - at least not design and definitely not in the ultimate implementation.
• It should not introduce another language either; instead use existing language runtimes as host environments and introduce ways to work with the code by adding metadata and the ability to programmatically refactor it.
• It should introduce a runtime environment that is live in the same way a LISP repl or Smalltalk IDE is. Compiled environments should get as close as possible to this ideal. This is fluent.
• It should allow for models of the code to be built at varying levels of abstraction from the code itself. These models would be used to understand code and build it. It should therefore subsume disparate technologies used to build working apps or make it easy to add new ones into the modeling platform.
• It should allow incremental adoption of itself. No big bang, lose-your-religion type steps because we're dealing with maintenance code and developers following RDD.
• By definition, therefore, all tools and techniques within should have models for incompleteness: incomplete code, design, architecture, relations, etc. This should be used to help comprehension of the code in "comfortable bite sizes" and to allow description of future code whose specifics are not yet known.
• It should have a native access to the history of the code so that comprehension can follow. This is the reason the previous post has the scm FFI.

So there you have it: the WHY and WHAT of Jack in one post.