Groovy DSL Voodoo... or why I think imperative programming makes more evolutionary sense

I recently had to consider the Groovy space for a project proposal as the client had decided on Groovy/Grails as the platform of choice. While evaluating the platform for the proposal, I naturally gravitated towards Groovy's DSL capabilities and started thinking about using it for my personal projects, obviously. Abu seems like a natural fit (why didnt I think of Groovy before, I wonder? I'd hit a wall with JRuby due to generics anyway); as did Jack.

Anyhoo, with all this background, I hit slideshare and found tons of decks from the language leads on the specific topic of DSLs. One example piqued my interest in particular: slide 106 of Paul King's deck, which I've replicated here as a gist:

I wanted to both understand how this worked and to replicate it by myself. Here's all the flailing I went about in trying to achieve those two goals:

As you can see, I failed miserably. My last trial had a near-working version, but left me completely disillusioned about Groovy's DSL capabilities - especially considering I came out of the process feeling that all the work was done by the single 'of' closure with the rest playing dummies to the Groovy language's optional syntax rules - especially because in my Englishy version, you could switch the order of the closures passed into 'of' and still have it work fine.

However, all of this didn't explain the "extra code" in the original, which left me with the nagging feeling that no language author would be trying to pull this much wool over people's eyes. So I went back and expanded the original expressions like so:

Now it made much better sense.

Mind you, I still haven't figured out how you go from a problem statement like "I want to express the ability to compute square roots in an English-y fashion" to the solution presented above. This is as far as I can get:

1. I'll obviously need a way to call Math.sqrt and then print it. This needs to be a function name that's english-y
2. An englishy description could be something like "Show the square root of 100". Using Groovy's existing rules, that makes "of" the candidate for the function name.
3. How now do we make the rest of the words work? Well, "show" and "square root" are exactly the two actions to be taken. So as long as I can define those as functions and compose them, I'm good.
4. How do I pass in the functions without naming them?

Obviously, its all the baggage of years of imperative programming knowledge holding me back. Somebody with more skill in functional programming might find this the intuitive enough to roll out.

But I have to wonder: There's too much magic going on here. The imperative approach to the stated requirement would be to define a grammar, write a parser and allow actual interpretation of such a language. Painstaking and error prone, sure; but explicit and clearly out of the programmer's head and into a language that is so dumb that there's no doubt as to what's happening.

This sounds like I'm propounding Worse Is Better in other words,  but there's something more: I realized that the cognitive overload required to understand the Groovy version is higher. More importantly, the cognitive overload to retain that understanding over time is even more so - for "normal" developers who have not walked the functional way for very long. That's the majority in today's world, at least.

More insidious, however, is the implied promise that this is real; that the Groovy DSL is actually English. Could I, for example, expect the slightly less polite "show the square_root of 100" to work? Or the even curt "square_root of 100"?  As an English speaker, why would I not?

As a programmer, I see why 'please' and 'the' are the required glue to make the english-y sentence work within Groovy's pretend-English world. Its extensible in that you could replace square_root with cube_root, for example; but not so that you could change the grammar of the sentence itself. That would require a different set of closures, like the ones you'd find in slide 105 of the same deck, for example. Note that in this version its 'the()' that is the active closure. But I fail to see why this should prevent me from expressing myself naturally as an English speaker.

This then, IMO, is the larger problem with DSLs. When you make something like its real-world counterpart, the human mind immediately taps into the huge body of latent real-world knowledge and tries to use it with that thing: although this DSL doesnt promise it, I immediately wanted to use English structure on it. It's all fine that your programming language has risen to talk your user's language, but has it captured enough of the user's world to go beyond "training wheel use"? To paraphrase Carl Sagan,: To make an apple pie DSL, you must make a universe that's up to scratch. 

How will the DSL author handle this? In general, not very well, I think. I realize I'm picking on a toy example here, but on extrapolation I cannot imagine a domain where all possible concepts and all of their possible combinations are easily enumerated; and then retain their meaning over time and across minds.

I begin to wonder then, at attempts like the VPRI's FONC, where one of the overarching ideas seems to be to create a multitude of DSLs to produce an OS in under ~20 KLOC; and at feeble attempts like my vaporware designs of Jack and Fluent: would they build a better world or is the imperative code bloat of Windows and Linux the more sustainable way?

Why new bottles with the same old wine works... everytime!

I attended the second BangaloreJS meet today. The sessions were good and so was the food; and I met some really interesting people so in all a fun time was had.

Before the sessions started, however, Jon - the meetup organizer - showed a quick demo of Yeoman: a new tool released by Paul Irish and some of his friends from Twitter that eases building client-side JS projects.

I had a senior moment when I finally realized what it was.

This was Rake in client-side JS clothing, which was ant in Ruby clothing, which was make in Java clothing, which was ... I dont know what came before make because that was waaay before I was born, but I know there was something else that all the cool kids left for make.

Yes, I realize this does way more than a simple build tool; much like rake does way more than just build code. And it did it in a way that seemed magical when it came out first, too. You could do all that with ant tasks/plugins but it was not the same thing. But that's besides the point. It was a build tool in its core.

Why do we as developers keep.reinventing.these.common.tools?

Answer: For the same reason tool vendors have different versions of the same tool by language.

See, I happened to be browsing JetBrains' website the other day and saw that they claim that their non-Java IDEs share the "same features as other Jetbrains IDEs" or something to that effect. It struck me that they surely have a generic base version of the IDE that is skinned and customized for each "vertical" i.e., language-specific IDE.

Why then do they have one tool per language?

Because there are very few polyglot programmers in the world. Majority of us stick to one language, maybe a platform with a few related ones. That's IT. So we want something that "speaks to us, knows our problems", not something that can boil the PL ocean. And why would we need all those generic, all-language features anyway, right?

So the vendors sell us the shiny new bottles. And we drink it up.

And that's the way it should be. Why should client-side JS devs (who are in an unprecedented hype cycle after a long winter) be grudged a "robust and opinionated client-side stack, comprised of tools and frameworks that can help developers quickly build beautiful web applications"?

Note to self: When you finally finish creating Fluent, remember to brand it by language. FluentC, FluentRuby, FluentJava all sound very usable :)

Tools are the bane of the Beginner

"Huh? Isn't that counter-intuitive?", you ask?
I realizing something particular as a result of some experiences and wrote that line down, but to me even it sounds counter-intuitive; so let me hasten to explain.

A beginner, for the purposes of this discussion, is somebody who's beginning something. This could be a novice starting to learn a skill; but it could also be an expert who's beginning a new project within his area of expertise.

A a beginner (thus defined) is hindered by the presence of tools and frameworks for two reasons: 

• They obscure the what and the how of the problem at hand by hiding it within ( usually for novice beginners).
• They prevent easy exploration of the why of the problem and its solution space through ceremony and preventing access(usually for experienced beginners).

Allow me to present the experiences that led to this realization. It all began one mundane work day in the not-too-distant past....

Story#1

My team had just got a bunch of freshers. They were recent engineering graduates (presumably with some exposure to programming) who have passed through the company's training program (which again presumably imparts further such exposure). We, however, found that they couldn't do some simple tasks like write code outside Eclipse. They didn't know how to deploy a web application except through the Eclipse plugin; had never debugged an application via logs and in fact didn't know  about webapps as an idea independent from "Tomcat". Their OO concepts were shaky at best, but they had implemented small-but-complete web applications using Tomcat, Struts and Hibernate and passed an EJB exam. When asked to build their study app the same from scratch using a command line, however, they were lost. When asked to build a different application (than the one they'd done) *using Eclipse*, they were similarly lost.

While a large portion of the blame should rightly lie in the teaching methodology (or lack of one), the tools and frameworks too, IMO, should bear some of it. "Deploying" for them meant clicking on the Tomcat icon in Eclipse, so they had no use of knowing what "web.xml" did nor did they know that it was no longer required. The same wizards and menu options that make the life of a practitioner easy actually obscure the underlying process (and why its required) to a beginner.

Story#2

The same group of freshers were slowly getting on track with (re)learning the basics of programming, when I thought it might be a good idea to instill in them at this "early age" the values of Test Driven Development. I immediately checked myself, however, because they'd have to learn JUnit and how to use it. On second thoughts, however, I realized that they didn't HAVE to use Junit or any such framework to do TDD. All they had to do was write a test before writing the actual code, have it fail, write the code and have it pass the test. The test could be code in the same function or in main() or as a series of calls to the program stored as a shell script or as JUnits. All of them are equally valid as "tests". We generally, however, recognize only the last of these as tests. The concept of TDD has been usurped by the concepts of the tools that implement it - to the extent that TDD doesn't seem have a life outside of those tools.

Tools, therefore, seem to be actively scuttling the consumption and adoption of Concepts, even when they were created explicitly for the reason of automating the repeated application of known concepts.

I personally have been struggling with this - the guilt of not doing TDD vs the allure of just seeing working code - especially when I'm beginning something and still feeling the problem and solution spaces out. In some cases, the solution space doesnt have readily available tools (BDD on the browser, anyone?) and in others there are tools but I'm still not ready to commit to them because I dont know what my solution is yet (should I build the parser first to figure out the syntax or the AST interpreter to see how it would run?). My liberation came when I declared that a test will be whatever I call a test for the situation at hand, not what some framework determines to be one. Since then the test-red-code-green-repeat cycle is a much more doable one.

Full Circle

Back to the initial "Huh?" moment from the beginning. Why then do we generally consider tools to be useful - especially for beginners? Tools are generally time-savers. They do one thing and they do it well; and that is their value. They do, however, have an "operating range" in which they're most useful. Below that, they're overkill and above, they're obstructive - as depicted in this highly accurate graph on problem size vs tool effectiveness:

So when we usually talk about tools being useful, we're talking about the useful operating range. Specifically for beginners, tools are solution accelerators at that range. The stories presented here represent the two ends of the spectrum, however, where tools are sub-optimal.

Note: I've glossed over frameworks in this discussion, but the concept is the same; or applicable even more so for frameworks. Frameworks by definition are a standard solution to a common problem, with room for customization so that application specifics can still be implemented. The framework is one because it has a known world view and exposes an interface that allows operations on that world view. The concept of operating range is well-ingrained, therefore; as are those of the limits on either side.So please read "tools/frameworks" wherever you see "tools" below.

So...

Armed with this framework for evaluating tools, we can start asking some interesting questions.

1. What is a good tool?
2. When are tools not required?
3. When are tools required?
4. How do we determine the operating range of a tool, then? 
5. What can we do to use tools more effectively?
6. What can tool builders do to make effective tools

Attempts at answers to these questions in part 2 of this article.

I visited JavaFXville and all I got was this post

I tried to write an FLV player in JavaFX today.

I tried very hard.

I didnt get very far. Or when I did, it was not obvious how far I actually had.

It all started with me looking for a trustworthy way of downloading Youtube videos - you known, the Randy Pausch kind. To share with my team easily.

Googling for Java and FLV showed that JavaFX spoke FLV natively. Awesome. I'd heard enough of JavaFX in its previous incarnation as F3 to think it was worth a shot for a weekend project. Not to mention this is the closest that one can get these days to simple declarative GUI.

Of course, I'm sadly backed up on the news front. Oracle, in its infinite sagacity, decided that the declarative DSL was to go; replaced by a Java API. So what looked like a simple sample in the 1.3 documentation morphed into this beast of boilerplate in 2.0.

But wait, there's FXML - the Oracle blessed replacement for FXScript, right? It IS XML, but its still declarative, right?

Wrong. Conjuring up the right incantations to get MediaView instantiate a Media object is obviously beyond me. I've never liked XML as a UI description language and this abomination only stokes that feeling.

So I give that up and get back to the POJO version; which works for the sample FLV they provide.

So I proceed to download a Youtube video using a Sourceforge-based downloader (also Java) and the bloody thing doesnt load. Maybe the downloader is broken?

I dunno. The code looks like its doing multiple passes on the embed element's params; and justifiably so because that value's a snakepit of multiple URL encode sessions. Security by obscurity, perhaps?

Ugh.

Its 4 in the AM, and all I have to show for my efforts is this blogpost.

Final nail in the coffin: I now read that JavaFX has spotty support for Linux. The whole intent of this was to create something in Java so that I can share it with my linux-using team.

FML.

I know everyone has to be a grasshopper first, but...

... could I do it matrix "I know Kung-fu" style , please?

I bull-in-a-china-shopped my way through SICP so that I can get to the metacircular evaluator bit as soon as possible, but now I see even more clearly that each chapter firmly builds on the previous one in this book.

As I read the implementations of eval and apply I realize I have to go back and read the foundational material before I can expect to just grok them. My kingdom for the knowledge packs from the Matrix :)! 

New paradigms in wiring code emerging

It seems to me that ideas happen at the same time to many people. I've been thinking of ways of wiring code other than stacking 'em one after the other, and I find Coplien talking about DCI (although in fairness his ideas are a bit old - I just discovered them now), and today this:

http://thorstenlorenz.wordpress.com/2011/07/23/dependency-injection-is-dead-long-live-verbs/

The title's a bit dramatic, and the content seems original but is probably not entirely new(Google and ACM search pending).

What excites me, however, is:

• Wiring code is being treated as a first-class problem
• Solutions are recognizing the problems of being in Nounland (thanks Yegge), but not knee-jerking into functionville entirely. Solutions are emerging that are somewhere between, where the common coder lives!

Dependency Hell: There has to be a better way

I have just given up on trying to install Open Head Tracker - an open source eye gaze tracker. I've been researching head tracking software to see if my aunt who's currently paralyzed can use something like this, and naturally started checking out the open source offerings available; and given that she's in India, I wanted to look for a Windows solution as that's easier to purchase, service etc for non-technical people.

So open gazer is based on technologies that are cross-platform: Python, Numpy, OpenCV and Qt. The original was tested on Ubuntu, and the author expresses the sentiment that since all the underlying software is cross platform, and the software itself is in python, it should work on other platforms.

He's right in principle. In practice he is so wrong its not funny.

Unlike the linux world, you usually install binaries on windows; you just do. Its a wimpy, windows-y thing. Sorry. All windows software comes that way - and then you deal with Registry hell :)

And although each of the packages above were available as binaries, they were just not a compatible set. 2 days of trial and error later, I got the Mingw compiled release of the right version of each package to be installed and tested individually.

That didn't mean they worked together to show up Open Head Tracker's display. I still get a segfault on cvResize (after debugging through the source - thank god for scripting languages!) and I have no clue why. The only option remaining is to disable use of SSE.

The only way to do that? Compile from source.

If my experience with trying to build Chrome is any indicator, that's another wild goose chase into build-time dependency hell.

There has to be a better way!

Sidebar: The last two days have been a heady roller coaster ride through Numpy and OpenCV code, however. I dont understand most of it - mainly because I was just trying to get Headtracker to work, but there's some cool stuff in there, including in Headtracker's deceptively small codebase. Qt's demos make it look really cool too - too bad the days of the Desktop are done now. 

TDD is an all-or-nothing proposition

I've been reading "Growing OO Software with tests" and trying to implement TDD practices in my projects, and I arrived at the sentiment that's the title of this post.

Let me explain.

Non-TDD is easy and alluring because you can choose to ignore certain errors. This is decidedly unprofessional, but in the early stages of the project its invaluable. When you're trying to get your feet wet, and feel around the solution space, you DO want to know the big things in the design you're missing, but you dont want to know EVERY thing you're missing.

Since TDD implies executing code, you do have to fix everything that you've missed before you get to the interesting bits; and that is why I say TDD is an all or nothing proposition - you have to fix each and every bug in your straw man code before you can get it work.

Usually, some of this extra burden is taken care of by the TDD framework that you're using - a typical TDD tool assumes a particular environment (eg, Rails), and has the support that makes most of this go away, but for environments that don't have such support - Javascript-in-browser or console apps come to mind - building the framework while building the app gets to be tedious and frustrating.

That's not to say that even with TDD tools this burden doesn't go away - its just lesser. It'd be much nicer if you could "ignore" errors that aren't crucial to early stage prototyping. For now, I solve the problem by having a prototyping phase/spike which isnt TDD based where I flesh out the solution concepts in throw away code.

Installs should be hudson-simple

Idly musing about the causes for Google Wave to reach the place it did, it stuck me that its new incarnation - the yet-to-be-unveiled "Apache Wave in a Box" - should be dead simple to install if it has any hope of adoption.

My gold standard for java apps that are dead simple to install? Hudson or now Jenkins.It should be as simple as downloading a jar and running it. Everything else should flow from there - including updates.

Reality Driven Development

I've started reading Growing OO Software - Guided by tests, and two paragraphs in Chapter 1 struck me as interesting constrasts:

The catch is that few developers enjoy testing their code. In many development

groups, writing automated tests is seen as not “real” work compared to adding

features, and boring as well. Most people do not do as well as they should at

work they find uninspiring.

and a few lines later:

If we write tests all the way through the development process, we can build

up a safety net of automated regression tests that give us the confidence to make

changes.

It seemed to me that the first is grounded in reality and the second aspires to a idyllic future.

What if we came up with a methodology that actually assumed reality, and while we're at it, the worst possible one? The concept is nothing new - for eg, network design assumes the worst always - the internet's architecture is replete with strategies against the untoward happening while expecting it to.

So, any methodology expecting to better the software development process should expect that:

• The average programmer doesn't want to write tests
• The code developed will outlive any one person involved in its creation and maintenance
• The architecture, design and implementation will always represent the forces that were in effect at the time of their being decided upon, and therefore will be exactly what those forces required them to be
• Forces will change over time, and will pull/push the architecture, design and implementation in ways not expected originally
• Architects, Designers and Developers are forces on their own right, and will do things "in their own way" despite any official or external mandate .
• Evolution of the software will be more akin to frankenstein-in than darwinian, ie, all software will gravitate towards a Big Ball of Mud
• Average developers will prefer tools to better practices, i.e prefer fixing instances of bad behavior to changing them
• In a large enough organization, average developers are not cross-functional. They have a niche and are very happy in it. The exceptions do prove the rule. 
• The average developer will tend to narrow the definition of his niche because his world is constantly expanding and its difficult to keep up.The only possible exception to this rule is interview time, when the developer will make an all out attempt to project an air of being a generalist.

I could keep going, but you get the general idea. That then, is Reality Driven Development.Nothing new here, I just gave a name to something we all know - kinda like Ajax :)

How to practice RDD, you ask? Well you already ARE - this is the status quo :).

If you're intent on changing that status quo for a better reality however, the first step is to accept the reality. This might be easier for the actual developers to see as that IS the reality, but for people intent on changing that reality it might be a little bit more difficult. I personally am someone trying desperately to close my eyes to this reality because it doesn't fit the ideal world of "how programming should be done". I'm guessing that proponents of Design Patterns, proponents of best practices of any kind, mature developers and TDD/ATDD/BDD practitioners would feel the same way. "If only we could get Joe Developer to see the light" seems to be the underlying sentiment; but accept we must.

Once we accept that this is how we actually build software, we can move in quite a few ways towards a better outcome, and again by extension from fault-tolerant network design, I present some ideas:

• Quality doesn't have to be absolute: However your app currently works, it does. Don't let your next step be 100% quality. Instead focus on the next 5% increment.
• A model of layers of reliable quality built over ones that aren't: Remember the OSI model where each layer did one thing right but was expected to do others not so well? And how layers above did those things right? This is an extension of that idea. I don't have exact suggestion yet on how this should be applied to the list of problems above, but it seems like this is the approach that any solution should adopt. 
• Support over prescription: This particularly addresses changes in behavior such as TDD and BDD. Asking developers to change their workflow on its head is not likely to be accepted except by those already predisposed to changing it. Instead, make the adoption easy by providing support. For eg, why not create a tool that records the outcome of any debug session as a junit test automatically instead of expecting the developer to hand-write the test?

I realize that the ideas above are not exactly fleshed out, but I'm alluding toward an approach to software development that's grounded in reality, and aims at improving the overall maturity and reliability over time. I don't mean something like CMM, however, because its interpretation has almost always meant handing off the quality responsibility to an external auditor. I'm leaning more towards something like the agile manifesto, but grounded in reality. 

Note on CMM and its interpretation: I have found that CMM is more often than not interpreted as an organization compliance initiative, not as a means to measure maturity and improve. This is exactly opposite of the CMM's stated intent, and therefore can be ascribed to flaws in the interpretation of the model. The most visible parts of the CMM machine, however are always big, up-front audits and compliance checks.Its no surprise, therefore, that the average developer treats the CMM process with suspicion, and its outcomes even more so.

Note on interpretation in general: TDD, Agile and such best practices suffer the same issue of the gap between espoused ideal vs interpretation of that ideal by practitioners. RDD is a response to this gap.

Maintainability of code

Most of my career has involved maintaining code built by others. I'd wager that most developers' career has been the same, even if they didn't want to admit it. Sometimes maintenance goes under the name of updating your own code, so people delude themselves into thinking they're not maintaining code, they're just creating the next version. From a maintainability POV, any code that's built is legacy.

And yet most (all?) methodologies of software development either address completely start-from-scratch greenfield development, or propose ideal practices that mature, passionate developers who are in the field for the love of programming.

The reality, however, is that most development involves enhancing or maintaining code - usually somebody else's, and mostly crufty; and in all likelihood the crufty code is from somebody who's presence in the industry is a happenstance rather than a planned event. If you're lucky it merely represents somebody's state of maturity at the time of writing, and that person has now improved.

A comprehensive methodology for maintainable software, therefore, must:

• To address the legacy code angle:
• Provide mechanisms to create "good enough" comprehension of the code
• But avoid attempts at large scale or totalitarian model-driven comprehension. Such attempts will always fail in the real world simply because there will always be something outside the model
• That is, allow for leaky abstractions
• That is, allow for manual steps. There will always be manual steps. The methodology should allow for that by exclusion or explicit statement.The former is easier.
• Help identify what changes and what doesn't. Easier: help identify what has changed and not in the past and let the humans extrapolate to the future.
• Provide a means to migrate from what is to what should be that's incremental.
• To address the maturity concern:
• Allow for different levels of maturity
• Allow for the ability to define "good enough" architecture, design and code; and the ability to easily enforce it
• Allow quick enough comprehension of these definitions
• Allow for gradual adoption, and a means to measure progress

The usual solution is to relegate this to Software Engineering, which typically spirals into talks of CMM and suchlike - process heavy, people agnostic.

The reality, however, is that software development is largely a human effort, and precisely because it lacks the usual shackles of other human endeavors. A mechanical, electrical or electronics engineer will always hit upon some natural limit. Not so the software engineer. His limits are the limits of the mind. If you can think it, you can make it.

And therein lies the problem. If you can think it, so can a multitude of other software engineers; and each such mind can think of at least one variation to the same problem using the same essential solution. This is why I believe we will not see Software ICs any time soon. Most process oriented methodologies tend to gravitate towards this concept, or to the equivalent one of "resources tasked to do X will repeatably produce output of increasing metric Y".

Meantime, in the real world, human programmers are finding ways to game that system. 

As software practitioners, what are we doing to better this? There seem to be promising starts with the BDD and (to a smaller extent) TDD movements, and (in a less focused, but generic scale) with the general move towards declarative programming. There're some incremental gains from tooling in general, but those gains are largely in the area of reducing the steps that the interface requires you to go through to achieve any particular task. There's also some progress in architecture analysis, validation and enforcing that constructs such as DSM and its ilk provide - if practiced regularly.

However, all of these lean toward the mature, self-aware programmer. By the time it reaches Joe Developer, its an organization mandate, not something he WANTS to do. So the gaming cycle begins. This time, however, its insidious because it projects the impression that code quality has improved. We therefore need tools and methods that "work in the trenches". I don't have a good enough answer, but here're are some interim steps that I can think of:

• Allow for easy documentation of what exists, and in such a way that the document follows changes. Working diagrams are a good way of documenting the subset of the code that's understood. 
• Use tagging as a means of collation of documentation created thus. Its easy and "good enough".
• Don't loose what you gain. Developers gain a lot of system knowledge during debugging. There's no easy way of saving that for posterity, so the knowledge is lost. Do not expect developers to write a wiki page - they don't have the time. This area is ripe for tools such as 
• bookmarking and code traversal trails being saved. 
• A "tweet my XApp-fu" app is what's required here.
• A way to share all of this that's searchable
• Make creating tests a 1-click affair. Any run of the software should be convertible into a test, especially a manual  during the normal write-compile-test cycle that the developer engages in.
• Allow cheating on setting up the context. In most production systems, its not easy to inject mocks or test doubles. Test automation should allow for as little of the system being mocked out as possible.
• Mindset changes:Somebody on the team needs to evangelize these:
• "Works on integration box" is less useful than "PROVABLY works on any box". This implies CI without requiring buying into the CI religion.
• Exemplar test data is better than just test data. Exemplars are test data sets that "stand for" a scenario, not a particular test run.