Idea extension: The dead letter box framework

This is an extension of the idea for the delicious replacement. The idea is to treat accesses to a website as messages dropped off - dead letter box style.

Dead letter boxes allow communication to happen without meeting using a pre-agreed signal or format. The idea is to use a web server's log file as a dead letter box for various types of messages. Like so:

404 GET /dlb/tweet/my_tweet_as_an_underscored_resource_name
404 GET /dlb/todo?task=do%20something&priority=high&context=@home
404 GET /dlb/post-blog-entry?title=A%20Blog%20Title&body=A%20uuencoded_string_as_long_as_log_will_take_goes_here

Again, the way it would work is that a job wakes up regularly, greps the web server logs for these error messages, and does what's actually expected from each of these.

Subversion of http's actual intentions? Maybe. But it sure produces a very decoupled way of working - no high capacity web server with custom code to handle all these operations need exist, and new operations can be added in a very decoupled way.

The best advantage of all? As long as you're able to hit a web server - you can tweet, log a todo, post to your blog, etc, and much more. No more need to be connected to each of these different sources.

Next: do the same with email, so that the same functions are available when you send an email. Although you could implement that as an actual email processsor.

Not entirely addressed yet: security. Maybe making it https would suffice?

Idea: An acceptance test framework for command line programs

This is an extension of the standard acceptance test framework pattern to the world of command line programs.
If you specify:

• a set of scripts to run, 
• a set of inputs and expected outputs, 

the framework runs them, and prints out the results of running the tests like so:

I have most of the framework working in another project that should be easy to extract out and make into a standalone project

The problem with TDD for custom development

I had an idea for a cool app today. I wrote up the initial set of requirements, and came up with the high level "here's how the guts will work" to convince myself it can be done, and naturally started thinking of the architecture.

Soon as I had enough of the pieces "working" in my head, I thought "well  great, let's write some code. But maybe I should do it the TDD way. Hmmm..."

Here's the problem: TDD tools don't exist for what I'm trying to do. Its not a webapp, and it most probably is not in Ruby or Java. Or maybe I should write it in one of those languages then? I dont know.

My point is that TDD is probably not appropriate when you're still doing exploratory architecting of the application itself. You want to see the core pieces work, or you want to figure out what the core pieces even are, TDD will get in the way for one or both of two reasons:

• You'll get hung up on the specification and miss out the architecting
• You'll put your actual project on hold while you build out the TDD tools :)

Now TDD purists would say the specification will extract the architecture or clarify it, but IMO TDD is at its best when the second D implies automation.

As I've been writing this, I'm thinking there's another alternative: Plan for TDD - decide when you'll start doing it and with what tools, and which parts of the app are appropriate to use it for - but dont do it just yet until you know the architecture itself.

Installs should be hudson-simple

Idly musing about the causes for Google Wave to reach the place it did, it stuck me that its new incarnation - the yet-to-be-unveiled "Apache Wave in a Box" - should be dead simple to install if it has any hope of adoption.

My gold standard for java apps that are dead simple to install? Hudson or now Jenkins.It should be as simple as downloading a jar and running it. Everything else should flow from there - including updates.

Idea: InstantQ

It struck me as I was waiting at the bus stop the other day that in urban living is a lot of standing in queues. The bus stop follows the honor system by default, and people are generous to a fault (at least where I live in Chicago), but could technology help?

What if you could start a queue using your smartphone when you arrive at the stop? Anybody arriving after you would look for your queue, and join it.When the bus arrives, it automatically notify each person in the order in which they joined the queue that it was their turn.

Now this might be too slow for a bus stop where once the bus arrives, everyone gets in in a few seconds, but any other place that involves waiting in a queue, this might help avoid queue anxiety.

And now for the commercial version: Any organization that needs to provide service to people has these elaborate systems that show the "Now serving" display could just buy the InstantQ software that would run on a PC and "host a queue". Anybody with a phone could join, and as long as we provide the ability to join a queue via text, we could cover most of the population.

Jack/Webster/Fluent: Use YAML as the text format

YAML seems like a nice fit for a text-based format for Jack or Webster/fluent. The things that attracted me to YAML are:

• Strings dont need to be quoted unless absolutely required. This is a huge advance over json
• YAML has references.

it still might not be the best way to represent code as data, but its close.

Reality Driven Development

I've started reading Growing OO Software - Guided by tests, and two paragraphs in Chapter 1 struck me as interesting constrasts:

The catch is that few developers enjoy testing their code. In many development

groups, writing automated tests is seen as not “real” work compared to adding

features, and boring as well. Most people do not do as well as they should at

work they find uninspiring.

and a few lines later:

If we write tests all the way through the development process, we can build

up a safety net of automated regression tests that give us the confidence to make

changes.

It seemed to me that the first is grounded in reality and the second aspires to a idyllic future.

What if we came up with a methodology that actually assumed reality, and while we're at it, the worst possible one? The concept is nothing new - for eg, network design assumes the worst always - the internet's architecture is replete with strategies against the untoward happening while expecting it to.

So, any methodology expecting to better the software development process should expect that:

• The average programmer doesn't want to write tests
• The code developed will outlive any one person involved in its creation and maintenance
• The architecture, design and implementation will always represent the forces that were in effect at the time of their being decided upon, and therefore will be exactly what those forces required them to be
• Forces will change over time, and will pull/push the architecture, design and implementation in ways not expected originally
• Architects, Designers and Developers are forces on their own right, and will do things "in their own way" despite any official or external mandate .
• Evolution of the software will be more akin to frankenstein-in than darwinian, ie, all software will gravitate towards a Big Ball of Mud
• Average developers will prefer tools to better practices, i.e prefer fixing instances of bad behavior to changing them
• In a large enough organization, average developers are not cross-functional. They have a niche and are very happy in it. The exceptions do prove the rule. 
• The average developer will tend to narrow the definition of his niche because his world is constantly expanding and its difficult to keep up.The only possible exception to this rule is interview time, when the developer will make an all out attempt to project an air of being a generalist.

I could keep going, but you get the general idea. That then, is Reality Driven Development.Nothing new here, I just gave a name to something we all know - kinda like Ajax :)

How to practice RDD, you ask? Well you already ARE - this is the status quo :).

If you're intent on changing that status quo for a better reality however, the first step is to accept the reality. This might be easier for the actual developers to see as that IS the reality, but for people intent on changing that reality it might be a little bit more difficult. I personally am someone trying desperately to close my eyes to this reality because it doesn't fit the ideal world of "how programming should be done". I'm guessing that proponents of Design Patterns, proponents of best practices of any kind, mature developers and TDD/ATDD/BDD practitioners would feel the same way. "If only we could get Joe Developer to see the light" seems to be the underlying sentiment; but accept we must.

Once we accept that this is how we actually build software, we can move in quite a few ways towards a better outcome, and again by extension from fault-tolerant network design, I present some ideas:

• Quality doesn't have to be absolute: However your app currently works, it does. Don't let your next step be 100% quality. Instead focus on the next 5% increment.
• A model of layers of reliable quality built over ones that aren't: Remember the OSI model where each layer did one thing right but was expected to do others not so well? And how layers above did those things right? This is an extension of that idea. I don't have exact suggestion yet on how this should be applied to the list of problems above, but it seems like this is the approach that any solution should adopt. 
• Support over prescription: This particularly addresses changes in behavior such as TDD and BDD. Asking developers to change their workflow on its head is not likely to be accepted except by those already predisposed to changing it. Instead, make the adoption easy by providing support. For eg, why not create a tool that records the outcome of any debug session as a junit test automatically instead of expecting the developer to hand-write the test?

I realize that the ideas above are not exactly fleshed out, but I'm alluding toward an approach to software development that's grounded in reality, and aims at improving the overall maturity and reliability over time. I don't mean something like CMM, however, because its interpretation has almost always meant handing off the quality responsibility to an external auditor. I'm leaning more towards something like the agile manifesto, but grounded in reality. 

Note on CMM and its interpretation: I have found that CMM is more often than not interpreted as an organization compliance initiative, not as a means to measure maturity and improve. This is exactly opposite of the CMM's stated intent, and therefore can be ascribed to flaws in the interpretation of the model. The most visible parts of the CMM machine, however are always big, up-front audits and compliance checks.Its no surprise, therefore, that the average developer treats the CMM process with suspicion, and its outcomes even more so.

Note on interpretation in general: TDD, Agile and such best practices suffer the same issue of the gap between espoused ideal vs interpretation of that ideal by practitioners. RDD is a response to this gap.

Concept: Treat requirements as external forces, especially non-functional requirements

Along the lines of my idea to use FEM-like analysis on code, I realized that it should be possible to model the things that cause the architecture and design of the software over time as physical forces.

Case in point: My team is currently working on adding multi-lingual and multi-currency support to a fairly large app. Adding this predictably requires changes across the board, and led me to visualize all the places we're pushing and pulling at the current design/architecture to make this happen.

Could this instead be considered a single external force acting on the elements of the system? Can we study the "material properties" of the code under such forces and "sum up" the forces?

If we could do that, it would certainly move us out of the current dark ages where cost of change is still measured as a list of things to change per a specific change plan.

Maintainability of code

Most of my career has involved maintaining code built by others. I'd wager that most developers' career has been the same, even if they didn't want to admit it. Sometimes maintenance goes under the name of updating your own code, so people delude themselves into thinking they're not maintaining code, they're just creating the next version. From a maintainability POV, any code that's built is legacy.

And yet most (all?) methodologies of software development either address completely start-from-scratch greenfield development, or propose ideal practices that mature, passionate developers who are in the field for the love of programming.

The reality, however, is that most development involves enhancing or maintaining code - usually somebody else's, and mostly crufty; and in all likelihood the crufty code is from somebody who's presence in the industry is a happenstance rather than a planned event. If you're lucky it merely represents somebody's state of maturity at the time of writing, and that person has now improved.

A comprehensive methodology for maintainable software, therefore, must:

• To address the legacy code angle:
• Provide mechanisms to create "good enough" comprehension of the code
• But avoid attempts at large scale or totalitarian model-driven comprehension. Such attempts will always fail in the real world simply because there will always be something outside the model
• That is, allow for leaky abstractions
• That is, allow for manual steps. There will always be manual steps. The methodology should allow for that by exclusion or explicit statement.The former is easier.
• Help identify what changes and what doesn't. Easier: help identify what has changed and not in the past and let the humans extrapolate to the future.
• Provide a means to migrate from what is to what should be that's incremental.
• To address the maturity concern:
• Allow for different levels of maturity
• Allow for the ability to define "good enough" architecture, design and code; and the ability to easily enforce it
• Allow quick enough comprehension of these definitions
• Allow for gradual adoption, and a means to measure progress

The usual solution is to relegate this to Software Engineering, which typically spirals into talks of CMM and suchlike - process heavy, people agnostic.

The reality, however, is that software development is largely a human effort, and precisely because it lacks the usual shackles of other human endeavors. A mechanical, electrical or electronics engineer will always hit upon some natural limit. Not so the software engineer. His limits are the limits of the mind. If you can think it, you can make it.

And therein lies the problem. If you can think it, so can a multitude of other software engineers; and each such mind can think of at least one variation to the same problem using the same essential solution. This is why I believe we will not see Software ICs any time soon. Most process oriented methodologies tend to gravitate towards this concept, or to the equivalent one of "resources tasked to do X will repeatably produce output of increasing metric Y".

Meantime, in the real world, human programmers are finding ways to game that system. 

As software practitioners, what are we doing to better this? There seem to be promising starts with the BDD and (to a smaller extent) TDD movements, and (in a less focused, but generic scale) with the general move towards declarative programming. There're some incremental gains from tooling in general, but those gains are largely in the area of reducing the steps that the interface requires you to go through to achieve any particular task. There's also some progress in architecture analysis, validation and enforcing that constructs such as DSM and its ilk provide - if practiced regularly.

However, all of these lean toward the mature, self-aware programmer. By the time it reaches Joe Developer, its an organization mandate, not something he WANTS to do. So the gaming cycle begins. This time, however, its insidious because it projects the impression that code quality has improved. We therefore need tools and methods that "work in the trenches". I don't have a good enough answer, but here're are some interim steps that I can think of:

• Allow for easy documentation of what exists, and in such a way that the document follows changes. Working diagrams are a good way of documenting the subset of the code that's understood. 
• Use tagging as a means of collation of documentation created thus. Its easy and "good enough".
• Don't loose what you gain. Developers gain a lot of system knowledge during debugging. There's no easy way of saving that for posterity, so the knowledge is lost. Do not expect developers to write a wiki page - they don't have the time. This area is ripe for tools such as 
• bookmarking and code traversal trails being saved. 
• A "tweet my XApp-fu" app is what's required here.
• A way to share all of this that's searchable
• Make creating tests a 1-click affair. Any run of the software should be convertible into a test, especially a manual  during the normal write-compile-test cycle that the developer engages in.
• Allow cheating on setting up the context. In most production systems, its not easy to inject mocks or test doubles. Test automation should allow for as little of the system being mocked out as possible.
• Mindset changes:Somebody on the team needs to evangelize these:
• "Works on integration box" is less useful than "PROVABLY works on any box". This implies CI without requiring buying into the CI religion.
• Exemplar test data is better than just test data. Exemplars are test data sets that "stand for" a scenario, not a particular test run. 

Webster/Fluent/Jack: Create a text-based syntax as well

I realized that while I'm trying to usher in better programmer tools, most programmers might already be comfortable with their existing ones. So to foster better adoption, the simpler idea might be to allow a text-based syntax as well.

Hopefully, they wont be able to escape the wily wares of better tools for too long!

Idea: Queues: a visualization of work queues

I have been thinking about the concepts mentioned in Product Development Flow, and visualizing a dashboard based on the Queues mentioned in that book. An agile team could be represented minimally by a queue - the list of tasks in its backlog, and how the team processes that list. If we had a visualization of this, it would be easy to represent how well/efficiently the team is working

Extending this downwards, the team's performance could be measured in different ways: as a summation of each team member's work queue (which would allow per team member productivity and efficiency estimation), or as a summation of work completed, or (even interesting) as a summation of the different workstreams it's currently undertaking.

Extending this upwards, a set of teams or the entire organization could be measured in terms of the queues for each team, and the attributes of each queue chosen to be measured.

Speaking in software terms, I'm thinking of creating:

• A system that allows the creation of a queue, and a visualization of it.
• The ability to create parent and child queues
• The ability to "sum up" based on arbitrary attributes of each queue