Branching by Abstraction

Ok Paul, I'll bite.

Some time ago I was working for a client which was experiencing merge hell, come to think of it all the large clients I've worked for routinely experience merge hell, hopefully this isn't correlated with my presence. At the time I mused that when someone creates two implementations of an interface they are effectively branching and so I
voiced that instead of using version control to branch some code perhaps we should Branch by Abstraction. I was working with Paul Hammant at the time and he's apparently ran with this idea and achieved some notable improvements with it.

I don't think BBA it is really that novel, it's just another way of saying 'Hey!, Separate your Concerns', but since a lot of coding-land seems to be engaged in a energetic fight to do the opposite it's probably good to have as many ways of stating the bleedin' obvious as possible.

I'm quite a lazy creature by habit so I've only drafted an article on this on C2, and intend (honest) to polish it up over time, here's a copy of the first draft:


Summary: Don't separate differing concerns by using a VCS, use an abstraction instead.

Long story: Conceive the changes of a project as providing alternate implementations of abstractions already present in the software, or of extracting out an abstraction and making it explicit and them providing an alternate implementation. By doing so the abstraction and the varying implementations provide the separation and branches are need much less.

Consider a possible project that aims to add more data to a business report, say a customer invoice. Suppose that the data being added is the date on which a transaction took place. The current CustomerInvoice might obtain a bunch of LineItems and render them. The new requirement is to add date information to the LineItem and display this on the invoice. At the same time as this project is going on bug fixes are being applied the existing invoice, so we can't just make changes to the invoice.

Step 1) Extract out abstractions needed by the project which are not currently explicit in the code.

1a) Extract the code which provides line items into an interface and implementation: LineItemProvider, NameAndCostLineItemProvider, make the implementation the default the CustomerInvoice uses. Deprecate the implementation with a note to its impending replacement. Test and release to production.

1b) Extract the code which renders line items into an interface and implementation: LineItemRender, NameAndCostLineItemRenderer, make implementation the default the CustomerInvoice uses. Deprecate the implementation with a note to its impending replacement. Test and release to production.

Since the changes in Step 1 should be expressible as a series of refactorings it should be possible to make the test step very short, or arguably unnecessary.

Step 2) Create alternate implementations of the abstractions created in step 1, or already existing before the project began. NameCostAndDateLineItemProvider, NameCostAndDateLineItemRenderer.

Step 3) Release the new implementations in a QA environment configured to use the new implementations.

Step 4) Release the new implementations in production configured to use them.

Optional step) Back out the implementations from production, revert back to the previous implementations at run time.


Whilst most of the project is going on (steps 2 & 3) bug fixes can still be applied to all other aspect of the system, and the dated-invoice project just receives these changes for free, as they are only changing the line item providing / rendering aspect. If a bug fix is needed in the line item providing / rending aspect the bug fix team easily become aware of the new implementation via the deprecation notice, alternately the dated-invoice team can monitor changes to the undated-implementation.

If another project starts up whilst the dated-invoices is chugging along then it will have the same steps, and will involve work on some other abstraction. Should another project need to do work on the same abstractions they too will have the deprecation as a notice to be careful and consider interaction with the ongoing work on dated-invoice, otherwise they can work merrily in ignorant.

So in short the encapsulation that abstractions provide can work for projects just as well as they work for code. When changes occur in other projects then as long as it doesn't involve the abstractions I'm working on in my project then I don't need to care.

Compare this with BranchByVcs. In this one creates a fork of a much larger set of code, probably all of it. Other projects / bug fixes will occur in the same code and later the two changes will be merged together, the merge will take place a series of poorly understood textual changes with an unspecified focus, additionally the merging of changes from some project is usually done by people not on the project.

Poetic / pathetic version control

Old codebase
Clearcase update
Flop

Anti-censorship

A colleague raised a question about whether he should self-censor to remain visible to people behind the 'great firewall of China' or retain prohibited material and 'deny' all his content, prohibited or not.

I offer the links below as my answer.


Appeasing Dictatorship: From Munich to Hong Kong: http://www.aynrand.org/site/News2?page=NewsArticle&id=5137

Save Tibet website:
http://www.savetibet.org/index.php

Tiananmen Square, Wikipedia article:
http://en.wikipedia.org/wiki/Tiananmen_Square_protests_of_1989

Taiwan independence, Wikipedia article:
http://en.wikipedia.org/wiki/Taiwan_independence

China Human Rights Fact Sheet:
http://www.christusrex.org/www1/sdc/hr_facts.html

Java mocking without injection, same line count as Mocha

The Ruby Mocha/Stubba mocking framework written by James Mead allows one to code collaboration expectations very clearly in very few lines.

Stephen Chu has expressed some disbelief that the same would be as readable in Java. However I've found that I can mock in Java with comparable clarity in the same number of lines as Ruby.

public class LaptopTest extends picounit.TestCase
  public void powerOnFailsWhenBatteryTooLow(Battery anyBatteryInstance) {
    should.call(anyBatteryInstance.meter()).andReturn(9);

    Laptop laptop = new Laptop();
    laptop.powerOn();

    verify.that(laptop.powerStatus()).isEqualTo(Power.Off);
  }
}

I've implemented many different ways of mocking in PicoUnit, all of them using some flavour of dependancy injection.

I haven't actually implemented the above one which requires much more infrastructure, it requires that PicoUnit modify every single class under test, specifically it requires all methods of all classes to have a pre-check injected into it (by altering the bytecode during class loading) for whether the method in question is being mocked and a subsequent redirect into PicoUnit if it is or a fall-through into the ordinary code if it isn't.

I've already done plenty of class bytecode manipulation / observation for PicoUnit, many of the features of PicoUnit would be impossible without this, there's so much bytecode stuff going on that PicoUnit has its own mini-framework for just this purpose, so I should be able to implement injectless mocking easily enough.

Convention, Configuration, and Communication Entropy

The ever more popular web framework, Ruby on Rails, has at its core an interesting idiom: Convention over Configuration. The idea goes as follows: for the most part a data model will map directly to a database schema, and for the most part the display and editing of the data will map directly onto the model, therefore: don't specify the mapping in configuration, assume the mapping exists 1-1.

The long established ideas of Algorithmic Information Theory back up these design choices very well, in particular the idea of Communication Entropy applies directly to this situation. The entropy associated with a given communication relates to the amount of information that one learns upon receiving the communication. So for instance when one learns that a table PERSON has a data model equivalent Person and a view person.rhtml then one learns very little because this 1-1 type mapping is very common.

In communication theory one uses the entropy of a message to compress the message, so for instance in English the letter E crops up very often, one learns very little from a message from the presence of E (you can test this by removing all the E's from this text, or indeed all the vowels, you will mostly likely find the text completely readable without them), therefore when we want to send an 'E' down a communication line we send the smallest encoding: 0, leaving the whole of the rest of the alphabet to begin with 1 (t = 101, a = 110, etc.). When one encodes English like this on average a letter can be communicated with about 2 bits. Without encoding the average would be around 5 bits, over twice as large.

Perhaps if the same ideas were applied to software we could also see a reduction in message size. In fact this idea underpins Domain Specific Languages (like those Rails uses): don't communicate that which you can take as a given.

The parallels between TDD derived code and Artificial Neural Networks

I'd like to draw an analogy between artificial neural networks and the code that often gets produced by poor TDD, i.e. TDD done with little respect paid to refactoring.

Artificial Neural Networks

An artificial neural network (ANN) is a graph like structure that maps some set of inputs to a set of outputs. The ANN can make very complex mappings due to having multiple layers and weightings between nodes in the network.

One example of an ANN is one which can recognise a human face. To obtain such a network one stimulates a randomized network with samples of a face and of non-faces. For each sample the network will produce an output. One takes the output and calculates the difference between it and the desired output and then that difference is back propogated through the network. Back propogation of the 'error' causes the network to learn, to produce smaller and smaller errors.

After training one has a very useful tool that can classify things sometimes more accurately than a human can. The faustain bargain one has agreed to is this: One doesn't know how the network classifies its input. The 'knowledge' is just a bunch of numbers, it's a black box.

Code produced by TDD

With TDD the tests take the same role as input to the code and expected output. The developer takes the difference in actual and expected output and back propogates changes to reduce this difference to zero.

At the point when all the tests pass it can be very tempting to move onto the next feature but if one succumbs to this temptation, and does so regularly then the resulting code will closely resemble the neural network, it will work, but you won't know how it works, it too will be a black box.

Danger of over-specificity

Once one has a trained network one might have the danger of having an over specific network: it can recognize all of the faces in the sample set but cannot recognize faces outside the set, even if we might feel those additional faces are too similar to warrant misclassification. To address this problem one usually divides the sample into two sets; one set being used for training and both being used to verify.

But with TDD there is no equivalent to having a sample of tests used to drive the code and additional tests used to verify the code. So not only will the code be a black box but it could very likely be an over specific black box.

Conclusion

I find the conclusion to all of this quite obvious: Refactor your code if you want to retain the ability to understand it! Even if you have great tests and great coverage you may still have a murky black box that nobody wants to work on.

(*) It may be that black box code cannot be tested with good tests and so the existence of black box code will be noticed by the developers as an increasing difficulty when trying to write good tests.