Writing good code requires you to perform experiments

By Confusion on Thursday 5 March 2009 07:52 - Comments (19)
Categories: Java, Software engineering, Views: 19.493

How often do you have to solve a small problem that is a part of a larger project and decide to take the time to perform some seperate experiments to solve the problem, before adding the partial solution to the whole? In the past, I hardly ever did that and everytime I encounter such a situation, I still have to resist the temptation to take the route that seems to be the shortest, but that has long proven to be the longest road to the solution.

As an example, suppose you need to extract some information from a string and decide the best way is to use a regular expression. You know that the problem isn't trivial and that you probably need a few attempts to get it right. Still, your initial inclination is to just put the regex in there and go through the motions of compiling the code, performing the steps required to invoke the code (click a button, enter some text, etc), find out that the regex doesn't work, modify it and go through the motions again and again and again, probably while making some other modifications. Because the cycles take relatively long, progress is slow and you start cursing every typo. If you recognize this, then I think you can become faster and happier at your problem solving by reading on.

The advantage of the 'intuitive' approach is that, if the solution is right, you have immediately come closer to solving the larger problem. However, if the solution fails, the iterations to be informed of your failure and to verify subsequent modifications may soon take more time than developing a seperate solution to the subproblem would have taken. For some reason, we always seem to underestimate the amount of work it takes to get it right and consequently we opt for the small change that we will have the correct solution the first time around.

Some of you may now be thinking that the obvious solution is to write a proper unit test and run that unit test after every modification, until the code passes the test. I agree that goes a long way, but usually the test is part of a larger number of classes and running all those tests takes times, especially if it bootstraps an entire Spring-Hibernate application or something of the like. In such as case, it still takes more time than is needed.

My solution is not to be afraid to experiment. It seems to cost too much time to create a new script or class, seperate from the larger project, provide the correct libraries, run this small project multiple times to get feedback on your solution and finally copy-paste the solution back into the project your are working on. However, my experience is that it is well worth the time you need to do this, because it prevents the endless cycles of building-deploying the entire application and getting it to provide feedback. I hardly write a regular expression or SQL query without first testing it seperately from the application. Now these example are given at the smallest level, but it also holds for somewhat larger design issues and even for issues people call 'software architecture'.

Another thing about these small experiments that I find a major advantage, is that it gives you the freedom to explore some avenues that you wouldn't dare put into the actual project. You can't just start switching libraries or refactoring relatively large parts of the code and changing too much is risky and frowned upon, even if you can easily revert your changes (assuming you are using a versioning system. If you aren't, stop reading, install Subversion and commit your code before continuining this article).

A third advantage of experimenting is that it encourages you to rewrite and polish the code that you are writing. You can go through more iterations in experiments 'outside' of the project, because you have a clear overview over the code involved. Modifications of code inside a larger project are less inviting to proper refactoring to do 'the right thing'.

Painters, writers, craftsmen, even philosophers: if famous ones are asked for the secret of their succes, they always advise exercise and experimentation. I hope I have explained that it also simply makes sense. If you don't want to take it from them, take it from rationale.

Why functional programming doesn't catch on

By Confusion on Saturday 21 February 2009 22:30 - Comments (61)
Categories: Java, Software engineering, Views: 84.130

Authorities in the field of software engineering have long lamented the fact that functional programming (FP) doesn't seem to catch on. Their sympathizers often blame that on shortcomings in the audience or their pointy-haired bosses, but that is the easy way out: to assume that someone that disagrees with you simply lacks the ability to understand you is a way of shielding yourself from the possiblity that you are the one with a shortcoming: the inability to provide a proper explanation.

I think that the actual reason for the limited adoption of FP languages is exactly that: the way the advantages are formulated and exemplified are completely obvious and compelling to their proponents, but those same explanations and examples utterly fail to convince most software engineers, because they simply do not appeal to the problems they are confronted with in their daily line of work.

I think this hypothesis is best exemplified by the seminal paper Why Functional Programming Matters. This paper sets out to
[..] demonstrate to the 'real world' that functional programming is vitally important,
[..]
[listing of advantages]
Such a catalogue of 'advantages' is all very well, but one must not be surprised if outsiders don't take it too seriously.
[..]
We must must find something to put in its place - something which not only explains the power of functional programming, but also gives a clear indication of what the functional programmer should strive towards.
and then goes on with explanations like
The definition of reduce can be derived just by parameterising the definition of sum, giving
(reduce f x) nil = x
(reduce f x) (cons a l) = f a ((reduce f x) l)
and applications to things like the Newton-Raphson approximation and tree pruning in artifical intelligence.

[I'm inserting an intentional dramatic silence here, in which you can see my facial expression turn to complete astonishment]

Now how in heavens name can one expect to convince the average software engineer of the tenets of functional programming with these kinds of examples? If it does one thing, it is scaring people away by seeming unnecessarily difficult and academic.

Every introduction to a programming language shows you the recursive method to calculate Fibonacci numbers. It's abstract, many people do not relate to it very well, but it's only a single example. However, the documentation for FP languages seem to consist solely of these kinds of highly mathematically inspired examples. No 'Address' class to be found there. Hasn't anyone written a functional equivalent of the Pet Store application to demonstrate the power of FP for the regular work that most of us do?

People that want to improve the world often overlook one fundamental problem: you cannot improve the world just by being right. You need to convince others of that fact if you want to exert influence. If you cannot convince them, find out why you cannot convince them. I think there is a bright future ahead for functional programming, as soon as someone stands up to convince the masses.

Update:
As not everyone reads the comments, I would like to highlight a few points made there:
  • The article I quote is 25 years old and was directed to people in academia. It would not be fair to conclude that all FP documentation is that hard to understand for someone without an academic background.
  • There are deliberate attempts at displaying the 'real life' value of FP, for instance in Real World Haskell and Practical Common Lisp.
  • Several people assert that FP is catching on, but it just isn't very visible (yet). This may be because those that use it consider it a competative advantage

Spring, Hibernate and leaky abstractions

By Confusion on Friday 20 February 2009 22:33 - Comments (3)
Categories: Java, Software engineering, Views: 6.403

The advantage of using frameworks/libraries like Spring and Hibernate is supposed to be that they take care of the nasty details and that you only need to think about what you wish to do on a much higher level. However, abstractions are always leaky and when you run into a problem, you still need to understand about the nitty-gritty details.

This week, I ran into the following problem: my Spring/Hibernate based application locked up. It was reproducible and seemed to depend on the number of queries that had taken place. So I started Tomcat with
-Xrunjdwp:transport=dt_socket,address=8000

told Eclipse to connect to that socket (debug remote application) and suspended operation after the application frooze. The result was clear: no more connections in the connection pool and an infinite wait until one would be available.

Now this application is reasonably old and has worked succesfully for quite a while. Everything seemed to be in order, until, after looking at the Javadoc of the HibernateDaoSupport class for the umpteenth time, it suddenly hit me: the latest DAO used getSession() instead of getSessionFactory().getCurrentSession() to obtain a session. While the first could very well be a shorthand for the latter, it most certainly isn't. I noticed the difference when I reviewed the differences when updating from Subversion and read the appropriate Javadoc, but I wasn't paying enough attention to realise that the difference between these ways of obtaining a Hibernate session was a nitty-gritty detail that was very important.

Using grails on an existing database

By Confusion on Thursday 12 February 2009 22:12 - Comments (3)
Categories: Java, Software engineering, Views: 18.414

I've encountered the following problem several times: I needed to inspect or alter data in a database, but there is no interface to this data. Todays problem was that I needed to edit some html in a SQL server database, but the so called 'Enterprise Manager' did not display these fields, complaining about the 'grid buffer size'. Of course it turned out to be impossible to find out how to increase that value and using the Query Manager to edit large text fields isn't very convenient either.

Now I could write some custom Java to perform the specific actions that were required, but I would be hard pressed to provide full CRUD capabilities within a reasonable period of time and I certainly wouldn't have a GUI. After searching a bit for frameworks to generate CRUD with editing controls for an existing database, I decided to give grails, a Ruby on Rails type framework based on the JVM language Groovy, a go.

Now firstly, it is important to realize that such frameworks are usually bent on specifying a domain model, which is then used to generate the database. When used with an existing database the generator code will often try to modify your database. In case of grails, by adding an 'id' and 'version' column. If you generate the skeleton for a grails application (see the quick start, there will be a database configuration file in grails-app/conf/DataSource.groovy. When you modify this file to add your configuration, be sure to comment the lines saying

Groovy:
1
dbCreate = "update" // or "create" or ...



The order of operation is as follows: create domain objects, generate default views and controllers and modify them appropriately. There is a tool called GRAG that can help you to reverse engineer your domain objects from your database and although it produced some odd results in my case, it will certainly help you on your way. A domain object may look like this:

Groovy:
1
2
3
4
5
6
7
8
class Email {
    static mapping = {
         table 'emails'
         id column:'email_id'
         version false
    }
    String content
}


The 'mapping' element is required because I'm not following the defaults: if the table was called 'email', the id column was called 'id' and the table would have a 'version' column, it would not have been necessary.

After having created domain objects, you need to run
grails generate-all Email

to generate a controller and a view. Follow with
grails run-app

and you've got a CRUD application running for this table and its two fields.

Of course, you'll often need a bit more than this, but for instance changing the input element for the html field to a large enough textarea and adding some functionality to properly (un)escape html was easy.

Is there more bad PHP than bad Java?

By Confusion on Sunday 8 February 2009 11:34 - Comments (7)
Categories: Java, Software engineering, Views: 4.305

Bad code is code that invites future bugs due to being incomprehensible or fragile. Although this is mostly the result of a lack of forethought and design, I think that in some programming languages it is much easier to write such code than in others.

I think there are three main reasons why a language may be more bad-code-prone than others: lacking namespaces, using dynamic typing and providing functional programming possibilities. Now I am not argueing that it is an inherent feature of such languages that they result in bad code. Rather, I think that these languages provide more, and easier, ways to shoot oneself in the foot. Dynamic typing and functional programming provide easy ways to circumvent the restrictions imposed by a proper design, while the lack of proper namespaces does not allow you to implement a proper design at all.

As a result, I think it requires more skill and thought to write good code in languages like PHP and Javascript then it does in a statically typed, 'restricted', language like Java. However, these are precisely the languages that less skilled programmers use more often, because they seem simpler. Consequently, there is necessarily more bad PHP than there is bad Java.