Testing; the key to heaven or hell?

Hey all,

This isn’t specifically about one programming language, or programming languages in general now that I think about it, it’s about testing your code. I didn’t want to stick this in the lounge because it makes less sense to put it there than it does here IMO .


This past week or so I’ve been thinking [more or less] exclusively about testing, what it means to be able to automatically test code and what the implications of this are.

I have to admit it’s a subject that I’m quite new to, so, if you would forgive me my ignorance here. If you have any comments or questions just shout out; even more so if you think I’m talking crap.


So is automated testing really the answer, and why is testing (in one form or another) being pushed so much lately?

Testing of course has always been emphasized (not just in software development), but this last year or so it seems to have shot up all over the place and blossomed. To be honest I’m not so sure I like the look or smell of these flowers!


The first thing that hits me is the obligatory “COOL”. At first it seems more than useful to have one piece of code test another piece; think about all the time I could save myself; not to mention the happy feeling of closure .

… But, and there always is one. This is only if the code doing the test can be guaranteed correct, which it can’t. Now I can almost see people writing tests for tests if we continue down this road .


I can’t help thinking that maybe I just don’t get it and I’m missing out. There’s a lot to consider, it’s a large subject and some of it is scary and or contradictory.

Can you imagine a world where code that is otherwise correct gets flagged as having errors because of a fault within a test case? The programmer may well be fooled into introducing bugs just to pass!

Even worse, what if some programmer, convinced that the code is right and the test wrong, adjusts some of the test cases?

Well then we’ve elevated a possibly simple problem to an uber bug – a bug that you’ll almost certainly never find in a large system because everything is telling you that the problem is elsewhere. After all, we’ve proven this code works fine haven’t we.

This sounds almost perverse don’t you think.

Whether or not this actually happens I can’t say, but I suspect that it has and does quite often. Given enough time I think we’ll start to hear about it but I wouldn’t hold your breath for now.


Interestingly many of the problems with common practices pointed out by advocates of test-driven development, are shared by tests.


It’s been said by many great programmers that comments are largely a bad idea for a few different reasons; here are two of the usual reasons that I agree with .

Comments duplicate knowledge that should already be present in the programs source.

Comments are destined to become outdated, and so cause more harm than good in the long run.


Well both of these reasons can certainly be applied to testing, maybe even more so.

For any test to be truly worth it’s weight in gold*, the level of detail has to be far, far, far greater than with comments, and this is detailed knowledge which must already exist in working code.

Given that the second also seems to be true. If good tests need to go into such detail, then updating them accurately is a major job.

If simple comments become outdated so easily then this implies that tests will certainly become stale at some point. That is of course if people didn’t insist on using these tests to drive development.

This brings me to my next point .


We as humans (and presumably everyone reading this is) can only think about things in finite detail. It’s easy to see how and why being able to hack on some code and have the computer tell you when you’re done is desirable – this is no substitute for reasoning .

It doesn’t work because these tests can only tell you if you’re right [or wrong] for the cases you give them.

If you want to write code that works you need to be able to reason about it, this is the only way that you can truly say a piece of code is correct; as in mathematics you need to prove it formally.

We have to acknowledge that if tests aren’t detailed enough, or are just plain wrong then the potential for evil is far greater than anything we’ve had previously. If we don’t trust our tests then they’re useless to us, but we can’t trust our code to be correct in general.

How can we then double the code base and reasonably trust the extra code to tell us when we’re wrong?

I don’t think we can but maybe it’s just me,

Thanks for reading.

Mark.


* that is if tests weighed anything .

By testing, I take it you mean unit testing. Obviously, you test your entire program in the end, but the Agile and XP, etc. methodologies brought unit testing to quite a spotlight. Approaching any hyped technology with skepticism is pragmatic. You do well to question it.

Do you remember when XML a few years ago was expected to solve all the computing problems, along with world poverty and the like? Like XML, while unit testing itself is a useful tool, the hype can be misleading.

It's best if you think of your code as black boxes being wired together. Unit tests are used to guarantee these black boxes function as advertised/designed/expected. They say nothing about performance, and they don't have a sanity check on your overall program design and logic. But especially in large projects, it helps to know when parts of your code severely malfunction.

Remember, unit tests are best designed to check black box functionality. That means you should avoid writing ultra specific implementation tests. Limit your code coverage to only what is necessary. That is why you don't need to bear the onus of testing every bit of your code. It is actually detrimental as you note.

You also realise that unit testing depends on the quality of the tests. By limiting yourself to black box functionality, you can avoid tripping over logic errors. And of course, the programmer should be wise enough to do so. As you note, poor unit tests can lead to even worse code.

I'll give you an example. Let's say I write a linked list class in C++. I'll check the overall functionality. Can i add objects, and retrieve them? What about trying to retrieve from an empty list? I don't care about implementation in my unit tests. Sure, I don't test everything. I don't have a perfect guarantee my code works. But that's not what unit testing is for. It's a sanity check that my code behaves to a degree like expected.

I do TDD (Test Driven Development) whenever possible, and the problems you cite are never an issue in practice.

Automated tests do not prove that a program works - that is not what they are for, and proponents of automated testing do not claim that. What they do do is alert you when the program breaks in certain ways. If a test fails then you know that either the code is broken or the test is wrong. You then use the same debugging skills that you would use anyway to figure out which it is, then fix one or the other. As you say, tests are not a substitute for reasoning.

If you write each test BEFORE you write the code it is testing then you get additional benefits on top of knowing when the code doesn't work. When you write a test for a piece of code that does not yet exist, you are focused on several questions that you would not be otherwise:
* what does this piece of code need to do?
* what data does it need to do that?
* What is the result of this code - how will I know when its done it?

These are all DESIGN questions - at this point you are not concerned about the implementation issues at all. TDD is a design methodology as much as a testing one. By writing the tests first you force each piece of code to be well structured so that it can be tested in isolation.

Also with TDD you get IMMEDIATE feedback when your code breaks - seconds or minutes instead of waiting days, weeks or months for someone else to find the bug (perhaps a customer after it has been deployed). The gains from this are immense.

Even if you do TDD you still want to do end-to-end testing since TDD will not catch bugs caused by the interaction between objects. You can do this with another automated tool such as Fit. You also want to have testers doing manual testing, since there are always things that cannot be tested automatically, such as usability.

If you want to improve your code and testing then I recommend reading Test Drive Development By Example by Kent Beck.

Dave

I'm a strong advocate of white box testing. Black box testing suffers from so many faults.

When you're writing lisp particularly, you spend a long time ensuring the code is lofically correct. You're thinking of everything as data, you're logically working with data when you're writing the code. Your testing stage becomes mixed with your coding stage.

Yeah, you gotta do a few testcases to make sure it still works, but a combination of the two is perfect.

Hey Dave,

Firstly, thanks very much for the excellent reply. This was just the kind of thing I was after .


I'm still learning about TDD – it’s in my nature to take a skeptical view to anything that seemingly contradicts good programming practices, even if it does purport to offer a lot.

In some ways it's seems like a good thing, in others it seems very bad. I've been writing simple tests for parts of my programs for a while now; one thing that I’ve noticed is that there is a lot of duplicated code, and no logical way to split it up and reuse it.

This may be a bad idea I don’t know. The way I write all my tests currently is as a logical description of the code being tested. Consequently I do feel that I’m duplicating a lot of information. Here I’m describing the purpose of the code logically; there I’m describing the purpose of the code functionally/declaratively (or however, depending on the language I’m using).


A simple example, I have a higher-order function called make-between? that produces specialized versions of a predicate for testing if a value is within the given boundaries. As input it takes two predicates, which it uses to work this out.



I’m sure you can imagine that when I write out the tests for these functions the tests are nearly identical. I can of course break these up to reduce the replication, but the break is unnatural.

I’ve been play designing a macro for this that will let me to write a base test, and extend or update that with new test cases. It does seem to be more trouble that it should be right now though.

Maybe I’ll change my mind?

I’ll check out the book you pointed to when the workload lets up,

Thanks again, you too LP,

Mark.

Hi guys, interesting thread so far.



I've been using Django (excellent web app framework for python) a lot recently, and one of the things they do with their tests is to use them as a starting point for documentation and examples they can show to users. For example: here's the many-to-many database API.

This is great - it's not repeating yourself so much as killing two birds (documentation/examples and testing) with one stone.

I also think one of the better uses of tests is regression testing of bug fixes. I'm sure we've all worked on/seen apps which have those stupid little bugs that keep popping up and being reintroduced. If there was a reg. test to make sure that bug was not reintroduced, then you can immediately see it and squash it.

--Simon

Here is how I handle that situation - other TDD programmers may do it differently.

If I add a test that is a variation on a previous test, I usually start by cutting and pasting a copy of the entire test, rename it, and change whatever is needed in the body for the new test. Once it passes I then go back and refactor all the common code out into a helper function, so that both tests then call the helper and passing it different test data.

If I then find I am writing lots of tests that all call the same helper function, then I refactor again and put the test data into an array and replace the tests with a single test that loops through the array and calls the helper function with different sets of data. this makes it very easy to add new test cases, but has the disadvantage that if one test case fails the others are not run. It also means that you cannot run individual tests (JUnit and the Python unittest module lets you run a single test by name - I dont know if the scheme unit test framework has that feature). Whether or not this is a problem depends on what you are testing - if the tests are fast enough and you always ensure that all tests pass 100% of the time then it is not a problem.

There is a wealth of resources online about TDD, including articles with example programming sessions. For starters the are a bunch on the ObjectMentor site.

Dave

Just my 2 cents, but i like testing the old fashion way; letting loose a group of fanatical users upon my code like a bunch of parahna.

As much as this is frowned on now (to costly to have experienced users wasting thier time testing), knowledgable users will know how to test an app in its entirety.

One of the worst things to run up against is to find out all your changes work (when tested individually be someone newbie) but when brought in into the scope of the tool and the users processes, that they become worthless.

Knowing what to test is an art that needs thought and it's very easy to write a useless test, to miss out on testing something or to test with the wrong data. QuickCheck is an interesting haskell testing library that removes the programmer to a level above - you provide a specification for a function, not the tests. The tests are generated and run for you with a range of random data including corner cases, fenceposts etc (the data can be controlled). Because haskell is a purely functional language with no side effects (apart from impure code) this effectively verifies the function - if your code does include side effects you're back to painstakingly crafting your own unit tests, and it's at that point that you realise how difficult it is to write a test which is really effective under all conditions unless you avoid side effects.

Nice posts guys,

Jamie: I think this will be one area where functional programming languages just come out on top in the long run. If there is one thing that could push them into mainstream it's probably this.

It's a bit of a shame that functional programming isn't very popular since functional languages tend to be far more beautiful than there imperative oppressors, in my honest opinion.

That said functional programming is becoming more and more common. That's always a plus .

Mark.