Bryan Says...

Rants, opinions, and other stuff I find interesting.

Just enough testing

It’s happening again: An extremely successful software developer with a very large following lists his rules for success, and people jump for joy at the occasion, and use that list to justify their practices. That sounds great, doesn’t it?

While reading, Testing like the TSA, I found myself nodding in agreement with what most David was suggesting. I didn’t agree with all of it, but that’s fine; there is definitely room for positive discussion and disagreement when it comes to solving problems with code. The best part about the blog post wasn’t the post itself. The comments and other discussion on sites like HackerNews are the real jewels.

I’d now like to think out loud about testing. We’ve come so far in the Ruby community, but in reality, we still have a long way to go.

Just Enough Testing

A question that I hear constantly is, “How much tests should we have?” I always reply with, “Just Enough.” What does that mean exactly? It boils down to your application or library should have just enough tests that you are confident that it works as it should. Do I think your code base has enough tests? I don’t know. You should know. There isn’t a magic number. You’ll know when you’ve found it. Velocities will increase and regressions will decrease.

There’s a difference between spiking (or prototyping) and writing code that is destined for production.

Too many times, we see a blog post or a book that echoes our thoughts, and we grasp on to it as if were a gospel. When I see a comment that basically says, “… and that’s why I don’t test”, I think we should mark that as a failure. There are plenty of times when all of us don’t test. Last year, while I was learning Clojure, I didn’t write one test (and I still don’t). When I was learning how to write Android apps, I didn’t write tests either. The reason for this is because I wasn’t sure what I was writing. It’s pretty hard to describe the behavior of something with tests. It is even harder when you don’t know the language of description.

This can also be applied to web developers who develop web apps constantly. Sometimes we don’t know what we are trying to build. A little exploratory code helps us understand the problem. What we are looking for is the correct question and you might have to write a little code before understanding that. I suggest that after you learn the right question, you throw the prototype or spike away, and start again using TDD. That might not always be possible due to time or budgetary constraints, so you might have to retrofit it with tests later. This isn’t optimal, but it’s a part of being a professional developer. You can’t learn this by reading blog posts or books.

As an aside, I almost feel confident when test driving Android apps. It isn’t perfect, but I feel like this is the beginning of a conversation rather than any type of a complete solution.

Testing for testing’s sake is a waste of time

You may remember back a few years ago when I spoke about TATFT. You might not have known it, but this was a short talk that was more parody than anything else. Many of you took this at face value. Doing that put you in the same place as taking the “Testing like the TSA” post at face value. You shouldn’t do that. The real value is understanding why we say what we say. I don’t write tests all the time. I do think constantly if I’m writing in a way to make things easily tested. I do write tests first most of the time. Keep in mind, I certainly don’t advocate that I or anyone else should write tests all the f#$@ time. That’s silly.

I rarely look at tools like code coverage or test to code ratios. That isn’t important to me. What I care about is is that the tests that I’ve written properly describe the behavior of the project. The times where code coverage and test to code ratios are important are as a metric over time. If these numbers are going down, it is a sign that code quality is decreasing. David pointed out in his post, “1:2 is a smell, above 1:3 is a stink.” You are going to have learn what’s good and bad on your own. Projects with different developers, with different problem domains will have different tolerances for smells.

TDD is hard

TDD is hard. Think about the first time you sat down to write tests first. What was the first test you wrote? Does that test still exist? I sure hope not. Red, Green, Refactor is a catchy slogan. It seems easy to write a failing test, write some code to make it pass, than refactor. It isn’t, unfortunately. There are some good books on the subject. Test Driven Development by Example (the Kent Beck book) is one people always talk about. Growing Object-Oriented Software, Guided by Tests is much better, and definitely belongs on everyone’s bookshelf.

You also have to remember that TDD is just a suggestion. Take what works for you and form your own opinions. I personally believe in Write a Test, Make it Pass, or Change the Message. This allows me an easier logical transition between Red and Green.

The one thing I can promise you is that with practice and anger, you’ll learn what does and does not work. The goal here is get working software that you can trust as fast as you can.

Don’t use tool X

I feel like this is always bad advice. Some people like RSpec and some don’t. Some people have had great results with Cucumber, while some flail with it. At one time, all of us knew nothing about code. We found a language we liked, and learned how to be productive with it. Sometimes tools aren’t productive because you aren’t using them right. Sometimes tools are just bad. I appreciate that we have a myriad of tools for our arsenals.

Don’t test your web framework

I don’t have much to say about this. Don’t test it until you absolutely have to. Think twice before writing tests for that validation. In our Rails projects, we should be thinking about behaviors of objects and interactions between objects. Your unit tests should reflect that. Your acceptance tests verify the orchestration of those units. Simple, right?

 

  • http://ngauthier.com Nick Gauthier

    Whenever you write an assertion, you are saying “this piece of code must do exactly this”. Make sure that’s what you really mean. Sometimes a test is really saying “I hope this is what it’s doing” or “Let me just check to make sure it’s …”. Those tests are generally less productive.

    I really like the idea of tests as a spec for the project. If a test must be changed, it’s because the spec of the project has changed. If you have to change a test without changing the “spec”, then the test was not appropriate, it was too specific.

    If you can change the “spec” (or behavior) of the code without having to change a test, then you were undertesting.

    Make sure your assertions are assertions.

    Thanks Bryan.

  • http://melriffe.info Mel Riffe

    First, thanks for the blog post. I don’t read a lot of blog posts these days but I have to agree with Brian Hogan: this is very well written.

    Second, I agree with a lot of your sentiments and I’d like to add: be pragmatic. Very nearly every Software Development-related question can be answered with, ‘It Depends.’ I don’t understand why folks don’t take the time to figure what works for them and their projects. I tried Cucumber. And while it’s a fine tool, for the project I was on it didn’t add value so I kicked it to the curb; I’m now using RSpec on that project. But I also have other projects where Test::Unit is the testing tool.

    I would also add you have to think of your project’s culture before introducing a new toolset. Again: Folks, take the time to figure out what works and doesn’t work.

    Third, I find conversations that state absolutes quickly degrade into noise. People should understand those posts are written in a context not their own. Now, if the post is meant to start a conversation, so be it. But make the intention clearly and early.

    I’m rambling now so I’m going to end my comment.

    Thanks Again!

  • http://melriffe.info Mel Riffe

    First, thanks for the blog post. I don’t read a lot of blog posts these days but I have to agree with Brian Hogan: this is very well written.

    Second, I agree with a lot of your sentiments and I’d like to add: be pragmatic. Very nearly every Software Development-related question can be answered with, ‘It Depends.’ I don’t understand why folks don’t take the time to figure what works for them and their projects. I tried Cucumber. And while it’s a fine tool, for the project I was on it didn’t add value so I kicked it to the curb; I’m now using RSpec on that project. But I also have other projects where Test::Unit is the testing tool.

    I would also add you have to think of your project’s culture before introducing a new toolset. Again: Folks, take the time to figure out what works and doesn’t work.

    Third, I find conversations that state absolutes quickly degrade into noise. People should understand those posts are written in a context not their own. Now, if the post is meant to start a conversation, so be it. But make the intention clearly and early.

    I’m rambling now so I’m going to end my comment.

    Thanks Again!

  • Anonymous

    Agree – I’m working on a codebase that is 75% tests of the “Let me just check to make sure…” nature. They say nothing about intentions, and generally make it harder to refactor rather than easier. There are also a slew of “specs” designed to prevent adding functionality even if it doesn’t affect existing behavior. These are supposed to be reminders for things like “remember to add a method of type X when you add a column of type Y,” except that its not always a hard and fast rule – instead it’s a shortcut to writing the appropriate specs before adding column Y. 
    The result is you never know if a spec is broken because the behavior changed, or because it was just brittle. This means every change to the code involves looking at a bunch of broken specs and figuring out why they broke. (And don’t even get me started on the fact that I have to wait 20 minutes for the suite to run…) Ugh. 

  • http://twitter.com/rubyhead Joon You

    The only thing I’ll say here is that although I agree that we shouldn’t say “don’t use x”, don’t use Cucumber.