||[Apr. 22nd, 2010|07:41 pm]
I'm going to nerd out for a wee while. Also, Open Source folks may want to cover their ears for this, as a rant ensues. (That said, I'm trying to keep it as high-level and readable for all as possible)
I have a problem with how the Open Source Community handles bugs.
Now, it should be said that I'm coming from a particular point of view. If you were to ask my company's HR dept, I am apparently a "professional" (although, purely in the sense that it's my job) bug-logger.
Due to various attempts to get a Linux distro working smoothly on my shiny-new home pc, I've been chasing down a performance regression that affects my machine. You can actually follow the conversation surrounding the bug here. I say "conversation" and a "conversation" is what a bug report should be; it's a communication tool. Obviously it shouldn't have anything flippant like "How's your day been?" or anything like that, but it is a space for various parties to put as much information relevant to the bug as possible.
It's my job to make clear, concise and coherent reports of things that break in the products I'm testing. I mostly get it right. This is the best (albeit long-winded) public example of a bug I wrote; taking the techy details away, it basically says, "This is what's wrong and this is how it's supposed to behave, and here's how you can reproduce that problem". It's enough information for someone to start drilling down into that bug and figuring out what the problem is for themselves. There are other reports that I've logged, but I can't show off the really interesting ones.
This is my job, by the way, in case anyone's ever wondered.
Of course, not all users will be as thorough when they're reporting bugs - mostly because they don't know what the developers want to see; that's forgivable. I honestly don't have a problem with that because it's open-source; anyone can pick up an open-source product and use it, and anyone can report a problem, even if they don't know how to do either.
What does irk me is the other side of the conversation; developers should give you a good example of what's going on. This is a decent example (I didn't have anything to do with it, I just did a quick search on the opensolaris bug search).
To me, there is a covenant. You give the developer a good description of what's going wrong, and the developer gives a good description of how it's going wrong, and how they are fixing it. However, that launchpad bug report, and in the underlying bug report, you see "Fixed it. I've put it into blah." - it tells us nothing. The worst thing is that it seems to be standard operating procedure.
You may be thinking, "So? They fixed it. What's the big deal if they don't show how they did it?". Yeah, they fixed it, but there is no dissemination of knowledge. Some of my best tricks for giving more useful and relevant information in my reports came from previous reports where developers dissected the information you gave them. If another developer working on a similar problem comes along, there should be at least some information there for them to get their bearings. But in the open source world, you don't seem to find it. What's the problem? I mean, it's not like there's confidential information that they can't discuss publicly or anything, IT'S OPEN SOURCE!
Conversations in reports are the long-term memory of the product development team. Members of that team will rotate, so you need a way to learn about how they fixed a problem, and NONE of it is there.
I can honestly say that reports of that quality make me cringe and die a little inside. I am sure that this was a subtle and complex issue that they fixed under the hood, and if another developer goes and plays with that, they could easily undo that good work if they don't know what's going on, or why something was done in a particular way. Basically, it's making more work for future developers if they break something.
You see, I can totally understand the reasons for the common complaint about there not being enough documentation for open source projects - it's often exactly the sort of thing you don't want developers writing (you want documentation to be written for a human) - but not even notes for themselves while they fix a documented problem?!?! I think it's actually a very dangerous thing.
Let's put it this way, I don't know if others are like me, but I'm reluctant to upgrade a program if I know that it works. The reason is, I've seen far too many regressions (the technical term for when a bug fix is undone) in software due to upgrades. This behaviour of not recording any lessons you learn, or how you learned them is ASKING for people to make the same mistake again.
It seems that some people think that a bug report should just have, "This is a problem", from the submitter and for the developer's reply to be, "It's fixed". I think those people are idiots. However, it seems that some developers are working to appease those people; maybe they think that it's good PR to just say that they magically fixed a problem without going into the details. For me, I find it scary. There's a time and a place to be high-level and non-technical, and a bug report isn't it.
The bottom line is, by the time a bug report is closed it should clearly document What happened, Under What Circumstances, optionally What Should Happen, How/Why it happened, What the potential solutions are and How it was fixed. If it doesn't have those, then it's not a good bug report, and the developers shouldn't be there because they're only making problems for the next person to work on it.