Subversion’s “read length line” error

When I did something stupid as a kid my parents would occasionally say words to the effect of “We’re not angry with you, just disappointed”. That’s how I feel right now with Subversion after finding out about this repository corruption issue which seems to have been lurking around for a while.

Finding such a bug in any system you rely on as heavily as a source control system is always going to hurt, and more so if you find about about it first hand. I’ve been unable to repair the repository in question, and although it’s not much of an issue as it was a recently created code branch so we can quite happily drop the history and start again – the bigger issue is doubting your source control system. It’s not a nice feeling.

If you’re still reading at this point, then you might like to ask…

What is the “read length line” error?

This particular error manifests itself because of a particular kind of corruption that has occurred in the repository. Malcolm Rowe provided a great analysis and description of the problem here: http://svn.haxx.se/dev/archive-2006-02/0473.shtml. To put it in a nutshell, the crux of the issue is that two handles are eventually opened to the transaction by mod_dav_svn, and one of them is freed later which ends up writing the remaining portion of it’s cache. The end result is that the representation is now malformed, and this extra data has invalidated a number of the offset specified later in the file. The “read length line” error is a result of the the backend trying to read from the wrong offset.

And then, your next questions might be…

What can you do about it?

At the moment, not much. The root cause of the problem is very elusive, making it extremely difficult to track, and more difficult to produce. If for some reason, you’ve come up with a reproduction recipe though, please emaildev@subversion.tigris.org, as we’d very interested in having such a recipe. However, I feel it’s mostly an issue with configuration, load, and some decision process that Apache2 makes when deciding to fork a new process to handle a request, coupled with some other error, making this a very difficult bug to reproduce.

 

What is the Subversion team doing about it?

The team is aware the problem exists, and is eager to get it resolved. There are on-going discussions of how to fix this issue, but without knowing where the problem actually resides it’s difficult to fix this issue properly. We may just end up masking the problem if we aren’t careful. That said, there are discussions right now about limiting the write handle to a transaction to just one process. That comes with some tradeoffs, mainly in terms of commit speed, as we may not have the option to pipeline the commit operation.

So it’s a nasty issue, and if you got here by googling because you’ve got a corrupt repository then you’ll probably want to try these links and hope they see you right:

Link: fsfsverify - jszakmeister’s techblog
Link: Repairing A Corrupt FSFS Repository

I’d really love to see this one fixed. For now I’m going to be splitting a few groups of code into their own repositories in order to minimise the loss if this ever happens again, as well as making a few minor revisions to the repository dumping/exporting/backup nightly processes

Oh, and crossing my fingers.

Tags: SubVersion

 Print | Posted on Thursday, August 05, 2010 11:39 PM |