Subversion’s “read length line” error

When I did something stupid as a kid my parents would occasionally say words to the effect of “We’re not angry with you, just disappointed”. That’s how I feel right now with Subversion after finding out about this repository corruption issue which seems to have been lurking around for a while.

Finding such a bug in any system you rely on as heavily as a source control system is always going to hurt, and more so if you find about about it first hand. I’ve been unable to repair the repository in question, and although it’s not much of an issue as it was a recently created code branch so we can quite happily drop the history and start again – the bigger issue is doubting your source control system. It’s not a nice feeling.

If you’re still reading at this point, then you might like to ask…

What is the “read length line” error?

This particular error manifests itself because of a particular kind of corruption that has occurred in the repository. Malcolm Rowe provided a great analysis and description of the problem here: http://svn.haxx.se/dev/archive-2006-02/0473.shtml. To put it in a nutshell, the crux of the issue is that two handles are eventually opened to the transaction by mod_dav_svn, and one of them is freed later which ends up writing the remaining portion of it’s cache. The end result is that the representation is now malformed, and this extra data has invalidated a number of the offset specified later in the file. The “read length line” error is a result of the the backend trying to read from the wrong offset.

 

And then, your next questions might be…

What can you do about it?

At the moment, not much. The root cause of the problem is very elusive, making it extremely difficult to track, and more difficult to produce. If for some reason, you’ve come up with a reproduction recipe though, please emaildev@subversion.tigris.org, as we’d very interested in having such a recipe. However, I feel it’s mostly an issue with configuration, load, and some decision process that Apache2 makes when deciding to fork a new process to handle a request, coupled with some other error, making this a very difficult bug to reproduce.

 

What is the Subversion team doing about it?

The team is aware the problem exists, and is eager to get it resolved. There are on-going discussions of how to fix this issue, but without knowing where the problem actually resides it’s difficult to fix this issue properly. We may just end up masking the problem if we aren’t careful. That said, there are discussions right now about limiting the write handle to a transaction to just one process. That comes with some tradeoffs, mainly in terms of commit speed, as we may not have the option to pipeline the commit operation.

So it’s a nasty issue, and if you got here by googling because you’ve got a corrupt repository then you’ll probably want to try these links and hope they see you right:

Link: fsfsverify - jszakmeister’s techblog
Link: Repairing A Corrupt FSFS Repository

I’d really love to see this one fixed. For now I’m going to be splitting a few groups of code into their own repositories in order to minimise the loss if this ever happens again, as well as making a few minor revisions to the repository dumping/exporting/backup nightly processes

Oh, and crossing my fingers.

Tags:

Posted on Thursday, August 5, 2010 11:39 PM |

Like this? Share it!

  • # re: Subversion’s “read length line” error
    Gravatar
    Commented on 8/19/2010 10:24 PM

    If you wouldn't mind providing some information about the the actual error you received, and the version of SVN you were running, I'd appreciate it. I try to keep a sense of whom this is affecting, and to do date, most everyone who has had a problem has been because the revision was created with an older version of SVN. If you're seeing the issue with a newer version of SVN, I'd really like to know that.

    Also, it sounds like you've started from scratch again, but I've had a great deal of success in repairing broken revisions. If you run across the problem again, shoot me an email. I'll be happy to take a look at the issue (within reason, I do have a day job).

    Thanks!

  • # re: Subversion’s “read length line” error
    Gravatar
    Commented on 8/19/2010 10:41 PM

    Hi John,

    Thanks for your comment - I'll flick you an email and give you a bit more detail.

    I did pick up from your site that you've had some good luck helping people out in your own time. As you say, I've started from scratch now but I'm still keen to submit whatever info I can if it helps pinpoint the bug (and having the repository repaired for historical purposes would still be very handy indeed).


    Cheers,

    Ross

Post a comment
Please add 7 and 8 and type the answer here:
Remember me?
Ensure the word in this box says 'orange':