Forums : Technical Issue Help

Ever since we migrated to the new Sourceforge platform back in March, our code updates to BRL-CAD (http://ohloh.net/p/brlcad) have been stuck. Please help get us unstuck.

We updated our enlistments, but no updates have been seen. An e-mail inquiry indicated that there is some timeout occurring. We have a repository with a LOT of complexity and history. In fact, ours is the world's oldest repo ever that is still active according to Robin Luckey.

Right now, we're over 57000 commits so Ohloh's view has fallen behind by more than 5000 commits since March. Is there anything we can do on our end?

sean over 10 years ago

Some background info, right now the enlistment is an svn:// repository download. We assume that this should be the fastest and most efficient method, but is it worth trying an http:// method?

If the download timeout is protocol-agnostic, it'll obviously only make things worse. If they're handled differently, I could see it making a difference.

sean over 10 years ago

Sean,

Let me take a look at the history and I'll get back to you this morning.

Thanks!

ssnow-blackduck over 10 years ago

Sean,

At the time (May) this was my assessment:

There are 55,207 revisions in this subversion. 25,000 revisions takes 2:40. 45,000 revisions takes 4:30. 50,000 revisions takes between 4:25 and 5:00. 51,000 revisions takes 5:24 or 4:46 (near-impossible). I suspect that loads on the subversion and the internet will make the numbers a little bit rubbery. (The total commits as of today is 57,018...)

Today, 51,000 commits took 4:35.

SourceForge imposes a timeout (unknown length but presumably five minutes plus?) for any request like this. You can see what we're up against. This is only to get a top-to-bottom log of the repository. This is probably the most demanding single step. The actual fetches are done commit-by-commit and probably will work OK.

Apart from moving to git, I'm at a loss what to suggest. We are up against a fundamental limit that we don't control. I'd be glad to discuss this with SF admins if you can get some interest.

Thanks!

ssnow-blackduck over 10 years ago

I can certainly raise this issue with the Sourceforge folks, but can you share what the actual commands being issued are? Is this literally an svn log URL or is it a --raw format lot or something else altogether?

If we gave you the log, could you kick-start the process on your end? I'm sure there are no restrictions if I extracted the log directly on Sourceforge.

I'm sure you've also considered a third option, having Ohloh's backend only retrieve N log options at a time, say 1000 at a time until there aren't 1000 more to retrieve. If you pointed me in the right direction, I'd be glad to take a look at the code. Wouldn't be more than a few lines of code to patch in a loop.

sean over 10 years ago

Sean,

I don't have any control of the goings-on behind the curtain. You could probably get some insight from our open-source SCM library at https://github.com/blackducksw/ohloh_scm

I'll e-mail you the exact command which is failing. I suspect there is some manipulation going on to get the entire log into a cache so that subsequent requests run almost instantly. Or, perhaps I am off-base? Either way, the command issued fails due to timeout with a not-so-helpful error message (subversion is full of those...)

Standby and thanks!

ssnow-blackduck over 10 years ago

To anyone following this discussion, the issue turned out to be Ohloh's infrastructure attempting to determine where an enlistment begins by looking through the commit log and ultimately failing.

In our case, what failed was an svn log call that was taking so long on the server side that the eventually the network connection would time out and fail. Either the server (Sourceforge hardware) was too slow, our history was too big (50k+ revisions), SVN is poorly optimized for reverse logs, or some combination thereof. The request was taking more than 5 minutes, so the network socket would shut down.

Ohloh's request asks svn to extract all commits in reverse order (-rHEAD:1) so that the backend server can just send the oldest (--limit 1). This is commonly documented as the way to obtain the first revision for a branch. However, sorting all commits in reverse order ends up taking too long. I created a pull request on Github that should fix this issue that will hopefully help some other big projects as well that might be running into this issue. It's not fast, but it should succeed by requesting the log and having the receiver identify the last entry.

Cheers!

sean over 10 years ago