Forums : Technical Issue Help

gentoo cvs enlistment failure

Main cvs repository for Gentoo is experiencing problems with update. Would you mind looking into it?

cvs url: :pserver:anonymous:@anoncvs.gentoo.org:/var/cvsroot

Also...sorry, but I was cleaning up some moved repositories and I accidentally removed/readded this one as well. Hope the debugging won't be too difficult

Af0dcc73c5c00a6b9139adb21c5b45a9?&s=32&rating=pg&d=https%3a%2f%2fwww

Stanislav Ochot...

over 4 years ago
 

Stanislav,

Am working on this issue now.

Found two enlistment errors. First one (bazaar) re-fetched the repo with no major problem. Looks like the style of repository was changed.

Second one is looking to be a major undertaking. Had to re-fetch this repo due to a local error here at ohloh and is running along slowly. Currently at 421/640851 commits read. Will probably take several days or longer to finish.

Thanks in advance for your patience.

Will report back here on the progress.

C06ad304d5a864f33eed63121a0407be?&s=32&rating=pg&d=https%3a%2f%2fwww

ssnow-blackduck

over 4 years ago
 

Not a problem. I know the cvs repo is huge (it's quite old so that's to be expected).

Thank you for working on this.

Af0dcc73c5c00a6b9139adb21c5b45a9?&s=32&rating=pg&d=https%3a%2f%2fwww

Stanislav Ochot...

over 4 years ago
 

It seems that the cvs import is again "in queue". It used to display progress yesterday. Maybe it's hitting some hard quota on time spent with checkout? Or just the web is not displaying progress anymore?

Af0dcc73c5c00a6b9139adb21c5b45a9?&s=32&rating=pg&d=https%3a%2f%2fwww

Stanislav Ochot...

over 4 years ago
 

Stanislav,

Will need to refer this problem "upstairs" to folks with more hands-on access. The processing did stop with a message "Runtime limit exceeded." The process waits about 10 hours for the next attempt but I'm not sure if it resumes where it left off. Will need to watch and wait.

Thanks again for your patience.

C06ad304d5a864f33eed63121a0407be?&s=32&rating=pg&d=https%3a%2f%2fwww

ssnow-blackduck

over 4 years ago
 

Stanislav:

Hello

So the CVS repository is going really, really, really slowly. I don't know why either, because if I checkout the CVS repo locally, the cvs checkout command only took 18 minutes.

We need to dig deeper into this to figure out what it's going so slow on our servers.

It may be a few days because we're right in the middle of some server architecture changes and some other high priority fixes. I'll reply here once I know more about the problem.

By the way, I've used Gentoo for a large number of years now and I use it as my primary OS both at home and at work. Gentoo rocks :)

23d04cec636a28b2a08bb925d6b84c6d?&s=32&rating=pg&d=https%3a%2f%2fwww

RobertSchultz

over 4 years ago
 

Hi Robert,

thank you for "digging deeper" :-)

As for the time...Don't worry about it. I just noticed the stats haven't been updating so I created this request. I think we'll be able to wait a few days/weeks for updates to come through.

Gentoo rocks :)

Great! I hope you'll keep using it for the time to come :-)

Af0dcc73c5c00a6b9139adb21c5b45a9?&s=32&rating=pg&d=https%3a%2f%2fwww

Stanislav Ochot...

over 4 years ago
 

I thought to ask again since it's been some time. Have you tried again? I know our CVS is big & old so it's not trivial, but it would be appreciated.

FYI: It's still not critical, I just don't want this to be completely forgotten :-)

Af0dcc73c5c00a6b9139adb21c5b45a9?&s=32&rating=pg&d=https%3a%2f%2fwww

Stanislav Ochot...

over 4 years ago
 

Hi Stanislav,

This CVS repository is far too large for Ohloh to process.

Our CVS importer is brute-force: we start from the oldest commit, and check out each revision in sequence. As Rob said, for this repository, this takes 15-20 minutes for each revision.

Some simple math shows that with 600,000 revisions in the repository, it will take several years for Ohloh to complete the import.

Your best bet for getting this onto Ohloh is to convert the repository to Git. Bzr and Hg will probably not work; they are an order of magnitude slower than Git when it comes the operations Ohloh exercises (mainly cat-file).

Robin

063ffe4b4061aa874f09216cfd60d8f2?&s=32&rating=pg&d=https%3a%2f%2fwww

Robin Luckey

over 4 years ago
 

We've identified code in the back end that was slowing all our job processes way way way down. We're easily running 10x as many jobs as we were before.

I've re-scheduled the Gentoo CVS job and hope that this time it runs faster thanks to our back end improvements. I'll keep an eye on it over the next day or two and let you know how it goes :)

Also, it looks like the bazaar repository is broken due to a transient Ohloh bug. I'm going to work on fixing that as well.

23d04cec636a28b2a08bb925d6b84c6d?&s=32&rating=pg&d=https%3a%2f%2fwww

RobertSchultz

over 4 years ago
 

Robin: There is a project within Gentoo to convert our infrastructure to git, but it's far from finished so CVS is here to stay for now.

Robert: Thanks! Glad to hear you managed to find the bugger that was slowing things down. Always nice to hear about performance optimisations :-D

Af0dcc73c5c00a6b9139adb21c5b45a9?&s=32&rating=pg&d=https%3a%2f%2fwww

Stanislav Ochot...

over 4 years ago
 

So, Robin knows a lot more about the Ohloh back end than I do. Also, I wasn't aware of how many revisions are involved. I also wasn't aware of the length of time it took per revision. So Robin's conclusion stands, it won't be possible to import this into Ohloh at the current time.

23d04cec636a28b2a08bb925d6b84c6d?&s=32&rating=pg&d=https%3a%2f%2fwww

RobertSchultz

over 4 years ago
 

OK, not good, but at least we know where we stand. Thanks for trying!

I guess we'll have to try harder to convert to git ASAP...

Af0dcc73c5c00a6b9139adb21c5b45a9?&s=32&rating=pg&d=https%3a%2f%2fwww

Stanislav Ochot...

over 4 years ago
 

OK. So we converted our CVS repo to git (read only, just so we can get the stats for now). But it failed again. It failed during download, which I assume is caused by the size: ~1.8G packed. Anything more we can do on our end?

Af0dcc73c5c00a6b9139adb21c5b45a9?&s=32&rating=pg&d=https%3a%2f%2fwww

Stanislav Ochot...

over 4 years ago
 

Oh, I have forgotten...the repo is: git://binaryparadise.com/gentoo-x86.git/

I guess that would have been easy to figure out, but better safe than sorry :-)

Af0dcc73c5c00a6b9139adb21c5b45a9?&s=32&rating=pg&d=https%3a%2f%2fwww

Stanislav Ochot...

over 4 years ago
 

Stanislav,

I have rescheduled the job for the new repo and will monitor the project. Have also found another failed job and will investigate.

Thanks!

C06ad304d5a864f33eed63121a0407be?&s=32&rating=pg&d=https%3a%2f%2fwww

ssnow-blackduck

over 4 years ago
 

Note that one HDD on machine where the repo is failed today (nice timing eh?). I'll monitor on my end as well :-)

Af0dcc73c5c00a6b9139adb21c5b45a9?&s=32&rating=pg&d=https%3a%2f%2fwww

Stanislav Ochot...

over 4 years ago
 

Stanislav,

I am unable to retrieve the repo at this time; seems the server is off-line.

Let me know when it returns... I'll try occasionally when I can.

Thanks!

C06ad304d5a864f33eed63121a0407be?&s=32&rating=pg&d=https%3a%2f%2fwww

ssnow-blackduck

over 4 years ago
 

OK, so we moved the converted git repo to our infrastructure server. Currently waiting in queue...so let's see what will happen.

It's probably among the biggest git repos you'll see (~8GB), so I wouldn't be surprised if it failed.

Af0dcc73c5c00a6b9139adb21c5b45a9?&s=32&rating=pg&d=https%3a%2f%2fwww

Stanislav Ochot...

about 4 years ago
 

not to scare you...1.8G :-)

Af0dcc73c5c00a6b9139adb21c5b45a9?&s=32&rating=pg&d=https%3a%2f%2fwww

Stanislav Ochot...

about 4 years ago
 

Stanislav,

I notice that one enlistment from the CVS server remains. It has failed due to a "No space left on device" error. I have hesitated to reschedule or re-fetch (oh-no!) the enlistment if it should have been deleted. It will prevent the full update from finishing. If, on the other hand, it is a useful enlistment, please let me know and I'll keep pushing on it.

Thanks!

C06ad304d5a864f33eed63121a0407be?&s=32&rating=pg&d=https%3a%2f%2fwww

ssnow-blackduck

about 4 years ago
 

Hmm, are talking about ":pserver:anonymous:@anoncvs.gentoo.org:/var/cvsroot gentoo" enlistment? Apparently there were some optimisations for speed that could cause this. I've removed the enlistment for now. Maybe we'll try to move it to git as well for stats.

Af0dcc73c5c00a6b9139adb21c5b45a9?&s=32&rating=pg&d=https%3a%2f%2fwww

Stanislav Ochot...

about 4 years ago
 

Stanislav,

Just so! Not exactly sad to see it go. The other three pending enlistments have started up and are processing now. I'll keep you informed here about their status for better or for worse. I do know that one of these enlistments is subject to a bug which may need me to be vigilant but will press on with all three.

Thanks!

C06ad304d5a864f33eed63121a0407be?&s=32&rating=pg&d=https%3a%2f%2fwww

ssnow-blackduck

about 4 years ago
 

Hmm, the "git://git-exp.overlays.gentoo.org/exp/gentoo-x86.git master" failed this time. I've done a full clone from this repo yesterday so I guess it's in workable state and configuration is able to cope with it. Perhaps logs on your end will tell us more

Af0dcc73c5c00a6b9139adb21c5b45a9?&s=32&rating=pg&d=https%3a%2f%2fwww

Stanislav Ochot...

about 4 years ago
 

Stanislav,

Was just there and jump-started the lazy jobs. The Bazaar enlistment is the one with the (our end) bug attached and I'll need to watch that one more closely. It was quite easy to revive and quick too (698 commits). It's performing line counts now and with fingers crossed I hope that it will be finished shortly. (It has...)

The other one that failed was indeed the one you cited and I finally decided it needed to be re-fetched. Nothing to do for it and now we wait for 2.3 Gb compressed to come down.

Thanks!

C06ad304d5a864f33eed63121a0407be?&s=32&rating=pg&d=https%3a%2f%2fwww

ssnow-blackduck

about 4 years ago
 

Post a Response