Main cvs repository for Gentoo is experiencing problems with update. Would you mind looking into it?
cvs url: :pserver:anonymous:@anoncvs.gentoo.org:/var/cvsroot
Also...sorry, but I was cleaning up some moved repositories and I accidentally removed/readded this one as well. Hope the debugging won't be too difficult
Am working on this issue now.
Found two enlistment errors. First one (bazaar) re-fetched the repo with no major problem. Looks like the style of repository was changed.
Second one is looking to be a major undertaking. Had to re-fetch this repo due to a local error here at ohloh and is running along slowly. Currently at 421/640851 commits read. Will probably take several days or longer to finish.
Thanks in advance for your patience.
Will report back here on the progress.
Not a problem. I know the cvs repo is huge (it's quite old so that's to be expected).
Thank you for working on this.
It seems that the cvs import is again "in queue". It used to display progress yesterday. Maybe it's hitting some hard quota on time spent with checkout? Or just the web is not displaying progress anymore?
Will need to refer this problem "upstairs" to folks with more hands-on access. The processing did stop with a message "Runtime limit exceeded." The process waits about 10 hours for the next attempt but I'm not sure if it resumes where it left off. Will need to watch and wait.
Thanks again for your patience.
So the CVS repository is going really, really, really slowly. I don't know why either, because if I checkout the CVS repo locally, the cvs checkout command only took 18 minutes.
We need to dig deeper into this to figure out what it's going so slow on our servers.
It may be a few days because we're right in the middle of some server architecture changes and some other high priority fixes. I'll reply here once I know more about the problem.
By the way, I've used Gentoo for a large number of years now and I use it as my primary OS both at home and at work. Gentoo rocks :)
thank you for "digging deeper" :-)
As for the time...Don't worry about it. I just noticed the stats haven't been updating so I created this request. I think we'll be able to wait a few days/weeks for updates to come through.
Gentoo rocks :)
Great! I hope you'll keep using it for the time to come :-)
I thought to ask again since it's been some time. Have you tried again? I know our CVS is big & old so it's not trivial, but it would be appreciated.
FYI: It's still not critical, I just don't want this to be completely forgotten :-)
This CVS repository is far too large for Ohloh to process.
Our CVS importer is brute-force: we start from the oldest commit, and check out each revision in sequence. As Rob said, for this repository, this takes 15-20 minutes for each revision.
Some simple math shows that with 600,000 revisions in the repository, it will take several years for Ohloh to complete the import.
Your best bet for getting this onto Ohloh is to convert the repository to Git. Bzr and Hg will probably not work; they are an order of magnitude slower than Git when it comes the operations Ohloh exercises (mainly cat-file).
We've identified code in the back end that was slowing all our job processes way way way down. We're easily running 10x as many jobs as we were before.
I've re-scheduled the Gentoo CVS job and hope that this time it runs faster thanks to our back end improvements. I'll keep an eye on it over the next day or two and let you know how it goes :)
Also, it looks like the bazaar repository is broken due to a transient Ohloh bug. I'm going to work on fixing that as well.
Robin: There is a project within Gentoo to convert our infrastructure to git, but it's far from finished so CVS is here to stay for now.
Robert: Thanks! Glad to hear you managed to find the bugger that was slowing things down. Always nice to hear about performance optimisations :-D
So, Robin knows a lot more about the Ohloh back end than I do. Also, I wasn't aware of how many revisions are involved. I also wasn't aware of the length of time it took per revision. So Robin's conclusion stands, it won't be possible to import this into Ohloh at the current time.
OK, not good, but at least we know where we stand. Thanks for trying!
I guess we'll have to try harder to convert to git ASAP...
OK. So we converted our CVS repo to git (read only, just so we can get the stats for now). But it failed again. It failed during download, which I assume is caused by the size: ~1.8G packed. Anything more we can do on our end?
Oh, I have forgotten...the repo is: git://binaryparadise.com/gentoo-x86.git/
I guess that would have been easy to figure out, but better safe than sorry :-)
I have rescheduled the job for the new repo and will monitor the project. Have also found another failed job and will investigate.
Note that one HDD on machine where the repo is failed today (nice timing eh?). I'll monitor on my end as well :-)
I am unable to retrieve the repo at this time; seems the server is off-line.
Let me know when it returns... I'll try occasionally when I can.
OK, so we moved the converted git repo to our infrastructure server. Currently waiting in queue...so let's see what will happen.
It's probably among the biggest git repos you'll see (~8GB), so I wouldn't be surprised if it failed.
not to scare you...1.8G :-)
I notice that one enlistment from the CVS server remains. It has failed due to a "No space left on device" error. I have hesitated to reschedule or re-fetch (oh-no!) the enlistment if it should have been deleted. It will prevent the full update from finishing. If, on the other hand, it is a useful enlistment, please let me know and I'll keep pushing on it.
Hmm, are talking about ":pserver:anonymous:@anoncvs.gentoo.org:/var/cvsroot gentoo" enlistment? Apparently there were some optimisations for speed that could cause this. I've removed the enlistment for now. Maybe we'll try to move it to git as well for stats.
Just so! Not exactly sad to see it go. The other three pending enlistments have started up and are processing now. I'll keep you informed here about their status for better or for worse. I do know that one of these enlistments is subject to a bug which may need me to be vigilant but will press on with all three.
Hmm, the "git://git-exp.overlays.gentoo.org/exp/gentoo-x86.git master" failed this time. I've done a full clone from this repo yesterday so I guess it's in workable state and configuration is able to cope with it. Perhaps logs on your end will tell us more
Was just there and jump-started the lazy jobs. The Bazaar enlistment is the one with the (our end) bug attached and I'll need to watch that one more closely. It was quite easy to revive and quick too (698 commits). It's performing line counts now and with fingers crossed I hope that it will be finished shortly. (It has...)
The other one that failed was indeed the one you cited and I finally decided it needed to be re-fetched. Nothing to do for it and now we wait for 2.3 Gb compressed to come down.