Forums : Suggestions for Ohloh 2.0

We now have a volunteer to investigate Darcs support for Ohloh over the weekend, time permitting. Any tips from th Ohloh team? Many thanks!

kowey almost 15 years ago

Hi kowey,

I really think it's great that there is interest in adding Darcs to ohloh_scm. Honestly, everyone here at Ohloh is excited about the prospect.

However, I want to be very careful to set expectations appropriately. This is a real crunch time for us, and we have no scheduled time available to work on this.

I suspect that a functioning Darcs adapter can be knocked out in a single evening. That's the easy part. The real price comes later, in debugging the endless variety of strange crashes and server performance sinks that arise in production deployment.

There's no bare minimum implementation that is useful, and if undertaken with that attitude it simply will not happen. We need some confidence that we aren't going to to be stuck solving a lot of problems after the rollout, and that we'll get good help from the community when we encounter them.

Regardless of our own scope intentions, the public is going to feed every Darcs repository in existence to this adapter, and will expect it to work -- even when given extremely large and/or invalid repositories. You'll be amazed at the variety of abuses people have committed with source control systems.

Also, this adapter code has to be well-behaved (low memory, low CPU), even when given extreme repositories, so that it will not interfere with our existing services.

This means that the adapter has to be very robust and efficient right from the starting gate, or it's not likely to be deployed.

So I'm not saying no, I'm saying yes if you really, really mean it and are willing to do some work to help us.

If after my stern warnings you're still up for it, I'm happy to slice out some time over the next few weeks and give it a shot. Here are some development pointers:

1. A similar OS will probably help a lot. While ohloh_scm should run on most any POSIX OS, we run Ubuntu 8.04 on our servers and OS X 10.5 on our development machines. Windows is a non-starter.

2. It will take some effort to get the ohloh_scm test suites passing on your development machine -- you have to install 4 different source control systems on your box, and few outside of Ohloh have ever run this code. Passing the unit tests are the first milestone. I am happy to help with this step over IRC.

3. Once you have the test suites passing, I would take a look at the Bazaar or Mercurial adapter (whichever is most similar to Darcs) and use it as a model. I would literally just copy and paste the entire lib/scm/adapters/hg directory and all of the test/unit/hg_* files, start renaming them to darcs, and go from there.

4. For the test suites, you'll need to prepare a couple of small sample repositories that exercise all of the key features, including whatever branching and merging possibilities may arise. Good coverage is essential -- I'll be a real stickler about this.

5. Finally, I know you all are Darcs users, but somehow you'll have to learn to like Git long enough to submit your patches to me, preferably via GitHub :-).

I'm happy to answer any questions, explain the big picture of how Ohloh uses this library, and help the process along. I hope there are some Darcs lovers who are willing to commit some time to this!

Thanks,

Robin

Robin Luckey almost 15 years ago

Great roadmap. I'm excited to see this moving forward, even if it's going to take us a while.

Simon Michael almost 15 years ago

Hi Robin,

Thanks your detailed response. I especially appreciated how upfront you were about your schedule and real world requirements. Stern is way better than flaky!

We want to make sure that Darcs support is something that you guys will be happy with. As Simon says, ohloh adds darcs support, followed by darcs support brings down ohloh would not be a good scenario. We're prepared to wait however long it takes for Darcs and the Darcs adapter before we even dream of getting it deployed.

This was a much needed dose of sobriety! It sounds like the best bet as far as the Darcs team is concerned is to revisit this question after August, that is, after our July release is out, and our Google Summer of Code project has been completed).

In the meantime, we will tackle the roadmap you proposed, asking questions along the way. I think the easy parts are (i) most of us use Linux and MacOS X too and (v) we'de be happy to send patches your way via Git.

Two questions from my incremental perspective: would it be useful to you to have a preliminary version of the Darcs adapter in the main ohloh_scm repository without it necessarily being deployed? I can understand that this could impose an unwanted maintenance cost, or maybe add unwanted pressure to deploy the adapter. But if it is possible, the extra visibility could be useful for making sure we complete the project. We could always maintain a branch for however long it takes.

Also: do you have any mechanisms that would allow us to pilot the use of the Darcs adapter in ohloh.net for a few small, key projects; without enabling it for the wider world?

kowey almost 15 years ago

Hi kowey,

Although I'm stern, I'm actually pretty optimistic -- this should be pretty straightforward, unless there are a lot of surprises. It really is simply a time issue, which is the only reason it's still on our to-do list. To answer your questions:

First, yes, I'm happy to periodically accept and distribute preliminary code. Even if the code's not ready for production, there's value in having it for testing, review, and getting lots of eyes on it.

Second, yes, we can run the code on selected Ohloh projects long before opening it up to the public. We did this for Bazaar and Mercurial. Once private local testing has run out of bugs, we can throw the code into the lair and see what interesting ways Ohloh will find to explode :-).

Robin

Robin Luckey almost 15 years ago

Half a year has passed. Any news on this front?

nomeata over 14 years ago

Sorry for not replying, I didn't get a notification of this until Eric pinged me.

I came across three problems:

• Issues with darcs not being history based. This was actually resolved when I met Eric at AngloHaskell, as we decided that darcs could be integrated into Ohloh as long as patch order was not changed.

• Slowness of communication. I'm used to working on-site, pair programming, and it was taking a long time to get questions answered that I'm used to resolving issues. This is a personal failing of me, as I'm more used to commercial work than open source work.

• Lack of time. Business pressure has forced me to focus on paid work, unfortunately.

That said, I have some test cases and some working code in a local git repo, that I could try to merge with recent code and push to a hosted branch.

Let me know how I should procede.

Ashley Moran over 14 years ago

I just thought I might add that this is being tracked on the Darcs tracker at http://bugs.darcs.net/issue1002

kowey over 14 years ago

Hi Ashley,

The easiest way to do this is probably for you to fork robin's github repo commit your patch and send us a pull request.

I'll try out your changes and give feedback.

Thanks for working on this.

Andy Verprauskus over 14 years ago

Hi Andy

Don't get your hopes up too much about what I've managed to get working :)

I am busy this week and this weekend, but I will squeeze this in in the next fortnight.

I'd love to get this working, but to do so efficiently I think it'd take me, you and Eric round a table for a day or two!

Ashley

Ashley Moran over 14 years ago

Because I promised to do something on this, I thought I should update you that I haven't been able to work on it. I've had a lot of work come up with pressing deadlines, so unfortunately it has had to take a back seat. But it's still pretty high up my current actions.

Ashley Moran over 14 years ago

Don't sweat it, I understand.

Andy Verprauskus over 14 years ago

There's been no news on this for a while now. Is it still in the cards?

Magnus Therning over 13 years ago

It's not off the cards, but it's a matter of me (assuming I do it) fitting it in. I found it difficult to do on my own because of the difference between darcs and timeline-based systems. It would be much more efficient to do it paired with Eric, but then it's a matter of finding the time to get together. (We came close to managing that recently but it didn't work out...)

Ashley Moran over 13 years ago

Now that ohloh is under new management – will this have an effect on the Darcs support?

nomeata over 13 years ago

nomeata, Thanks for your question. Alas, the mechanics of adding darcs support did not change with the change of ownership. Robin's process and list of requirements for adding support for a particular SCM still likely represent the best outline of how to get it done. That said, we're looking for the low hanging fruit for making some substantive updates to ohloh. We're evaluating what would be most valuable and doable. Are there members of the darcs team ready to tackle this?

Rich Sands

Ohloh Team over 13 years ago

I think it should be useful

OliverOwen over 13 years ago

Yes, there are. I have set up a how-to/status document and sent a call for help. Darcs-on-ohloh fans, now is the time to help out (with a little light ruby coding - quite refreshing). This could be done quickly.

Simon Michael over 13 years ago

Preliminary darcs support is now passing all tests, and I have submitted a pull request. I'm sure there are naivetes in this code, but hopefully it's ready for some testing at Ohloh. Let us know what else we can do to help.

Simon Michael over 13 years ago

Simon:

I'm not sure Robin maintains that fork of ohloh_scm anymore.

I've forked it to: https://github.com/blackducksw/ohloh_scm

That's the official version of the ohloh_scm code that ohloh.net will be using.

I've entered a ticket on our side to evaluate the code changes you've made on your fork, test them, and once everything is all set pull those changes into the official branch and push live to ohloh.

Having just got the ohloh code about a month ago we are still learning how all of ohloh works and getting used to the code.

We are also busy preparing to migrate the servers from Chicago to Boston which will take place in the next few weeks.

Due to these other activities we probably won't be able to look into adding darcs support until sometime early next year.

I'll reply here once I know more about 'when'.

By the way, I saw you pop into the IRC #ohloh channel on Freenode :)

RobertSchultz over 13 years ago

Hi Robert, thanks for the update. Sorry to hear that there might be a few months' more delay on this. :-( We all hope your transition goes more easily than expected.

Simon Michael over 13 years ago

Is there anything we can do to merge these patches into ohloh trunk without waiting for the servers to get moved from Chicago to Boston? What's the process of getting code merged -- does it require RobertSchultz to review it? Can we add documentation, tests, refactoring, a code walk-through or something to make this easier for him? Can someone else merge it?

zooko over 13 years ago

zooko:

We just finished migrating the servers from Chicago to Boston last week.

For the rest of the year and through the holidays we will be focusing on fixing enlistments, critical Ohloh bugs and re-organizing/updating our server hardware/software.

On our plate at the start of next year are two major problems with Ohloh's core. One has to do with how we count source code incorrectly for GIT repositories that branch and merge a lot. The other has to do with a scaling problem on Ohloh's back end. Fixing these two problems are multi-week efforts.

However we clearly see how excited you are to have Ohloh support Darcs :)

So I am going to squirrel a few days away in the first couple weeks of January to look into pulling in the Darcs support code from the ohloh_scm GitHub fork and enhancing Ohloh to support that in the UI.

I hope I can complete it in that time.

Either way, I'll post here again then with an update :)

RobertSchultz over 13 years ago

Any update on this? I have an interest in this feature, as I'm currently looking for employment, and I'd like to be able to conveniently use Ohloh to list and auto-update my contributions to darcs-based projects.

Robin Green about 13 years ago

Since my last update, we've fixed a few critical Ohloh bugs.

We've also fixed one of the two major problems I mentioned, incorrectly counting source code for GIT repositories.

Over the past 2-3 weeks or so we have been examining Ohloh's back end and how we can modify it to scale better. We are currently not automatically adding projects from Github or Code.Google and we'd like to do this but we need make sure the back end can handle it first.

So no progress has been made on adding darcs support, sorry.

RobertSchultz about 13 years ago