Configure Hudson CI Server to automatically disable itself when main SVN repositories become unreachable

batmat — Sun, 16 Jan 2011 12:57:00 +0100

Note: I wrote this post some months ago, and just made it public since the problem making it impossible to use was fixed some weeks ago. In the meantime, you should also be aware that Hudson has recently been renamed to Jenkins, and its new house is now http://jenkins-ci.org/

Sometimes, we encounter erratic issues accessing our subversion repositories. Even apart from the server upgrade information that just dont reach the interested people, but only managers who didn't forward (since there're obviously not the ones that use the dev server...), we also have random problems like everyone.

The problem

When SVN becomes unreachable, every one starts receiving mails about it from Hudson... For example, last week-end I received 6000+ emails about that. So, I wrote this small script to update all our jobs to not run during both the night and the week-end.

But sure, this won't solve everything. For example, if the server goes down during a working-day, and you're not in front of your computer for some reason. When coming back to your box, you might discover the big amount of mails from Hudson, or even from the devs if you're in charge of operating the CI server.

So I've been looking for a way to just automatically disable Hudson builds when a problem is detected.

The solution

For some days now, I've been playing with the Hudson script console since I discovered how greatly powerful it can be.

My starting point was the hudson command used to prepare a shutdown. How to do it through the groovy console? I gave it here in one tweet: hudson.model.Hudson.instance.doQuietDown(). Once I found this, I just had to find a way to interact with the SVN inside the groovy/hudson console system and build around it a small groovy script.

After some struggle about how to programmatically use SVNKit (Subversion pure Java API), and then how to use an anonymous class with Groovy, I was done.

Here's the resulting script:

import hudson.model.*

import org.tmatesoft.svn.core.*

import org.tmatesoft.svn.core.wc.*



String[] repoToCheck = ['svn://svn/scle', 'svn://svn:3691/pgih']



class MyHandler implements ISVNDirEntryHandler

{

  def void handleDirEntry(SVNDirEntry dirEntry)

  {

    // nothing

  } 

}

                

org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryFactoryImpl.setup();

Map<String, Throwable> problematicRepos = new LinkedHashMap<String, Throwable>();

for(String repo:repoToCheck)

{

  SVNURL url = SVNURL.parseURIDecoded(repo);



  SVNClientManager clientManager = SVNClientManager.newInstance();



  SVNLogClient c = clientManager.getLogClient();

  try

  {

    // Special groovy anonymous class construct

    def handler = new MyHandler()

    c.doList(url, SVNRevision.UNDEFINED, SVNRevision.HEAD, false, false, handler);

  }

  catch (Exception e)

  {

    problematicRepos.put(repo, e);

  }

}



if(!problematicRepos.isEmpty())

{

  for(Map.Entry<String, Throwable> entry:problematicRepos.entrySet())

  {

    println("Problem accessing \""+entry.getKey()+"\"");

    String s = entry.getValue();

    println(s)

  }

  println("Disabling hudson build")

  hudson.model.Hudson.instance.doQuietDown()

  return 1

}

else

{

  println("No problems with repos");

}

How to install and configure it

Install the Groovy Plugin for Hudson. This way, you'll be able to add job directly written in Groovy. Then create a job that will run every minute! ("* * * * *") and put the script above inside an "Execute system Groovy script".

Then, configure the notification you like. It's probably a good idea to target admin email when this jobs fails. That's what I did.

Important note: there used to be a difference of behaviour with classloading between "groovy script console" and "groovy system script" in a job. This made the script above unable to work. The good news if that it was fixed with Hudson 1.352 and HUDSON-6068. So the bad news is that you can't use this technique if you're using an older version (time to upgrade? ;-)).

Possible improvements

Sure the script isn't perfect, here's a few thought of what's currently missing:

At the moment, watched repositories are explicitly declared in the script. Maybe it would be more interesting to iterate through the whole list of jobs to find the actual used repositories. But I feel this approach might be a problem: if only one job is failing, maybe the team HAS to be notified. You're not going to disable the whole hudson instance (or even cluster) for only one job failing, right?
About svn :
- this is the only supported scm in the script above
- Only the "svn" protocol is supposed to be supported. If you want to access http exposed repos, then you might have to also initialize the webdav subsystem by calling DAVRepository.setup() first.
Groovify the script a bit. I first wrote it in Java, and then adapted it a bit to be more groovy-like, but my experience in Groovy is only about two weeks, so please be indulgent.

Hope this helps!

[Hudson] How to set a private maven repository by job and easily be able to delete them

batmat — Fri, 09 Oct 2009 10:17:00 +0200

When building maven projects with hudson, there's some common best practices about maven repositories handling :

isolate maven repositories between jobs
regularly purge repositories

The problem

The basic way to do it is to activate the hudson per-job option : "Use private Maven repository". But the thing is you have to do it for EVERY new job you add. There is no way inside hudson to activate it globally.

Documented solution

If you look at hudson help for this option, you'll see a link to a simple solution that specify the repository directly in the maven settings.xml file. The tip is to redefine the localRepository tag inside settings with this special value :

<localRepository>${env.WORKSPACE}/m2_repo</localRepository>

This way, you're done with the first best practice : isolate maven repositories. But not yet with the regularly purge repositories one. Actually, using this option will put the m2_repo inside each hudson job workspace. So, finding and deleting them could become a bit cumbersome. You'd have to cron something like find . -name m2_repo -exec rm -rf "{}" \;.

Even better

As you might have understood, I was not thoroughly satisfied with this solution. I wanted to be able to really easily delete the repositories. So I just changed the option above to have them all inside the same root directory under ~/.m2/repositories, one per job.

Quite simple in fact, instead, just use :

<localRepository>/some/path/.m2/repositories/${env.JOB_NAME}/repository</localRepository>

This way, the only thing you have to put in the cron job is rm -rf /some/path/.m2/repositories/. A bit more straightforward, isn't it?

Hope this helps.

Blogounage - hudson

Configure Hudson CI Server to automatically disable itself when main SVN repositories become unreachable

The problem

The solution

How to install and configure it

Possible improvements

[Hudson] How to set a private maven repository by job and easily be able to delete them

The problem

Documented solution

Even better