New plugin to distribute maven jobs steps

[François Petitit]

Hello,

we are interesting with some colleagues in developping a Hudson plug-in that would discover modules dependencies in a Maven job and then would be able to distribute the build of independant modules in different Hudson nodes.
The main goal is to maximize build parallelization.

What do you think about?
Is there already a way to obtain the same result?

Thank a lot for your feedbacks,

--
François Petitit
Consultant at OCTO Technology

[Kohsuke Kawaguchi-2]

François Petitit wrote:
> Hello,
>
> we are interesting with some colleagues in developping a Hudson plug-in that
> would discover modules dependencies in a Maven job and then would be able to
> distribute the build of independant modules in different Hudson nodes.
> The main goal is to maximize build parallelization.
>
> What do you think about?
> Is there already a way to obtain the same result?

I thought about this some time ago, and the key to achieve this is to
swap in a custom artifact resolution code into Maven, so that artifacts
of one module built in one place can be sent over to the build of
another module.

If anyone wants to tackle it, by all means please do it.


>
> Thank a lot for your feedbacks,
>
> --
> *François Petitit*
> Consultant at OCTO Technology <http://www.octo.com>
>


--
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

[Jorg Heymans]

See also here http://old.nabble.com/Proposal-after-the-fact%3A-Experimental-multithreading-support-ts26277579.html
for some more insights.

Jorg

On Tue, Nov 10, 2009 at 12:27 AM, Kohsuke Kawaguchi
<[hidden email]> wrote:

> François Petitit wrote:
>> Hello,
>>
>> we are interesting with some colleagues in developping a Hudson plug-in that
>> would discover modules dependencies in a Maven job and then would be able to
>> distribute the build of independant modules in different Hudson nodes.
>> The main goal is to maximize build parallelization.
>>
>> What do you think about?
>> Is there already a way to obtain the same result?
>
> I thought about this some time ago, and the key to achieve this is to
> swap in a custom artifact resolution code into Maven, so that artifacts
> of one module built in one place can be sent over to the build of
> another module.
>
> If anyone wants to tackle it, by all means please do it.
>
>
>>
>> Thank a lot for your feedbacks,
>>
>> --
>> *François Petitit*
>> Consultant at OCTO Technology <http://www.octo.com>
>>
>
>
> --
> Kohsuke Kawaguchi
> Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[François Petitit]

Thanks a lot for your answer.

As a first step we are thinking that artifacts of one module built in Hudson node should be deployed in a central repository, typically the central repository of a company, and so it will be accessible for other Hudson nodes during the build.

The maven goal of most of our builds is "deploy", that's why it is not bad for us to deploy in the central repository.

Do you have any suggestion?

Thanks in advance.

2009/11/10 Kohsuke Kawaguchi <[hidden email]>

François Petitit wrote:
> Hello,
>
> we are interesting with some colleagues in developping a Hudson plug-in that
> would discover modules dependencies in a Maven job and then would be able to
> distribute the build of independant modules in different Hudson nodes.
> The main goal is to maximize build parallelization.
>
> What do you think about?
> Is there already a way to obtain the same result?

I thought about this some time ago, and the key to achieve this is to
swap in a custom artifact resolution code into Maven, so that artifacts
of one module built in one place can be sent over to the build of
another module.

If anyone wants to tackle it, by all means please do it.

>
> Thank a lot for your feedbacks,
>
> --
> *François Petitit*

> Consultant at OCTO Technology <http://www.octo.com>
>


--
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

[François Petitit]

Thanks for your answer.

What this link propose is a great thing for build performance, indeed.

I just think that our evolution suggestion is not about multithreading of maven builds, but the distribution of independant steps of a maven build in different Hudson nodes.
The goal is to distribute the build on different computers, and not in different threads in the same computer.

What do you think?

François

2009/11/11 Jorg Heymans <[hidden email]>

See also here http://old.nabble.com/Proposal-after-the-fact%3A-Experimental-multithreading-support-ts26277579.html
for some more insights.

Jorg

On Tue, Nov 10, 2009 at 12:27 AM, Kohsuke Kawaguchi
<[hidden email]> wrote:
> François Petitit wrote:
>> Hello,
>>
>> we are interesting with some colleagues in developping a Hudson plug-in that
>> would discover modules dependencies in a Maven job and then would be able to
>> distribute the build of independant modules in different Hudson nodes.
>> The main goal is to maximize build parallelization.
>>
>> What do you think about?
>> Is there already a way to obtain the same result?
>
> I thought about this some time ago, and the key to achieve this is to
> swap in a custom artifact resolution code into Maven, so that artifacts
> of one module built in one place can be sent over to the build of
> another module.
>
> If anyone wants to tackle it, by all means please do it.
>
>
>>
>> Thank a lot for your feedbacks,
>>
>> --
>> *François Petitit*
>> Consultant at OCTO Technology <http://www.octo.com>
>>
>
>
> --
> Kohsuke Kawaguchi
> Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Fabrizio Giudici]

François Petitit wrote:

> Thanks a lot for your answer.
>
> As a first step we are thinking that artifacts of one module built in
> Hudson node should be deployed in a central repository, typically the
> central repository of a company, and so it will be accessible for
> other Hudson nodes during the build.
>
> The maven goal of most of our builds is "deploy", that's why it is not
> bad for us to deploy in the central repository.
>
> Do you have any suggestion?
>
> Thanks in advance.

Consider that an alternate deployment repo would be important for
staging, even when one is not interested in distributing a build. But,
since it seems a related matter, you could probably reach this
additional goal with a low effort.

What I'd like to see is that we can enable Maven to deploy stuff in a
local temp repository overriding the one declared in the pom ("local"
might be in simple case a file storage on a single computer when Hudson
is running, but also the central repo of a company as you said), and
eventually at the end of the Maven build step the artifacts are only
left on this local repo. Further Maven steps might deploy additional
stuff to this repo (for instance, it's known -
http://jira.codehaus.org/browse/MASSEMBLY-94 - that the
assembly:assembly goal has got troubles and in the end it should be
executed by a second Maven invocation). A final step could publish the
stuff in the temp repo to the official one. I've recently achieved this
sequence by exclusively working with the pom (you can see it at
http://hudson.tidalwave.it:8080/hudson/job/ForceTen%20Release/configure)
but I'd like to see it offered by Hudson, as I wouldn't need to keep
that pom section in sync among my several projects.

--
Fabrizio Giudici - Java Architect, Project Manager
Tidalwave s.a.s. - "We make Java work. Everywhere."
weblogs.java.net/blog/fabriziogiudici - www.tidalwave.it/people
[hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[John-Mason P. Shackelford]

So back on (c.f.
http://n4.nabble.com/build-distribution-levels-td387304.html) it was
suggested that the functionality which parses the parent POM could
(optionally) represent multi-module builds as separate jobs so that
this didn't need to be done manually, which would work so long as each
job ended with a deploy the same repository--probably a working
repository set aside for this particular purpose so that deploys to
the repository to which teams would ordinarily publish known-good
artifacts could be delayed until all modules where known to build
properly.

So we'd need:
1. An option to explode a multi-module project into separate jobs
2. An easy way to cascade removal of module jobs when we remove the
parent POM job.
3. To constrain module jobs to deploy to the working repo upon completion
4. What else?

Will we need to fiddle with dependency version numbers or will this
'just work' with both SNAPSHOT and release versions deployed to our
working repo?

Lets finalize design and divide up the work so we can get this done.

John-Mason Shackelford

Release Engineering
Pearson

2510 North Dodge St.
Iowa City, IA 52245
ph. 319-354-9200x6214
[hidden email]
http://pearsonschool.com

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]