New release plan

[Kohsuke Kawaguchi]

As you know, lately we are suffering a quality problem in releases.
There are multiple things we need to do to fix that, such as more tests
and early regression testing with plugins, but it's also clear that we
need to adjust the release process.

So this is the proposal to make that adjustment.

- Let's shoot for once-a-week release cycle, on Friday evening pacific
   time.

- We allow commits freely in Sat, Sun, Mon, and Tue. Tuesday evening
   is RC cut off.

- We reserve Wed, Thu, and Fri as the quiet period.
   We make commits to the core conservatively, and I'll use this period
   to deploy RC bits to my production Hudson to be a guinea pig.

- This should give us more time to write tests.


I know there will be all sorts of different ways to do releases, and I
don't want to argue which is better than which. So my main question is,
putting my dictator hat on, is this something that people can live with?

--
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

[Peter Reilly-2]

For plugins or just for hudson core?

Peter

On Tue, Mar 10, 2009 at 7:05 PM, Kohsuke Kawaguchi
<[hidden email]> wrote:

>
> As you know, lately we are suffering a quality problem in releases. There
> are multiple things we need to do to fix that, such as more tests and early
> regression testing with plugins, but it's also clear that we need to adjust
> the release process.
>
> So this is the proposal to make that adjustment.
>
> - Let's shoot for once-a-week release cycle, on Friday evening pacific
>  time.
>
> - We allow commits freely in Sat, Sun, Mon, and Tue. Tuesday evening
>  is RC cut off.
>
> - We reserve Wed, Thu, and Fri as the quiet period.
>  We make commits to the core conservatively, and I'll use this period
>  to deploy RC bits to my production Hudson to be a guinea pig.
>
> - This should give us more time to write tests.
>
>
> I know there will be all sorts of different ways to do releases, and I don't
> want to argue which is better than which. So my main question is, putting my
> dictator hat on, is this something that people can live with?
>
> --
> Kohsuke Kawaguchi
> Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Kohsuke Kawaguchi]

Peter Reilly wrote:
> For plugins or just for hudson core?

Just Hudson core.

>
> Peter
>
> On Tue, Mar 10, 2009 at 7:05 PM, Kohsuke Kawaguchi
> <[hidden email]> wrote:
>>
>> As you know, lately we are suffering a quality problem in releases. There
>> are multiple things we need to do to fix that, such as more tests and early
>> regression testing with plugins, but it's also clear that we need to adjust
>> the release process.
>>
>> So this is the proposal to make that adjustment.
>>
>> - Let's shoot for once-a-week release cycle, on Friday evening pacific
>> ?time.
>>
>> - We allow commits freely in Sat, Sun, Mon, and Tue. Tuesday evening
>> ?is RC cut off.
>>
>> - We reserve Wed, Thu, and Fri as the quiet period.
>> ?We make commits to the core conservatively, and I'll use this period
>> ?to deploy RC bits to my production Hudson to be a guinea pig.
>>
>> - This should give us more time to write tests.
>>
>>
>> I know there will be all sorts of different ways to do releases, and I don't
>> want to argue which is better than which. So my main question is, putting my
>> dictator hat on, is this something that people can live with?
>>
>> --
>> Kohsuke Kawaguchi
>> Sun Microsystems ? ? ? ? ? ? ? ? ? http://weblogs.java.net/blog/kohsuke/
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

--
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

[Dean Yu]

Hi Kohsuke,
  Is there a reason you don't want to create a branch on Tuesday evening
instead of cutting off commits? It seems ironic that we don't try to
practice CI for building Hudson itself.
  In the end you're our dictator and I'll go along with whatever you
decide. :)

  -- Dean

> -----Original Message-----
> From: Kohsuke Kawaguchi [mailto:[hidden email]]
> Sent: Tuesday, March 10, 2009 12:05 PM
> To: [hidden email]
> Subject: New release plan
>
>
> As you know, lately we are suffering a quality problem in releases.
> There are multiple things we need to do to fix that, such as
> more tests and early regression testing with plugins, but
> it's also clear that we need to adjust the release process.
>
> So this is the proposal to make that adjustment.
>
> - Let's shoot for once-a-week release cycle, on Friday evening pacific
>    time.
>
> - We allow commits freely in Sat, Sun, Mon, and Tue. Tuesday evening
>    is RC cut off.
>
> - We reserve Wed, Thu, and Fri as the quiet period.
>    We make commits to the core conservatively, and I'll use
> this period
>    to deploy RC bits to my production Hudson to be a guinea pig.
>
> - This should give us more time to write tests.
>
>
> I know there will be all sorts of different ways to do
> releases, and I don't want to argue which is better than
> which. So my main question is, putting my dictator hat on, is
> this something that people can live with?
>
> --
> Kohsuke Kawaguchi
> Sun Microsystems                  
> http://weblogs.java.net/blog/kohsuke/
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Kohsuke Kawaguchi]

Dean Yu wrote:
> Hi Kohsuke,
>   Is there a reason you don't want to create a branch on Tuesday evening
> instead of cutting off commits? It seems ironic that we don't try to
> practice CI for building Hudson itself.

Yeah, we should do that. I guess I could automate the branch handling on
my side, and the merging shouldn't be too painful. changelog.html needs
to be changed a bit so that we can list changes in N+2.

And I think we do practice CI for building Hudson --- there's Hudson on
Hudson as you can see on http://hudson.glassfish.org/job/hudson/

But my feeling is that our current test coverage is just not high enough
to catch all the problems this way.


>   In the end you're our dictator and I'll go along with whatever you
> decide. :)

--
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

[Tom Huybrechts]

On Tue, Mar 10, 2009 at 8:45 PM, Kohsuke Kawaguchi
<[hidden email]> wrote:

> Dean Yu wrote:
>>
>> Hi Kohsuke,
>>  Is there a reason you don't want to create a branch on Tuesday evening
>> instead of cutting off commits? It seems ironic that we don't try to
>> practice CI for building Hudson itself.
>
> Yeah, we should do that. I guess I could automate the branch handling on my
> side, and the merging shouldn't be too painful. changelog.html needs to be
> changed a bit so that we can list changes in N+2.
>
> And I think we do practice CI for building Hudson --- there's Hudson on
> Hudson as you can see on http://hudson.glassfish.org/job/hudson/
>
> But my feeling is that our current test coverage is just not high enough to
> catch all the problems this way.
>

I prefer the original idea (a 'voluntary' restraint on committing
during the quiet period) over branching each time.
I'd also like to suggest we take this time to review commits others
have done. Not every problem (e.g. backward compatibility) will be
caught by the tests we have now, but some extra eyeballs will surely
help. Perhaps it would help to send a mail (automatically) every time
the quiet period starts stating changes that have been made since the
last release ? Or a link to a Fisheye page where can you can see them
all at a glance.

Tom


>
>>  In the end you're our dictator and I'll go along with whatever you
>> decide. :)
>
> --
> Kohsuke Kawaguchi
> Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Dean Yu]

> -----Original Message-----
> From: Tom Huybrechts [mailto:[hidden email]]
> Sent: Tuesday, March 10, 2009 12:47 PM
> To: [hidden email]
> Subject: Re: New release plan
>
> I prefer the original idea (a 'voluntary' restraint on
> committing during the quiet period) over branching each time.

I have seen software projects that take this approach. Every single one of them runs into the problem where the software gets completely destabilized from a stampeding herd of checkins after a period of not being able to check in. This is exactly the type of problem that continuous integration is supposed to solve in the first place.

I'm an infrequent committer at best, so if this is how people want to do it, I'll go along, but it will become a problem as more people contribute to the core.

> I'd also like to suggest we take this time to review commits
> others have done. Not every problem (e.g. backward
> compatibility) will be caught by the tests we have now, but
> some extra eyeballs will surely help. Perhaps it would help
> to send a mail (automatically) every time the quiet period
> starts stating changes that have been made since the last
> release ? Or a link to a Fisheye page where can you can see
> them all at a glance.

  +1 on this idea.

  -- Dean

>
> Tom
>
>
> >
> >>  In the end you're our dictator and I'll go along with
> whatever you
> >> decide. :)
> >
> > --
> > Kohsuke Kawaguchi
> > Sun Microsystems                  
> > http://weblogs.java.net/blog/kohsuke/
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Eric Lefevre-Ardant]

What about translations in the core? would we put the same restraints?

Personally, I am very inconsistent in the delays between 2 commits to the French translation; sometimes a couple of days, sometimes one month. I don't see myself paying much attention to days when I should commit. Not that I don't want to.

Eric

2009/3/10 Kohsuke Kawaguchi <[hidden email]>

Peter Reilly wrote:

For plugins or just for hudson core?

Just Hudson core.

Peter

On Tue, Mar 10, 2009 at 7:05 PM, Kohsuke Kawaguchi
<[hidden email]> wrote:

As you know, lately we are suffering a quality problem in releases. There
are multiple things we need to do to fix that, such as more tests and early
regression testing with plugins, but it's also clear that we need to adjust
the release process.

So this is the proposal to make that adjustment.

- Let's shoot for once-a-week release cycle, on Friday evening pacific
?time.

- We allow commits freely in Sat, Sun, Mon, and Tue. Tuesday evening
?is RC cut off.

- We reserve Wed, Thu, and Fri as the quiet period.
?We make commits to the core conservatively, and I'll use this period
?to deploy RC bits to my production Hudson to be a guinea pig.

- This should give us more time to write tests.


I know there will be all sorts of different ways to do releases, and I don't
want to argue which is better than which. So my main question is, putting my
dictator hat on, is this something that people can live with?

--
Kohsuke Kawaguchi
Sun Microsystems ? ? ? ? ? ? ? ? ? http://weblogs.java.net/blog/kohsuke/

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

--

Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

[Wim Rosseel]

Wel I can second the inconsistent delays between commits for the Dutch translations. However I wouldn't have a problem with the suggestion to have a quiet period. After all I find it a lot more important that we avoid regression problems in releases.
Hudson currently is seeing quite a wide usage, but I feel that people do tend to be weary about upgrading due to upgrade problems and plugin incompatibilities.

So +1 on the idea

Nam-tor Ozhika kluterek t'sha'sutenivaya - k'ish she-tor etek s'nezhak - isan utvau vah sha'kakhartayek.

On Wed, Mar 11, 2009 at 9:57 AM, Eric Lefevre-Ardant <[e-mail]> wrote:

What about translations in the core? would we put the same restraints?

Personally, I am very inconsistent in the delays between 2 commits to the French translation; sometimes a couple of days, sometimes one month. I don't see myself paying much attention to days when I should commit. Not that I don't want to.

Eric

2009/3/10 Kohsuke Kawaguchi <[hidden email]>

Peter Reilly wrote:

For plugins or just for hudson core?

Just Hudson core.

Peter

On Tue, Mar 10, 2009 at 7:05 PM, Kohsuke Kawaguchi
<[hidden email]> wrote:

As you know, lately we are suffering a quality problem in releases. There
are multiple things we need to do to fix that, such as more tests and early
regression testing with plugins, but it's also clear that we need to adjust
the release process.

So this is the proposal to make that adjustment.

- Let's shoot for once-a-week release cycle, on Friday evening pacific
?time.

- We allow commits freely in Sat, Sun, Mon, and Tue. Tuesday evening
?is RC cut off.

- We reserve Wed, Thu, and Fri as the quiet period.
?We make commits to the core conservatively, and I'll use this period
?to deploy RC bits to my production Hudson to be a guinea pig.

- This should give us more time to write tests.


I know there will be all sorts of different ways to do releases, and I don't
want to argue which is better than which. So my main question is, putting my
dictator hat on, is this something that people can live with?

--
Kohsuke Kawaguchi
Sun Microsystems ? ? ? ? ? ? ? ? ? http://weblogs.java.net/blog/kohsuke/

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

--

Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

[BrandonB]

Dean Yu wrote:

I have seen software projects that take this approach. Every single one of them runs into the problem where the software gets completely destabilized from a stampeding herd of checkins after a period of not being able to check in. This is exactly the type of problem that continuous integration is supposed to solve in the first place.

I have also witnessed another problem with the "quiet period" approach on another large scale project I work on - everyone usually rushes to check tons of changes in to the trunk before the weekly "lockdown". Creating a release branch mitigates this somewhat - people can still freely check in changes at their own pace, but they just may not make the weekly release.

As far as merging goes - the release branches should not have to be frequently merged back to the trunk, unless running automated tests against the release branch reveals that patches are required to create a stable release. If you're anti-merging, you can choose to simply not make the release, apply the patches directly to the trunk and wait for the next week's release (or try again with a new, later, release branch).

[BrandonB]

Whether the decision is to go with a "quiet period" or branches, I make the following suggestions:

1) Add code coverage checks (Cobertura or Clover) to the http://hudson.glassfish.org/job/hudson/ job, against the existing tests the Hudson core already has. This should identify weaknesses in coverage, and prompt the team/users to start contributing more tests.

2) Before a Hudson core release is made, compile the core against the latest release of all plugins. Wherever possible, run the plugin unit tests against the new core. If, for whatever reason, the hudson.glassfish.org server cannot run plugin unit tests (eg, it might be hard to run the AccuRev plugin tests without AccuRev server or licenses), notify (automatically?) the plugin owners to run their unit tests against the about-to-be-released core.

3) Do any of the plugins have Hudson jobs on a public Hudson instance? If not, can they be added to hudson.glassfish.org? If so, can they be moved there?

4) All plugins should have code coverage (Cobertura or Clover) checks against their unit tests.

5) When the core release is made, publish a document (somewhere, wiki, release notes, part of the changelog, don't know) stating which versions of plugins were tested against the new core. I believe some of the plugins already document which versions of the core *they* were compiled/tested against when they are released - we need the reverse.

6) In fact, including the information from #5 in the 'Updates' and/or 'Available' tabs of the 'Manage Plugins' table would be extremely useful. (I'm envisioning another column stating something to the effect of "This version of Hudson was tested againstplugin release #X.Y.) This can help people identify whether they *should* upgrade plugins, instead of just crossing their fingers and upgrading blindly (or manually comparing release dates of core and plugins).

[Andrew Bayer]

I agree wholeheartedly on the plugin-related suggestions - it seems like a lot of the upgrade issues are due to plugin compatibility issues, and I've seen at least one bug against a plugin that was a result of a version of that plugin being run with a version of Hudson that couldn't support it. Unit-testing plugins against new cores seems like it would help with revealing some of those problems, and having some sort of information on what version of Hudson core a particular plugin version is known to work with, etc, would be a big help as well.

A.

On Wed, Mar 11, 2009 at 9:50 AM, BrandonB <[hidden email]> wrote:

Whether the decision is to go with a "quiet period" or branches, I make the
following suggestions:

1) Add code coverage checks (Cobertura or Clover) to the
http://hudson.glassfish.org/job/hudson/ job, against the existing tests the
Hudson core already has. This should identify weaknesses in coverage, and
prompt the team/users to start contributing more tests.

2) Before a Hudson core release is made, compile the core against the latest
release of all plugins. Wherever possible, run the plugin unit tests against
the new core. If, for whatever reason, the hudson.glassfish.org server
cannot run plugin unit tests (eg, it might be hard to run the AccuRev plugin
tests without AccuRev server or licenses), notify (automatically?) the
plugin owners to run their unit tests against the about-to-be-released core.

3) Do any of the plugins have Hudson jobs on a public Hudson instance? If
not, can they be added to hudson.glassfish.org? If so, can they be moved
there?

4) All plugins should have code coverage (Cobertura or Clover) checks
against their unit tests.

5) When the core release is made, publish a document (somewhere, wiki,
release notes, part of the changelog, don't know) stating which versions of
plugins were tested against the new core. I believe some of the plugins
already document which versions of the core *they* were compiled/tested
against when they are released - we need the reverse.

6) In fact, including the information from #5 in the 'Updates' and/or
'Available' tabs of the 'Manage Plugins' table would be extremely useful.
(I'm envisioning another column stating something to the effect of "This
version of Hudson was tested againstplugin release #X.Y.) This can help
people identify whether they *should* upgrade plugins, instead of just
crossing their fingers and upgrading blindly (or manually comparing release
dates of core and plugins).
--
View this message in context: http://www.nabble.com/New-release-plan-tp22441320p22459412.html

Sent from the Hudson dev mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Kohsuke Kawaguchi]

Eric Lefevre-Ardant wrote:
> What about translations in the core? would we put the same restraints?
> Personally, I am very inconsistent in the delays between 2 commits to the
> French translation; sometimes a couple of days, sometimes one month. I don't
> see myself paying much attention to days when I should commit. Not that I
> don't want to.

I can't think of any past incident where l10n/i18n commits resulted in a
problematic release, so I think you should consider the repository open
for commits all the time.

(And if do a release branch, this point will be moot anyway)

> Eric
>
> 2009/3/10 Kohsuke Kawaguchi <[hidden email]>
>
>> Peter Reilly wrote:
>>
>>> For plugins or just for hudson core?
>>>
>>
>> Just Hudson core.
>>
>>
>>
>>> Peter
>>>
>>> On Tue, Mar 10, 2009 at 7:05 PM, Kohsuke Kawaguchi
>>> <[hidden email]> wrote:
>>>
>>>>
>>>> As you know, lately we are suffering a quality problem in releases. There
>>>> are multiple things we need to do to fix that, such as more tests and
>>>> early
>>>> regression testing with plugins, but it's also clear that we need to
>>>> adjust
>>>> the release process.
>>>>
>>>> So this is the proposal to make that adjustment.
>>>>
>>>> - Let's shoot for once-a-week release cycle, on Friday evening pacific
>>>> ?time.
>>>>
>>>> - We allow commits freely in Sat, Sun, Mon, and Tue. Tuesday evening
>>>> ?is RC cut off.
>>>>
>>>> - We reserve Wed, Thu, and Fri as the quiet period.
>>>> ?We make commits to the core conservatively, and I'll use this period
>>>> ?to deploy RC bits to my production Hudson to be a guinea pig.
>>>>
>>>> - This should give us more time to write tests.
>>>>
>>>>
>>>> I know there will be all sorts of different ways to do releases, and I
>>>> don't
>>>> want to argue which is better than which. So my main question is, putting
>>>> my
>>>> dictator hat on, is this something that people can live with?
>>>>
>>>> --
>>>> Kohsuke Kawaguchi
>>>> Sun Microsystems ? ? ? ? ? ? ? ? ? http://weblogs.java.net/blog/kohsuke/
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>>
>>>
>>
>> --
>> Kohsuke Kawaguchi
>> Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/
>>
>

--
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

[Kohsuke Kawaguchi]

Dean Yu wrote:

>> -----Original Message-----
>> From: Tom Huybrechts [mailto:[hidden email]]
>> Sent: Tuesday, March 10, 2009 12:47 PM
>> To: [hidden email]
>> Subject: Re: New release plan
>>
>> I prefer the original idea (a 'voluntary' restraint on
>> committing during the quiet period) over branching each time.
>
> I have seen software projects that take this approach. Every single one
> of them runs into the problem where the software gets completely
> destabilized from a stampeding herd of checkins after a period of not
> being able to check in. This is exactly the type of problem that
> continuous integration is supposed to solve in the first place.

I think I like branching better, too, not so much because of the dumping
changes effect (since the stabilization period is still just 3 days),
but because it creates confusion like what changes can be committed and
what cannot, as can be seen by e-mails from Eric.

I think merging back the release branch shouldn't that much of a
problem, again given the short lifespan of the release branch.

> I'm an infrequent committer at best, so if this is how people want to do it, I'll go along, but it will become a problem as more people contribute to the core.
>
>> I'd also like to suggest we take this time to review commits
>> others have done. Not every problem (e.g. backward
>> compatibility) will be caught by the tests we have now, but
>> some extra eyeballs will surely help. Perhaps it would help
>> to send a mail (automatically) every time the quiet period
>> starts stating changes that have been made since the last
>> release ? Or a link to a Fisheye page where can you can see
>> them all at a glance.
>
>   +1 on this idea.

OK. I should be able to automate this.

>
>   -- Dean
>
>>
>> Tom
>>
>>
>> >
>> >> ?In the end you're our dictator and I'll go along with
>> whatever you
>> >> decide. :)
>> >
>> > --
>> > Kohsuke Kawaguchi
>> > Sun Microsystems ? ? ? ? ? ? ? ? ?
>> > http://weblogs.java.net/blog/kohsuke/
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

--
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

[Ulli Hafner]

I'm not a frequent committer to Hudson core but I would prefer using a release branch... E.g., I'm working on Hudson not on a daily basis: so if I have some time it would be sad if I could not check in my changes just because its end of the week.

Improve the Hudson core stability:
- Since we don't have a testing team the only way I see is to increase the test coverage.
- Maybe we can set up a team of users and plug-in developers that act as kind of beta tester: if a given number of group members confirms that the release works for their use cases then flag a release as stable and make it available via update center (otherwise you need to download manually).

Improve the interaction testing between Hudson and plugins:
- It would help if we could build AND deploy the plug-ins in a public Hudson. This should work at least for the reporting plug-ins. (This would additionally be quite usefule as a live-experience for new users, better than static Wiki pages:-)

Ulli

> -----Original Message-----
> From: Kohsuke Kawaguchi [mailto:[hidden email]]
> Sent: Dienstag, 10. März 2009 20:46
> To: [hidden email]
> Subject: Re: New release plan
>
> Dean Yu wrote:
> > Hi Kohsuke,
> >   Is there a reason you don't want to create a branch on Tuesday
> > evening instead of cutting off commits? It seems ironic
> that we don't
> > try to practice CI for building Hudson itself.
>
> Yeah, we should do that. I guess I could automate the branch
> handling on my side, and the merging shouldn't be too
> painful. changelog.html needs to be changed a bit so that we
> can list changes in N+2.
>
> And I think we do practice CI for building Hudson --- there's
> Hudson on Hudson as you can see on
> http://hudson.glassfish.org/job/hudson/
>
> But my feeling is that our current test coverage is just not
> high enough to catch all the problems this way.
>
>
> >   In the end you're our dictator and I'll go along with whatever you
> > decide. :)
>
> --
> Kohsuke Kawaguchi
> Sun Microsystems                  
> http://weblogs.java.net/blog/kohsuke/
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Andrew Bayer]

This all brings up another question I had - what's the release plan for plugins? Are plugins only released when the owner of that plugin chooses? What if there's no active owner? Right now, we end up with situations where, for example, the build promotion plugin had a major bug in it that was fixed over a week ago, but there hasn't been a new release of that plugin yet. People are actually still submitting new bug reports of that same bug.

A.

On Wed, Mar 11, 2009 at 11:11 AM, Hafner Ullrich <[hidden email]> wrote:

I'm not a frequent committer to Hudson core but I would prefer using a release branch... E.g., I'm working on Hudson not on a daily basis: so if I have some time it would be sad if I could not check in my changes just because its end of the week.

Improve the Hudson core stability:
- Since we don't have a testing team the only way I see is to increase the test coverage.
- Maybe we can set up a team of users and plug-in developers that act as kind of beta tester: if a given number of group members confirms that the release works for their use cases then flag a release as stable and make it available via update center (otherwise you need to download manually).

Improve the interaction testing between Hudson and plugins:
- It would help if we could build AND deploy the plug-ins in a public Hudson. This should work at least for the reporting plug-ins. (This would additionally be quite usefule as a live-experience for new users, better than static Wiki pages:-)

Ulli

> -----Original Message-----
> From: Kohsuke Kawaguchi [mailto:[hidden email]]

> Sent: Dienstag, 10. März 2009 20:46
> To: [hidden email]

> Subject: Re: New release plan
>
> Dean Yu wrote:
> > Hi Kohsuke,
> >   Is there a reason you don't want to create a branch on Tuesday
> > evening instead of cutting off commits? It seems ironic
> that we don't
> > try to practice CI for building Hudson itself.
>
> Yeah, we should do that. I guess I could automate the branch
> handling on my side, and the merging shouldn't be too
> painful. changelog.html needs to be changed a bit so that we
> can list changes in N+2.
>
> And I think we do practice CI for building Hudson --- there's
> Hudson on Hudson as you can see on
> http://hudson.glassfish.org/job/hudson/
>
> But my feeling is that our current test coverage is just not
> high enough to catch all the problems this way.
>
>
> >   In the end you're our dictator and I'll go along with whatever you
> > decide. :)
>
> --
> Kohsuke Kawaguchi
> Sun Microsystems
> http://weblogs.java.net/blog/kohsuke/
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Alan Harder-2]

A lot of good feedback/ideas on this thread.  Giving my +1s to:

- Make compiling and testing all plugins part of the regular release
process (catch problems before release, and encourage plugin developers
to add unit tests to ensure continued compatibility of their plugins.. I
did!)

- Avoiding missed/delayed contributions because I happen to have some
time during a quiet period.

However, I'm not fond of tons of branches and a lot of merging work for
kohsuke either..
It does make sense that Hudson does CI releases, being a CI platform..
this part has never bothered me.  In fact, it is clear that the project
/relies/ on many people using releases of unknown stability in order to
find and report issues.  While an updated release process can definitely
improve stability/compatibility of releases, clearly many issues will
still be discovered by users.

So, I guess my preference is not to add branches/quiet period for every
release but rather a periodic focus on a more stable release... say,
once per month or maybe every 10 releases.  Create a release branch
prior to the next "stability focused" release and spend some time
looking at the issue tracker for possible improvements to be merged into
the branch.

That's my input into kohsuke's decision process.. of course I too will
be fine with whatever decision is made.  Just running plugin tests
before releases will be a great improvement.

    - Alan



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Kohsuke Kawaguchi]

BrandonB wrote:
> Whether the decision is to go with a "quiet period" or branches, I make the
> following suggestions:
>
> 1) Add code coverage checks (Cobertura or Clover) to the
> http://hudson.glassfish.org/job/hudson/ job, against the existing tests the
> Hudson core already has. This should identify weaknesses in coverage, and
> prompt the team/users to start contributing more tests.

Yes, I need to do this.

> 2) Before a Hudson core release is made, compile the core against the latest
> release of all plugins. Wherever possible, run the plugin unit tests against
> the new core. If, for whatever reason, the hudson.glassfish.org server
> cannot run plugin unit tests (eg, it might be hard to run the AccuRev plugin
> tests without AccuRev server or licenses), notify (automatically?) the
> plugin owners to run their unit tests against the about-to-be-released core.

+1.

> 3) Do any of the plugins have Hudson jobs on a public Hudson instance? If
> not, can they be added to hudson.glassfish.org? If so, can they be moved
> there?

Currently there's a single build that builds the entire Hudson including
all the plugins.

If we create one job per plugin, I need a new feature in Hudson to
manage that.

> 4) All plugins should have code coverage (Cobertura or Clover) checks
> against their unit tests.

Right, I believe we get this with the same effort to do your #1.

> 5) When the core release is made, publish a document (somewhere, wiki,
> release notes, part of the changelog, don't know) stating which versions of
> plugins were tested against the new core. I believe some of the plugins
> already document which versions of the core *they* were compiled/tested
> against when they are released - we need the reverse.

Isn't it better to just bake that information in Hudson itself or update
center metadata, so that the user gets a reassuring "tested" icon or
something on UC?


> 6) In fact, including the information from #5 in the 'Updates' and/or
> 'Available' tabs of the 'Manage Plugins' table would be extremely useful.
> (I'm envisioning another column stating something to the effect of "This
> version of Hudson was tested againstplugin release #X.Y.) This can help
> people identify whether they *should* upgrade plugins, instead of just
> crossing their fingers and upgrading blindly (or manually comparing release
> dates of core and plugins).

OK, so you've already thought about that, then.

--
Kohsuke Kawaguchi
Sun Microsystems                   http://weblogs.java.net/blog/kohsuke/

[John McNair]

On Tue, Mar 10, 2009 at 3:05 PM, Kohsuke Kawaguchi <[hidden email]> wrote:

As you know, lately we are suffering a quality problem in releases. There are multiple things we need to do to fix that, such as more tests and early regression testing with plugins, but it's also clear that we need to adjust the release process.

So this is the proposal to make that adjustment.

- Let's shoot for once-a-week release cycle, on Friday evening pacific
 time.

- We allow commits freely in Sat, Sun, Mon, and Tue. Tuesday evening
 is RC cut off.

- We reserve Wed, Thu, and Fri as the quiet period.
 We make commits to the core conservatively, and I'll use this period
 to deploy RC bits to my production Hudson to be a guinea pig.

I only have a handful of commits on Hudson to my credit, but I thought I'd share a little bit of experience that I don't see mentioned here.  Many people have offered good suggestions.  Increasing test coverage and its measurement is always a good idea.  More frequent CI is good.

But still there is the question of how to use the SCM to its best advantage.  Others have pointed out the problem with quiet periods.  A good SCM should never be off.  That is, developers should always be able to commit _somewhere_.  Someone else pointed out the potential pain with temporary release branches.  It can cause merging pain.

I haven't seen anyone mention the notion of topic branches.  That's git terminology I suppose, but git doesn't own the idea.  Bascially, all work should be done in small isolated branches.  One of the major sources of instability for a project like this is that you want developers to be able to commit frequently, but also be able to release frequently.  So what happens to all those half-implemented features or bug fixes.  Some features never even land at all.  So why not keep them off trunk until they are done?

New feature?  Branch.  New bug fix?  Branch.  Then code would only be shipped on purpose, i.e. when the developer believes it to be complete.  If necessary, you can still have a release branch created a couple of days before release.  If work is done in branches, then critical fixes can be merged to trunk and to the release branch while trunk only accepts less critical features.  If done this way, release branches never have to have a big bang merge back into trunk.

This allows the community to:
- Always have a place to commit
- Ship and prioritize code on purpose rather than whatever is a work in progress
- Deal with conflicts in more isolated settings
- Avoid big bang merges
- Always code against known stable trunk plus developer's own small changeset

Just some food for thought.  As I mentioned I haven't committed much code here, but perhaps I can spur some additional constructive debate.

--
John McNair
[hidden email]

[Daniel Lindner-2]

Hafner Ullrich wrote:
> I'm not a frequent committer to Hudson core but I would prefer using a release branch... E.g., I'm working on Hudson not on a daily basis: so if I have some time it would be sad if I could not check in my changes just because its end of the week.
>
> Improve the Hudson core stability:
> - Since we don't have a testing team the only way I see is to increase the test coverage.
> - Maybe we can set up a team of users and plug-in developers that act as kind of beta tester: if a given number of group members confirms that the release works for their use cases then flag a release as stable and make it available via update center (otherwise you need to download manually).
>  
I'm no committer to hudson core (yet), so please don't weight my opinion
as much as the active developer's ones.
But i really like the idea of "beta testers". I will happily play this
role with my few hudson installations. Perhaps it is possible to let
each hudson administrator choose in the update center which "flavor" of
hudson (stable, testing, bleeding edge - reminds me of debian a bit) he
wants to obtain. The ones that do beta tests run "testing". Really early
adopters run "bleeding edge". All others run "stable" and have the
confidence of double-checked stability, as the early and beta testers
run real-world hudsons just like them. The only difference between the
user groups would be that testers have earlier access to new features
but are capable of dealing with some bugs.
Perhaps this idea can be integrated with your anticipated branch management.

Sincerely,
Daniel


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]