Skip to main content

[Hudson-Dev] Re: Releasing ... there are many test failures

  • From: Winston Prakash <winston.prakash@...>
  • To: dev@...
  • Cc: Jason van Zyl <jason@...>
  • Subject: [Hudson-Dev] Re: Releasing ... there are many test failures
  • Date: Sat, 05 Feb 2011 12:20:31 -0800
  • Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type; b=U7y+ZLP0ST+wELIAhxGIfMiSpGko3c+Xv1wk3uG3/ICROj0xDuPBIxQuTs0VI8nFqv kKXLfsPGN9CZA2+rBMybBviT9MzZBPVeMNL58CU8tLpDenvj+0xDN1p2swIMZLI4BOOv KEzpnvDDjGr9/Y70xP/dnYwPpc2A0y4rZrh1Y=

Sounds good. May be we should think about

- Having minimal tests in each of the modules as sanity tests to facilitate incremental continuous builds after each check in. Build should finish with in minutes.

- Keep all tests with good code coverage in the Test harness module. The test harness module should be built along with full clean build of all the modules nightly. May take several hours for the build to finish.

Should we split the test harness module in to various modules corresponding to the source modules, that way it would be easy to pin point the modules for which the tests are failing.

If I'm correct, Maven 3 has about 8000 tests compared to 800 or so in Hudson. That is 10x more. We should think about adding lots more to Hudson over next several months to improve the stability and code coverage.

- Winston
I'm working on a newly created on release VM and I have test failures on Ubuntu and Redhat when run from the top-level. I don't think these are new so I'm not sure how previous releases were made, but I was thinking of putting the test suite in a separate profile for now or just running them separately like we do for Maven. I can't find much on the release process but running the tests that take an hour are more like integration tests to me and probably shouldn't be run as part of the release process. 

At any rate here are the failures I'm getting. They seem to be primarily with the Maven 3.x integration.

Ubuntu:

Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.474 sec

Results :

Failed tests: 
  testPageDecoratorIsListedInPage(hudson.pages.SystemConfigurationTestCase)
  testMaven3Unstable(hudson.maven.MavenBuildSurefireFailedTest)
  testTarGzMaven3(hudson.maven.RedeployPublisherTest)
  testTarGzUniqueVersionTrueMaven3(hudson.maven.RedeployPublisherTest)
  testMultiModMavenNonRecursive(hudson.maven.Maven3BuildTest)
  testSimpleMaven3Build(hudson.maven.Maven3BuildTest)
  testSimpleMaven3BuildRedeployPublisher(hudson.maven.Maven3BuildTest)
  testSiteBuildWithForkedMojo(hudson.maven.Maven3BuildTest)
  testMavenSeveralModulesInDirectory(hudson.maven.Maven3BuildTest)
  testMavenWithDependencyVersionInEnvVar(hudson.maven.Maven3BuildTest)

Tests run: 843, Failures: 10, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Hudson :: Main .................................... SUCCESS [2.243s]
[INFO] Hudson remoting layer ............................. SUCCESS [27.740s]
[INFO] Hudson CLI ........................................ SUCCESS [3.971s]
[INFO] Hudson core ....................................... SUCCESS [1:39.703s]
[INFO] Hudson Maven 2 PluginManager interceptor .......... SUCCESS [2.118s]
[INFO] Hudson Maven2 CLI agent ........................... SUCCESS [2.649s]
[INFO] Hudson Maven3 Interceptor ......................... SUCCESS [3.267s]
[INFO] Hudson Maven3 CLI Agent ........................... SUCCESS [1.688s]
[INFO] Maven Integration plugin .......................... SUCCESS [12.339s]
[INFO] Hudson war ........................................ SUCCESS [13.373s]
[INFO] Test harness for Hudson and plugins ............... FAILURE [55:03.463s]
[INFO] Hudson UI sample plugin ........................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 57:56.221s
[INFO] Finished at: Sat Feb 05 00:25:35 EST 2011
[INFO] Final Memory: 69M/989M
[INFO] ------------------------------------------------------------------------

Redhat:

Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 53.697 sec

Results :

Failed tests: 
  testMultiModMavenNonRecursive(hudson.maven.Maven3BuildTest)
  testSimpleMaven3Build(hudson.maven.Maven3BuildTest)
  testSimpleMaven3BuildRedeployPublisher(hudson.maven.Maven3BuildTest)
  testSiteBuildWithForkedMojo(hudson.maven.Maven3BuildTest)
  testMavenSeveralModulesInDirectory(hudson.maven.Maven3BuildTest)
  testMavenWithDependencyVersionInEnvVar(hudson.maven.Maven3BuildTest)
  testTarGzMaven3(hudson.maven.RedeployPublisherTest)
  testTarGzUniqueVersionTrueMaven3(hudson.maven.RedeployPublisherTest)
  testMaven3Unstable(hudson.maven.MavenBuildSurefireFailedTest)
  testPageDecoratorIsListedInPage(hudson.pages.SystemConfigurationTestCase)
  testFreeStyleAntOnSlave(hudson.tasks.EnvVarsInConfigTasksTest)
  test1(hudson.tasks._ant.AntTargetAnnotationTest)
  testBuildAxisInAnt(hudson.matrix.MatrixProjectTest)
  testAnt(hudson.tools.ToolLocationNodePropertyTest)

Tests in error: 
  testMulticast(hudson.UDPBroadcastThreadTest)

Tests run: 843, Failures: 14, Errors: 1, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Hudson :: Main .................................... SUCCESS [43.892s]
[INFO] Hudson remoting layer ............................. SUCCESS [1:02.841s]
[INFO] Hudson CLI ........................................ SUCCESS [27.932s]
[INFO] Hudson core ....................................... SUCCESS [2:23.262s]
[INFO] Hudson Maven 2 PluginManager interceptor .......... SUCCESS [13.517s]
[INFO] Hudson Maven2 CLI agent ........................... SUCCESS [1.565s]
[INFO] Hudson Maven3 Interceptor ......................... SUCCESS [32.697s]
[INFO] Hudson Maven3 CLI Agent ........................... SUCCESS [1.201s]
[INFO] Maven Integration plugin .......................... SUCCESS [26.235s]
[INFO] Hudson war ........................................ SUCCESS [23.193s]
[INFO] Test harness for Hudson and plugins ............... FAILURE [41:15.711s]
[INFO] Hudson UI sample plugin ........................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 49:09.077s
[INFO] Finished at: Sat Feb 05 11:00:10 CST 2011
[INFO] Final Memory: 62M/255M
[INFO] ------------------------------------------------------------------------


Thanks,

Jason

----------------------------------------------------------
Jason van Zyl
Founder,  Apache Maven
http://twitter.com/jvanzyl
---------------------------------------------------------

the course of true love never did run smooth ...

 -- Shakespeare






[Hudson-Dev] Releasing ... there are many test failures

Jason van Zyl 02/05/2011

[Hudson-Dev] Re: Releasing ... there are many test failures

Winston Prakash 02/05/2011

[Hudson-Dev] Re: Releasing ... there are many test failures

Jason van Zyl 02/05/2011

[Hudson-Dev] Re: Releasing ... there are many test failures

Winston Prakash 02/05/2011

[Hudson-Dev] Re: Releasing ... there are many test failures

Jerome Lacoste 02/05/2011

[Hudson-Dev] Re: Releasing ... there are many test failures

Winston Prakash 02/05/2011

[Hudson-Dev] Re: Releasing ... there are many test failures

Jerome Lacoste 02/06/2011

[Hudson-Dev] Re: Releasing ... there are many test failures

Jerome Lacoste 02/07/2011

[Hudson-Dev] Re: Releasing ... there are many test failures

Jason van Zyl 02/07/2011
 
 
Close
loading
Please Confirm
Close