VTK/SoftwareQuality/NightlyExpectedBuildTestFailures: Difference between revisions

From KitwarePublic
< VTK
Jump to navigationJump to search
Line 53: Line 53:
[http://open.cdash.org/testSummary.php?project=11&date=2013-02-23&name=vtkInteractionWidgetsCxx-TestContourWidget2 vtkInteractionWidgetsCxx-TestContourWidget2]
[http://open.cdash.org/testSummary.php?project=11&date=2013-02-23&name=vtkInteractionWidgetsCxx-TestContourWidget2 vtkInteractionWidgetsCxx-TestContourWidget2]
| 1
| 1
| [http://open.cdash.org/testDetails.php?test=178390042&build=2823952 Small image regression]
| [http://open.cdash.org/testDetails.php?test=178390042&build=2823952 Regression defect, small]
| Unassigned
| Unassigned
|-
|-
Line 59: Line 59:
[http://open.cdash.org/testSummary.php?project=11&date=2013-02-23&name=vtkRenderingOpenGLCxx-TestGlyph3DMapperOrientationArray vtkRenderingOpenGLCxx-TestGlyph3DMapperOrientationArray]
[http://open.cdash.org/testSummary.php?project=11&date=2013-02-23&name=vtkRenderingOpenGLCxx-TestGlyph3DMapperOrientationArray vtkRenderingOpenGLCxx-TestGlyph3DMapperOrientationArray]
| 1
| 1
| AIX only [http://open.cdash.org/testDetails.php?test=178422932&build=2824162 Image regression]
| Regression defect [http://open.cdash.org/testDetails.php?test=178422932&build=2824162 AIX only]
| Unassigned
| Unassigned
|-
|-
Line 71: Line 71:
[http://open.cdash.org/testSummary.php?project=11&date=2013-02-23&name=vtkRenderingVolumePython-TestFixedPointRayCasterNearestCropped vtkRenderingVolumePython-TestFixedPointRayCasterNearestCropped]
[http://open.cdash.org/testSummary.php?project=11&date=2013-02-23&name=vtkRenderingVolumePython-TestFixedPointRayCasterNearestCropped vtkRenderingVolumePython-TestFixedPointRayCasterNearestCropped]
| 1
| 1
| [http://open.cdash.org/testDetails.php?test=178384129&build=2823951 Image regression]
| [http://open.cdash.org/testDetails.php?test=178384129&build=2823951 Regression defect]
AIX exit
AIX exit
| Unassigned
| Unassigned
Line 90: Line 90:
[http://open.cdash.org/testSummary.php?project=11&date=2013-02-23&name=vtkInfovisLayoutCxx-TestCirclePackLayoutStrategy vtkInfovisLayoutCxx-TestCirclePackLayoutStrategy]
[http://open.cdash.org/testSummary.php?project=11&date=2013-02-23&name=vtkInfovisLayoutCxx-TestCirclePackLayoutStrategy vtkInfovisLayoutCxx-TestCirclePackLayoutStrategy]
| 1
| 1
| [http://open.cdash.org/testDetails.php?test=178377163&build=2823849 Major regression failure]
| [http://open.cdash.org/testDetails.php?test=178377163&build=2823849 Regression defect, major]
| Unassigned
| Unassigned
|-
|-
Line 108: Line 108:
[http://open.cdash.org/testSummary.php?project=11&date=2013-02-23&name=vtkRenderingOpenGLCxx-TestTextureRGBADepthPeeling vtkRenderingOpenGLCxx-TestTextureRGBADepthPeeling]
[http://open.cdash.org/testSummary.php?project=11&date=2013-02-23&name=vtkRenderingOpenGLCxx-TestTextureRGBADepthPeeling vtkRenderingOpenGLCxx-TestTextureRGBADepthPeeling]
| 1
| 1
| [http://open.cdash.org/testDetails.php?test=178390201&build=2823952 Intermittent regression failure]
| [http://open.cdash.org/testDetails.php?test=178390201&build=2823952 Regression defect, Intermittent]
| Unassigned
| Unassigned
|-
|-

Revision as of 16:36, 24 February 2013

The VTK Nightly Expected builds show the status of VTK on VTK supported platforms at a fixed time each day. Although the Gerrit Review Process and Continuous Builds attempt to catch build defects early, it is the Nightly Expected builds that show the health of VTK on a broader collection of systems. Recurring nightly failures make it difficult for a developer to see if her changes have affected the spectrum of platforms.

This experiment uses the DMAIC methodology of the Six Sigma management process to "Define", "Measure", "Analyze", "Improve" and "Control" to resolve these issues.

The basic methodology (from Wikipedia) consists of the following five steps:

  • Define process goals that are consistent with customer demands and VTK's strategy.
  • Measure key aspects of the current process and collect relevant data.
  • Analyze the data to verify cause-and-effect relationships. Determine what the relationships are, and attempt to ensure that all factors have been considered.
  • Improve or optimize the process.
  • Control to ensure that any deviations from target are corrected before they result in defects. Set up pilot runs to establish software quality, move on to production, set up control mechanisms and continuously monitor the process.

Define

Keep the number of VTK Nightly Expected recurring test failures(defects) to 0. When the defects are above 0, developers find it difficult to notice that their changes introduce new defects.

Measure

Recent summaries:

2013-02-19

2013-02-20

2013-02-21

2013-02-23

Analyze

Test Failing Defect Assigned To

vtkFiltersParallelStatisticsCxx-MPI-TestRandomPMomentStatisticsMPI

1 Memory Leak Unassigned

vtkRenderingVolumePython-volTM3DCropRegions

1 Failing since Feb 4 Unassigned

vtkFiltersGeometryCxx-TestImageDataToUniformGrid

1 AIX only Unassigned

vtkInteractionWidgetsCxx-TestContourWidget2

1 Regression defect, small Unassigned

vtkRenderingOpenGLCxx-TestGlyph3DMapperOrientationArray

1 Regression defect AIX only Unassigned

vtkParallelMPIPython-TestPProbe

1 Python syntax Unassigned Existing gerrit topic

vtkRenderingVolumePython-TestFixedPointRayCasterNearestCropped

1 Regression defect

AIX exit

Unassigned

vtkRenderingParallelCxx-MPI-TestClientServerRendering

1 Failed to connect Unassigned

JavaDeleteRegression

1 Unknown Unassigned

vtkInfovisLayoutCxx-TestCirclePackLayoutStrategy

1 Regression defect, major Unassigned

JavaDeleteAndGC

1 Unknown Unassigned

vtkInteractionStylePython-TestStyleTrackballCamera

1 AIX Major regression failure Unassigned

vtkRenderingOpenGLCxx-TestTextureRGBADepthPeeling

1 Regression defect, Intermittent Unassigned

vtkInteractionWidgetsCxx-TestPickingManagerWidgets

1 Unknown Unassigned

vtkRenderingVolumePython-TestFixedPointRayCasterLinearCropped

1 Unknown Unassigned

vtkRenderingOpenGLCxx-TestTextActor3DAlphaBlending

1 Unknown Unassigned

vtkRenderingVolumeCxx-TestHAVSVolumeMapper

1 Unknown Unassigned

vtkRenderingOpenGLCxx-TestGlyph3DMapperMasking

1 Unknown Unassigned

vtkInteractionWidgetsCxx-TestImplicitPlaneWidget2b

1 Unknown Unassigned

vtkIONetCDFCxx-SLACMultipleModes

1 Unknown Unassigned

vtkFiltersProgrammablePython-progGlyphsBySource

1 Unknown Unassigned

vtkRenderingOpenGLCxx-TestTexturedBackground

1 Unknown Unassigned

JavaManualGC

1 Unknown Unassigned

vtkInteractionStylePython-TestStyleJoystickCamera

1 Unknown Unassigned

vtkChartsCorePython-TestLinePlotColors

1 Unknown Unassigned

vtkRenderingVolumeTcl-volTM3DCompressedCropRegions

1 Unknown Unassigned

vtkRenderingVolumePython-volTM3DCompressedCropRegions

1 Unknown Unassigned

vtkIOGeometryCxx-TestWindBladeReader

1 Unknown Unassigned

vtkRenderingOpenGLCxx-TestOpacity

1 Unknown Unassigned

vtkFiltersGeneralCxx-TestGradientAndVorticity

1 Unknown Unassigned

vtkRenderingVolumePython-TestFixedPointRayCasterNearest

1 Unknown Unassigned

vtkImagingCorePython-TestCheckerboard

1 Unknown Unassigned

vtkIOExportCxx-TestGL2PSExporterRaster-VerifyRasterizedPNG

1 Unknown Unassigned

vtkFiltersGeometryCxx-TestStructuredGridConnectivity

1 Unknown Unassigned

vtkInteractionWidgetsPython-TestInteractorEventRecorder

1 Unknown Unassigned

vtkRenderingVolumePython-TestFixedPointRayCasterLinear

1 Unknown Unassigned

vtkInteractionStylePython-TestStyleTrackballActor

1 Unknown Unassigned

vtkInteractionWidgetsCxx-TestTextWidget

1 Unknown Unassigned

JavaGCRegression

1 Unknown Unassigned

vtkRenderingParallelCxx-MPI-TestParallelRendering

1 Unknown Unassigned

vtkFiltersGeneralCxx-TestIntersectionPolyDataFilter2

1 Unknown Unassigned

vtkInteractionStylePython-TestStyleJoystickActor

1 Unknown Unassigned

JavaRegression

1 Unknown Unassigned

vtkChartsCoreCxx-TestScatterPlotMatrix

1 Unknown Unassigned

vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DGeometryMaterial

2 Unknown Unassigned

vtkRenderingCorePython-TestOpacity2

2 Unknown Unassigned

vtkRenderingVolumeCxx-TestProjectedHexahedra

2 Unknown Unassigned

vtkFiltersSelectionCxx-TestLinearSelector3D

2 Unknown Unassigned

vtkFiltersHybridTcl-3DMorph

2 Unknown Unassigned

vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DAxisCut

2 Unknown Unassigned

vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DUnstructured

2 Unknown Unassigned

vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DClip

2 Unknown Unassigned

vtkRenderingVolumePython-volTM2DRotateClip

2 Unknown Unassigned

vtkFiltersHyperTreeCxx-TestHyperTreeGridTernarySphereMaterial

2 Unknown Unassigned

vtkRenderingOpenGLCxx-TestGaussianBlurPass

2 Unknown Unassigned

vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DGeometry

2 Unknown Unassigned

vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DContourMaterial

2 Unknown Unassigned

vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DUnstructuredMaterial

2 Unknown Unassigned

vtkFiltersGeneralCxx-TestDensifyPolyData

2 Unknown Unassigned

vtkRenderingCoreTcl-TestOpacity2

2 Unknown Unassigned

vtkIOMPIParallelCxx-TestPWindBladeReader

2 Unknown Unassigned

vtkGUISupportQtCxx-TestQtDebugLeaksView

2 Unknown Unassigned

vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DCut

2 Unknown Unassigned

vtkRenderingOpenGLCxx-TestTranslucentLUTTextureDepthPeeling

2 Unknown Unassigned

vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DAxisCutMaterial

2 Unknown Unassigned

vtkFiltersHybridPython-3DMorph

2 Unknown Unassigned

vtkRenderingCoreTcl-CamBlur

2 Unknown Unassigned

vtkRenderingVolumeTcl-TestFixedPointRayCasterLinearCropped

2 Unknown Unassigned

vtkRenderingOpenGLCxx-TestTranslucentLUTDepthPeelingPass

2 Unknown Unassigned

vtkFiltersFlowPathsCxx-TestBSPTree

2 Unknown Unassigned

vtkRenderingOpenGLCxx-TestTranslucentLUTDepthPeeling

3 Unknown Unassigned

vtkChartsCoreCxx-TestLinePlot3D

3 Regression defects Unassigned

vtkIOExportCxx-TestGL2PSExporterVolumeRaster-VerifyRasterizedPNG

3 Regression defects that differ in various machines Unassigned

vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DContour

4 Unknown Unassigned

vtkCommonCoreTcl-TestSetGet

5 Timeout Unassigned

Improve

The following Gerrit reviews addressed the defects.

Control

Once reduced to 0, developer diligence is needed to keep the defects to 0.