japex
  1. japex
  2. JAPEX-21

more statistical values (median, percentiles)

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: current
    • Fix Version/s: milestone 1
    • Component/s: renderer
    • Labels:
      None
    • Environment:

      Operating System: All
      Platform: Macintosh

    • Issuezilla Id:
      21

      Description

      Right now japex only reports/collects the total time taken and the number of rounds and means
      calculated. For non-trivial usecases percentiles (e.g. the median) are of interest as well. If one single run
      took 91 ms and the other 9 take 1 second, the arithmetric mean will be 10 and the geometric mean 1.

      An IMHO easier to understand value would be the 90% value, that is the time in which 90% of the runs did
      complete (1).

        Activity

        Hide
        Santiago Pericas-Geertsen added a comment -

        Thanks for the feedback. Japex currently computes two types of means, what I call vertical and horizontal
        means. Horizontal means are computed across all the test cases in your test suite, while vertical means
        are computed across complete runs (when japex.runsPerDriver > 1). Which means are you referring to in
        your description?

        As for the case of a single run taking 91 ms and all others taking 1 s (1000 ms), if that is the case, I'd say
        the system isn't in steady state and other parameters (such as warmups) need to be adjusted first. But
        perhaps your use is different and just need to understand it better.

        Show
        Santiago Pericas-Geertsen added a comment - Thanks for the feedback. Japex currently computes two types of means, what I call vertical and horizontal means. Horizontal means are computed across all the test cases in your test suite, while vertical means are computed across complete runs (when japex.runsPerDriver > 1). Which means are you referring to in your description? As for the case of a single run taking 91 ms and all others taking 1 s (1000 ms), if that is the case, I'd say the system isn't in steady state and other parameters (such as warmups) need to be adjusted first. But perhaps your use is different and just need to understand it better.
        Hide
        Mirko Friedenhagen added a comment -

        I have to confess that I am abusing japex a litte as I try to run integrative tests with it. This is where the
        much higher deviations in my measurements come from. With its tight integration into Hudson, it would
        be nice to have percentiles in japex.

        Show
        Mirko Friedenhagen added a comment - I have to confess that I am abusing japex a litte as I try to run integrative tests with it. This is where the much higher deviations in my measurements come from. With its tight integration into Hudson, it would be nice to have percentiles in japex.

          People

          • Assignee:
            Santiago Pericas-Geertsen
            Reporter:
            Mirko Friedenhagen
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated: