[Pellet-users] some pellet2 experiments: performance issues

[Evren Sirin]

Thank you very much for the detailed description of the problems and
tests you have run. We appreciate good use cases a lot. I've created a
ticket for these issues [1] and we'll investigate the problems you
described. Hopefully, the next RC will resolve these issues.

Cheers,
Evren

[1] http://cvsdude.com/trac/clark-parsia/pellet-devel/ticket/226


On Mon, Jan 12, 2009 at 5:53 AM, Wolfgang May
<may at informatik.uni-goettingen.de> wrote:
>
> Hi,
>
> by testing our more intricated examples (for the simple ones
> there were no measurable differences), I got the following
> results from pellet2 vs pellet1:
>
> * performance of pellet2 much slower than pellet1,
> * highly varying runtime for the same problem,
> * sometimes it does not terminate after long time.
>
> Downloaded versions from:
>  Version 31.Oct.07: (pellet1)
>  Version 30.Jan.08: (pellet1)
>  Version 1.Sept.08: (pellet1)
>  Version 13.Nov.08: (pellet2 RC3)
>  Version 6.Jan.09: (pellet2 RC4)
> We run pellet embedded in a Jena environment.
>
> Computer: Laptop 1400MHz, 1.5GB
>
> A zip with the sample files can be found at
>  http://user.informatik.uni-goettingen.de/~may/PelletTest/
>
> *** Example 1: "Big Cities"
> Database-style use case: simple TBox, large ABox,
> relatively simple query based on owl:Restrictions:
>
> (note: uncomment bigcities-owl11.n3 in the from line for older
>  versions)
>
> Version 31.Oct.07: (pellet1)
>     31min2, 38min, 38min
> Version 30.Jan.08: (pellet1)
>     32min, 37min, 39min
> Version 1.Sept.08: (pellet1)
>     567min, 572min, 574min, 578min
> Version 13.Nov.08: (pellet2 RC3): quite different results:
>     26min, 53min, 71min, 74min, 83min,
>     124min, 250min, 288min, 393min,
>     two times not terminated after >600min
> Version 6.Jan.09: (pellet2 RC4)
>     two times not terminated after 823min and 3620min.
>
>
> Tests on a second computer with 2600MHz and 4GB:
>
> Version 31.Oct.07: (pellet1)
>     20min50 ... 24min28 (6 runs)
> Version 30.Jan.08: (pellet1)
>     19min16 .. 23min41 (5 runs)
> Version 1.Sept.08: (pellet1)
>     227min, 360min, 361min, 369min, 370min, 374min, 379min
> Version 13.Nov.08: (pellet2 RC3): quite different results:
>     95min, 187min, 985min, 1224min, 6737min, 8092min
> Version 6.Jan.09: (pellet2 RC4)
>     124min, 2267min
>
> For this example the performace drop was already before the last
> version of pellet1.
>
> *** Example 2: "italian-professors"
> This example uses the same data, but with a different query.
> (The example has been motivated by an example ontology by Enrico
> Franconi).
> There is also a slower performance since the late versions of
> pellet1, but only +25%.
>
> Version 31.Oct.07: (pellet1)
>  14min22 ... 14min30 (5 runs)
> Version 30.Jan.08: (pellet1)
>  14min17 ... 14min44 (5 runs)
> Version 1.Sept.08: (pellet1)
>  18min13 ... 18min46 (5 runs)
> Version 13.Nov.08: (RC3)
>  18min55 ... 18min49 (5 runs), 2 runs in 18.78 and 20.05seconds(!)
> Version 6.Jan.09: (RC4)
>
>
> *** Example 3: "Fishpuzzle"
> Combinatorial puzzle; very small TBox, small ABox,
>  heavy reasoning.
>
> Version 31.Oct.07: (pellet1)
>  4min49, 5min17, 7min14, 8min45, 9min17, 10min55,
>  13min25, 32min17, 46min10, 138min24, 168min32
> Version 30.Jan.08: (pellet1)
>  10min21, 10min24, 13min33, 15min11,
>  23min49,29min44, 38min36,
>  142min12, 142min44, 143min14, 143min43
> Version 1.Sept.08: (pellet1)
>  2min50, 3min08, 4in17, 4min28, 4min36, 5min03, 7min01,
>  13min50, 202min24
> Version 13.Nov.08: (RC3)
>  2min47, 5min14, 5min49, 15min57, 16min17,
>  28min45, 30min11, 31min13, 36min07, 36min43,
>  39min36, 42min17, 42min27, 45min46,
>  58min32, 70min4, 79min52, 84min51
> Version 6.Jan.09: (RC4)
>  5min27, 5min35,
>  29min07, 29min39, 30min17, 34min24, 39min04,
>  67min57, 71min55, 75min58,
>  once it did not terminate after 690 minutes.
>
>
> Tests on a second computer with 2600MHz and 4GB:
>
> Version 1.Sept.08: (pellet1)
>  1min40, 1min45, 5min25, 5min27, 45min59, 79min38, 227min52
> Version 13.Nov.08: (RC3)
>  3min26, 6min37, 6min55,
>  12min58, 15min98, 18min34, 18min44
> Version 6.Jan.09: (RC4)
>  15min03, 18min42, 19min08
>
>
> Cheers,
> Wolfgang
> _______________________________________________
> Pellet-users mailing list
> Pellet-users at lists.owldl.com
> http://lists.owldl.com/mailman/listinfo/pellet-users
> _______________________________________________
>
> Sponsored by Clark & Parsia, LLC http://clarkparsia.com/
>