Benchmark Problems

ASU

Event Calculus Answer Set Programming

Benchmark Problems

We have compared the performance of different systems on 14 benchmark commonsense reasoning problems.

*Commuter includes events with duration and EC_dur.lp has to be used instead of EC.lp [DEC.lp].

**ThielscherCircuit includes causal constraints and ECCausal.lp has to be used with EC.lp [DEC.lp].

We notice that adding 'option renaming off' to the event calculus description reduces the numbers of variables and clauses, and
time spent when executed using the DEC reasoner.

Click other problems to see the results.

<A: number of atoms, C: number of clauses, R: number of ground rules>
Problem (max time)	DEC Reasoner	ECASP with	ECASP with	ECASP with
Problem (max time)	DEC Reasoner	LPARSE + CMODELS	GRINGO + CLASP(D)	CLINGO
BusRide (2)	Can't handle	0.04	0.02	Can't handle
BusRide2-ea.e	Disjunctive	(0.03+0.01)	(0.02+0.00)	Disjunctive
BusRide2-ea.lp	Events Axioms	A:29/R:216/C:31	A:83/R:115	Rules
BusRide (40)	Can't handle	7.84	0.48	Can't handle
BusRide40-ea.e	Disjunctive	(7.12+0.72)	(0.39+0.09)	Disjunctive
BusRide40-ea.lp	Events Axioms	A:709/R:130973/C:714	A:3408/R:51705	Rules
ChessBoard (2)	0.00	0.03	0.02	Can't handle Disjunctive Rules
ChessBoard2.e	(0.00+0.00)	(0.02+0.01)	(0.01+0.01)
ChessBoard2-ea.e	A:27/C:52	A:38/R:151/C:55	A:65/R:84
ChessBoard2-ea.lp	A:27/C:52	A:38/R:151/C:55	A:65/R:84
ChessBoard (40)	0.10	0.16	0.05	Can't handle Disjunctive Rules
ChessBoard40.e	(0.10+0.00)	(0.12+0.04)	(0.03+0.02)
ChessBoard40-ea.e	A:369/C:812	A:736/R:2595/C:1391	A:1483/R:1730
ChessBoard40-ea.lp	A:369/C:812	A:736/R:2595/C:1391	A:1483/R:1730
CoinToss (4)	0.00	0.03	0.02	0.01
CoinToss4.e	(0.00+0.00)	(0.02+0.01)	(0.01+0.01)
CoinToss4-ea.e	A:25/C:66	A:58/R:195/C:109	A:82/R:119
CoinToss4-ea.lp	A:25/C:66	A:58/R:195/C:109	A:82/R:119
CoinToss (40)	0.00	0.39	0.06	0.04
CoinToss40.e	(0.00+0.00)	(0.22+0.17)	(0.05+0.01)
CoinToss40-ea.e	A:205/C:606	A:1462/R:5019/C:3646	A:2782/R:4476
CoinToss40-ea.lp	A:205/C:606	A:1462/R:5019/C:3646	A:2782/R:4476
*Commuter (15)	Can't handle	493.55	44.67	28.91
Commuter15-ea.e	Compound	(446.34+47.21)	(37.92+6.75)
Commuter15-ea.lp	Events	A:4913/R:7383943/C:4952	A:24698/R:5381620
DeadOrAlive (3)	0.00	0.03	0.02	0.02
DeadOrAlive3.e	(0.00+0.00)	(0.02+0.01)	(0.01+0.01)
DeadOrAlive3-ea.e	A:36/C:76	A:125/R:188/C:0	A:70/R:98
DeadOrAlive3-ea.lp	A:36/C:76	A:125/R:188/C:0	A:70/R:98
DeadOrAlive (40)	0.30	0.22	0.06	0.05
DeadOrAlive40.e	(0.20+0.10)	(0.19+0.03)	(0.04+0.02)
DeadOrAlive40-ea.e	A:369/C:890	A:3066/R:4470/C:0	A:2533/R:3652
DeadOrAlive40-ea.lp	A:369/C:890	A:3066/R:4470/C:0	A:2533/R:3652
Happy (2)	0.00	0.09	0.02	0.02
Happy2.e	(0.00+0.00)	(0.03+0.06)	(0.02+0.00)
Happy2-ea.e	A:24/C:46	A:910/R:1557/C:0	A:38/R:63
Happy2-ea.lp	A:24/C:46	A:910/R:1557/C:0	A:38/R:63
Happy (40)	0.10	0.10	0.04	0.04
Happy40.e	(0.10+0.00)	(0.09+0.01)	(0.03+0.01)
Happy40-ea.e	A:328/C:730	A:1246/R:1893/C:0	A:754/R:1197
Happy40-ea.lp	A:328/C:730	A:1246/R:1893/C:0	A:754/R:1197
KitchenSink (5)	0.20	0.33	0.05	0.04
KitchenSink_M5.e	(0.20+0.00)	(0.30+0.03)	(0.04+0.01)
KitchenSink_M5-ea.e	A:102/C:593	A:1512/R:3816/C:0	A:1158/R:1828
KitchenSink_M5-ea.lp	A:102/C:593	A:1512/R:3816/C:0	A:1158/R:1828
KitchenSink (25)	71.10	43.40	2.47	2.04
KitchenSink_M25.e	(70.70+0.40)	(37.30+6.1)	(1.73+0.74)
KitchenSink_M25-ea.e	A:1014/C:12109	A:123452/R:482018/C:0	A:114968/R:179195
KitchenSink_M25-ea.lp	A:1014/C:12109	A:123452/R:482018/C:0	A:114968/R:179195
KitchenSink (40)	Ran over 1 hour	268.53	15.04	12.81
KitchenSink_M40-ea.e	and	(224.62+43.91)	(10.28+4.76)
KitchenSink_M40-ea.lp	didn't terminate	A:705747/R:2912813/C:0	A:676343/R:1042040
RussianTurkey (4)	0.00	0.03	0.02	0.01
RussianTurkey4.e	(0.00+0.00)	(0.02+0.01)	(0.01+0.01)
RussianTurkey4-ea.e	A:35/C:87	A:55/R:172/C:64	A:82/R:110
RussianTurkey4-ea.lp	A:35/C:87	A:55/R:172/C:64	A:82/R:110
RussianTurkey (40)	0.10	0.20	0.04	0.04
RussianTurkey40.e	(0.10+0.00)	(0.14+0.06)	(0.04+0.00)
RussianTurkey40-ea.e	A:287/C:807	A:1171/R:3349/C:1513	A:1963/R:2800
RussianTurkey40-ea.lp	A:287/C:807	A:1171/R:3349/C:1513	A:1963/R:2800
StolenCar (2)	0.00	0.03	0.02	0.01
StolenCar2.e	(0.00+0.00)	(0.02+0.01)	(0.01+0.01)
StolenCar2-ea.e	A:10/C:20	A:55/R:172/C:64	A:82/R:110
StolenCar2-ea.lp	A:10/C:20	A:55/R:172/C:64	A:82/R:110
StolenCar (40)	0.20	1.63	0.15	0.1
StolenCar40.e	(0.20+0.00)	(0.91+0.72)	(0.12+0.03)
StolenCar40-ea.e	A:162/C:1144	A:1651/R:23200/C:9854	A:1930/R:19027
StolenCar40-ea.lp	A:162/C:1144	A:1651/R:23200/C:9854	A:1930/R:19027
StuffyRoom (2)	0.00	0.03	0.02	0.02
StuffyRoom2.e	(0.00+0.00)	(0.02+0.01)	(0.02+0.00)
StuffyRoom2-ea.e	A:27/C:57	A:83/R:122/C:0	A:47/R:68
StuffyRoom2-ea.lp	A:27/C:57	A:83/R:122/C:0	A:47/R:68
StuffyRoom (40)	0.20	0.11	0.04	0.04
StuffyRoom40.e	(0.10+0.10)	(0.10+0.01)	(0.04+0.00)
StuffyRoom40-ea.e	A:369/C:931	A:1956/R:2603/C:0	A:1464/R:1902
StuffyRoom40-ea.lp	A:369/C:931	A:1956/R:2603/C:0	A:1464/R:1902
Supermarket (12)	0.00	0.05	0.03	0.03
Supermarket12.e	(0.00+0.00)	(0.05+0.00)	(0.03+0.00)
Supermarket12-ea.e	A:104/C:752	A:471/R:620/C:0	A:315/R:458
Supermarket12-ea.lp	A:104/C:752	A:471/R:620/C:0	A:315/R:458
Supermarket (40)	0.10	0.14	0.05	0.05
Supermarket40.e	(0.10+0.00)	(0.12+0.02)	(0.04+0.01)
Supermarket40-ea.e	A:328/C:2488	A:2461/R:2946/C:0	A:1969/R:2448
Supermarket40-ea.lp	A:328/C:2488	A:2461/R:2946/C:0	A:1969/R:2448
**ThielscherCircuit (1)	0.20	0.05	0.03	0.03
ThielscherCircuit1.e	(0.20+0.00)	(0.05+0.00)	(0.03+0.00)
ThielscherCircuit1-ea.e	A:68/C:1114	A:346/R:569/C:0	A:108/R:140
ThielscherCircuit1-ea.lp	A:68/C:1114	A:346/R:569/C:0	A:108/R:140
**ThielscherCircuit (20)	5.30	0.55	0.09	0.07
ThielscherCircuit20.e	(5.00+0.30)	(0.49+0.06)	(0.07+0.02)
ThielscherCircuit20-ea.e	A:714/C:21254	A:4336/R:11399/C:0	A:1743/R:5669
ThielscherCircuit20-ea.lp	A:714/C:21254	A:4336/R:11399/C:0	A:1743/R:5669
**ThielscherCircuit (40)	11.10	2.58	0.33	0.22
ThielscherCircuit40.e	(10.50+0.60)	(2.28+0.30)	(0.24+0.09)
ThielscherCircuit40-ea.e	A:1394/C:42454	A:10096/R:54519/C:0	A:4812/R:35458
ThielscherCircuit40-ea.lp	A:1394/C:42454	A:10096/R:54519/C:0	A:4812/R:35458
WalkingTurkey (3)	Can't handle	0.03	0.02	0.02
WalkingTurkey3-ea.e	Effect	(0.02+0.01)	(0.01+0.01)
WalkingTurkey3-ea.lp	Constraints	A:112/R:150/C:0	A:64/R:96
WalkingTurkey (40)	Can't handle	0.13	0.04	0.03
WalkingTurkey40-ea.e	Effect	(0.12+0.01)	(0.04+0.00)
WalkingTurkey40-ea.lp	Constraints	A:2156/R:2527/C:0	A:1664/R:2029
Yale (3)	0.00	0.02	0.01	0.01
Yale3.e	(0.00+0.00)	(0.02+0.00)	(0.01+0.00)
Yale3-ea.e	A:28/C:64	A:91/R:127/C:0	A:56/R:78
Yale3-ea.lp	A:28/C:64	A:91/R:127/C:0	A:56/R:78
Yale (40)	0.00	0.15	0.04	0.03
Yale40.e	(0.00+0.00)	(0.13+0.02)	(0.03+0.01)
Yale40-ea.e	A:287/C:767	A:2201/R:3057/C:0	A:1872/R:2613
Yale40-ea.lp	A:287/C:767	A:2201/R:3057/C:0	A:1872/R:2613