G. Charwat, G. Ianni, T. Krennwallner, M. Kronegger, A. Pfandler, C. Redl, M. Schwengerer, L. Spendier, J. P. Wallner, G. Xiao:
"VCWC: A Versioning Competition Workflow Compiler";
Talk: 12th International Conference on Logic Programming and Nonmonotonic Reasoning (LPNMR 2013), Corunna, Spain; 2013-09-15 - 2013-09-19; in: "12th International Conference on Logic Programming and Nonmonotonic Reasoning (LPNMR 2013)", T. Son, P. Cabalar (ed.); Springer, 8148 (2013), 233 - 238.

System competitions evaluate solvers and compare state-of-the-art implementations on benchmark sets in a dedicated and controlled computing environment comprising of multiple hosts. An important task for running a competition is the benchmark execution platform that schedules the workload on available benchmark machines, keeps track of failed and finished jobs, and calculates the competition statistics and solver ranking. In this paper we present VCWC, the Versioning Competition Workflow Compiler. This tool takes as input the participating solvers and dedicated benchmark sets and generates a workflow description for executing all necessary (sub-)tasks for generating the final solver rankings and statistics. As jobs may fail during the execution, VCWC supports a gradual refinement of the competition workflow and allows to add or update solvers, instances, benchmarks, or further runs after the machinery has been brought up. We introduce an abstract model for a competition and present the implementation and system architecture for VCWC. Based on this we report how VCWC is used for the Answer Set Programming Competition 2013.

Benchmarks, Competition

Project Head Thomas Eiter:
Evaluierung von ASP Programming mit Externen Zugriffen

Project Head Thomas Eiter:
Modulare HEX-Programme