Capability Matching

The benchmark

In Metrics-Based Comparison of OWL and XML for Representing and Querying Cognitive Radio Capabilities (Applied Sciences, 2022), the same team built both stacks and ran them head to head. On the semantic side: DeVISor, our semantic matcher, over an Apache Jena Fuseki triple store, with BaseVISor doing the inference — “a forward-chaining inference engine specialized to handle facts in the form of RDF triples with support for OWL 2 RL and XML Schema Datatypes.” On the XML side: BaseX, an XML database with an XQuery 3.1 processor. Device descriptions were generated synthetically in RDF/XML — which is valid XML and valid RDF, so literally the same files fed both stacks — across datasets of 1000, 2000, 3000 and 4000 devices, with four batches of 5000 queries. The OWL side was then run not once but six times, against a descending chain of ontologies from which defining axioms were progressively stripped.

The result: a recall story, not a precision story

The XML/XQuery matcher returned results that were “approximately 34% less complete” than the OWL-based matcher running the richest test ontology. Concretely: at 4000 devices, across 5000 queries, its average recall was 65.75%.

In operational terms: roughly one qualifying radio in three is never found. Not misidentified — never found. And that distinction is the point, because precision was above 93% for every approach tested, including XML. The XML matcher does not return the wrong radios; it silently fails to return the right ones. Anyone who tells you semantics makes matching “more accurate” has not read the numbers. It makes matching more complete, and in a spectrum-sharing network the radios you never found are the capacity you never used.

The single most useful control in the study is the last rung of the ladder. Take the OWL ontology and strip all of its axioms out — leave the vocabulary and nothing else — and the OWL stack's results become exactly identical to XML's. The authors' conclusion is the one sentence we would put on our own door: “shallow use of OWL does not buy much in terms of the quality of matching.” The semantics is the difference. An ontology that is only a taxonomy is an XML schema wearing a hat.

And what it costs — which we state, because the paper does

You do not get completeness for free, and pretending otherwise is how a technical evaluation is lost.

• OWL loses on device-registration time in every single configuration tested. Registering a device into the semantic store also means running the inference engine and inserting the inferred facts, so the ingest path is strictly more expensive — and it gets more expensive the richer the ontology gets. In the authors' words: “the registration process in the XML-based approach takes less time, regardless of the number of devices.” The reasoning is paid at ingest. That is a design choice, not an accident: you pay once, when a radio joins, so that every subsequent query is complete.
• We do not claim SPARQL is faster than XQuery. The OWL side did return queries faster in this study, but the authors refuse to attribute that to the language, and so do we: “we cannot conclude that XQuery is less efficient than SPARQL. Apart from the query language, the query response time also depends on other factors, such as the query processor, query optimization, dataset structure, etc.” It is a result about DeVISor and BaseX, not about SPARQL and XQuery.
• The ground truth is the OWL matcher's own output under the strongest ontology, B0 — a limitation the authors disclose plainly, along with their reason: “it would be better to have an independent source of ground truth, the simple fact is that such a source does not exist.” So this is not an independent adversarial head-to-head, and we will never present it as one. What it measures rigorously is how much of a formally-defined expected answer set each approach recovers.
• The data is synthetic and the scale is a laptop. Device descriptions and queries were machine-generated (real capability data is not shareable, for privacy and security reasons), the largest dataset is 4000 devices, and everything ran on one MacBook Pro. This is not a field result and we do not dress it up as one.

The honest headline is the authors' own: the results “demonstrate the benefits of using a more specific ontology to improve the quality of the matching results at the cost of sacrificing some level of performance.” That trade — completeness bought with reasoning time, and only realized if the ontology is genuinely rich — is exactly the trade we are in business to engineer. The evaluation source code is public, at github.com/YankeeChen/evaluator; the raw metric results are in the repository. Check us.