Standards Conformance Testing

Our Approach: Lift the Data, Then Check Meaning

The architecture has been the same since the first paper, and it has two moves. First, lift the data: automatically convert the vendor XML into a language that has formal semantics, so a machine can reason about what it means rather than only how it is shaped. Then check meaning with declarative rules held as data, separate from the validator’s code — so one engine enforces any rule, and a new rule is a new file, not a new build.

The lift: XML to OWL, with a generic lifter

“The process of semantic lifting, is the process of converting non-semantic data (e.g. XML) to a language with formal semantics (e.g. OWL).” (The comma splice is the paper’s.) The payoff is stated in one line: “XML data can be automatically lifted to OWL and then processed using standards-based inference engines, which can detect conflicts not detectable at the XML level.” And the choice of OWL is not aesthetic — “Using a language with formal semantics guarantees that inference is sound and complete, and with OWL, tractable.”

The lifter VIStology built is deliberately standard-agnostic. Three off-the-shelf converters were tried first and none produced a usable ontology for TMATS or MDL, so the team wrote its own: “The lifter is entirely generic, i.e. it can work with any XML schema and data, thus can be applied to any T&E XML-based standard.” That genericity is the point: a new revision of a standard means re-running the converter, not rewriting an interpreter.

The rules, and the honest history of the engine underneath them

What runs over the lifted graph is a set of declarative constraints, and here the record spans two engines across two eras — a distinction we keep because a reviewer with the PDFs will keep it too.

ITC’15 (Phase I). The constraints were authored as rules in VIStology’s own rule language and executed by VIStology’s reasoner: “The constraint validation rules were developed in the BVR language supported by the BaseVISor inference engine”. That prototype was named xVISor.
TACL (Phase II). The later work moved the constraint substrate onto an open W3C standard — SHACL, with SHACL-SPARQL and OWL — still executed by xVISor. xVISor is not BaseVISor, and the TACL work does not run on BVR; in the TACL paper, SHACL appears dozens of times and BaseVISor and BVR appear not at all. We keep the two straight.

The constant across both eras is the architecture, not the engine: lift the data to formal semantics, express the standard’s rules declaratively, and keep them out of the validator’s code. The paper states the payoff positively — “Declarative rules that are supported by this technology have formal semantics, yet they are not hardcoded and can be altered without making any changes to the validation software” — which “promotes a loose coupling between the T&E standardization efforts and the development of the validation software.” That the later substrate is an open standard only sharpens the claim: the whole stack is standards-based, top to bottom.

Two honesties belong right here. The ITC’15 result is feasibility, not performance — it “showed that the proposed approach is feasible”, demonstrated by a prototype “successfully tested with MDL and TMATS data”, with no benchmark of any kind. And the demonstrated work touched TMATS and MDL only; DDML and IHAL are named as IRIG context, never validated.

TACL: A Constraint Language a Person Can Read

If a standard’s rules are going to live as data, someone has to be able to write them. That is what TACL is for. In the ITC’18 paper’s one-sentence definition, “we present TACL, a T&E extension to W3C Shape Constraints Language (SHACL) for formulating constraints on configurations represented in MDL and TMATS, independently of any configuration software.” The acronym is the paper’s own pun: “we are proposing T&E Extension for SHACL, or TACL”.

The choice of SHACL is deliberate, and it is the standards story in miniature. SHACL is a real, adopted W3C standard for validating RDF graphs, and XSD is strictly weaker than it: “All XSD constraints can be expressed in SHACL, but many SHACL constraints cannot be expressed in XSD because the latter is a lot less expressive.” The paper shows the gap on a single cross-field rule: “Note that :Shape2 involves relating the values of two different properties, which cannot be expressed in a language like XML Schema.” TACL then rides on SHACL exactly the way telemetry XML rides on XML: “TACL bears a similar relationship to SHACL as MDL does to XML — a customization of a standard language to a specific domain that can be processed with the tools developed for the core language.”

Why a constraint language beats a hand-coded tree-walk

The concrete argument for TACL is resilience to schema change, and the paper makes it by putting the same constraint two ways. Written directly against the XML tree — in XPath — a rule is a long, brittle scope expression that navigates the document structure; when the schema moves, the expression breaks. Written in TACL, the same rule refers to meaning — a measurement belonging to a network node named DAU1 — not to tree position, so a schema revision leaves it untouched. The paper’s claim for it is measured in words, not numbers: “It exhibits much better resilience to changes in the XML schemas than the languages that refer directly to the XML trees.” The complex SHACL-SPARQL that makes this work is hidden behind readable, reusable component names, so an engineer can “utilize the power of OWL inference to introduce a high-level of abstraction in constraint definitions” without ever seeing the machinery.

Figure 1. The same rule, two ways. One constraint from the TACL paper — the device named DAU1 may only sample at powers of two — encoded first in XPath (a deep tree-walk that breaks whenever the schema moves, with the paper’s own three objections beneath it) and then in TACL (a SHACL shape that targets the meaning — a measurement of the node named DAU1 — and so survives a schema revision). That contrast is the case for a constraint language. TACL is a proposed W3C SHACL extension executed by xVISor (not BaseVISor); it won ITC’s telemetry-standards paper award but is not itself an adopted standard.

What TACL is, and — just as important — what it is not

TACL is a proposal. Not a standard, not an adopted one, not “the” T&E constraint language. The paper says so in the plainest possible terms, and we quote it wherever the word “standard” appears near TACL on this page:

“Note that TACL is just a proposal and that it requires more involvement from the wider community before it can be subject to standardization. The primary purpose of this work is to convince the community of its value and to show how this approach would benefit all stakeholders.”

Everything around it is consistent with “proposal, early days.” No requirements document even existed yet: “While the T&E community largely agrees there is a need for a constraint language, no formal document has been published to date to formally specify the requirements for such language.” The nearest standards-body activity was a requirements-gathering task the paper expected to fall short: “There is an ongoing RCC-TG task with the purpose of collecting such requirements from various US military bases, although it will not address the vendor constraints, and undoubtedly be incomplete.” And uptake was thin, in the authors’ own words: “Being fully aware of this challenge, VIStology and SwRI have been encouraging various stakeholders to participate in the process of requirements elicitation, with a rather limited success so far.” SHACL is the W3C standard here; TACL is the proposed T&E extension to it, and none of SHACL’s standing transfers to TACL.

One credential is real, and worth stating with its limit attached. The ITC’18 TACL paper won the Lawrence Rauch Telemetry Standards Paper Award — per telemetry.org (the International Foundation for Telemetering), not the paper itself. It is an award for a paper. It is not standardization, and we do not let it stand in for adoption.

Its empirical content is a proof of concept, and no more: “A proof of concept TACL engine has been successfully developed and applied to MDL/TACL configurations”, where “All of the constraints described in this paper were implemented and tested against input XML data, including the MDL examples that are bundled with the schema.” The engine is named once, and it is not BaseVISor: “The tests were performed with xVISor, VIStology’s TACL constraints validation engine, which includes a semantic lifter that automatically converts the XML input to RDF/OWL.”

What We Did Not Measure

A technical evaluator asks one question about a page like this inside a minute: how fast is it, and how far does it scale? The honest answer is that we do not know, and neither do the papers.

Not one of the four sources behind this page reports an evaluation of any kind. No timing. No throughput. No scale study. No accuracy, precision or recall. No baseline comparison. No benchmark. We would rather say so plainly than have a reviewer open the PDFs and say it for us.

The only performance content that exists anywhere is qualitative, and we quote it with its hedges intact:

ITC’19, on the notification path: “The processing time of the remaining four notifications does not depend on the size of the configuration, and in our experiments, met the user expectations”, and “changes made in the UI were almost instantly reflected by validation messages.” A user impression, not a measurement.
The ITEA’19 deck’s single performance line is a hallway verdict from a demo audience: “Good performance (according to the audience).” The parenthetical is load-bearing, and it always travels with the phrase.

So the claim is exactly the one the authors make: feasibility, and proof of concept. ITC’15 “showed that the proposed approach is feasible”; TACL was demonstrated as “A proof of concept TACL engine”. The scaling contrast between re-export and notifications is an argument — import/export “does not scale”, notifications are “expected” to — not a measured result. TACL and the Validation Protocol are proposals, not adopted standards. And xVISor is an SBIR research prototype — not a product, not deployed, not fielded.

The architecture claim, though, does not need a stopwatch. That a standard’s rules can be authored in a readable language, held as data, executed by a general engine, and revised without rebuilding the validator — that is settled by construction. What a benchmark would tell you is whether this particular prototype is fast enough for your configurations, and that is a number we would rather go and measure on your data than quote you one we never took.

Standards: What We Did, and What We Did Not

T&E is a regulated, government-maintained domain, and in that domain the provenance of a claim matters as much as the claim. So here is the record, fenced on both sides.

What we did

We engaged the standards community for years, and in the open. VIStology presented its Air Force SBIR work — semantic validation, and the proposed TACL constraint language — to the RCC Telemetry Group’s data-multiplex committee across nine meetings from 2014 to 2019, took part in the iNET MDL user community, and published three peer-reviewed ITC papers plus a workshop presentation, every one cleared for public release. We proposed two artifacts to that community: TACL, a backward-compatible extension of the W3C SHACL standard, and a Validation Protocol for the engine-to-manager interface. In the ITC’18 paper’s own framing of the endgame: “we are proposing that only TACL is standardized, allowing vendors to develop their own user-friendly interfaces to interact with their users, but to require that TACL is used in any exchange crossing their systems’ boundaries.”

What we did not

We did not author IRIG 106, and we do not claim it. IRIG 106 — and its XML components TMATS, DDML and IHAL — is the Range Commanders Council Telemetry Group’s standard; VIStology built a validation layer on top of it, not the standard itself. On MDL, we keep the paper’s own tense: MDL is only “anticipated to be incorporated into IRIG”, not yet part of it. The demonstrated work touched TMATS and MDL only — DDML and IHAL are named as context, never validated.

We were an invited SBIR presenter and participant at the RCC Telemetry Group, not a member — the members are the U.S. government ranges — and we did not chair or drive its committees, which are government-chaired, and in which a competing SBIR contractor (KBSI) was pursuing the validation-handbook work alongside us. Nine registration forms are evidence of attendance, not of leadership.

And TACL and the Validation Protocol are proposals, full stop — not adopted standards, not IRIG-approved, with no RCC document number. The TACL paper says it “requires more involvement from the wider community before it can be subject to standardization”, and the Validation Protocol is something that “should be standardized within the RCC community.” If you have read a T&E capability statement that blurs any of these lines — authorship of the standard, membership of the RCC, an adopted TACL — it was not this one.

Proof Points

Standards proposal

TACL, on an Open W3C Substrate

TACL is “a T&E extension to W3C Shape Constraints Language (SHACL)”, so the constraint substrate is an adopted W3C standard with its own toolchain, and TACL inherits its resilience: “It exhibits much better resilience to changes in the XML schemas than the languages that refer directly to the XML trees.” The ITC’18 paper won the Lawrence Rauch Telemetry Standards Paper Award (per telemetry.org). Boundaries: TACL itself is a proposal — in the paper’s words “just a proposal” needing more community involvement “before it can be subject to standardization” — none of SHACL’s W3C standing transfers to it, and a paper award is not standardization.

Selected Publications

Semantic Validation of T&E XML Data

Moskal, J.J., Kokar, M.M. & Morgan, J.

Proceedings of the International Telemetering Conference (ITC 2015), International Foundation for Telemetering

View Publication

Introducing TACL – A Proposal for a New Standard T&E Constraint Language

Moskal, J.J., Whittington, A., Kokar, M.M. & Abbott, B.

Proceedings of the International Telemetering Conference (ITC 2018), International Foundation for Telemetering