Data Integration for T&E

The Challenge: Why This Bites Hardest in a Multi-Vendor System

The pain is worst exactly where T&E most wants to go: mixing the best hardware from different vendors. “This greatly complicates the validation process of XML data used in testing, because the process cannot be fully automated with standard schema processors. This applies to large multi-vendor systems in particular, where XML data might contain conflicts that cannot be detected with off-the-shelf validators.” A single vendor cannot see the whole picture, and the ITC’19 paper names the gap precisely:

“In multi-vendor T&E systems, a single hardware vendor cannot anticipate the dependencies on the settings from hardware manufactured by other vendors, or the systemic constraints that are specific to a particular customer.”

Those systemic constraints are concrete and unforgiving — a total bandwidth, weight or power budget across the whole network; a value that must be globally unique; an RF MAC address that, in the ITEA’19 deck’s words, “cannot be reused across different Radio Access Networks”. And because no standard language exists to state them, vendors do the only thing they can: “vendors do not expose the constraints of their devices in a formal language. Instead, they hardcode them into their proprietary configuration software that is shipped with the hardware.” The result, in the deck’s blunt summary, is that “Multi-vendor systems are expensive, impractical” and “In practice, they are often avoided via vendor lock-in”.

The Missing Piece, and the Multi-Vendor Payoff

A language and an engine are two of three pieces. The third is the interface — how a configuration tool tells a validation engine what changed, and how the engine tells it what is wrong. The ITC’19 paper calls that interface the missing puzzle piece, and argues it should be standardized too.

The setup is a clean separation of concerns: SwRI’s iNET System Manager is the tool an engineer drives, and VIStology’s xVISor is the validation engine behind it. The hardware side of that picture is already standardized; the software side is not: “While the interaction between the System Manager and the T&E hardware is fully standardized under the iNET program, the interaction between the two software components, termed Validation Protocol, is not and is the primary subject of this paper.” Close that gap, and “the Validation Protocol is the last puzzle piece to turn the entire architecture into a truly vendor-independent, loosely-coupled, standards-based T&E configuration system.”

The reason to keep the engine separate at all is the multi-vendor payoff, stated as a market argument: “Vendors of the different implementations of the validation engine can compete in terms of performance, cost, or platform requirements, etc., reducing the overall cost of the system.” And the constraints, once written in a shared language, become portable: “Constraints written in such a language can be validated by a third party validation engine, without relying on any particular vendor’s software.”

What was actually built — and what was only proposed

Here the vision and the result have to be told apart, because the deck that argues the vision is careful to, and so are we. The vision is one shared standard replacing a wall of proprietary tools — the deck literally titles it “Tomorrow”, with “Vendor-neutral, third-party configuration software and constraint engines” and “Multi-vendor systems become practical”. That is the goal, well argued.

What was demonstrated is narrower and real: two parties, not an ecosystem. “At the 2018 ITC conference, VIStology successfully demonstrated an integration of the xVISor TACL Engine with the SwRI’s reference implementation of the iNET System Manager” — one validation engine, one configuration tool, with a live cross-device constraint violation (an already-used RF MAC) surfaced inside the System Manager’s own GUI. No independent third-party vendor was integrated. The multi-vendor ecosystem was motivated and proposed; a single two-party integration was built in support of it.

The design choice underneath it was how to keep the engine’s view of the configuration in sync. Re-exporting the entire MDL/TMATS file on every edit was rejected — “The major downside of this approach is that, in practice, it does not scale.” A notification-based scheme that sends only the delta was preferred: “the notification-based approach to conveying the configuration state, while demanding some development effort on the part of the configuration manager, is expected to yield the desired performance characteristics. Thus, it is preferred over the integration based on importing and exporting MDL/TMATS XML files.” Note the modal verb — expected, not measured.

The 2018 demonstration used Redis as its message bus, and the authors are candid that this was an expedient rather than the design: the notifications “too closely resembled the manager’s internal data structures, making the integration more tightly-coupled”. The proposed protocol recommends the opposite — notifications independent of any proprietary data model, referring only to MDL/TMATS, plus a heartbeat and manager-controlled constraint sets — and the whole thing, in the paper’s words, “should be standardized within the RCC community.” Proposed. Not adopted.

Proof Points

Demonstrated

xVISor Inside SwRI’s System Manager

“At the 2018 ITC conference, VIStology successfully demonstrated an integration of the xVISor TACL Engine with the SwRI’s reference implementation of the iNET System Manager” — the genuinely built artifact behind this page. A live cross-device constraint (an RF MAC already used in another radio access network) was flagged inside the System Manager’s own GUI. Boundaries: it is two parties, not an ecosystem — one engine, one configuration tool, no independent third-party vendor integrated; the message bus was Redis, which the authors call too tightly-coupled and explicitly not the proposed design; and there is no performance measurement of any kind.

Subcontract

Prime, with the iNET Lab as Sub

The Phase II funding line, verbatim from the deck: “VIStology (lead) and SwRI (sub) are funded via a Phase II SBIR (AF141-227)”. VIStology is the prime; Southwest Research Institute is the subcontractor, supplying iNET/MDL domain expertise. Separately — and this is a distinct fact — SwRI builds and operates the iNET System Manager and the iNET System Integration Lab, which is the software xVISor was integrated with. Boundaries: SwRI is not a customer; Boeing was a briefed prospect with no contract; and the government engineers who supplied data and use cases (Bruce Lipe, Air Force Test Center; William A. Malatesta, NAVAIR) are thanked, not customers.

Selected Publications

Validation Protocol – The Missing Puzzle Piece

Moskal, J.J., Whittington, A., Morgan, J., Kokar, M.M. & Abbott, B.

Proceedings of the International Telemetering Conference (ITC 2019), Vol. 55, International Foundation for Telemetering

View Publication

Building Multi-Vendor T&E Systems in iNET

Moskal, J.J., Kokar, M.M., Abbott, B., Whittington, A. & Morgan, J.

23rd Test & Training Instrumentation Workshop (T&TIW / ITEA), 2019 — presentation

View Slides

See the full list of VIStology publications →