Knowledge Retrieval & Relevance

The analyst's problem, in the analyst's own terms

The operational version of this was set out with us by the U.S. Army. In Understanding the Role of Context in the Interpretation of Complex Battlespace Intelligence (FUSION 2006), lead-authored by Gerald M. Powell of U.S. Army RDECOM CERDEC I2WD, then at Fort Monmouth, New Jersey, the persona is an intelligence officer for a mechanized infantry brigade, working two distinct jobs at once: answering a commanding officer's priority intelligence requirement — “Will the enemy attack the bridge with artillery fire in the next 24 hours?” — and the open-ended question of situation development, “what is the enemy doing?” In both, the paper says, “the primary challenge remains the same: how to quickly analyze and interpret massive volumes of intelligence reports and select all of those that are relevant, credible and require further attention.”

Its verdict on the automation that exists is blunt, and it was written in 2006: “effective processes for automated analysis at these higher levels tend to be highly-specific point solutions based upon simplifying assumptions that render them brittle and inapplicable to novel situations.”

Goals define relevance

The principle is stated most sharply in the IEEE Intelligent Systems article Kokar co-authored with Endsley: “Without goals, sensed data has no independent meaning, making strictly bottom-up information integration and fusion nearly impossible. Goals define information's relevance (separating signal from noise) and let meaning be established regarding that information.” Bottom-up fusion alone cannot tell you what matters, because mattering is not a property of the data.

That principle appears in the representation and in the running system, and it is worth being precise about which does what. In the Situation Theory Ontology, relevance is modeled: every situation must name its relevant objects and its relevant relations, focus is a sub-kind of relevance, and the user's query — the Goal — supplies the perspective that determines them. In SAWA, relevance is configured: the Situation Management Component reasons from the operator's standing relation to the subset of rules the Relation Monitoring Agent needs to load and the subset of objects and attributes the Event Management Component needs to watch. Theory in one, running mechanism in the other.

Both stop short of the question an evaluator asks next. Narrowing the rules and the data before you reason is not the same as knowing, after you have reasoned, exactly which facts a particular answer stood on. The flagship names that gap and does not close it: the authors write that they “would like to be able to derive” the other individuals relevant to a situation, and stop. Closing it took a separate paper.

Relevance is not a ranking. It is a proof obligation.

In Inferring Relations and Individuals Relevant to a Situation (CogSIMA 2014), relevance stops being a matter of judgment and becomes a matter of proof. The definition is one sentence: “Relevant entities with respect to a given query Q are those entities that are necessary for proving [that the situation entails Q].” An entity is relevant to a question exactly when the proof of the answer needs it. Not a score, not a similarity, not a ranked list of hits — the relevant set is whatever the derivation used. The authors chose that definition deliberately, “to avoid any arbitrariness in the definition of relevance.”

The mechanism is worth stating plainly, because it is unusual: the rules reason about the rules. Rules are annotated so that a rule's head asserts the relevance of the triples that rule consumed, and a metarule relates one rule to another — “a rule R2 is relevant to rule R1 if a triple that appears in the head of R2 (i.e. is asserted by R2) appears in the body of R1.” Chaining that backwards from the answer collects the individuals and the relations the derivation actually leaned on. The proof tree is the relevance criterion, and the system walks it. Rules and metarules alike ran on BaseVISor — the engine our flagship names, and the engine we still ship — which that paper describes as an OWL 2 RL reasoner. (The flagship describes the same engine at R-Entailment. Each paper is quoted at the profile it states; we do not backdate one onto the other.) And the reason rules are in the picture at all is stated in a line: “While OWL can describe the individuals, the individuals cannot be explicitly identified. Therefore, we use rules to extend OWL's capabilities.”

What the worked example shows — and the example is a civilian toy, which we will not dress up as anything else — is the part that generalizes. The query named two individuals. The proof of the answer pulled in two more that the query never mentioned, and five relations nobody had asked about, because the relation the question actually turned on was not on file and had to be derived from others. The relevant facts are not the ones you asked about. You cannot know in advance which individuals will matter. Only the proof knows.

The limits, as the authors state them. There is no evaluation — no dataset, no baseline, no timing, no scale; the evidence is a single worked example. The rules are not even printed: “The full text of the rules could not be included due to the lack of space.” This is a published mechanism, not a productized relevance engine, and we do not offer it as one. And minimality is open — the authors' own closing question is “whether the set of relevant individuals and relevant relations is the smallest possible.” Relevance here is derived. It is not claimed to be minimal, and we do not claim it either.

The payoff is not search. It is what crosses the link.

The motivation for deriving relevance was never a better result list. It was bandwidth. Two agents have to exchange a situation over a link that may not carry much; the receiving agent is itself an inference engine, so it does not need everything the sender knows — it needs the facts from which it can rebuild the rest. The paper's statement of the payoff: “not all of the knowledge that an agent has needs to be sent over a possibly low bandwidth link to another agent. Instead, the sending of only the relevant facts should allow the other agent to reconstruct the description of the situation.”

Do not ship the knowledge base. Ship the facts the answer depended on, and let the far end derive the closure. The flagship and the Endsley article both frame the shape of that argument — agents that share an ontology need exchange only what the other cannot infer for itself — and this is the work that supplies the missing operator: which facts those are. Read against a tactical network that is denied, degraded, intermittent or limited, the difference is between a picture you cannot afford to transmit and a handful of facts that reconstitute it at the far end. That reading is ours; what the paper says is “narrow bandwidth.”

Asking a knowledge base is not asking a database

Once the facts are in a formal model, questions are answered with SPARQL — and the difference from a query over a database is the whole reason the model exists: “the answer to a knowledge base query may include facts that are inferred as well as facts that have been explicitly asserted.” You are not retrieving what somebody remembered to store. You are retrieving what the knowledge, taken together, entails — including the relations nobody wrote down because nobody could have.

Queries reach across situations as well as within one, and they run against history as well as the present: SAWA's triple store retained every event and answered questions about past states. It also supported what-if queries — assert a hypothetical fact, query, then retract it along with everything deduced from it. Suppose that route is cut: who loses their supply line? is a branch-plan question answered by the same rules that do the monitoring. The authors flagged their store as too inefficient for near-real-time use and were rebuilding it; we cite the mechanism, not a latency.

Explainability was a requirement here before it was a mandate

Relevance and retrieval only pay off if the analyst believes the answer. The FUSION 2006 paper — written with the Army — makes that a system requirement, and its formulation has aged better than most things written about AI in 2006:

“An analyst is not going to accept an interpretation from a black box if it is not clear how the interpretation was obtained and what it is based on. … If the system is unable to provide some transparency into its reasoning process and assure the user its interpretations are plausible, the system is unlikely to be trusted and thus unlikely to be used.”

That is a requirement we can cite rather than promise. It is also, structurally, what an ontology-and-rules architecture gives you for free: every derived relation is the consequence of a named rule firing over facts that are themselves either observed or derived, so the chain from a conclusion back to the assertion that produced it is not a feature to be added — it is how the thing works. And it is precisely the chain the relevance rules above walk.