The question that is never asked

A question that is technically trivial and professionally explosive

There are questions that would be technically trivial for a language model to answer and that must, for exactly that reason, never be answered. Such a question actually came up during the trial of JiCo, not as a thought experiment: somebody asked the assistant which people in the system belong to a particular religious denomination. Purely technically that would have been no problem — the information was partly there in free-text fields, from which an answer could have been derived. Professionally and legally, that is precisely the moment at which a system is allowed to refuse, no matter how impressive its detection performance is otherwise.

Religious affiliation belongs to the special categories of personal data that both Swiss data protection law and the European General Data Protection Regulation subject to heightened requirements. The usual reflex at this point would be to simply mask the detected term, as one would do with a telephone number. That very reflex would be too weak here.

Why masking alone is not enough here

The difference from a telephone number lies in the nature of the question itself, not in the individual data value. A masked telephone number in an answer is harmless — the actual information somebody wanted to draw from the answer was not the number itself anyway. A question that explicitly asks for a special category is something fundamentally different: even if every single name in the answer were masked, the genuinely sensitive information would remain — a list, a count, a distribution of people according to a characteristic that enjoys special protection. The counting itself is already the problem here, not only the names.

That is why a stricter, different rule applies to such a question: a request that recognisably asks for a special category of personal data is held back as a whole, not merely filtered in part. JiCo does not answer it at all — regardless of what the answer might have looked like in detail, regardless of whether the person asking had good or bad intentions. The intention behind a question cannot be assessed reliably in any case; the category of the question can.

A boundary that can be extended without rewriting the code

Religion and denomination, including common abbreviations, were the first concrete case on which this rule was developed — but the category has not stayed limited to that. The same list of especially sensitive characteristics can be extended by further categories, health data or ethnic origin for instance, without much having to be changed elsewhere in the system. That is a deliberate design decision: the boundary should be able to grow with new insights, instead of needing a separate, standalone set of rules for every new category.

What of this appears in the record — and what expressly does not

A question that is held back does not vanish without trace either, but it leaves a different kind of trace than an ordinary redaction. Every masking of a special category appears as a type and a count in the record, in the panel and in the metrics — but never the actual value behind it. Anyone who wants to check afterwards whether and how often such cases occurred will find a number. Anyone who tried to reconstruct the genuinely sensitive information from the record itself will find nothing there that would serve that purpose.

Why this rule sits in the assistant, not only in a policy

One could object that a ban on asking for special categories could just as well be formulated as a usage policy — a sentence in the manual that everybody signs before being allowed to use the system. Such policies have their place, but they rely on every single person remembering them in every single moment, including when a question was actually meant harmlessly in the heat of research and nobody intended to cross a boundary.

A rule that sits technically in the assistant itself does not need that memory. It takes effect regardless of whether the person asking has read, understood or simply forgotten the policy at that moment — and it takes effect even when the question was not asked out of bad intent but out of harmless everyday curiosity. That is exactly the difference between a rule whose observance one hopes for and a rule that actually holds, no matter who happens to be at the other end of the question.

Why this one finding weighs more than a thousand ordinary test cases

A single concrete test case like this often weighs more in practice than a hundred abstract test scenarios somebody thinks up at a desk without a real system ever having been confronted with them. The reason is simple: a test case designed on the drawing board checks what the developer imagined somebody might ask. A test case that was actually asked shows what somebody really asked — with all the secondary meanings and consequences that may not have been considered at design time.

This one question from the trial phase made visible a rule that could in theory have been formulated beforehand, but was only actually formulated because of the concrete occasion. That is no coincidence, but the real value of testing a system not only at a desk but against real usage before it is put into production — and then phrasing the resulting rule generally enough that it also covers the next case, the one not yet thought of.