What we deliberately do not measure

The metric that deliberately never stands on its own

There is one metric that looks, at first glance, like the most obvious measure of success there could be: how long does a proposal wait before a human decides on it? Shorter is better, one might think — the faster the decision falls, the smoother operations run, the happier the people using the tool. JiCo does measure this time. But it is never read on its own anywhere, in no report and on no dashboard.

If the number of decisions rises at the same time as the time per individual decision falls, that is not a success to be celebrated — it is a warning sign that means precisely the opposite. At that point nothing is really being decided any more, things are merely being clicked through, and the approval itself has degenerated in daily practice into a formality, a click that happens but no longer checks anything. The counter-metric for this is called approval fatigue, and in the internal set of metrics it deliberately stands directly next to speed, not somewhere below it in a hidden second line.

When a metric improves by dismantling control

The same basic principle applies to a second metric: the hit rate, meaning the share of proposals that a human approved unchanged, without anything having to be corrected. If this rate falls over time, it means the model is increasingly proposing things that are not usable in that form. The only permissible answer to that is to sharpen the professional glossary and the stored conventions so that the model understands the actual task better in future.

What is explicitly not a permissible answer: loosening the approval itself so that more proposals automatically pass as "suitable" and the number improves again in purely arithmetical terms. That would have rescued the metric and destroyed the actual product at the same time — both in the same single measure, without anything having had to be decided in between. A metric that only improves by softening one of the system's fundamental rules is by definition not an improvement. It is precisely the loss that the whole system was meant to prevent.

Three things that are deliberately missing — and why

Anyone looking through JiCo's set of metrics for certain values that seem obvious at first glance will deliberately come away empty-handed in three places.

There is, first, no metric for individual time saved per employee. Such a figure would in effect be a performance measurement of the individual — and that is exactly what JiCo must under no circumstances become. The audit trail does record who made a particular decision, because the evidential value of an action requires that. But that does not make it a performance statistic, and technically it is not read or evaluated like one either — there is no analysis that sets decision times against individual people.

There is, second, no detection rate for the PII gate — for exactly the reason that has already been explained at length elsewhere: a genuine rate would need a gold standard, and for free text that simply does not exist. Only the number of actual hits is counted, never a rate that would put a figure on what is systematically missed. Such a number would be invented, however convincing it might look on a management dashboard.

And there is, third, no rating of model quality in points or stars, of the kind one knows from other AI products. There is only one deliberately simple question that is actually answered: did a human find this particular proposal usable, yes or no? That is the only rating that counts in the end — not how elegant or polished the wording was, but whether it served the matter at hand.

Throughput is not the same as activity

The most basic of all metrics follows the same principle, that what is measured is what was actually brought about, not what merely happened. Throughput counts executed actions only — approved changes that actually arrived in Jira — not the sheer quantity of proposals a model formulated in the same period.

The difference sounds like a technical nicety at first, but in practice it is decisive: a system could very easily be set up to produce an impressive number of proposals, hardly any of which end up leading to a real change — on paper that would look like high activity, in reality it would be nothing but noise. The volume of proposals is therefore explicitly not an achievement in itself. Only what a human ultimately judged to be right and approved counts as what the system really accomplished.

Where the numbers come from at all — and the one rule that applies to all of them

Every single metric in JiCo is derived from the same audit trail that has to be kept for evidential purposes anyway. There is no second, separate store of data created and maintained only for measurement, and no additional collection that would have to be set up specifically for a dashboard.

And for every single one of these numbers the same rule applies without exception: no metric may be allowed to improve by softening one of the system's fundamental rules somewhere. A number that only gets better through less control is strictly speaking not an improvement, however good it may look at first glance. It is precisely the loss that avoiding was the whole point of the exercise.