How to tell if it is working, before the renewal
4 min readUpdated
Usage metrics rise during any rollout and fall afterwards, so they indicate very little. The measures that matter are whether requests to the data team fell, whether questions are being repeated by different people, whether anyone presents an answer from the tool in a meeting that matters, and whether a second team adopted it without the original sponsor pushing.
Why logins mislead
Every rollout produces a usage spike. People are curious, the sponsor is watching, and there is often a launch email. Three months later the number is lower and nobody can say whether that means adoption failed or the novelty simply wore off.
Four measures that mean something
- Did requests to the data team go down?The single best signal, and the easiest to measure, because your data team already tracks its queue. If the backlog is unchanged, nothing has moved regardless of usage.
- Are different people asking the same questions?Repetition across people means the answers are trusted enough to be reused. Repetition by one person often means they did not believe the first answer.
- Has an answer appeared in a meeting that matters?Somebody putting a number in front of a leadership team is a stronger trust signal than any dashboard view count.
- Did a second team adopt it unprompted?Adoption that requires continuous pushing from the sponsor is not adoption. It is a favour, and it stops when the sponsor moves on.
The uncomfortable one
Ask your data team, privately, whether their week is better. If the answer is that they now spend it correcting the tool's answers, you have added work rather than removed it, and the definitional groundwork was skipped.