We deleted a dashboard we trusted

We ran a Grafana dashboard against our production database for about nine months, and last week we deleted it. We ask the database questions in English now, and a job posts the numbers we need pushed at us into Slack every weekday morning. The dashboard was more reliable than the AI, and we switched anyway.

Why we trusted the graphs

If you work somewhere with a data team, or a biz ops team, and someone there builds you a graph, you trust it. You trust that they checked it. You trust they know which rows everybody leaves out of the count. It says the same thing every time you open it, and when it is wrong it is wrong the same way for everyone looking at it.

That's the real value of a pre-built graph. It's not flexible, but it's reliable.

Think of spending a Saturday organizing the house and most of it goes fine: utensils in one place, batteries and chargers in another. Then you get to the last armful. A tape measure and a key that fits no lock in the house. It's late, neither one belongs in anything you just built, and inventing a category for two objects feels ridiculous. So it goes in the bottom drawer.

Six months later the bottom drawer is the one you open first.

Our dashboard went the same way. We had 24 total graphs. The sections we set up in October held, and most of what we wondered about after that ended up in Miscellaneous.

Every one of those was a pull request. Somebody spent time, wrote the query, picked a chart, and matched the filters to the graphs around it so that two numbers on one screen would still agree. Then the next question arrived and we did it again.

So the dashboard answered the questions we had in October, and the pile at the bottom was everything since. What we wanted was a way to ask one of those without building something somebody would have to maintain a year later.

What replaced it is less reliable

In May we put a read-only server in front of the same database and started asking it questions in plain English. It's more flexible in every direction. We can ask things we never asked before, or get an answer about one customer in one week without building anything first.

And sometimes it tells you something, and you read it and think that can't be right. So you say so. And it says, "You're absolutely right, I made a mistake."

That happens often enough to matter. We have not solved it, and anything heading into a decision still gets checked against the database directly.

We switched anyway

Because we weren't opening the dashboard.

Nobody says let's go look at the dashboard and answer this. The question you actually have almost never maps onto one graph. You want a specific customer in a specific week, or a trend but only across accounts over a certain size, or last month's number split a way nobody thought to split it.

You can get close with filters and date pickers. We did that for months. It works, and it is tedious, and tedious is enough to stop somebody asking at all. Most of what we ask now is small questions nobody would have built a graph for.

Some things should still be pushed at you

You shouldn't have to ask whether a number crossed a threshold, and nobody checks every day. So we kept that part and moved it into Slack. A job runs each weekday morning and posts a summary. Its queries live in the repo instead of being written fresh each time, so the same questions get asked every day and the answers come back the same way.

It's in the product now

The same is true for anyone using Frigade. You get a dashboard with your stats, and an Insights page that groups what people are asking into themes, and both are useful.

The assistant in that dashboard can also query your data directly. Ask how your assistant did last week, or which answers people gave a thumbs down to, or whether the change you shipped moved anything. It comes back with links to the actual questions and users, so you can go look for yourself.

We added strong protections in place so we can't read specific user data or query against stuff we shouldn't be touching. All on top of a read-only layer so we couldn't mess up our database.

It's the same trade we made internally, for the same reason. You have questions about your own product that we were never going to guess.