When Reality Becomes an Estimate
There is a strange thing that happens to data during a crisis. At first, the indicators deteriorate. Then, eventually, our ability to measure those indicators deteriorates too.
That creates a paradox: the more rapidly reality changes, the more urgently we need accurate information, and the harder that information becomes to collect directly.
That is when observation begins to give way to inference.
Take Lebanon. Years of economic crisis, institutional weakness, infrastructure deterioration, conflict and displacement have made some forms of conventional measurement increasingly difficult. The IMF has repeatedly pointed to weaknesses in the availability, timeliness and reliability of key Lebanese economic statistics. The World Bank, when trying to understand more recent economic conditions, has supplemented conventional information with high-frequency indicators, including night-time satellite lights.
The important point is not the satellite itself. It is what happens when the normal ways of observing a country begin to weaken. We start reconstructing reality from whatever evidence remains: administrative records, phone surveys, rapid assessments, remote monitoring, transaction data, mobility information, models, proxies and expert judgment.
All of these can be useful. Sometimes they are the only practical way to understand what is happening. The problem begins when the distinction between what was directly observed and what was inferred, modeled or reconstructed becomes less visible.
This matters especially during crises because the thing we are trying to measure is itself changing. People move, businesses close, facilities stop operating, informal work expands and administrative systems weaken. A sampling frame designed six months earlier may no longer resemble the population currently living there.
So the challenge is not simply that information becomes harder to collect.
Reality itself becomes harder to observe.
Anyone working in monitoring, evaluation or research has encountered some version of this. You establish a baseline around a known population, known businesses, known facilities and a functioning system. Then something significant happens: war, displacement, economic collapse or a natural disaster.
When you return to measure change, the statistical universe you started with may have changed underneath you. You can still collect information, triangulate across sources and produce useful analysis. But you are no longer looking at reality through the same window.
This is where modern data systems are both powerful and slightly dangerous.
We have become very good at filling gaps. When surveys become difficult, we use administrative data. When physical access becomes impossible, we use remote monitoring. When no single source is sufficient, we combine several. When information remains incomplete, we model.
That adaptability is a strength. It allows policymakers, researchers and humanitarian actors to understand situations that might otherwise remain largely invisible.
But it also creates a subtle risk:
The better we become at estimating reality, the easier it becomes to forget that we estimated it.
A number can travel a long way. It may begin in a methodology, then move into a dataset, a report, an executive summary, a presentation, a dashboard and eventually a policy or funding decision. At every stage, some methodological context can disappear.
Eventually, what survives is often just the number: clean, precise and easy to communicate.
The problem is not necessarily the number. It is that the person seeing it may have very little sense of how far it sits from direct observation.
This does not mean estimates are bad. Quite the opposite. In a crisis, waiting for perfect information can be far more damaging than making a reasonable decision with imperfect evidence.
The real question is whether the distance between direct observation and the final number remains visible enough to the person using it.
Not all numbers represent the same relationship with reality. One may come from a representative survey completed last month. Another may come from partial administrative records. Another may rely heavily on proxies or models.
All can be legitimate. But they should not necessarily carry the same implied level of certainty.
And once they are placed next to each other on a dashboard, they often do.
Perhaps what major crisis indicators need is not another global index, but better visibility into how we know what we claim to know.
For any important figure, I would want to understand what was directly observed, what was inferred, what could not be observed, how recent the underlying evidence is and how much confidence should reasonably be placed in the result.
That information does not have to overwhelm the reader with methodology. It simply needs to stop disappearing.
This becomes even more important as AI enters the picture. We are becoming much better at generating useful estimates from fragmented information. Models can reconcile multiple sources, detect patterns and help analysts make sense of environments where conventional data collection has become difficult.
That will be enormously valuable.
But it will also make it easier to produce a complete-looking picture from incomplete reality.
The more sophisticated our ability to fill information gaps becomes, the more important it is to preserve the distinction between what we observed and what we reconstructed.
There is an old phrase: you cannot manage what you cannot measure.
Crisis environments complicate that. Sometimes we have to manage what we cannot fully measure. We estimate it, triangulate it, use proxies and make the best possible decision with the evidence available.
That is often exactly what responsible analysis requires.
But we should never confuse our ability to estimate reality with our ability to see it.
Comments