evaluation in recommender systems - part A

One of the interesting things about working in real-time bidding is that you are not really building models.

You are building policies.

That distinction matters.

In many classical prediction problems, the model is trying to estimate something that exists independently of the model itself.

If you are building a location algorithm using Kalman filtering, for example, your algorithm is trying to estimate the true position of an object. The object is either there or it is not. Your model may estimate its position more accurately or less accurately, but the actual position does not change because of the quality of your model.

The same is mostly true in fraud detection (mostly, but that's an issue for a different day).

When your model decides that something is fraud, the decision usually does not change the underlying truth. The transaction was either fraudulent or it was not. Your prediction may affect what happens next, but it does not change the original event.

Real-time bidding, recommendation systems, and similar domains are different.

When your model decides that a user is high-value for a certain video, ad, product, or offer, and you act on that decision, you are not just observing the world anymore.

You are changing it.

You show the user something. The user responds to what you showed them. Their behavior becomes part of your data. That data then feeds back into your algorithm, your training process, and your offline evaluation.

This is especially clear in recommendation systems.

“Their behavior becomes part of your data”

A YouTube user is not choosing from billions of videos. They are choosing from a tiny list selected by the system. Sometimes they are not even choosing at all; they are responding to what was auto-played.

So when the user clicks, watches, skips, buys, or churns, what exactly are we measuring?

Their true preference?

The quality of the model?

The effect of the policy?

The bias introduced by what we chose to expose them to?

This is why offline evaluation becomes so hard in these systems. The data is not a neutral record of user intent. It is the result of previous decisions made by the system.

Traditionally, the clean way to test this is A/B testing.

And A/B testing works precisely because it evaluates policies on live users. You are no longer asking:

“Would the model have predicted the past correctly?”

You are asking

“What happens when this policy actually controls what users see?”

That is much closer to the real question.

But it is also expensive.

Because when you run an A/B test in these systems, you are experimenting on the actual environment. Real users see real recommendations. Real ads are bought. Real inventory is spent. Real creators, advertisers, sellers, or publishers are affected.

If the policy is worse, users may get worse content. They may watch less, click less, buy less, or churn. Advertisers may overspend. Creators may lose exposure. Marketplaces may distort supply and demand. And the company may lose money during the experiment itself.

So the obvious question becomes:

“Can we evaluate these models offline well enough to reduce the cost of A/B testing?”

Can we use historical data, simulation, replay, counterfactual evaluation, or carefully designed offline metrics to filter out the weak candidates — and only expose real users to the models that are truly promising?

We'll discuss it further in part B