Ranking Hotel Listings on Expedia
May 2022 ~ MSc Course "Data Mining Techniques"
Length: 1mo (at 0.25 FTE)
Programming language: Python (Pandas, time, datetime, scikit-learn, LightGBM)
Data: Approximately five million hotel search queries, containing the
attributes of the retrieved properties, user details, and whether the visitor clicked on
and booked the hotel
Problem description:
Rank hotel listings by their likelihood of being booked
Approach:
First, the data was explored, investigating its shape, followed by the type, missing values,
and distribution of its features. Secondly, the missing entries were filled in, new variables were
derived, and the categorical features were one-hot-encoded. Next, the data was split into the
train (90%) and validation (10%) sets. Lastly, two LightGBM models were designed, a regressor
(pointwise approach) and a ranker (listwise approach).
Results:
After testing different target variables and encoding techniques for the categorical features,
multiple combinations of hyperparameters were compared using a 4-fold Cross-Validation method.
The NDCG@5 of the regressor and ranker estimators were evaluated at 0.350 and 0.375, respectively,
on the validation data. Next, the models with the best hyperparameters were trained on all the
data (including the held-out set) and evaluated on the competition test set at 0.333 and 0.366,
respectively, suggesting a slight overfit on the validation data. The resulted NDCG@5 scores were
interpreted in relation to the random baseline ranking (0.156). Accordingly, the outcomes
indicate that the developed models are over two times as good as listing properties randomly.
In other words, Expedia would satisfy their customers twice as much by using one of the two
models instead of randomly displaying listings.
Finally, in the picture below, one can see the feature importance of the final LGBMRanker
model. The most important variable is by far the property id, implying that specific properties
are significantly more likely to be booked than others. This also means the model would struggle to rank
hotels it has never seen before, hence, ranking such properties accurately would require retraining the
model without the property id, relying instead on more general property attributes.