Dragos Pop
Data Science & Engineering

Jan 2022 ~ FedEx Express

Length:   2mo (at 0.5 FTE)

Programming language:   Python (Pandas, NumPy, Matplotlib, Plotly, scikit-learn, LightGBM)

Data:   Over 600.000 shipped parcels having 742 features, including the target variable (whether the parcel was held awaiting correct information)

Problem description:
Develop a LightGBM model for classifying whether a shipment will be held awaiting correct information

Approach:
Since the data was already cleaned previously, the first step involved slicing it into train, validation, and test sets, using the validation rows to monitor the model's performance during hyperparameter tuning. Then, a LightGBM model was created, and its hyperparameters were first tuned through a Random Search. The options in the grid were selected according to the official LightGBM documentation, considering the balance between overfitting, performance, and training speed. Afterward, the best parameters obtained from the Random Search were put in a new grid next to other values in their close vicinity. The respective grid was used to optimize the hyperparameters another time through Grid Search. Using the best hyperparameters from Grid Search, the model was fitted again until the learning curve of the validation data plateaued, which showed the number of iterations needed for the loss to converge. Finally, the model was retrained using both the train and validation data together.

Results:
An accuracy of 81% was observed after evaluating the predictions on the test data. However, the company uses a custom, greater-is-better score for this problem called clearance score. The LightGBM achieved a clearance score of 2.08, outperforming the deployed Neural Network by 53%, meaning fewer misclassified shipments. Moreover, the LightGBM model was trained in only 45 seconds, while the Neural Network took approximately 22 minutes. Accordingly, the former was 29 times faster, enabling cheaper retraining once deployed.

• Previous project Next project

• Home