An engine removed without warning grounds an aircraft and breaks a maintenance plan. This project, delivered by a team of three during my Master of Engineering (MEng) in Big Data & Machine Learning at Efrei Paris, asked whether the sensor readings of a single flight can tell a planner that an engine should go to the shop within its next 30 flights. On 100 engines held back until the very end, the model caught 21 of the 25 that were truly close to failure and raised 3 false alarms on the other 75.
Those figures are the smaller part of the deliverable. Anyone can fit a classifier on NASA’s C-MAPSS FD001 simulation in an afternoon. The work that matters is being able to defend each step to the person who will spend money on the answer. This case study follows the decisions in the order we took them, including what we rejected and why. The same sequence applies to any machine learning solution built on a company’s own data.
| Decision | We chose | We rejected | Deciding evidence |
|---|---|---|---|
| Target | Alarm within 30 flights | Exact remaining life | A planner books a slot, not a cycle count |
| Validation | Whole engines held out | Random rows | 97% of random validation rows sit beside a training row |
| Inputs | 14 sensors | 21 sensors, or the 12 strongest | 7 never move; removing the weak 2 hurt every model |
| Outliers | Kept | Removed | They cluster in the final 30 flights |
| Yardstick | Average precision | Accuracy | Never raising an alarm scores 85% |
| Model | Logistic regression | Boosting, SVM, k-NN | A tie within fold noise, broken by a rule set in advance |
| Final check | One pass on the test set | Switching to the test winner | A second look turns the test set into a selection set |
Frame the Decision Before the Model
The dataset invites you to predict remaining useful life as a number. We turned it into a yes-or-no question: is this engine within 30 flights of failure? A maintenance planner acts on that question, and a slot in the shop does not depend on whether 27 flights remain or 29.
The window itself is a trade-off we tested. Ten flights would leave 7 positive engines in the test set, too few to evaluate anything. A longer horizon blurs the signal: the strongest sensor separates the two classes by 2.50 standard deviations at 30 flights and by 1.85 at 75.
We also restricted the input to one flight’s snapshot, with no history. It costs some accuracy, and it means an operator can apply the model to the latest flight record without building a time-series pipeline first.
Read the Data Before Trusting It
Ten quality checks passed and nothing needed imputing, so the risk lay elsewhere. The training engines run until they fail; the test engines are cut off at an arbitrary point. Test engine 3 stops at cycle 126 with 69 flights left, and failure comes at 195. Anyone who treats the last row as the end of life is wrong on every test engine, so the true labels have to come from the separate answer file.
Lifetimes ranged from 128 to 362 cycles. Cycle 128 is the last flight of one engine and about a third of the life of another, so age alone cannot make this call, and we excluded the cycle counter from the inputs.
Decide How the Model Will Be Graded
This is the most consequential choice in the project, and it happens before any model exists. Consecutive flights of one engine are almost identical. Split the data by row and the model is graded on engines it has already studied.
- Engines on both sides
- Validation rows beside a training row
Split by row: all 100 engines appear on both sides, and 97% of validation rows sit beside a training row.
We used five-fold cross-validation grouped by engine, with scaling fitted inside each training fold. Every validation fold holds 20 engines the model has never seen, which is the situation it would meet on a new aircraft.
Let the Physics Narrow the Inputs
FD001 simulates one fault, degradation of the high-pressure compressor, and the sensors around it respond first. Screening removed what cannot carry information and kept the rest for testing, instead of ranking sensors by correlation and cutting the bottom.
Ps30, HPC outlet static pressure: kept. The strongest single signal. Every statistical outlier it produced fell within 30 cycles of failure, so outliers stayed in.
Two sensors tested that discipline. Core speed and its corrected twin correlate weakly with remaining life, and at 0.96 with each other, so a tidy analysis would drop them. Their spread near failure is three times the healthy spread, though, a pattern a correlation cannot see. We wrote down the rule before running the experiment: drop them if at least four of five models lose less than 0.002 average precision. Across 150 fits, all five lost between 0.018 and 0.030, and the 12-sensor set won in 0 of 15 paired folds. They stayed.
Outliers got the same treatment. The interquartile rule flagged 2,431 rows, 11.8% of the training set. For the two strongest sensors, every flagged row fell within 30 flights of failure. These are the events we are trying to detect, so they stayed in as well.
The textbook cleaning step would have deleted the failures.
Choose a Yardstick the Business Can Live With
A model that never raises an alarm scores 85% accuracy and prevents nothing, so accuracy could not select anything here. We compared models on average precision, which measures how well they rank engines by risk and is independent of any alarm setting.
Class weights showed why that separation matters. Weighting the rare class 5.7 to 1 took logistic regression from 84% to 95% recall and from 88% to 73% precision, with average precision unchanged. The weights moved the operating point without improving the ranking. That is a policy choice, and it should be made in the open.
Compare Broadly, Then Select by Rule
We ran seven model families, tuned them over 240 fits and re-scored them on fresh fold seeds. The leaders finished level, and picking the top score would have meant choosing noise. So the selection rule was fixed before the results existed: among models within 0.005 of the best, take the simplest.
First look. Seven families on sensible defaults. Naive Bayes leads; the SVM looks like the worst choice. Naive Bayes leads at 0.952.
Logistic regression won, and simplicity has a commercial value. The trained model is a scaler, 14 weights and a threshold in a small JSON file. The project website runs it directly in the browser, an operator could embed it in an existing maintenance system without hosting a machine learning runtime, and an engineer who asks why an engine was flagged can read the answer from its coefficients.
The alarm setting came last, from out-of-fold predictions: 0.592, the point that maximizes the F2 score, which counts recall above precision. The optimum is broad, as F2 moves only from 0.892 to 0.893 against the default 0.5. We report that plainly, because it means 0.592 has no special authority.
Open the Test Set Once
The 100 test engines were scored a single time, after everything above was frozen.
| Measure | Result | 95% interval |
|---|---|---|
| Failures caught (recall) | 21 of 25 (84%) | 67–96% |
| Alarms that were real (precision) | 21 of 24 (87.5%) | 73–100% |
| Ranking quality (ROC AUC) | 0.978 | 0.95–0.995 |
| Average precision | 0.944 | 0.87–0.99 |
Naive Bayes then posted a higher test score, 0.962 against 0.944. We did not switch. Choosing a model after seeing the test result makes the test set part of the selection, and the number would stop meaning anything. With 25 positive engines, one engine moves recall by 4 points, and a business case should be sized on the lower bound.
- Caught
- /25
- False alarms
- /75
- Cost
Alarm at 0.592: 21 of 25 failures caught, 4 missed, 3 false alarms, cost 44 inspections. Engine 64: Failure missed.
All seven errors lie within 12 flights of the 30-flight boundary: the misses have between 18 and 28 flights left, and the false alarms between 34 and 38. The model fails where the label itself is an arbitrary line, and it behaves as a graded signal when applied to every flight in the test histories:
| Flights left | 0–10 | 11–20 | 21–30 | 31–40 | 41–50 | 51–75 | Over 75 |
|---|---|---|---|---|---|---|---|
| Share of flights with an alarm | 100% | 95% | 71% | 32% | 12% | 3% | 1% or less |
Two findings belong in any handover. First, precision depends on the population: the 87.5% applies to a fleet where a quarter of engines are near failure, and scoring every flight, where only 2.5% are, brings it to 62%. Second, the threshold is a business parameter. If a missed failure costs 20 inspections, the delivered setting costs the equivalent of 104 across this fleet and the lowest-cost setting costs 37. That comparison uses hindsight, so it shows sensitivity rather than a tuned result, and it shows that one workshop with the maintenance controller could move the outcome more than another round of modeling.
From Simulation to Service
FD001 is a simulation with one operating condition and one fault mode, the validation reused the training engines for several choices, and the test set is small. None of these results transfer to a real fleet as they stand. Putting the model into service with an operator would take four steps:
- Price the error. Agree the cost of an unscheduled removal against an inspection, per fleet, with maintenance and finance.
- Run in shadow mode. Score live flights for one maintenance cycle and compare alarms with what the shop actually found, with no change to operations.
- Refit on the operator’s data. Add operating conditions and fault modes, and move to nested validation.
- Hand over the control. Give planners the threshold, a monitor for sensor drift and a named owner for recalibration.
The same discipline applies wherever a score triggers a costly intervention, from industrial pumps to rail bogies and credit reviews.