$ ./guess_the_languageAI & ML
AI & ML · medium · #5

What is wrong with this preprocessing order?

scaler = StandardScaler().fit(X)      # all rows
X_scaled = scaler.transform(X)
X_tr, X_te, y_tr, y_te = train_test_split(X_scaled, y)
  1. Test-set stats leak into training
  2. The scaler should run after fitting
  3. StandardScaler needs labels too
  4. train_test_split cannot take arrays
Fitting the scaler on all rows lets test-set means and variances influence training; fit it on the training split only.
Play AI & ML ▸