作用：最大程度地遏制模型的过拟合和欠拟合。

验证数据集

为什么需要验证数据集：
1.由于测试数据并不是真实数据，因此训练出的模型在预测真实值时，可能出现过拟合和欠拟合。
2.训练出的模型依赖测试集，当测试集发生变化，模型需要重新训练
验证数据集的具体作用：模拟真实环境
1.充当测试数据集
2.真正的测试数据集变成了真实数据集
image.png

交叉验证

随机取验证数据集：避免验证数据有大量极端值
image.png

编码

  1. # 1.准备数据
  2. import numpy as np
  3. from sklearn import datasets
  5. digits = datasets.load_digits()
  6. X = digits.data
  7. y = digits.target
  9. # 2.分割数据
  10. from sklearn.model_selection import train_test_split
  11. X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=666)
  13. # 3.建模
  14. from sklearn.neighbors import KNeighborsClassifier
  16. best_k, best_p, best_score = 0, 0, 0
  17. for k in range(2, 11):
  18.     for p in range(1, 6):
  19.         knn_clf = KNeighborsClassifier(weights="distance", n_neighbors=k, p=p)
  20.         knn_clf.fit(X_train, y_train)
  21.         score = knn_clf.score(X_test, y_test)
  22.         if score > best_score:
  23.             best_k, best_p, best_score = k, p, score
  25. print("Best K =", best_k)  # 3
  26. print("Best P =", best_p)  # 4
  27. print("Best Score =", best_score)  # 0.986091794159
  29. # 4.使用交叉验证
  30. from sklearn.model_selection import cross_val_score
  32. knn_clf = KNeighborsClassifier()
  33. cross_val_score(knn_clf, X_train, y_train)  # array([ 0.98895028,  0.97777778,  0.96629213])
  35. best_k, best_p, best_score = 0, 0, 0
  36. for k in range(2, 11):
  37.     for p in range(1, 6):
  38.         knn_clf = KNeighborsClassifier(weights="distance", n_neighbors=k, p=p)
  39.         scores = cross_val_score(knn_clf, X_train, y_train)  # 交叉验证：随机选取验证数据集和建模
  40.         score = np.mean(scores)
  41.         if score > best_score:
  42.             best_k, best_p, best_score = k, p, score
  44. print("Best K =", best_k)   # 2
  45. print("Best P =", best_p)   # 2
  46. print("Best Score =", best_score)  # 0.982359987401
  48. best_knn_clf = KNeighborsClassifier(weights="distance", n_neighbors=2, p=2)
  49. best_knn_clf.fit(X_train, y_train)
  50. best_knn_clf.score(X_test, y_test)  # 0.98052851182197498
  52. # 5.网格搜索
  53. # 网格搜索内置了交叉验证
  54. from sklearn.model_selection import GridSearchCV
  56. param_grid = [
  57.     {
  58.         'weights': ['distance'],
  59.         'n_neighbors': [i for i in range(2, 11)], 
  60.         'p': [i for i in range(1, 6)]
  61.     }
  62. ]
  64. grid_search = GridSearchCV(knn_clf, param_grid, verbose=1)
  65. grid_search.fit(X_train, y_train)  
  66. # Fitting 3 folds for each of 45 candidates, totalling 135 fits
  67. # 训练集分成3组，一共45个模型，共训练135次
  69. grid_search.best_score_  # 0.98237476808905377
  70. grid_search.best_params_  # {'n_neighbors': 2, 'p': 2, 'weights': 'distance'}
  72. best_knn_clf = grid_search.best_estimator_
  73. best_knn_clf.score(X_test, y_test)  # 0.98052851182197498
  75. # 6.cv参数
  76. # 训练集分成几组
  77. cross_val_score(knn_clf, X_train, y_train, cv=5)  
  78. # array([ 0.99543379,  0.96803653,  0.98148148,  0.96261682,  0.97619048])
  80. grid_search = GridSearchCV(knn_clf, param_grid, verbose=1, cv=5)

k-folds交叉验证

  • 定义：把训练集分成k份，称为k-folds cross validation
  • 优点：模型更精确

  • [x] 缺点：每次训练k个模型，整体性能慢了k倍

    LOO-CV

    Leave-One-Out Cross Validation

  • [x] 定义：训练集分成m份，m-1份用于训练，剩下的1份用于验证。

  • 优点：完全不受随机影响，最接近模型真正性能指标，常用于学术、论文中来保证模型的准确度。
  • 缺点：计算量巨大