建模過程中分類變數的處理（筆記一）
- 2020 年 3 月 3 日
- 筆記
本文的內容來自參考書《Python機器學習基礎教程》第四章數據表示與特徵工程第一小節的內容
自己最淺顯的理解：數學建模是基於數學表達式，數學表達式只認數字（連續變數），不認字元（分類變數）；那麼如何將我們收集到的數據中的字元轉換成數字，科學家起了一個比較高端的名字叫做特徵工程（feature engineering） 比如這一小節中使用到的示例數據：1994年美國成年人的收入，此數據集的任務是預測一名工人的收入是高於50,000美元還是低於50,000美元。數據集中的變數包括：
- age
- workclass
- educatiuon
- gender
- hours-per-week
- occupation
- income
其中age(年齡)和hours-per-week(每周工作時長)便是連續特徵；而workclass（工作類型）、education(教育程度)、gender(性別)和occupation(職業)都是分類變數。 那麼如何處理這種情況，一種解決辦法是使用one-hot編碼（或者叫做N取一編碼，也叫作虛擬變數dummy variable）。虛擬變數背後的思想就是將一個分類變數替換為一個或多個新特徵，新特徵取值為0，1，對於數學公式而言0，1兩個值是有意義的。比如數據集
seq | gender | income | hours-per-week |
|---|---|---|---|
1 | Male | 50,000 | 50 |
2 | Female | 60,000 | 40 |
經過轉換就變成另外的格式
seq | Male | Female | income | hours-per-week |
|---|---|---|---|---|
1 | 1 | 0 | 50,000 | 50 |
2 | 0 | 1 | 60,000 | 40 |
python中實現這種轉換法的一種方式是使用pandas中的 get_dummies() 函數
接下來是重複書中的案例
第一步：下載數據集
使用搜索引擎搜索adult.data關鍵詞，找到下載地址 http://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data 可以選擇將其複製到文本文件中，也可以選擇使用python將其抓取下來，這應該是python爬蟲一個非常簡單的案例
- python抓取程式碼
from urllib.request import urlopen html = urlopen("http://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data") adult_data = html.read() adult_data = adult_data.decode('utf-8') fw = open('adult.data',"w",encoding = "utf-8") fw.write(adult_data) fw.close()參考文獻
- https://blog.csdn.net/xman4code/article/details/80989601
- https://www.jianshu.com/p/cfbdacbeac6e
第二步：數據處理與建模
import pandas as pd df = pd.read_csv('adult.data',header=None,index_col=False, names = ['age','workclass','fnlwgt','education','education-num', 'marital-status','occupation','relationship','race','gender', 'capital-gain','capital-loss','hours-per-week','native-country','income']) df.head()輸出結果
age workclass fnlwgt education education-num 0 39 State-gov 77516 Bachelors 13 1 50 Self-emp-not-inc 83311 Bachelors 13 2 38 Private 215646 HS-grad 9 3 53 Private 234721 11th 7 4 28 Private 338409 Bachelors 13 marital-status occupation relationship race gender 0 Never-married Adm-clerical Not-in-family White Male 1 Married-civ-spouse Exec-managerial Husband White Male 2 Divorced Handlers-cleaners Not-in-family White Male 3 Married-civ-spouse Handlers-cleaners Husband Black Male 4 Married-civ-spouse Prof-specialty Wife Black Female capital-gain capital-loss hours-per-week native-country income 0 2174 0 40 United-States <=50K 1 0 0 13 United-States <=50K 2 0 0 40 United-States <=50K 3 0 0 40 United-States <=50K 4 0 0 40 Cuba <=50K
選擇特定的變數
df = df[['age','workclass','education','gender','hours-per-week','occupation','income']]
檢查分類數據是否存在異常使用到value_counts()函數：顯示唯一值及其出現次數
for char in list(df.columns): if df[char].dtypes == "object": print(df[char].value_counts())
輸出結果
Private 22696 Self-emp-not-inc 2541 Local-gov 2093 ? 1836 State-gov 1298 Self-emp-inc 1116 Federal-gov 960 Without-pay 14 Never-worked 7 Name: workclass, dtype: int64 HS-grad 10501 Some-college 7291 Bachelors 5355 Masters 1723 Assoc-voc 1382 11th 1175 Assoc-acdm 1067 10th 933 7th-8th 646 Prof-school 576 9th 514 12th 433 Doctorate 413 5th-6th 333 1st-4th 168 Preschool 51 Name: education, dtype: int64 Male 21790 Female 10771 Name: gender, dtype: int64 Prof-specialty 4140 Craft-repair 4099 Exec-managerial 4066 Adm-clerical 3770 Sales 3650 Other-service 3295 Machine-op-inspct 2002 ? 1843 Transport-moving 1597 Handlers-cleaners 1370 Farming-fishing 994 Tech-support 928 Protective-serv 649 Priv-house-serv 149 Armed-Forces 9 Name: occupation, dtype: int64 <=50K 24720 >50K 7841 Name: income, dtype: int64
可以從結果中看到workclass和occupation變數中包括 「 ？」，接下來刪除包含問號的行
df = df[df['occupation'] != "?"] df = df[df['workclass'] != "?"]
參考文獻
- https://www.cnblogs.com/cocowool/p/8421997.html
使用get_dummies()函數對分類變數進行轉換
df_dummies = pd.get_dummies(df) print("Features after get_dummies: n", list(df_dummies.columns))輸出結果
Features after get_dummies: ['age', 'hours-per-week', 'workclass_Federal-gov', 'workclass_Local-gov', 'workclass_Private', 'workclass_Self-emp-inc', 'workclass_Self-emp-not-inc', 'workclass_State-gov', 'workclass_Without-pay', 'education_10th', 'education_11th', 'education_12th', 'education_1st-4th', 'education_5th-6th', 'education_7th-8th', 'education_9th', 'education_Assoc-acdm', 'education_Assoc-voc', 'education_Bachelors', 'education_Doctorate', 'education_HS-grad', 'education_Masters', 'education_Preschool', 'education_Prof-school', 'education_Some-college', 'gender_Female', 'gender_Male', 'occupation_Adm-clerical', 'occupation_Armed-Forces', 'occupation_Craft-repair', 'occupation_Exec-managerial', 'occupation_Farming-fishing', 'occupation_Handlers-cleaners', 'occupation_Machine-op-inspct', 'occupation_Other-service', 'occupation_Priv-house-serv', 'occupation_Prof-specialty', 'occupation_Protective-serv', 'occupation_Sales', 'occupation_Tech-support', 'occupation_Transport-moving', 'income_<=50K', 'income_>50K']
接下來訓練邏輯斯蒂回歸分類模型
features = df_dummies.ix[:,'age':'occupation_Transport-moving'] # 這個語句不太明白 X = features.values Y = df_dummies['income_>50K'].values print("X.shape: {} Y.shape:{}".format(X.shape,Y.shape)) #輸出 X.shape: (30718, 41) Y.shape:(30718,)使用 ix()時遇到
C:UsersmingyAppDataLocalContinuumanaconda3libsite-packagesipykernel_launcher.py:1: DeprecationWarning: .ix is deprecated. Please use .loc for label based indexing or .iloc for positional indexing See the documentation here: http://pandas.pydata.org/pandas-docs/stable/indexing.html#ix-indexer-is-deprecated """Entry point for launching an IPython kernel.
訓練模型
from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split X_train,X_test,y_train,y_test = train_test_split(X,Y,random_state=0) logreg = LogisticRegression() logreg.fit(X_train,y_train) #輸出結果 LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True, intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1, penalty='l2', random_state=None, solver='liblinear', tol=0.0001, verbose=0, warm_start=False) print("Test score:{:.2f}".format(logreg.score(X_test,y_test))) #輸出結果 Test score:0.81
