用于在PySpark上多次更改一列的链withColumn
用于在PySpark上多次更改一列的链withColumn
提问于 2018-08-25 19:18:24
我使用的是UCI的成人年收入。
我有一个数据帧，在一列中有一个类别变量，我想将它分组到不同的类别中(一些常见的特征工程)。
df.groupBy('education').count().show()提供：
+------------+-----+
| education|count|
+------------+-----+
| 10th| 1223|
| Masters| 2514|
| 5th-6th| 449|
| Assoc-acdm| 1507|
| Assoc-voc| 1959|
| 7th-8th| 823|
| 9th| 676|
| HS-grad|14783|
| Bachelors| 7570|
| 11th| 1619|
| 1st-4th| 222|
| Preschool| 72|
| 12th| 577|
| Doctorate| 544|
|Some-college| 9899|
| Prof-school| 785|
+------------+-----+我想将以下类别放在特定的组中，这样：
dropout = ['Preschool', '1st-4th', '5th-6th', '7th-8th', '9th', '10th', '11th', '12th']
community_college = ['Assoc-acdm', 'Assoc-voc', 'Some-college']
masters = ['Prof-school']为此，我可以执行以下操作：
from pyspark.sql.functions import when, col
df = df.withColumn('education', when(col('education').isin(dropout), 'Dropout').otherwise(df['education']))
df = df.withColumn('education', when(col('education').isin(community_college), 'Community_college').otherwise(df['education']))
df = df.withColumn('education', when(col('education') == 'Prof-school', 'Masters').otherwise(df['education']))获取：
+-----------------+-----+
| education|count|
+-----------------+-----+
| Masters| 3299|
| HS-grad|14783|
| Bachelors| 7570|
| Dropout| 5661|
| Doctorate| 544|
|Community_college|13365|
+-----------------+-----+有没有可能把这些withColumn链接起来？我尝试了以下几种方法，但都没有成功：
df = df.withColumn('education', when(col('education').isin(dropout), 'Dropout').otherwise(df['education']))\
.withColumn('education', when(col('education').isin(community_college), 'Community_college').otherwise(df['education']))\
.withColumn('education', when(col('education') == 'Prof-school', 'Masters').otherwise(df['education']))回答 1
Stack Overflow用户
回答已采纳
发布于 2018-08-25 23:13:16
是的，有通过链接when()。
df = df.withColumn('education', when(col('education').isin(dropout), 'Dropout')\
.when(col('education').isin(community_college), 'Community_college')\
.when(col('education') == 'Prof-school', 'Masters') \
.otherwise(df['education']))页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/52016771
复制相关文章
相似问题
腾讯云开发者
