从pyspark数据帧创建Numpy矩阵
从pyspark数据帧创建Numpy矩阵
提问于 2017-11-29 22:33:33
我有一个pyspark dataframe child，它的列如下：
lat1 lon1
80 70
65 75我正在尝试使用IndexedRowMatrix将其转换为numpy矩阵，如下所示：
from pyspark.mllib.linalg.distributed import IndexedRow, IndexedRowMatrix
mat = IndexedRowMatrix(child.select('lat','lon').rdd.map(lambda row: IndexedRow(row[0], Vectors.dense(row[1:]))))但这让我大错特错。我希望避免转换为pandas dataframe来获取矩阵。
错误：
Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 33.0 failed 4 times, most recent failure: Lost task 0.3 in stage 33.0 (TID 733, ebdp-avdc-d281p.sys.comcast.net, executor 16): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/data/02/yarn/nm/usercache/mbansa001c/appcache/application_1506130884691_56333/container_e48_1506130884691_56333_01_000017/pyspark.zip/pyspark/worker.py", line 174, in main
process()回答
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/47555231
复制相关文章
相似问题
腾讯云开发者
