我正在通过Java MapReduce程序尝试Hbase - bulkLoad。我在Eclipse中运行我的程序。
但是我得到了以下错误:
12/06/14 20:04:28 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
12/06/14 20:04:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java cla
使用hadoop流:
hadoop jar hadoop/contrib/streaming/hadoop-streaming-1.0.3.jar -file mapper.rb -mapper mapper.rb -file reducer.rb -reducer reducer.rb -input textfile.txt -output output
假设我所在的目录是“/home/user/sei/ textfile.txt”和
1) is in the same folder as the directory I am currently in
2) I did not use -
我已经安装了pyspark,但没有单独安装任何hadoop或spark版本。
显然,在Windows下,pyspark需要访问Hadoop的winutils.exe来做一些事情(例如,将文件写入磁盘)。当pyspark想要访问winutilis.exe时,它会在HADOOP_HOME环境变量(用户变量)指定的文件夹的bin目录中查找它。因此,我将winutils.exe复制到pyspark (.\site-packages\pyspark\bin)的bin目录中,并将HADOOP_HOME指定为.\site-packages\pyspark\。这解决了获取错误消息的问题:Failed to l