我想创建一个RDD (实际的,不是模拟的),它在单元测试中包含模拟的元素(使用Mockito)。
我的尝试是:
lazy val sc = SparkContext.getOrCreate()
val myRDD = sc.parallelize(Seq( (Mockito.mock(classOf[ImmutableBytesWritable]), Mockito.mock(classOf[Result])) ))
其中ImmutableBytesWritable
和Result
来自HBase接口。我有org.apache.spark.SparkException: Task not serializable
有没有可能实现我的目标?谢谢!
发布于 2018-08-10 18:20:32
默认情况下,Mockito mock是不可序列化的,这就是出现错误的原因。
要创建可序列化的mock,您必须显式定义它:
mock = Mockito.mock(
classOf[ImmutableBytesWritable],
Mockito.withSettings().serializable()
)
同样的事情也应该应用到你的Result
模拟中。
如果你得到一个java.lang.ClassNotFoundException: org.apache.hadoop.hbase.io.ImmutableBytesWritable$MockitoMock$...
异常,你可能需要使用:
import org.mockito.mock.SerializableMode
mock = Mockito.mock(
classOf[ImmutableBytesWritable],
Mockito.withSettings().serializable(SerializableMode.ACROSS_CLASSLOADERS)
)
最后,你应该有类似这样的东西:
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.apache.hadoop.hbase.io.ImmutableBytesWritable
import org.apache.hadoop.hbase.client.Result
import org.mockito.Mockito
import org.mockito.mock.SerializableMode
object Test extends App {
val conf = new SparkConf()
.setMaster("local[2]")
.setAppName("test")
lazy val sc = new SparkContext(conf)
val mockImmutableBytesWritable = Mockito.mock(
classOf[ImmutableBytesWritable],
Mockito.withSettings().serializable(
SerializableMode.ACROSS_CLASSLOADERS
)
)
val mockResult = Mockito.mock(
classOf[Result],
Mockito.withSettings().serializable(
SerializableMode.ACROSS_CLASSLOADERS
)
)
val myRDD = sc.parallelize(Seq((mockImmutableBytesWritable, mockResult)))
println(myRDD.count)
sc.stop()
}
https://stackoverflow.com/questions/51774731
复制相似问题