我目前正试图在Google Colab的多个tpu核心上运行一些代码,但当同步代码(xm.rendezvous)在目标函数的末尾被调用时,我似乎得到了一个错误,但现在当同步代码在顶部时。下面是一个例子:
# "Map function": acquires a corresponding Cloud TPU core, creates a tensor on it,
# and prints its core
def simple_map_fn(index, flags):
# xm.rendezvous('init') # place rendezvous here instead of at the bottom works fine.
# Acquires the (unique) Cloud TPU core corresponding to this process's index
device = xm.xla_device()
ordinal = xm.get_ordinal()
local_ordinal = xm.get_ordinal()
print(f"index {index}, process device {device}, local ordinal {local_ordinal}, ordinal {ordinal}")
# Barrier to prevent master from exiting before workers connect.
xm.rendezvous('leave')
# Spawns eight of the map functions, one for each of the eight cores on
# the Cloud TPU
flags = {}
xmp.spawn(simple_map_fn, args=(flags,), nprocs=8, start_method='fork')当我在Google Colab中运行上面的代码时,就像在这个notebook中一样,我得到以下错误:
Exception in device=TPU:7: tensorflow/compiler/xla/xla_client/mesh_service.cc:294 : Failed to meet rendezvous 'leave': Socket closed (14)你知道为什么当会合被放在目标函数的底部时会失败吗?
发布于 2020-06-10 08:59:59
经过仔细的调查,我发现当google colab实例作为“高内存”实例运行时,这个问题不会发生。我的结论是,这个错误最有可能的原因是内存之外的错误。
https://stackoverflow.com/questions/62293934
复制相似问题