these 7 derivations will be built: /nix/store/81k9sm7bzkhcqfs9k3cr0p41qr5bvhk0-python3.9-jax-0.3.23.drv /nix/store/lfh96z1q2zm5s94yh96l2iamyigjhj26-python3.9-chex-0.1.5.drv /nix/store/i9rgalibn8jx1kp3hqwghiqg9my9ccy2-python3.9-optax-0.1.4.drv /nix/store/ywkwlwdh37xy4ikan4srll1bsmr9haml-tensorflow-gpu-2.10.1.drv /nix/store/kg91xfl228in2afz7bkk7mjm1abicw2k-python3.9-tensorflow-gpu-2.10.1.drv /nix/store/qgqwbizb9cd80rdbry4p870j1gc0qaw4-python3.9-flax-0.6.1.drv /nix/store/wwdna2ndarbkc1vjqs34q0frpyziy4vf-python3.9-vqgan-jax-unstable-2022-04-20.drv error: Cannot build '/nix/store/ywkwlwdh37xy4ikan4srll1bsmr9haml-tensorflow-gpu-2.10.1.drv'. Reason: builder failed with exit code 137. Output paths: /nix/store/0sxzqhpqzz812gzyq3c3p4hzhq3pk4g8-tensorflow-gpu-2.10.1 /nix/store/h0yynv6awil0qdi21rig7w32x1x61chi-tensorflow-gpu-2.10.1-python Last 25 log lines: > WARNING: All illegal access operations will be denied in a future release > [146 / 2,427] Compiling llvm/lib/Demangle/ItaniumDemangle.cpp; 15s local ... (48 actions, 47 running) > [242 / 2,427] Compiling mlir/tools/mlir-tblgen/OpDefinitionsGen.cpp; 32s local ... (48 actions, 47 running) > [1,929 / 3,200] Compiling llvm/lib/TableGen/Record.cpp; 47s local ... (48 actions, 47 running) > [2,079 / 3,200] Compiling llvm/lib/TableGen/Record.cpp; 73s local ... (48 actions, 47 running) > [2,683 / 6,828] Compiling llvm/lib/Support/VirtualFileSystem.cpp; 28s local ... (48 actions, 47 running) > [3,052 / 6,828] Compiling tensorflow/core/protobuf/meta_graph.pb.cc; 60s local ... (48 actions, 47 running) > [3,126 / 6,828] Compiling mlir/lib/IR/BuiltinAttributes.cpp; 57s local ... (48 actions, 47 running) > [3,240 / 6,828] Compiling mlir/lib/Dialect/PDLInterp/IR/PDLInterp.cpp; 89s local ... (47 actions, 46 running) > [3,771 / 7,385] Compiling tensorflow/core/ir/ops.cc; 102s local ... (48 actions, 47 running) > [7,849 / 12,428] Compiling llvm/utils/TableGen/GlobalISelEmitter.cpp; 125s local ... (48 actions, 47 running) > [8,118 / 12,695] Compiling tensorflow/core/common_runtime/executor.cc; 93s local ... (48 actions, 47 running) > [8,324 / 12,877] Compiling mlir/lib/IR/AsmPrinter.cpp; 92s local ... (48 actions, 47 running) > [8,569 / 13,217] Compiling tensorflow/core/common_runtime/gpu/gpu_device.cc; 132s local ... (48 actions, 47 running) > [8,924 / 13,364] Compiling tensorflow/core/transforms/constant_folding/pass.cc; 204s local ... (48 actions, 47 running) > [9,205 / 13,579] Compiling tensorflow/core/transforms/graph_compactor/pass.cc; 118s local ... (48 actions, 47 running) > [9,523 / 13,797] Compiling tensorflow/core/grappler/optimizers/loop_optimizer.cc; 120s local ... (48 actions, 47 running) > [10,656 / 15,899] Compiling tensorflow/stream_executor/cuda/cuda_dnn.cc; 138s local ... (48 actions, 47 running) > [11,201 / 16,445] Compiling tensorflow/compiler/mlir/hlo/lib/Dialect/mhlo/IR/hlo_ops.cc; 203s local ... (48 actions, 47 running) > [11,688 / 17,533] Compiling tensorflow/compiler/mlir/hlo/lib/Dialect/mhlo/IR/hlo_ops.cc; 420s local ... (48 actions, 47 running) > [12,271 / 18,456] Compiling tensorflow/compiler/mlir/tensorflow/ir/tf_ops_a_m.cc; 569s local ... (48 actions, 47 running) > [14,056 / 22,093] Compiling tensorflow/compiler/mlir/hlo/lib/Dialect/mhlo/transforms/hlo_legalize_to_lhlo.cc; 186s local ... (48 actions, 47 running) > [14,878 / 23,647] Compiling external/nccl_archive/src/collectives/device/sum_u8_all_reduce.cu.cc; 212s local ... (48 actions running) > [15,040 / 23,728] Compiling external/nccl_archive/src/collectives/device/sum_u8_all_reduce.cu.cc; 586s local ... (48 actions running) > /nix/store/cp65c8nk29qq5cl1wyy5qyw103cwmax7-stdenv-linux/setup: line 1442: 3320 Killed BAZEL_USE_CPP_ONLY_TOOLCHAIN=1 USER=homeless-shelter bazel --batch --output_base="$bazelOut" --output_user_root="$bazelUserRoot" build --curses=no -j $NIX_BUILD_CORES "${copts[@]}" "${host_copts[@]}" "${linkopts[@]}" "${host_linkopts[@]}" $bazelFlags --config=opt //tensorflow/tools/pip_package:build_pip_package //tensorflow/tools/lib_package:libtensorflow For full logs, run: nix log /nix/store/ywkwlwdh37xy4ikan4srll1bsmr9haml-tensorflow-gpu-2.10.1.drv error: Cannot build '/nix/store/kg91xfl228in2afz7bkk7mjm1abicw2k-python3.9-tensorflow-gpu-2.10.1.drv'. Reason: 1 dependency failed. Output paths: /nix/store/g1qk48dl0ahfsj13344fzpxshjkk997x-python3.9-tensorflow-gpu-2.10.1 /nix/store/j79wmxlsp0chwgngcb8lc9jfcv2hnnwx-python3.9-tensorflow-gpu-2.10.1-dist error: Cannot build '/nix/store/81k9sm7bzkhcqfs9k3cr0p41qr5bvhk0-python3.9-jax-0.3.23.drv'. Reason: builder failed with exit code 1. Output paths: /nix/store/d4hmm3n160rc4h5a5svrim5r8wcd2n4c-python3.9-jax-0.3.23-dist /nix/store/nly6ywi9licvihy06rl2jvp2jf69wm7l-python3.9-jax-0.3.23 Last 25 log lines: > ........................................................................ [ 61%] > ........................................................................ [ 61%] > ........................................................................ [ 62%] > ........................................................................ [ 63%] > ........................................................................ [ 63%] > .............s.......................................................... [ 64%] > ........................................................................ [ 64%] > ........................................................................ [ 65%] > ........................................................................ [ 66%] > ........................................................................ [ 66%] > ........................................................................ [ 67%] > ........................................................................ [ 67%] > ........................................................................ [ 68%] > ..............................s......................................... [ 69%] > ............s..s[gw3] node down: Not properly terminated > F > replacing crashed worker gw3 > gw0 [11972] / gw1 [11972] / gw2 [11972] / gw4 [11972]sssss.s.ssssssssssssssssssssssss..ssssssssssssssssssssssssssssssssssssssss.sssssssssss.s................................sssss.....sssss..ss.sssssssssss..........sssssssssss.ssssssssssss.sss.........................sss.s......ssssss.......s.sss.ssssssssssssssssssss.ssss.......s.ss.s.s.................................s.........s.....................ss..s..........................................s...................................................ss.s.s....ss..........................s...s..s.......s................................s.................s.......ss..............................s....s.........s..........................s......s....s......s.......s..sssss....ssssssssssssssssssssssssssss...ss.........s...s........ss...s...........s..........s......................................ss.ss..ss......s...s..s...s.......ssssssssssssssssss.........ss.sssssssss.s...s...ss.ss..s........s..s......s............................ss......................ssssssssssssssssss..ssss.....sssss....ssssssssssssssssssssssssssss...ss........s.......ss..s.....s............ss........s.......ss..s....sssss.ss...s..s.s...s......s.....................sssss.ss.............s..s.s...s...........s...s..ss........s..s......s.s....s.....ss.....s...s.sssss....ssssssssssssssssssssssssssss..........................................s..ss.....................s..................s...s...........s.s......s.....ss...........s..ss...ss..s..........................s............................ssss.......ssss................................ss..........ss.........................ss....s.........................s...........ss..s.............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ss....s..ss...........................................s..................................ss..sssss...ss..s.s............s.s...........ss.s.sss....ss..sssss...ss.s.s.s............s.s...........sss.s.sssssssssss..........ssssssssss.........................ssssssss.s..s.sssss..ss.ss........s.......s.ss.sssss.s....................................s...................................................................................................................................................................................................................s........s...........................s...................ssssssss....s............................ssssssssss.........................................................................................s..........ss....................................................................................................................................................................................s.ssssssssss.ssssssssss.ss...ssssssssssssssssss...........................................................................ss..............sss............................... > =================================== FAILURES =================================== > ___________________________ tests/lax_scipy_test.py ____________________________ > [gw3] linux -- Python 3.9.16 /nix/store/30xlnfzi3g3liciakaa5za905grmp81j-python3-3.9.16/bin/python3.9 > worker 'gw3' crashed while running 'tests/lax_scipy_test.py::LaxBackedScipyTests::test_spectral_dac_eigh_linear_size=256_dtype=complex64_termination_size=1' > =========================== short test summary info ============================ > FAILED tests/lax_scipy_test.py::LaxBackedScipyTests::test_spectral_dac_eigh_linear_size=256_dtype=complex64_termination_size=1 > ========== 1 failed, 11006 passed, 965 skipped in 8979.92s (2:29:39) =========== For full logs, run: nix log /nix/store/81k9sm7bzkhcqfs9k3cr0p41qr5bvhk0-python3.9-jax-0.3.23.drv error: Cannot build '/nix/store/lfh96z1q2zm5s94yh96l2iamyigjhj26-python3.9-chex-0.1.5.drv'. Reason: 1 dependency failed. Output paths: /nix/store/hf2f1gx8qprxvajng374v5qpm715b73c-python3.9-chex-0.1.5-dist /nix/store/n84n0h43fxfwhar1chg21sqzl71sgibc-python3.9-chex-0.1.5 error: Cannot build '/nix/store/i9rgalibn8jx1kp3hqwghiqg9my9ccy2-python3.9-optax-0.1.4.drv'. Reason: 1 dependency failed. Output paths: /nix/store/0a78irl2nlhm7cw3kwdzg6fs22axs31x-python3.9-optax-0.1.4-dist /nix/store/ixlqmj308q4x78z9qygpxx2mgh06hq4l-python3.9-optax-0.1.4 /nix/store/nx9hsa8m36h5727lbnzm51nz22qb8lcb-python3.9-optax-0.1.4-testsout error: Cannot build '/nix/store/qgqwbizb9cd80rdbry4p870j1gc0qaw4-python3.9-flax-0.6.1.drv'. Reason: 3 dependencies failed. Output paths: /nix/store/fis6ysnz610p8nls81qh2lav7wabya04-python3.9-flax-0.6.1-dist /nix/store/n7l03ban1dpz966da3nbg6c4qyr29jfz-python3.9-flax-0.6.1 error: Cannot build '/nix/store/wwdna2ndarbkc1vjqs34q0frpyziy4vf-python3.9-vqgan-jax-unstable-2022-04-20.drv'. Reason: 2 dependencies failed. Output paths: /nix/store/hiy8g83lr852kawzxzg58wgy9jj3qyn6-python3.9-vqgan-jax-unstable-2022-04-20 /nix/store/zvy4r0rr5vh5gfa1nis70ss6pil649dm-python3.9-vqgan-jax-unstable-2022-04-20-dist