Failing unit test on macOS 10.15 with nixpkgs 19.09 (beta)

[danieldk]

The failure part of the log. Environment file below (but pretty boring), same setup works on Linux.

=================================== FAILURES ===================================
7: ______________________________ test_graph_lookup _______________________________
7:
7: ff_lib = <module '35d08339424fb7a799b84234e2d9d0a3'>
7:
7:     def test_graph_lookup(ff_lib):
7:         embeddings = ff_lib.ff_embeddings()
7: >       init = ff_lib.initialize_ff_embeddings(embeddings, "tests/data/test.fifu", False)
7:
7: ../../tests/test_graph_mode.py:8:
7: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
7: <string>:172: in initialize_ff_embeddings
7:     ???
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:527: in _apply_op_helper
7:     preferred_dtype=default_dtype)
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:1224: in internal_convert_to_tensor
7:     ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py:305: in _constant_tensor_conversion_function
7:     return constant(v, dtype=dtype, name=name)
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py:246: in constant
7:     allow_broadcast=True)
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py:284: in _constant_impl
7:     allow_broadcast=allow_broadcast))
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/tensorflow/python/framework/tensor_util.py:573: in make_tensor_proto
7:     append_fn(tensor_proto, proto_values)
7: tensorflow/python/framework/fast_tensor_util.pyx:134: in tensorflow.python.framework.fast_tensor_util.AppendBoolArrayToTensorProto
7:     ???
7: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
7:
7:     def append(self, value):
7:       """Appends an item to the list. Similar to list.append()."""
7: >     self._values.append(self._type_checker.CheckValue(value))
7: E     UnboundLocalError: local variable 'self' referenced before assignment
7:
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/google/protobuf/internal/containers.py:255: UnboundLocalError
7: ___________________________ test_graph_lookup_masked ___________________________
7:
7: ff_lib = <module '35d08339424fb7a799b84234e2d9d0a3'>
7:
7:     def test_graph_lookup_masked(ff_lib):
7:         embeddings = ff_lib.ff_embeddings()
7: >       init = ff_lib.initialize_ff_embeddings(embeddings, "tests/data/test.fifu", True)
7:
7: ../../tests/test_graph_mode.py:41:
7: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
7: <string>:172: in initialize_ff_embeddings
7:     ???
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:527: in _apply_op_helper
7:     preferred_dtype=default_dtype)
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:1224: in internal_convert_to_tensor
7:     ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py:305: in _constant_tensor_conversion_function
7:     return constant(v, dtype=dtype, name=name)
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py:246: in constant
7:     allow_broadcast=True)
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py:284: in _constant_impl
7:     allow_broadcast=allow_broadcast))
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/tensorflow/python/framework/tensor_util.py:573: in make_tensor_proto
7:     append_fn(tensor_proto, proto_values)
7: tensorflow/python/framework/fast_tensor_util.pyx:134: in tensorflow.python.framework.fast_tensor_util.AppendBoolArrayToTensorProto
7:     ???
7: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
7:
7:     def append(self, value):
7:       """Appends an item to the list. Similar to list.append()."""
7: >     self._values.append(self._type_checker.CheckValue(value))
7: E     UnboundLocalError: local variable 'self' referenced before assignment
7:
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/google/protobuf/internal/containers.py:255: UnboundLocalError
7: ______________________________ test_graph_errors _______________________________
7:
7: ff_lib = <module '35d08339424fb7a799b84234e2d9d0a3'>
7:
7:     def test_graph_errors(ff_lib):
7:         embeddings = ff_lib.ff_embeddings()
7:         tuebingen_unmasked = ff_lib.ff_lookup(embeddings, "Tübingen", mask_empty_string=False, mask_failed_lookup=False,
7:                                               embedding_len=100)
7:         ber_bad_shape = ff_lib.ff_lookup(embeddings, "Berlin", mask_empty_string=False, mask_failed_lookup=False,
7:                                          embedding_len=10)
7:         assert ber_bad_shape.shape == (10,)
7:         empty_unmasked = ff_lib.ff_lookup(embeddings, "", mask_empty_string=False, mask_failed_lookup=False,
7:                                           embedding_len=100)
7:
7:         with tf.Session() as sess:
7:             with pytest.raises(tf.errors.UnknownError):
7: >               sess.run([ff_lib.initialize_ff_embeddings(embeddings, "foo.fifu", False)])
7:
7: ../../tests/test_graph_mode.py:65:
7: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
7: <string>:172: in initialize_ff_embeddings
7:     ???
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:527: in _apply_op_helper
7:     preferred_dtype=default_dtype)
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/tensorflow/python/framework/ops.py:1224: in internal_convert_to_tensor
7:     ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py:305: in _constant_tensor_conversion_function
7:     return constant(v, dtype=dtype, name=name)
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py:246: in constant
7:     allow_broadcast=True)
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py:284: in _constant_impl
7:     allow_broadcast=allow_broadcast))
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/tensorflow/python/framework/tensor_util.py:573: in make_tensor_proto
7:     append_fn(tensor_proto, proto_values)
7: tensorflow/python/framework/fast_tensor_util.pyx:134: in tensorflow.python.framework.fast_tensor_util.AppendBoolArrayToTensorProto
7:     ???
7: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
7:
7:     def append(self, value):
7:       """Appends an item to the list. Similar to list.append()."""
7: >     self._values.append(self._type_checker.CheckValue(value))
7: E     UnboundLocalError: local variable 'self' referenced before assignment
7:
7: /nix/store/hpi9pphfy2pg3w5j4yhl1bqbv7slclvh-python3-3.7.4-env/lib/python3.7/site-packages/google/protobuf/internal/containers.py:255: UnboundLocalError
7: ----------------------------- Captured stderr call -----------------------------
7: WARNING: Logging before flag parsing goes to stderr.
7: W0925 14:35:32.100250 4597292352 deprecation_wrapper.py:119] From /Users/daniel/git/finalfusion-tensorflow/tests/test_graph_mode.py:63: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
7:
7: 2019-09-25 14:35:32.102395: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
7: ------------------------------ Captured log call -------------------------------
7: WARNING  tensorflow:deprecation_wrapper.py:119 From /Users/daniel/git/finalfusion-tensorflow/tests/test_graph_mode.py:63: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
7: ============================== 3 failed in 1.07s ===============================
7/7 Test #7: graph-mode .......................***Failed    5.75 sec

86% tests passed, 1 tests failed out of 7

Total Test time (real) =   9.89 sec

The following tests FAILED:
	  7 - graph-mode (Failed)
Errors while running CTest

shell.nix:

with import <nixpkgs> {};

stdenv.mkDerivation rec {
  name = "tensorflow-op-env";
  env = buildEnv { name = name; paths = buildInputs; };

  nativeBuildInputs = [
    cargo
    cmake
  ];

  buildInputs = [
    darwin.Security
    (python3.withPackages (ps: with ps; [
      pytest
      setuptools
      tensorflow-bin
    ]))
  ];

  propagatedBuildInputs = [
  ];
}

[sebpuetz]

This is pretty strange and seems to originate in tensorflow or protobuf? tensorflow/tensorflow#23410 sounds similar but the issue is supposedly solved since 1.13/1.14

If I'm reading the logs correctly, then this error originates in the following method:

def AppendBoolArrayToTensorProto(tensor_proto, nparray):
  cdef long i, n
  n = nparray.size
  for i in range(n):
    tensor_proto.bool_val.append(nparray.item(i))

which ends up calling the method in protobuf:

  def append(self, value):
    """Appends an item to the list. Similar to list.append()."""
    self._values.append(self._type_checker.CheckValue(value))
    if not self._message_listener.dirty:
      self._message_listener.Modified()

https://github.com/protocolbuffers/protobuf/blob/b9f405ae46036860a4e73e167bee3800dfe53a9e/python/google/protobuf/internal/containers.py#L255-L259

So something is going wrong in that method call that leaves self uninitialized?

I don't have a machine with MacOS 10.15 and travis-ci only supports up to 10.14, I'm not sure on how to proceed with this issue

[danieldk]

This may be a 10.15 issue, Apple is notorious for breaking things between releases and since this is Cython code there may be some weird interaction between Tensorflow (compiled with older macOS) and something that was compiled with 10.15.

But that's just a wild guess.

I might also try with macOS outside Nix, but I dread setting up all the stuff, since the machine is only half functional anyway since the update to 10.15 (that's what you get for running betas). Let's see what Patricia finds with her Mac.

[twuebi]

I might also try with macOS outside Nix, but I dread setting up all the stuff, since the machine is only half functional anyway since the update to 10.15 (that's what you get for running betas). Let's see what Patricia finds with her Mac.

No failing unit tests on 10.14.6