[tensorflow] backend.convert_to_numpy fails for MirroredStrategy (ValueError: object __array__ method not producing an array)

[crohkohl]

This call fails for a MirroredStrategy in a multi-gpu tensorflow setting with the following exception:

[ValueError: object __array__ method not producing an array

keras/keras/callbacks/learning_rate_scheduler.py

Line 58 in 66b1c0c

backend.convert_to_numpy(self.model.optimizer.learning_rate)

I modified backend.convert_to_numpy() as follows:

def convert_to_numpy(x):
    if isinstance(x, tf.SparseTensor):
        x = tf.sparse.to_dense(x)
    try:
        return np.array(x)
    except ValueError:
        return x.numpy()

>>> tf.__version__
'2.14.0'
>>> keras_core.__version__
'0.1.7'

[fchollet]

Thanks for the report. Can you provide a code snippet (or a Colab) to reproduce the issue?

[crohkohl]

I have built the latest master version. The following commit demonstrates the error:

master...crohkohl:keras:patch-2

[qlzh727]

Thanks for the example, let me take a look

[qlzh727]

Should be addressed by #18675

[gauravbrills]

Is there a workaround for this facing this on running keras on tensorflow backend tensorflow version on tpu

TensorFlow: 2.14.0
Keras-core: 0.1.7
KerasNLP: 0.6.3

error

ValueError                                Traceback (most recent call last)
Cell In[26], line 24
     21 model = build_model()
     23 # Start training the model
---> 24 history = model.fit(
     25     train_ds,
     26     epochs=CFG.epochs,
     27     validation_data=valid_ds,
     28     callbacks=callbacks,
     29     steps_per_epoch=int(len(train_df) / CFG.batch_size / CFG.replicas),
     30 )
     32 # Find the epoch with the best validation accuracy
     33 best_epoch = np.argmax(model.history.history['val_auc'])

File /usr/local/lib/python3.10/site-packages/keras_core/src/utils/traceback_utils.py:123, in filter_traceback.<locals>.error_handler(*args, **kwargs)
    120     filtered_tb = _process_traceback_frames(e.__traceback__)
    121     # To get the full stack trace, call:
    122     # `keras_core.config.disable_traceback_filtering()`
--> 123     raise e.with_traceback(filtered_tb) from None
    124 finally:
    125     del filtered_tb

File /usr/local/lib/python3.10/site-packages/keras_core/src/backend/tensorflow/core.py:97, in convert_to_numpy(x)
     95 if isinstance(x, tf.SparseTensor):
     96     x = tf.sparse.to_dense(x)
---> 97 return np.array(x)

ValueError: object __array__ method not producing an array

[qlzh727]

Please update to the latest keras version and try again. Thanks.

[gauravbrills]

but keras-core is : 0.1.7 and keras is 2.14.0 which is relatively latest

[qlzh727]

the #18675 is only applied to keras-team/keras but not keras-team/keras-core.