KerasCV Fix-it ⚒️

[divyashreepathihalli]

The main goal is to stabilize KerasCV's integration with Keras 3. The list of outstanding items listed below needs to be adresses to make KerasCV feature complete with Keras 3 is listed below. Please feel free to take up anything here.
Here is the migration guide to move to Keras 3.
If you are taking up an example, do comment down below before you submit the PR.

High level process

• update .py file here - https://github.com/keras-team/keras-io/tree/master/examples/vision
• from keras-io/scripts/ run python autogen.py add_example ../examples/vision/python_file_name
  more contributing details here
• if the example needs import tensorflow as tf - i.e it is a TF only example - please state that is the PR decription
• tensorflow_datasets can be used with Keras 3 code, so if you encounter this, you can leave it as it is and it works.

Update Keras IO examples - [Priority 1]

• learnable_resizer Learnable resizer to keras3 keras-io#1688
• mixup Update mixup keras-io#1680
• perceiver_image_classification [TF only] Update perceiver_image_classification example to Keras 3.0 keras-io#1701
• shiftvit [TF only]
• bit [TF only] Migrating Image Classification using BigTransfer (BiT) example to Keras 3 keras-io#1694
• simsiam @aditya02shah [TF only] Migrating SimSiam Example to Keras-3 keras-io#1697
• generative/dcgan_overriding_train_ste DCGAN overriding training step- Keras 3 Migration (Only Tensorflow Backend) keras-io#1693
• forwardforward
• nnclr @aditya02shah
• handwriting_recognition
• [x ] xray_classification_with_tpus
• edsr
• [ x] mobilevit
• yolov8
• video_classification - Migrate video classification example to keras 3.0 keras-io#1674
• nl_image_search
• supervised-contrastive-learning
• depth_estimation
• fully_convolutional_network Migrating Fully Convolutional Network example to Keras 3 keras-io#1691
• vivit
• randaugment @sachinprasadhs Update RandAugment example to Keras 3 keras-io#1683 (comment)
• consistency_training
• basnet_segmentation
• barlow_twins
• adamatch
• pointnet update pointnet to keras 3 keras-io#1679
• vit_small_ds
• retinanet
• temporal_latent_bottleneck
• near_dup_search
• 3D_image_classification
• swin_transformers
• focal_modulation_network

Generative

• generative/molecule_generation
• generative/gan_ada
• generative/conditional_gan Conditional GAN migration to Keras 3[TF Only Backend] keras-io#1705
• generative/dreambooth
• generative/fine_tune_via_textual_inversion
• generative/finetune_stable_diffusion

[ID6109]

I'm excited to take these on. I'll be sending PRs on these after a week. Thanks!

[soma2000-lang]

Will work on a few of these @ID6109 which ones will you be working on?

[divyashreepathihalli]

@soma2000-lang and @ID6109 Thank you for you interest. I wanted to add an update. We have decided to keep the preprocessing layers TF only for now. So, I have updated the list above.

[ID6109]

I see @divyashreepathihalli. Then perhaps it was good that we waited a bit to initiate the conversions. @soma2000-lang, how about you start from the bottom? I'll begin by starting from the top.

[soma2000-lang]

Sure @ID6109 works for me.

[sami-bg]

perceiver_image_classification uses the LAMB optimizer, implemented in TensorFlow Addons, as recommended from the paper. tfa is deprecated, so this particular example would need a keras-3 reimplementation of LAMB.

@soma2000-lang @ID6109 I'll be starting from the middle

[divyashreepathihalli]

@sami-bg yeah, let us leave the examples with custom implementation to the last.

[sachinprasadhs]

@divyashreepathihalli , For Handwriting recognition, ctc_batch_cost and ctc_decode is deprecated, any alternative that can be used?

[sachinprasadhs]

@sayakpaul , For Near-duplicate image search to migrate to Keras 3 is it possible to provide the pre-trained model saved in .keras format here bit_model = tf.keras.models.load_model("flower_model_bit_0.96875")

[sachinprasadhs]

Mixup seems to be migrated to Keras 3 already here keras-team/keras-io#1595

[divyashreepathihalli]

Thanks Sachin! marked it as done

[sachinprasadhs]

I will work on randaugment

[divyashreepathihalli]

I will work on randaugment

Thank you I will mark it as yours.

[divyashreepathihalli]

@IMvision12 in this example here - can you please add the details on what dataset you are using for the YOLO example here - https://keras.io/examples/vision/yolov8/
How to set it up and this example colab should run end to end ? - right now it cannot because of missing path to data.

[sineeli]

Hi @divyashreepathihalli,

I am trying to migrate retinanet notebook, are there any alternatives that can be used for tf.image functions for example tf.image.flip_right_left. I tried but could find any related in keras_cv.

Thanks

[divyashreepathihalli]

Hi @divyashreepathihalli,

I am trying to migrate retinanet notebook, are there any alternatives that can be used for tf.image functions for example tf.image.flip_right_left. I tried but could find any related in keras_cv.

Thanks

@sineeli you can use the preprocessing layers for that
for example

random_flip = keras_cv.layers.RandomFlip(rate=1.0, mode="horizontal")
fliped_image=random_flip(image)

We also have additional image ops here - https://keras.io/api/ops/image/

[sineeli]

@divyashreepathihalli it seems keras_cv.layers.multi_class_non_max_suppression does not support multi-backend as per the code here and retinanet notebook uses tf.image.combined_non_max_suppression, it's parallel is keras_cv.layers. multi_class_non_max_suppression.

[IMvision12]

@IMvision12 in this example here - can you please add the details on what dataset you are using for the YOLO example here - https://keras.io/examples/vision/yolov8/ How to set it up and this example colab should run end to end ? - right now it cannot because of missing path to data.

Hi @divyashreepathihalli you can use this to load data

!gdown 19CWdNL3ePq9XIgMeb_RjA641iGra3nWm
!unzip data.zip

and update the code for loading the data

# Path to images and annotations
path_images = "./data/images/"
path_annot = "./data/annotations/"

[divyashreepathihalli]

thank you @IMvision12!

[aditya02shah]

Hi @divyashreepathihalli
I am migrating the simsiam example. It uses an implementation of ResNet20 that will need to be updated too.
Thanks

[sachinprasadhs]

Learnable resizer: keras-team/keras-io#1688

[sitamgithub-MSIT]

@divyashreepathihalli I would like to take up the Efficient Object Detection with YOLOV8 and KerasCV example.

[divyashreepathihalli]

@divyashreepathihalli I would like to take up the Efficient Object Detection with YOLOV8 and KerasCV example.

Thanks @sitamgithub-MSIT - please also take a look at this comment for updating the example - #2211 (comment)

[sitamgithub-MSIT]

I tried the shiftvit [TF only] (A Vision Transformer without Attention) example with Keras 3. After a few changes, it is working fine. I will make a PR soon regarding that.

[pranavvp16]

I have migrated the x_ray_classification_tpu example to keras3. keras-team/keras-io#1756

[sitamgithub-MSIT]

@divyashreepathihalli Apart from these examples, there are some examples I tried out for which a problem appeared: a missing dataset. For example, the public dataset link does not exist right now. Or if data is taken from the GCP bucket, it does not exist. Should I mark them then anywhere?

[sitamgithub-MSIT]

Also, some of our examples, I worked I see the PR already exists, but somehow it's been a month since the contributor added those changes to the example.py file. It is not merged due to the fact that .ipynb and .md files are not generated. What should we do in those cases?

[divyashreepathihalli]

@divyashreepathihalli Apart from these examples, there are some examples I tried out for which a problem appeared: a missing dataset. For example, the public dataset link does not exist right now. Or if data is taken from the GCP bucket, it does not exist. Should I mark them then anywhere?

Yes, I do know about this one, can you share what dataset you are talking about? We might be able to find a mirror site.

[divyashreepathihalli]

Also, some of our examples, I worked I see the PR already exists, but somehow it's been a month since the contributor added those changes to the example.py file. It is not merged due to the fact that .ipynb and .md files are not generated. What should we do in those cases?

Please do respond on that PR that you would like to take it up.

[sitamgithub-MSIT]

@divyashreepathihalli Apart from these examples, there are some examples I tried out for which a problem appeared: a missing dataset. For example, the public dataset link does not exist right now. Or if data is taken from the GCP bucket, it does not exist. Should I mark them then anywhere?

Yes, I do know about this one, can you share what dataset you are talking about? We might be able to find a mirror site.

Classification with Gated Residual and Variable Selection Networks In this example, the dataset currently does not exist.

How to train a Keras model on TFRecord files In this example, the GCS bucket does not exist at present.

[sitamgithub-MSIT]

@divyashreepathihalli It is possible to replace certain TF ops or numpy ops used in older examples that were converted to Keras 3 with equivalent Keras ops without experiencing any issues with the code's overall functionality. So, are those in need of any modifications? Since the examples are ultimately operating properly with those unaltered tf or np ops using the designated backends.

[divyashreepathihalli]

@sitamgithub-MSIT we should be replaceing all tf ops with keras ops and the functionality should still be the same. I did not quite understand your second question? do you have an example?

[divyashreepathihalli]

@sitamgithub-MSIT > Classification with Gated Residual and Variable Selection Networks In this example, the dataset currently does not exist.
found this one https://archive.ics.uci.edu/static/public/20/census+income.zip
How to train a Keras model on TFRecord files In this example, the GCS bucket does not exist at present. - not sure

[sitamgithub-MSIT]

@sitamgithub-MSIT we should be replaceing all tf ops with keras ops and the functionality should still be the same. I did not quite understand your second question? do you have an example?

For example, this section of code from this keras 3 code example Simple MNIST convnet

# Scale images to the [0, 1] range
x_train = x_train.astype("float32") / 255
x_test = x_test.astype("float32") / 255
# Make sure images have shape (28, 28, 1)
x_train = np.expand_dims(x_train, -1)
x_test = np.expand_dims(x_test, -1)
print("x_train shape:", x_train.shape)
print(x_train.shape[0], "train samples")
print(x_test.shape[0], "test samples")

Here, np.expand_dims can be replaced with ops.expand_dims. I tried and worked fine with every backend.

Another one from this keras 3 code example is Image classification via fine-tuning with EfficientNet

size = (IMG_SIZE, IMG_SIZE)
ds_train = ds_train.map(lambda image, label: (tf.image.resize(image, size), label))
ds_test = ds_test.map(lambda image, label: (tf.image.resize(image, size), label))

Here is also keras.ops.image.resize, which can be used.

So, that I am asking, these examples are already marked as keras 3 on the website, so should I change this ops and let it be as it is?

[sitamgithub-MSIT]

@divyashreepathihalli for this code (x_train, y_train), (x_test, y_test) = keras.datasets.cifar100.load_data() getting this error Exception: URL fetch failure on https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz: 503 -- Service Unavailable.

Update: Fixed

[sitamgithub-MSIT]

Data-Parallel Training with KerasNLP and tf.distribute This is an example using a dataset that shows 403: Forbidden. Giving the message "Access Denied.".

Executed Notebook: https://colab.research.google.com/drive/1RbY321lKSodVP3Iy2vVq4kRlTUpfflMp#scrollTo=BfVweZvmOVpI

cc: @divyashreepathihalli

[divyashreepathihalli]

@sitamgithub-MSIT kerasNLP would be a better place for this question.

[sitamgithub-MSIT]

@sitamgithub-MSIT KerasNLP would be a better place for this question.

Ok, I will raise an issue then in KerasNLP. Also here in this example CycleGAN, instance normalization is applied, and Keras does not have any support as of now. So, I replaced this with a batch normalization layer in Keras. Another thing is that the author proposed to run the model for only one epoch, as it takes nearly 7 to 8 minutes for one epoch. So I did that and made a few changes to the checkpoint file name so the weights could be stored. But the author here did the inference step with the same model architecture that ran over 90 epochs. As the model is not in.weights.h5 or.keras format, I cannot perform the inferencing. So I need your guidance on this. Also, here is the notebook I ran for reference.

Executed Notebook: https://colab.research.google.com/drive/1NkXKivVipH2XrbbVlDJh0rgUogtr6yKZ?usp=sharing