The training did not converge in the custom dataset

[Shu-Ang]

I have a custom dataset of 4000 rows of data. BUT the training did NOT converge (epoch=100,lr=0.001)
Here is my config:

TRAIN:
  ENABLE: True
  DATASET: ava
  BATCH_SIZE: 8
  EVAL_PERIOD: 1
  CHECKPOINT_PERIOD: 2
  AUTO_RESUME: True
  CHECKPOINT_FILE_PATH: /data/zhangshuang/workspace/SlowFast/checkpoints/SLOWFAST_4x16_R50.pkl
  CHECKPOINT_TYPE: caffe2
DATA:
  NUM_FRAMES: 32  # α * T = 8 * 4
  SAMPLING_RATE: 2  #τ
  TRAIN_JITTER_SCALES: [256, 320]
  TRAIN_CROP_SIZE: 224
  TEST_CROP_SIZE: 224
  # TEST_CROP_SIZE: 256
  INPUT_CHANNEL_NUM: [3, 3]
  PATH_TO_DATA_DIR: '/data/zhangshuang/workspace/myAVA/Dataset'
DETECTION:
  ENABLE: True
  ALIGNED: True
AVA:
  BGR: False
  DETECTION_SCORE_THRESH: 0.5
  FRAME_DIR: /data/zhangshuang/workspace/myAVA/Dataset/rawframes
  FRAME_LIST_DIR: /data/zhangshuang/workspace/myAVA/Dataset/frame_lists
  ANNOTATION_DIR: /data/zhangshuang/workspace/myAVA/Dataset/annotations
  TRAIN_GT_BOX_LISTS: ["train.csv"]
  TRAIN_PREDICT_BOX_LISTS: []
  TEST_PREDICT_BOX_LISTS: [val.csv]
  EXCLUSION_FILE: train_excluded_timestamps.csv
  LABEL_MAP_FILE: action_list.pbtxt
  GROUNDTRUTH_FILE: val.csv
SLOWFAST:
  ALPHA: 8
  BETA_INV: 8
  FUSION_CONV_CHANNEL_RATIO: 2
  FUSION_KERNEL_SZ: 7
RESNET:
  ZERO_INIT_FINAL_BN: True
  WIDTH_PER_GROUP: 64
  NUM_GROUPS: 1
  DEPTH: 50
  TRANS_FUNC: bottleneck_transform
  STRIDE_1X1: False
  NUM_BLOCK_TEMP_KERNEL: [[3, 3], [4, 4], [6, 6], [3, 3]]
  SPATIAL_DILATIONS: [[1, 1], [1, 1], [1, 1], [2, 2]]
  SPATIAL_STRIDES: [[1, 1], [2, 2], [2, 2], [1, 1]]
NONLOCAL:
  LOCATION: [[[], []], [[], []], [[], []], [[], []]]
  GROUP: [[1, 1], [1, 1], [1, 1], [1, 1]]
  INSTANTIATION: dot_product
  POOL: [[[1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2]]]
BN:
  USE_PRECISE_STATS: False
  NUM_BATCHES_PRECISE: 200
SOLVER:
  BASE_LR: 0.001
  MOMENTUM: 0.9
  WEIGHT_DECAY: 1e-7
  OPTIMIZING_METHOD: sgd
  MAX_EPOCH: 100
MODEL:
  NUM_CLASSES: 8
  ARCH: slowfast
  MODEL_NAME: SlowFast
  LOSS_FUNC: bce
  DROPOUT_RATE: 0.5
  HEAD_ACT: sigmoid
TEST:
  ENABLE: False
  DATASET: ava
  BATCH_SIZE: 8
DATA_LOADER:
  NUM_WORKERS: 4
  PIN_MEMORY: True
NUM_GPUS: 1
NUM_SHARDS: 1
RNG_SEED: 0
OUTPUT_DIR: /data/zhangshuang/workspace/SlowFast/output/SLOWFAST_4*16_R50_epoch100_lr0.001

GPU: 0
TENSORBOARD:
  ENABLE: True
  LOG_DIR: /data/zhangshuang/workspace/SlowFast/output/SLOWFAST_4*16_R50_epoch100_lr0.001/log
  CLASS_NAMES_PATH: /data/zhangshuang/workspace/SlowFast/configs/AVA/action_list.json
  CONFUSION_MATRIX:
    ENABLE: True
    SUBSET_PATH: /data/zhangshuang/workspace/SlowFast/configs/AVA/action_list.txt

and the loss curve is shown in the figure