Multicam DLC project support

[CBroz1]

Description

Several lab members are using Daniel's branch for a multi-cam DLC project. This PR fetches from upstream and resolves conflicts.

Testing needed to make multicam compatible with NWB 2.5 spatial series

Checklist:

• This PR should be accompanied by a release: no
• (If release) I have updated the CITATION.cff
• I have updated the CHANGELOG.md
• I have added/edited docs/notebooks to reflect the changes

[emreybroyles]

Hi, I began testing this branch, and I get an error when creating a project involving videos that have 2 camera per epoch. Our goal for this branch is for us to be able to specify the camera_name (or, alternatively, video_file_num) from the video to specify which video to use per epoch. Currently, this branch is not recognizing the camera_name input in video_list (see error below), which Daniel's (dpeg22) add-multi-camera branch can do.

Minimal working code:

camera_name = "HomeBox_camera"
video_list = [
    {"nwb_file_name": "SC3820230610_.nwb", "epoch": 12, "camera_name" : camera_name},
    {"nwb_file_name": "SC3820230611_.nwb", "epoch": 13, "camera_name" : camera_name},
    {"nwb_file_name": "SC3820230607_.nwb", "epoch": 13, "camera_name" : camera_name},
    
    {"nwb_file_name": "SC100220231003_.nwb", "epoch": 12, "camera_name" : camera_name},
    {"nwb_file_name": "SC100220231005_.nwb", "epoch": 13, "camera_name" : camera_name},
    {"nwb_file_name": "SC100220231008_.nwb", "epoch": 12, "camera_name" : camera_name},
    
    {"nwb_file_name": "SC100020230910_.nwb", "epoch": 13, "camera_name" : camera_name},
    {"nwb_file_name": "SC100020230911_.nwb", "epoch": 12, "camera_name" : camera_name},
    {"nwb_file_name": "SC100020230912_.nwb", "epoch": 9, "camera_name" : camera_name},
    
    {"nwb_file_name": "SC100120230904_.nwb", "epoch": 13, "camera_name" : camera_name},
    {"nwb_file_name": "SC100120230906_.nwb", "epoch": 13, "camera_name" : camera_name},
    {"nwb_file_name": "SC100120230909_.nwb", "epoch": 12, "camera_name" : camera_name},
]

team_name = "sc_eb"
project_name = "topHome_multicam_test"
frames_per_video = 50
bodyparts = ["nose", "greenLED", "redLED_C", "earL", "earR", "spine1", "spine2", "spine3",
             "spine4", "spine5", "tailBase", "tailMid", "tailTip", "forelimbL", "forelimbR"]
project_key = sgp.DLCProject.insert_new_project(
    project_name=project_name,
    bodyparts=bodyparts,
    lab_team=team_name,
    frames_per_video=frames_per_video,
    video_list=video_list,
    skip_duplicates=True,
)

Error:

Found 2 videos for {'nwb_file_name': 'SC3820230610_.nwb', 'epoch': 12}
Found 2 videos for {'nwb_file_name': 'SC3820230611_.nwb', 'epoch': 13}
Found 2 videos for {'nwb_file_name': 'SC3820230607_.nwb', 'epoch': 13}
Found 2 videos for {'nwb_file_name': 'SC100220231003_.nwb', 'epoch': 12}
Found 2 videos for {'nwb_file_name': 'SC100220231005_.nwb', 'epoch': 13}
Found 2 videos for {'nwb_file_name': 'SC100220231008_.nwb', 'epoch': 12}
Found 2 videos for {'nwb_file_name': 'SC100020230910_.nwb', 'epoch': 13}
Found 2 videos for {'nwb_file_name': 'SC100020230911_.nwb', 'epoch': 12}
Found 2 videos for {'nwb_file_name': 'SC100020230912_.nwb', 'epoch': 9}
Found 2 videos for {'nwb_file_name': 'SC100120230904_.nwb', 'epoch': 13}
Found 2 videos for {'nwb_file_name': 'SC100120230906_.nwb', 'epoch': 13}
Found 2 videos for {'nwb_file_name': 'SC100120230909_.nwb', 'epoch': 12}

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[9], line 6
      3 frames_per_video = 50
      4 bodyparts = ["nose", "greenLED", "redLED_C", "earL", "earR", "spine1", "spine2", "spine3",
      5              "spine4", "spine5", "tailBase", "tailMid", "tailTip", "forelimbL", "forelimbR"]
----> 6 project_key = sgp.DLCProject.insert_new_project(
      7     project_name=project_name,
      8     bodyparts=bodyparts,
      9     lab_team=team_name,
     10     frames_per_video=frames_per_video,
     11     video_list=video_list,
     12     skip_duplicates=True,
     13 )

File ~/Src/spyglass/src/spyglass/position/v1/position_dlc_project.py:268, in DLCProject.insert_new_project(cls, project_name, bodyparts, lab_team, frames_per_video, video_list, project_directory, output_path, set_permissions, **kwargs)
    264 skeleton_node = None
    265 # If dict, assume of form {'nwb_file_name': nwb_file_name,
    266 # 'epoch': epoch, "video_file_num": num} and pass to get_video_path
    267 # to reference VideoFile table for path
--> 268 videos = cls.add_video_files(
    269     video_list=video_list,
    270     output_path=output_path,
    271     add_to_files=False,
    272 )
    273 from deeplabcut import create_new_project
    275 config_path = create_new_project(
    276     project_name,
    277     lab_team,
   (...)
    281     multianimal=False,
    282 )

File ~/Src/spyglass/src/spyglass/position/v1/position_dlc_project.py:375, in DLCProject.add_video_files(cls, video_list, config_path, key, output_path, add_new, add_to_files, **kwargs)
    371 if all(isinstance(n, Dict) for n in video_list):
    372     videos_to_convert = [
    373         get_video_path(video_key) for video_key in video_list
    374     ]
--> 375     videos = [
    376         check_videofile(
    377             video_path=video[0],
    378             output_path=output_path,
    379             video_filename=video[1],
    380         )[0].as_posix()
    381         for video in videos_to_convert
    382     ]
    383 # If not dict, assume list of video file paths
    384 # that may or may not need to be converted
    385 else:
    386     videos = []

File ~/Src/spyglass/src/spyglass/position/v1/position_dlc_project.py:376, in <listcomp>(.0)
    371 if all(isinstance(n, Dict) for n in video_list):
    372     videos_to_convert = [
    373         get_video_path(video_key) for video_key in video_list
    374     ]
    375     videos = [
--> 376         check_videofile(
    377             video_path=video[0],
    378             output_path=output_path,
    379             video_filename=video[1],
    380         )[0].as_posix()
    381         for video in videos_to_convert
    382     ]
    383 # If not dict, assume list of video file paths
    384 # that may or may not need to be converted
    385 else:
    386     videos = []

File ~/Src/spyglass/src/spyglass/position/v1/dlc_utils.py:379, in check_videofile(video_path, output_path, video_filename, video_filetype)
    355 """
    356 Checks the file extension of a video file to make sure it is .mp4 for
    357 DeepLabCut processes. Converts to MP4 if not already.
   (...)
    375     paths to converted video file(s)
    376 """
    378 if not video_filename:
--> 379     video_files = pathlib.Path(video_path).glob(f"*.{video_filetype}")
    380 else:
    381     video_files = [pathlib.Path(f"{video_path}/{video_filename}")]

File ~/anaconda3/envs/spyglass-position/lib/python3.9/pathlib.py:1082, in Path.__new__(cls, *args, **kwargs)
   1080 if cls is Path:
   1081     cls = WindowsPath if os.name == 'nt' else PosixPath
-> 1082 self = cls._from_parts(args, init=False)
   1083 if not self._flavour.is_supported:
   1084     raise NotImplementedError("cannot instantiate %r on your system"
   1085                               % (cls.__name__,))

File ~/anaconda3/envs/spyglass-position/lib/python3.9/pathlib.py:707, in PurePath._from_parts(cls, args, init)
    702 @classmethod
    703 def _from_parts(cls, args, init=True):
    704     # We need to call _parse_args on the instance, so as to get the
    705     # right flavour.
    706     self = object.__new__(cls)
--> 707     drv, root, parts = self._parse_args(args)
    708     self._drv = drv
    709     self._root = root

File ~/anaconda3/envs/spyglass-position/lib/python3.9/pathlib.py:691, in PurePath._parse_args(cls, args)
    689     parts += a._parts
    690 else:
--> 691     a = os.fspath(a)
    692     if isinstance(a, str):
    693         # Force-cast str subclasses to str (issue #21127)
    694         parts.append(str(a))

TypeError: expected str, bytes or os.PathLike object, not NoneType

[edeno]

Hey @emreybroyles, thanks for testing this out (and all your other work on the dlc stuff). @samuelbray32 , when you get back, can you work with @emreybroyles to resolve these differences?

[edeno]

@CBroz1, would it make sense to port these changes over to a branch of spyglass so that people can work on it?

[CBroz1]

@CBroz1, would it make sense to port these changes over to a branch of spyglass so that people can work on it?

Sure - I fetched upstream, resolved conflicts, and made a new branch on the org. We can work collaboratively there

[samuelbray32]

Quick comment here: I think the error above was just due to how the fetch upstream commit was resolved.