Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feat/coca #263

Open
wants to merge 167 commits into
base: main
Choose a base branch
from
Open

Feat/coca #263

wants to merge 167 commits into from

Commits on May 7, 2024

  1. feat: add basic webdataset

    @spravil
    spravil committed May 7, 2024
    Configuration menu
    Copy the full SHA
    d909ae0 View commit details
    Browse the repository at this point in the history

  2. fix: dim of cls token

    @spravil
    spravil committed May 7, 2024
    Configuration menu
    Copy the full SHA
    e233676 View commit details
    Browse the repository at this point in the history

  3. feat: simple console logging

    @spravil
    spravil committed May 7, 2024
    Configuration menu
    Copy the full SHA
    9986691 View commit details
    Browse the repository at this point in the history

  4. fix: add attention mask to cross entropy loss

    @spravil
    spravil committed May 7, 2024
    Configuration menu
    Copy the full SHA
    c47b6c1 View commit details
    Browse the repository at this point in the history

  5. feat: allow multiple loss functions

    @spravil
    spravil committed May 7, 2024
    Configuration menu
    Copy the full SHA
    70823e1 View commit details
    Browse the repository at this point in the history

  6. fix: register nce loss

    @spravil
    spravil committed May 7, 2024
    Configuration menu
    Copy the full SHA
    0c87d91 View commit details
    Browse the repository at this point in the history

  7. feat: add dataloader for webdataset

    @spravil
    spravil committed May 7, 2024
    Configuration menu
    Copy the full SHA
    b652a7d View commit details
    Browse the repository at this point in the history

  8. chore: add config

    @spravil
    spravil committed May 7, 2024
    Configuration menu
    Copy the full SHA
    b0e933a View commit details
    Browse the repository at this point in the history

  9. feat: add nicer logging to wandb

    @spravil
    spravil committed May 7, 2024
    Configuration menu
    Copy the full SHA
    d8d5a5f View commit details
    Browse the repository at this point in the history

  10. fix: hardcoded batches in web loader

    @spravil
    spravil committed May 7, 2024
    Configuration menu
    Copy the full SHA
    4ea65c8 View commit details
    Browse the repository at this point in the history

  11. chore: update coca config

    @spravil
    spravil committed May 7, 2024
    Configuration menu
    Copy the full SHA
    dfe88c9 View commit details
    Browse the repository at this point in the history

  12. fix: rebase

    @spravil
    spravil committed May 7, 2024
    Configuration menu
    Copy the full SHA
    5a3e844 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2024

  1. fix: print only on main rank in component factory

    @spravil
    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    043384d View commit details
    Browse the repository at this point in the history

  2. fix: total loss average logging

    @spravil
    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    3cd9244 View commit details
    Browse the repository at this point in the history

  3. fix: cuda env and run script

    @spravil
    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    b09d20e View commit details
    Browse the repository at this point in the history

  4. chore: update coca config

    @spravil
    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    e09745d View commit details
    Browse the repository at this point in the history

  5. fix: print parameters and done only on main rank

    @spravil
    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    5a74dee View commit details
    Browse the repository at this point in the history

  6. chore: update coca wds config

    @spravil
    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    f7b725c View commit details
    Browse the repository at this point in the history

  7. fix: tokenizer config of coca

    @spravil
    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    c7308e2 View commit details
    Browse the repository at this point in the history

  8. fix: add multinode splitter to webdataset

    @spravil
    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    32d0b19 View commit details
    Browse the repository at this point in the history

  9. fix: webdataset slow loading

    @spravil
    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    55c039f View commit details
    Browse the repository at this point in the history

  10. fix: add batching

    @spravil
    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    966d237 View commit details
    Browse the repository at this point in the history

  11. fix: add more options to webloader

    @spravil
    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    dacf639 View commit details
    Browse the repository at this point in the history

  12. fix: webloader

    @spravil
    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    b40ecd5 View commit details
    Browse the repository at this point in the history

  13. fix: dataset factory

    @spravil
    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    a9ce132 View commit details
    Browse the repository at this point in the history

  14. fix: webdataset

    @spravil
    spravil committed May 9, 2024
    Configuration menu
    Copy the full SHA
    63ef47c View commit details
    Browse the repository at this point in the history

Commits on May 10, 2024

  1. fix: loss accumulation

    @spravil
    spravil committed May 10, 2024
    Configuration menu
    Copy the full SHA
    d6d84dc View commit details
    Browse the repository at this point in the history

  2. fix: loss average for eval

    @spravil
    spravil committed May 10, 2024
    Configuration menu
    Copy the full SHA
    3d04f78 View commit details
    Browse the repository at this point in the history

  3. refactor: remove unused code from coca collator

    @spravil
    spravil committed May 10, 2024
    Configuration menu
    Copy the full SHA
    e65a3cd View commit details
    Browse the repository at this point in the history

  4. fix: coca collator

    @spravil
    spravil committed May 10, 2024
    Configuration menu
    Copy the full SHA
    622570d View commit details
    Browse the repository at this point in the history

  5. fix: loss normalization

    @spravil
    spravil committed May 10, 2024
    Configuration menu
    Copy the full SHA
    efedc77 View commit details
    Browse the repository at this point in the history

  6. feat: add clip loss

    @spravil
    spravil committed May 10, 2024
    Configuration menu
    Copy the full SHA
    eba23a9 View commit details
    Browse the repository at this point in the history

  7. fix: use clip loss in coca

    @spravil
    spravil committed May 10, 2024
    Configuration menu
    Copy the full SHA
    9bdc830 View commit details
    Browse the repository at this point in the history

  8. chore: update coca webdataset config

    @spravil
    spravil committed May 10, 2024
    Configuration menu
    Copy the full SHA
    b3aff91 View commit details
    Browse the repository at this point in the history

  9. chore: update coca webdataset config

    @spravil
    spravil committed May 10, 2024
    Configuration menu
    Copy the full SHA
    dffe644 View commit details
    Browse the repository at this point in the history

Commits on May 13, 2024

  1. fix: gradient accumulation

    @spravil
    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    d8bdebc View commit details
    Browse the repository at this point in the history

  2. fix: normalize cls token of coca

    @spravil
    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    244fac9 View commit details
    Browse the repository at this point in the history

  3. feat: add weight option to loss config

    @spravil
    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    0524acd View commit details
    Browse the repository at this point in the history

  4. fix: log weighted loss

    @spravil
    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    df48b62 View commit details
    Browse the repository at this point in the history

  5. feat: add cosine scheduler with warmup

    @spravil
    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    2945a03 View commit details
    Browse the repository at this point in the history

  6. fix: loss logging

    @spravil
    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    2d0e3f3 View commit details
    Browse the repository at this point in the history

  7. feat: add local clip loss

    @spravil
    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    1c0993f View commit details
    Browse the repository at this point in the history

  8. fix: clip loss

    @spravil
    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    0e3b239 View commit details
    Browse the repository at this point in the history

  9. fix: add barrier to eval

    @spravil
    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    7459ee0 View commit details
    Browse the repository at this point in the history

  10. feat: print global batch size

    @spravil
    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    1c0f456 View commit details
    Browse the repository at this point in the history

  11. fix: force integer for rank env variable

    @spravil
    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    a04bf97 View commit details
    Browse the repository at this point in the history

  12. fix: validation set loading

    @spravil
    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    9086af6 View commit details
    Browse the repository at this point in the history

  13. fix: webdataset splitter

    @spravil
    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    351990a View commit details
    Browse the repository at this point in the history

  14. feat: add drop_last to webloader

    @spravil
    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    c90df9d View commit details
    Browse the repository at this point in the history

  15. fix: val dataset with webdataset

    @spravil
    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    860b8fd View commit details
    Browse the repository at this point in the history

  16. refactor: evaluator

    @spravil
    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    962c4ca View commit details
    Browse the repository at this point in the history

  17. fix: nodesplitter in webdataset

    @spravil
    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    2d1ea92 View commit details
    Browse the repository at this point in the history

  18. feat: add wandb grouping

    @spravil
    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    1fb4fba View commit details
    Browse the repository at this point in the history

  19. chore: update coca config

    @spravil
    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    22c77cd View commit details
    Browse the repository at this point in the history

  20. fix: coca config

    @spravil
    spravil committed May 13, 2024
    Configuration menu
    Copy the full SHA
    6d980e7 View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2024

  1. feat: add universal multimodal dataset

    @spravil
    spravil committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    dfb7884 View commit details
    Browse the repository at this point in the history

  2. feat: add flatten_dict function

    @spravil
    spravil committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    da27f0d View commit details
    Browse the repository at this point in the history

  3. fix: webdataset

    @spravil
    spravil committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    016bed3 View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2024

  1. fix: web dataset integration

    @spravil
    spravil committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    faeec41 View commit details
    Browse the repository at this point in the history

  2. chore: add todo statement

    @spravil
    spravil committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    1c6fcff View commit details
    Browse the repository at this point in the history

  3. chore: merge remote-tracking branch 'origin/main' into exp/vision_lan… 

    …guage_coca
    @spravil
    spravil committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    e90a8d5 View commit details
    Browse the repository at this point in the history

  4. chore: update start script

    @spravil
    spravil committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    42f469b View commit details
    Browse the repository at this point in the history

  5. chore: merge remote-tracking branch 'origin/main' into exp/vision_lan… 

    …guage_coca
    @spravil
    spravil committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    03082b8 View commit details
    Browse the repository at this point in the history

  6. fix: coca webdataset config

    @spravil
    spravil committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    78efe14 View commit details
    Browse the repository at this point in the history

  7. feat: extend vision transformer model to video data

    @SogolHaghighat @spravil
    SogolHaghighat authored and spravil committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    deb0788 View commit details
    Browse the repository at this point in the history

  8. test: extend vision transformer test to video data

    @SogolHaghighat @spravil
    SogolHaghighat authored and spravil committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    4a9695d View commit details
    Browse the repository at this point in the history

  9. test: add and update config for testing vision transformer with image… 

    … and video
    @SogolHaghighat @spravil
    SogolHaghighat authored and spravil committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    f04243a View commit details
    Browse the repository at this point in the history

  10. feat: add video transforms

    @spravil
    spravil committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    cad8d11 View commit details
    Browse the repository at this point in the history

  11. chore: add video config

    @spravil
    spravil committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    3653922 View commit details
    Browse the repository at this point in the history

  12. fix: video coca

    @spravil
    spravil committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    3b72cd2 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. feat: add conformer audio encoder

    @manasMauryax
    manasMauryax authored and mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    0c0317a View commit details
    Browse the repository at this point in the history

  2. feat: make CoCa audio compatible

    @manasMauryax
    manasMauryax authored and mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    5f63246 View commit details
    Browse the repository at this point in the history

  3. test: change config and dummy dataset for E2E CoCa test

    @manasMauryax
    manasMauryax authored and mmaurya committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    857867e View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2024

  1. fix: webdataset with multiple dataset builders

    @spravil
    spravil committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    3f10bfa View commit details
    Browse the repository at this point in the history

  2. fix: vision transformer config

    @spravil
    spravil committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    599ed66 View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2024

  1. refactor: add pyav dependency

    @SogolHaghighat
    SogolHaghighat committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    2d3c3fa View commit details
    Browse the repository at this point in the history

  2. fix: block_size for video coca

    @SogolHaghighat
    SogolHaghighat committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    88ae41f View commit details
    Browse the repository at this point in the history

  3. test: add more test cases and merge video and image tests for vision … 

    …transformer
    @SogolHaghighat
    SogolHaghighat committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    fe23a69 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2024

  1. fix: hard coded num_frames in video transform

    @SogolHaghighat
    SogolHaghighat committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    256f6b2 View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. refactor: use decord for loading videos

    @sthoduka
    sthoduka committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    28a3b82 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2024

  1. fix: incorrect variable usage and audio input shape

    @manasMauryax
    mmaurya authored and manasMauryax committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    c0e8d1f View commit details
    Browse the repository at this point in the history

  2. fix: to avoid torch.tensor(tensor)

    @manasMauryax
    mmaurya authored and manasMauryax committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    83accc1 View commit details
    Browse the repository at this point in the history

  3. fix: add argument to ignore padding indices

    @manasMauryax
    mmaurya authored and manasMauryax committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    54906d6 View commit details
    Browse the repository at this point in the history

  4. test: uptate tests to comply with changes

    @manasMauryax
    mmaurya authored and manasMauryax committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    67d3778 View commit details
    Browse the repository at this point in the history

  5. chore: add configs 

    These can help run audio-only, vision-only or audio-vision experiments!
    @manasMauryax
    mmaurya authored and manasMauryax committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    0b1698d View commit details
    Browse the repository at this point in the history

  6. feat: allow masking of "pad" keys

    @manasMauryax
    mmaurya authored and manasMauryax committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    b89bff8 View commit details
    Browse the repository at this point in the history

  7. feat: implement Conformer from scratch

    @manasMauryax
    mmaurya authored and manasMauryax committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    d31d0a7 View commit details
    Browse the repository at this point in the history

  8. test: fix to comply to changes

    @manasMauryax
    mmaurya authored and manasMauryax committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    4c42b27 View commit details
    Browse the repository at this point in the history

  9. test: remove deprecated test

    @manasMauryax
    mmaurya authored and manasMauryax committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    1e09561 View commit details
    Browse the repository at this point in the history

  10. chore: fix configs to comply to changes

    @manasMauryax
    mmaurya authored and manasMauryax committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    68f6e5e View commit details
    Browse the repository at this point in the history

  11. fix: accelerate import

    @thomaschhh @manasMauryax
    thomaschhh authored and manasMauryax committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    94efdb8 View commit details
    Browse the repository at this point in the history

  12. refactor: introduce global constants

    @thomaschhh @manasMauryax
    thomaschhh authored and manasMauryax committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    b13c8cb View commit details
    Browse the repository at this point in the history

  13. fix: constant renaming

    @thomaschhh @manasMauryax
    thomaschhh authored and manasMauryax committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    70652f1 View commit details
    Browse the repository at this point in the history

  14. fix: disable mamba imports

    @manasMauryax
    mmaurya authored and manasMauryax committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    f07cfda View commit details
    Browse the repository at this point in the history

  15. chore: update audio coca arrow dataset config

    @manasMauryax
    mmaurya authored and manasMauryax committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    11a28e2 View commit details
    Browse the repository at this point in the history

  16. feat: add audio transform

    @thomaschhh @manasMauryax
    thomaschhh authored and manasMauryax committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    ced54c8 View commit details
    Browse the repository at this point in the history

  17. fix: cross entropy loss ignore index

    @thomaschhh @manasMauryax
    thomaschhh authored and manasMauryax committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    8ad8e86 View commit details
    Browse the repository at this point in the history

  18. fix: prepare_sample

    @thomaschhh @manasMauryax
    thomaschhh authored and manasMauryax committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    31d72d0 View commit details
    Browse the repository at this point in the history

  19. chore: apply changes from origin/feat/audio_coca

    @thomaschhh @manasMauryax
    thomaschhh authored and manasMauryax committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    3a14b95 View commit details
    Browse the repository at this point in the history

  20. Merge branch 'exp/vision_language_coca' into feat/coca

    @sthoduka
    sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    68a920f View commit details
    Browse the repository at this point in the history

  21. Merge branch 'main' into feat/coca

    @sthoduka
    sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    789f4f4 View commit details
    Browse the repository at this point in the history

  22. chore: add separate vision and audio configs

    @spravil @sthoduka
    spravil authored and sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    2335dcd View commit details
    Browse the repository at this point in the history

  23. fix: coca n_query parameter

    @spravil @sthoduka
    spravil authored and sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    420041b View commit details
    Browse the repository at this point in the history

  24. fix: copy paste error

    @manasMauryax @sthoduka
    manasMauryax authored and sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    9236a46 View commit details
    Browse the repository at this point in the history

  25. feat: allow for training all modalities

    @thomaschhh @sthoduka
    thomaschhh authored and sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    edbc1ac View commit details
    Browse the repository at this point in the history

  26. fix: revert back to multiple builders assertion

    @manasMauryax @sthoduka
    manasMauryax authored and sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    845f961 View commit details
    Browse the repository at this point in the history

  27. refactor: copy decord from github exp/vision_languauge_coca branch

    @manasMauryax @sthoduka
    manasMauryax authored and sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    35e6812 View commit details
    Browse the repository at this point in the history

  28. feat: add video-audio-text sample generation

    @manasMauryax @sthoduka
    manasMauryax authored and sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    f6f7023 View commit details
    Browse the repository at this point in the history

  29. fix: video-audio-text forward pass

    @manasMauryax @sthoduka
    manasMauryax authored and sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    21c9ee4 View commit details
    Browse the repository at this point in the history

  30. fix: gathered embeddings

    @manasMauryax @sthoduka
    manasMauryax authored and sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    9a78231 View commit details
    Browse the repository at this point in the history

  31. fix: use torchaudio to load audio from videos too

    @sthoduka
    sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    44fa306 View commit details
    Browse the repository at this point in the history

  32. refactor: simplify if statement

    @manasMauryax @sthoduka
    manasMauryax authored and sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    a2112da View commit details
    Browse the repository at this point in the history

  33. refactor: improve readability

    @manasMauryax @sthoduka
    manasMauryax authored and sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    8c12b8a View commit details
    Browse the repository at this point in the history

  34. feat: add first draft of single batch mixed modality

    @manasMauryax @sthoduka
    manasMauryax authored and sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    efae855 View commit details
    Browse the repository at this point in the history

  35. fix: load audio from video file only if it exists, and behave the sam… 

    …e way as audio-only datasets
    @sthoduka
    sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    c114506 View commit details
    Browse the repository at this point in the history

  36. fix: keep original dimension for audio when averaging channels

    @sthoduka
    sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    414031a View commit details
    Browse the repository at this point in the history

  37. fix: group input_ids based on modality; the order is determined by th… 

    …e order of modalities in the config file.

e.g.:
samples['input_ids'].shape = (10, max_length)
images = samples['input_ids'][:4], audio = samples['input_ids'][4:]
samples['images'].shape = (4, C, W, H), samples['audio'].shape = (6, L, N)
samples['attention_mask'] = (10, max_length)
    @sthoduka
    sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    7770b1c View commit details
    Browse the repository at this point in the history

  38. refactor: separate forward pass for audio-image and video-audio

    @manasMauryax @sthoduka
    manasMauryax authored and sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    d025dd4 View commit details
    Browse the repository at this point in the history

  39. fix: audio-image forward o/p for contrastive loss

    @manasMauryax @sthoduka
    manasMauryax authored and sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    8826758 View commit details
    Browse the repository at this point in the history

  40. refactor: revert back to wds.torch_audio

    @manasMauryax @sthoduka
    manasMauryax authored and sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    5435d4b View commit details
    Browse the repository at this point in the history

  41. fix: only collect text samples once for case where a single dataset h… 

    …as video and audio in the coca collator
    @sthoduka
    sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    7c3c4ba View commit details
    Browse the repository at this point in the history

  42. refactor: coca: separate vision into image and video, and refactor fo… 

    …rward function to be simpler
    @sthoduka
    sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    bc28749 View commit details
    Browse the repository at this point in the history

  43. fix: webdataset builder: remove constraint that all dataset builders … 

    …have the same type (required if we want to train with several datasets of different modalities)
    @sthoduka
    sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    c8b9f65 View commit details
    Browse the repository at this point in the history

  44. fix: add config parameter for video-audio-text dataset as a special case

    @sthoduka
    sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    ea8910c View commit details
    Browse the repository at this point in the history

  45. fix: only use audio from video-audio sample if specified

    @sthoduka
    sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    aa66bc6 View commit details
    Browse the repository at this point in the history

  46. chore: remove comment

    @thomaschhh @sthoduka
    thomaschhh authored and sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    9efecdb View commit details
    Browse the repository at this point in the history

  47. fix: set correct type for mixing ratios (float)

    @sthoduka
    sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    6d53b39 View commit details
    Browse the repository at this point in the history

  48. fix: gather all in clip loss

    @spravil @sthoduka
    spravil authored and sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    21d1751 View commit details
    Browse the repository at this point in the history

  49. fix: use maximum batch size of samples as batch length

    @sthoduka
    sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    a014b25 View commit details
    Browse the repository at this point in the history

  50. fix: webdataset: use a fixed round robin sampling strategy to get a f… 

    …ixed number of samples per modality in every batch
    @sthoduka
    sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    1788b15 View commit details
    Browse the repository at this point in the history

  51. refactor: make batch_size mandatory only if using multiple builders

    @sthoduka
    sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    7f31684 View commit details
    Browse the repository at this point in the history

  52. chore: remove unused configs; add new config

    @sthoduka
    sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    2e84732 View commit details
    Browse the repository at this point in the history

  53. fix: misc fixes after merging main into feat/coca

    @sthoduka
    sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    1664744 View commit details
    Browse the repository at this point in the history

  54. chore: remove comment and unused file

    @sthoduka
    sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    f31be51 View commit details
    Browse the repository at this point in the history

  55. refactor: out_put -> output

    @sthoduka
    sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    91934fd View commit details
    Browse the repository at this point in the history

  56. fix: reset cumulated losses using function

    @sthoduka
    sthoduka committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    cdbb956 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. fix: scaled weight initialization for residual layers of coca

    @sthoduka
    sthoduka committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    9bcc475 View commit details
    Browse the repository at this point in the history

  2. refactor: replace CosineAnnealingWithWarmupLR with OneCycleLR

    @sthoduka
    sthoduka committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    25130cd View commit details
    Browse the repository at this point in the history

  3. refactor: set weight decay groups for coca

    @sthoduka
    sthoduka committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    a97f448 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. fix: update path for coca tokenizer

    @sthoduka
    sthoduka committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    3b89853 View commit details
    Browse the repository at this point in the history

  2. chore: use built-in types

    @thomaschhh
    thomaschhh committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    8f7a114 View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. chore: refactor loss related items to match main

    @manasMauryax
    manasMauryax committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    3eb77a3 View commit details
    Browse the repository at this point in the history

  2. docs: add docstrings and type hints for audio-related classes and fun… 

    …ctions
    @thomaschhh
    thomaschhh committed Sep 24, 2024
    8 Configuration menu
    Copy the full SHA
    3366bc9 View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. test: update coca model test and add coca collator test

    @sthoduka
    sthoduka committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    61cfe51 View commit details
    Browse the repository at this point in the history

  2. refactor: verify correctness of coca model config

    @sthoduka
    sthoduka committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    56ea087 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. feat: multiple loss functions

    @manasMauryax
    manasMauryax committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    8f5aea4 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'feat/multiple_loss_functions' into feat/coca

    @manasMauryax
    manasMauryax committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    f1f0fe5 View commit details
    Browse the repository at this point in the history

  3. test: add more tests for loss functions

    @manasMauryax
    manasMauryax committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    c291fcc View commit details
    Browse the repository at this point in the history

  4. revert: add back default values for NCELoss

    @sthoduka
    sthoduka committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    f1dbe91 View commit details
    Browse the repository at this point in the history

  5. refactor: use composition to wrap the pytorch DataLoader using LLMDat… 

    …aLoader, so that both LLMDataLoader and WebLoader inherit only from DataLoaderIF
    @sthoduka
    sthoduka committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    146682f View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. refactor: rename WebLoader to WebDataLoader

    @sthoduka
    sthoduka committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    b985b9b View commit details
    Browse the repository at this point in the history

  2. docs: update docs for WebDataLoader and MultimodalWebDataset

    @sthoduka
    sthoduka committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    9113f8a View commit details
    Browse the repository at this point in the history

  3. docs: update docs for coca model

    @sthoduka
    sthoduka committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    2a1cfa1 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. docs: update docs for vision transforms and make video transform para… 

    …meters configurable
    @sthoduka
    sthoduka committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    b9bfaea View commit details
    Browse the repository at this point in the history

  2. test: add test for webdataset dataset and dataloader

    @sthoduka
    sthoduka committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    d6da583 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2024

  1. Merge branch 'main' into feat/coca

    @sthoduka
    sthoduka committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    8312ed6 View commit details
    Browse the repository at this point in the history

  2. docs: add docstring to WebDataloader, typehints to _init_modality

    @davidkaczer
    davidkaczer committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    152ebf2 View commit details
    Browse the repository at this point in the history

  3. fix: mask creation for audio inputs

    @manasMauryax
    manasMauryax committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    0353969 View commit details
    Browse the repository at this point in the history

  4. test: add tests for audio_transformer

    @manasMauryax
    manasMauryax committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    a12ff8a View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2024

  1. docs: misc. docstrings and type hints for VideoTransform, web dataset 

    builder etc.
    @sthoduka
    sthoduka committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    79c6c6b View commit details
    Browse the repository at this point in the history

  2. fix: update simple progress subscriber config

    @sthoduka
    sthoduka committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    23267e8 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. refactor: rename norm layers for easier regex for weight initializati… 

    …on and decay groups
    @sthoduka
    sthoduka committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    29352b7 View commit details
    Browse the repository at this point in the history

  2. test: fix weight initialization and weight decay tests for coca

    @sthoduka
    sthoduka committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    2aa4fc0 View commit details
    Browse the repository at this point in the history

  3. fix: update directory name for getting started example

    @sthoduka
    sthoduka committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    c84bbda View commit details
    Browse the repository at this point in the history

  4. docs: update changelog with info about CoCa PR

    @sthoduka
    sthoduka committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    e33076f View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2024

  1. chore: fix linting

    @sthoduka
    sthoduka committed Oct 12, 2024
    Configuration menu
    Copy the full SHA
    71a5bd1 View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2024

  1. docs: fix minor docstring inconsistencies

    @thomaschhh
    thomaschhh committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    de9baab View commit details
    Browse the repository at this point in the history