VAE caching & bucketing

LICENSE

@@ -1,117 +1,201 @@
-Copyright (c) 2019, NVIDIA Corporation. All rights reserved.
-
-
-Nvidia Source Code License-NC
-
-=======================================================================
-
-1. Definitions
-
-“Licensor” means any person or entity that distributes its Work.
-
-“Work” means (a) the original work of authorship made available under
-this license, which may include software, documentation, or other
-files, and (b) any additions to or derivative works  thereof
-that are made available under this license.
-
-“NVIDIA Processors” means any central processing unit (CPU),
-graphics processing unit (GPU), field-programmable gate array (FPGA),
-application-specific integrated circuit (ASIC) or any combination
-thereof designed, made, sold, or provided by NVIDIA or its affiliates.
-
-The terms “reproduce,” “reproduction,” “derivative works,” and
-“distribution” have the meaning as provided under U.S. copyright law;
-provided, however, that for the purposes of this license, derivative
-works shall not include works that remain separable from, or merely
-link (or bind by name) to the interfaces of, the Work.
-
-Works are “made available” under this license by including in or with
-the Work either (a) a copyright notice referencing the applicability
-of this license to the Work, or (b) a copy of this license.
-
-"Safe Model" means ShieldGemma-2B, which is a series of safety
-content moderation models designed to moderate four categories of
-harmful content: sexually explicit material, dangerous content,
-hate speech, and harassment, and which you separately obtain
-from Google at https://huggingface.co/google/shieldgemma-2b.
-
-
-2. License Grant
-
-2.1 Copyright Grant. Subject to the terms and conditions of this
-license, each Licensor grants to you a perpetual, worldwide,
-non-exclusive, royalty-free, copyright license to use, reproduce,
-prepare derivative works of, publicly display, publicly perform,
-sublicense and distribute its Work and any resulting derivative
-works in any form.
-
-3. Limitations
-
-3.1 Redistribution. You may reproduce or distribute the Work only if
-(a) you do so under this license, (b) you include a complete copy of
-this license with your distribution, and (c) you retain without
-modification any copyright, patent, trademark, or attribution notices
-that are present in the Work.
-
-3.2 Derivative Works. You may specify that additional or different
-terms apply to the use, reproduction, and distribution of your
-derivative works of the Work (“Your Terms”) only if (a) Your Terms
-provide that the use limitation in Section 3.3 applies to your
-derivative works, and (b) you identify the specific derivative works
-that are subject to Your Terms. Notwithstanding Your Terms, this
-license (including the redistribution requirements in Section 3.1)
-will continue to apply to the Work itself.
-
-3.3 Use Limitation. The Work and any derivative works thereof only may
-be used or intended for use non-commercially and with NVIDIA Processors,
-in accordance with Section 3.4, below. Notwithstanding the foregoing,
-NVIDIA Corporation and its affiliates may use the Work and any
-derivative works commercially. As used herein, “non-commercially”
-means for research or evaluation purposes only.
-
-3.4  You shall filter your input content to the Work and any derivative
-works thereof through the Safe Model to ensure that no content described
-as Not Safe For Work (NSFW) is processed or generated. You shall not use
-the Work to process or generate NSFW content. You are solely responsible
-for any damages and liabilities arising from your failure to adequately
-filter content in accordance with this section.  As used herein,
-“Not Safe For Work” or “NSFW” means content, videos or website pages
-that contain potentially disturbing subject matter, including but not
-limited to content that is sexually explicit, dangerous, hate,
-or harassment.
-
-3.5 Patent Claims. If you bring or threaten to bring a patent claim
-against any Licensor (including any claim, cross-claim or counterclaim
-in a lawsuit) to enforce any patents that you allege are infringed by
-any Work, then your rights under this license from such Licensor
-(including the grant in Section 2.1) will terminate immediately.
-
-3.6 Trademarks. This license does not grant any rights to use any
-Licensor’s or its affiliates’ names, logos, or trademarks, except as
-necessary to reproduce the notices described in this license.
-
-3.7 Termination. If you violate any term of this license, then your
-rights under this license (including the grant in Section 2.1) will
-terminate immediately.
-
-4. Disclaimer of Warranty.
-
-THE WORK IS PROVIDED “AS IS” WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WARRANTIES OR CONDITIONS OF
-MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR
-NON-INFRINGEMENT. YOU BEAR THE RISK OF UNDERTAKING ANY ACTIVITIES
-UNDER THIS LICENSE.
-
-5. Limitation of Liability.
-
-EXCEPT AS PROHIBITED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO LEGAL
-THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE
-SHALL ANY LICENSOR BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY DIRECT,
-INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF
-OR RELATED TO THIS LICENSE, THE USE OR INABILITY TO USE THE WORK
-(INCLUDING BUT NOT LIMITED TO LOSS OF GOODWILL, BUSINESS INTERRUPTION,
-LOST PROFITS OR DATA, COMPUTER FAILURE OR MALFUNCTION, OR ANY OTHER
-DAMAGES OR LOSSES), EVEN IF THE LICENSOR HAS BEEN ADVISED OF THE
-POSSIBILITY OF SUCH DAMAGES.
-
-=======================================================================
+                             Apache License
+                       Version 2.0, January 2004
+                    http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+  "License" shall mean the terms and conditions for use, reproduction,
+  and distribution as defined by Sections 1 through 9 of this document.
+
+  "Licensor" shall mean the copyright owner or entity authorized by
+  the copyright owner that is granting the License.
+
+  "Legal Entity" shall mean the union of the acting entity and all
+  other entities that control, are controlled by, or are under common
+  control with that entity. For the purposes of this definition,
+  "control" means (i) the power, direct or indirect, to cause the
+  direction or management of such entity, whether by contract or
+  otherwise, or (ii) ownership of fifty percent (50%) or more of the
+  outstanding shares, or (iii) beneficial ownership of such entity.
+
+  "You" (or "Your") shall mean an individual or Legal Entity
+  exercising permissions granted by this License.
+
+  "Source" form shall mean the preferred form for making modifications,
+  including but not limited to software source code, documentation
+  source, and configuration files.
+
+  "Object" form shall mean any form resulting from mechanical
+  transformation or translation of a Source form, including but
+  not limited to compiled object code, generated documentation,
+  and conversions to other media types.
+
+  "Work" shall mean the work of authorship, whether in Source or
+  Object form, made available under the License, as indicated by a
+  copyright notice that is included in or attached to the work
+  (an example is provided in the Appendix below).
+
+  "Derivative Works" shall mean any work, whether in Source or Object
+  form, that is based on (or derived from) the Work and for which the
+  editorial revisions, annotations, elaborations, or other modifications
+  represent, as a whole, an original work of authorship. For the purposes
+  of this License, Derivative Works shall not include works that remain
+  separable from, or merely link (or bind by name) to the interfaces of,
+  the Work and Derivative Works thereof.
+
+  "Contribution" shall mean any work of authorship, including
+  the original version of the Work and any modifications or additions
+  to that Work or Derivative Works thereof, that is intentionally
+  submitted to Licensor for inclusion in the Work by the copyright owner
+  or by an individual or Legal Entity authorized to submit on behalf of
+  the copyright owner. For the purposes of this definition, "submitted"
+  means any form of electronic, verbal, or written communication sent
+  to the Licensor or its representatives, including but not limited to
+  communication on electronic mailing lists, source code control systems,
+  and issue tracking systems that are managed by, or on behalf of, the
+  Licensor for the purpose of discussing and improving the Work, but
+  excluding communication that is conspicuously marked or otherwise
+  designated in writing by the copyright owner as "Not a Contribution."
+
+  "Contributor" shall mean Licensor and any individual or Legal Entity
+  on behalf of whom a Contribution has been received by Licensor and
+  subsequently incorporated within the Work.
+
+2. Grant of Copyright License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  copyright license to reproduce, prepare Derivative Works of,
+  publicly display, publicly perform, sublicense, and distribute the
+  Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  (except as stated in this section) patent license to make, have made,
+  use, offer to sell, sell, import, and otherwise transfer the Work,
+  where such license applies only to those patent claims licensable
+  by such Contributor that are necessarily infringed by their
+  Contribution(s) alone or by combination of their Contribution(s)
+  with the Work to which such Contribution(s) was submitted. If You
+  institute patent litigation against any entity (including a
+  cross-claim or counterclaim in a lawsuit) alleging that the Work
+  or a Contribution incorporated within the Work constitutes direct
+  or contributory patent infringement, then any patent licenses
+  granted to You under this License for that Work shall terminate
+  as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the
+  Work or Derivative Works thereof in any medium, with or without
+  modifications, and in Source or Object form, provided that You
+  meet the following conditions:
+
+  (a) You must give any other recipients of the Work or
+      Derivative Works a copy of this License; and
+
+  (b) You must cause any modified files to carry prominent notices
+      stating that You changed the files; and
+
+  (c) You must retain, in the Source form of any Derivative Works
+      that You distribute, all copyright, patent, trademark, and
+      attribution notices from the Source form of the Work,
+      excluding those notices that do not pertain to any part of
+      the Derivative Works; and
+
+  (d) If the Work includes a "NOTICE" text file as part of its
+      distribution, then any Derivative Works that You distribute must
+      include a readable copy of the attribution notices contained
+      within such NOTICE file, excluding those notices that do not
+      pertain to any part of the Derivative Works, in at least one
+      of the following places: within a NOTICE text file distributed
+      as part of the Derivative Works; within the Source form or
+      documentation, if provided along with the Derivative Works; or,
+      within a display generated by the Derivative Works, if and
+      wherever such third-party notices normally appear. The contents
+      of the NOTICE file are for informational purposes only and
+      do not modify the License. You may add Your own attribution
+      notices within Derivative Works that You distribute, alongside
+      or as an addendum to the NOTICE text from the Work, provided
+      that such additional attribution notices cannot be construed
+      as modifying the License.
+
+  You may add Your own copyright statement to Your modifications and
+  may provide additional or different license terms and conditions
+  for use, reproduction, or distribution of Your modifications, or
+  for any such Derivative Works as a whole, provided Your use,
+  reproduction, and distribution of the Work otherwise complies with
+  the conditions stated in this License.
+
+5. Submission of Contributions. Unless You explicitly state otherwise,
+  any Contribution intentionally submitted for inclusion in the Work
+  by You to the Licensor shall be under the terms and conditions of
+  this License, without any additional terms or conditions.
+  Notwithstanding the above, nothing herein shall supersede or modify
+  the terms of any separate license agreement you may have executed
+  with Licensor regarding such Contributions.
+
+6. Trademarks. This License does not grant permission to use the trade
+  names, trademarks, service marks, or product names of the Licensor,
+  except as required for reasonable and customary use in describing the
+  origin of the Work and reproducing the content of the NOTICE file.
+
+7. Disclaimer of Warranty. Unless required by applicable law or
+  agreed to in writing, Licensor provides the Work (and each
+  Contributor provides its Contributions) on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+  implied, including, without limitation, any warranties or conditions
+  of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+  PARTICULAR PURPOSE. You are solely responsible for determining the
+  appropriateness of using or redistributing the Work and assume any
+  risks associated with Your exercise of permissions under this License.
+
+8. Limitation of Liability. In no event and under no legal theory,
+  whether in tort (including negligence), contract, or otherwise,
+  unless required by applicable law (such as deliberate and grossly
+  negligent acts) or agreed to in writing, shall any Contributor be
+  liable to You for damages, including any direct, indirect, special,
+  incidental, or consequential damages of any character arising as a
+  result of this License or out of the use or inability to use the
+  Work (including but not limited to damages for loss of goodwill,
+  work stoppage, computer failure or malfunction, or any and all
+  other commercial damages or losses), even if such Contributor
+  has been advised of the possibility of such damages.
+
+9. Accepting Warranty or Additional Liability. While redistributing
+  the Work or Derivative Works thereof, You may choose to offer,
+  and charge a fee for, acceptance of support, warranty, indemnity,
+  or other liability obligations and/or rights consistent with this
+  License. However, in accepting such obligations, You may act only
+  on Your own behalf and on Your sole responsibility, not on behalf
+  of any other Contributor, and only if You agree to indemnify,
+  defend, and hold each Contributor harmless for any liability
+  incurred by, or claims asserted against, such Contributor by reason
+  of your accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
+APPENDIX: How to apply the Apache License to your work.
+
+  To apply the Apache License to your work, attach the following
+  boilerplate notice, with the fields enclosed by brackets "[]"
+  replaced with your own identifying information. (Don't include
+  the brackets!)  The text should be enclosed in the appropriate
+  comment syntax for the file format. We also recommend that a
+  file or class name and description of purpose be included on the
+  same "printed page" as the copyright notice for easier
+  identification within third-party archives.
+
+Copyright 2024 Junsong Chen, Jincheng Yu, Enze Xie
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

README.md

@@ -224,6 +224,25 @@ bash train_scripts/train.sh \
   --train.train_batch_size=8
 ```
 
+Local training with bucketing and VAE embedding caching:
+
+```bash
+# Prepare buckets and cache VAE embeds
+python train_scripts/make_buckets.py \
+    --config=configs/sana_config/1024ms/Sana_1600M_img1024.yaml \
+    --data.data_dir=[asset/example_data] \
+    --data.buckets_file=buckets.json
+
+# Start training with cached VAE embeddings and bucketing
+bash train_scripts/train_local.sh \
+    configs/sana_config/1024ms/Sana_1600M_img1024.yaml \
+    --data.buckets_file=buckets.json \
+    --train.train_batch_size=30
+```
+
+Using the AdamW optimizer, training with a batch size of 30 on 1024x1024 resolution consumes ~48GB VRAM on an NVIDIA A6000 GPU.
+Each training iteration takes ~7.5 seconds.
+
 # 💻 4. Metric toolkit
 
 Refer to [Toolkit Manual](asset/docs/metrics_toolkit.md).

diffusion/utils/config.py

@@ -23,6 +23,7 @@ def __str__(self):
 @dataclass
 class DataConfig(BaseConfig):
     data_dir: List[Optional[str]] = field(default_factory=list)
+    buckets_file: str = "buckets.json"
     caption_proportion: Dict[str, int] = field(default_factory=lambda: {"prompt": 1})
     external_caption_suffixes: List[str] = field(default_factory=list)
     external_clipscore_suffixes: List[str] = field(default_factory=list)