Skip to content

Commit

Permalink
"Added some documentation for what the huggingface directory is for"
Browse files Browse the repository at this point in the history
  • Loading branch information
John-Peters-UW committed Aug 28, 2024
1 parent fd4e72b commit 6fc00e5
Show file tree
Hide file tree
Showing 6 changed files with 35 additions and 0 deletions.
5 changes: 5 additions & 0 deletions huggingface/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
This directory is to maintain the 🤗 support of METL.

Herein are a few files to facilitate uploading the wrapper to 🤗. First, combine_files.py takes all of the files in the METL directory, barring files that have test or _.py (think, innit.py here) and combines them into a single file. combine_files.py also appends the huggingface wrapper code itself (stored in huggingface_code.py) onto the bottom of the script.

This script then gets auto-updated to 🤗 after formatting it by running the push_to_hub.py script. Some additional small comments are included in the top of each file repeating these responsibilities.
8 changes: 8 additions & 0 deletions huggingface/combine_files.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,11 @@
"""
This script combines all of the files in the metl directory into one file so that it can be uploaded automatically to huggingface.
Files ending with _.py and that contain test in the filename will not be included. This script automatically generates the required imports from the files as well.
Regardless of changes to metl, as long as necessary files that may be added don't contain test or _.py, this should work as intended.
"""

import argparse
import os

Expand Down
7 changes: 7 additions & 0 deletions huggingface/huggingface_code.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,10 @@
"""
This file contains the actual wrapper for METL.
Above the delimiter for this file: #$> we have included imports and shell functions
which prevent python (and other linters) from complaining this file has erros.
"""


from transformers import PretrainedConfig, PreTrainedModel

def get_from_uuid():
Expand Down
4 changes: 4 additions & 0 deletions huggingface/huggingface_wrapper.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
"""
This is an example wrapper after running the script combine_files. This is NOT the version on huggingface. That gets compiled live after ever PR or push to main.
"""

import collections
import copy
import enum
Expand Down
6 changes: 6 additions & 0 deletions huggingface/print_colab_dropdown.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
"""
Utility script for generating a list that can be pasted into the google colab when more models are uploaded to zenodo and added to the METL IDENT_UUID_MAP.
This pulls from huggingface, so wait for that action to finish first before running this script and uploading the colab notebook.
"""

from transformers import AutoModel

def main():
Expand Down
5 changes: 5 additions & 0 deletions huggingface/push_to_hub.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
"""
Basic minimal script for uploading the generated file from combine_files.py onto huggingface.
Requires the action to have access to the HF_TOKEN secret.
"""

from huggingface_wrapper import METLConfig, METLModel
from huggingface_hub import login
import os
Expand Down

0 comments on commit 6fc00e5

Please sign in to comment.