Islander

This repository is the official implementation for the paper Metric Mirages in Cell Embeddings.

Please contact [email protected] or [email protected] if you have any questions.

Citation

@article {Islander,
	author = {Hanchen Wang and Jure Leskovec and Aviv Regev},
	title = {Metric Mirages in Cell Embeddings},
	doi = {10.1101/2024.04.02.587824},
	publisher = {Cold Spring Harbor Laboratory},
	URL = {https://www.biorxiv.org/content/early/2024/04/02/2024.04.02.587824}
	journal = {bioRxiv},
	year = {2024},
}

---

Usage

We include scripts and logs to reproduce the results in the scripts folder. You can also follow the step-by-step instructions below:

Step 0: Set up the environment.

conda env create -f env.yml

NOTE: The default setup uses GPU-compiled packages (for PyTorch, JAXlib, etc.). Please adjust them according to your local CUDA version or switch to the CPU version as needed. The calculation of scGraph scores does not require GPU access.

Step 1: Preprocessing. Data can be downloaded from:

Brain	Breast	COVID	Eye	FetalGut	FetalLung	Heart	Lung	Pancreas	Skin
Paper	Paper	Paper	Paper	Paper	Paper	Paper	Paper	Paper	Paper
Data	Data	Data	Data	Data	Data	Data	Data	Data	Data

We applied quality control to each dataset by filtering out cell profiles with fewer than 1,000 reads or fewer than 500 detected genes. Genes present in fewer than five cells were also excluded. Normalization was performed using Scanpy, where each cell’s read counts were scaled to a total of 10,000, followed by a log1p transformation:

# download via "wget -O data/breast/local.h5ad https://datasets.cellxgene.cziscience.com/b8b5be07-061b-4390-af0a-f9ced877a068.h5ad"
adata = sc.read_h5ad(dh.DATA_RAW_["breast"])
adata.X = adata.raw.X
adata.layers["raw_counts"] = adata.raw.X
del adata.raw
uh.preprocess(adata)

[Output]
filtered out 9954 cells that have less than 1000 counts
filtered out 865 cells that have less than 500 genes expressed
filtered out 3803 genes that are detected in less than 5 cells
=============================================================================
29431 genes x 703512 cells after quality control.
=============================================================================
normalizing by total count per cell
    finished (0:00:06): normalized adata.X and added    'n_counts', counts per cell before normalization (adata.obs)

The top 1000 highly variable genes are selected through:

sc.pp.highly_variable_genes(adata, subset=True, flavor="seurat_v3", n_top_genes=1000)

Then metadata is saved as JSON files. See the minimal example: Process_Breast.ipynb.

Step 2: Run Islander and benchmark with scIB

cd ${HOME}/Islander/src

export LR=0.001
export EPOCH=10
export MODE="mixup"
export LEAKAGE=16
export MLPSIZE="128 128"
export DATASET_List=("lung" "lung_fetal_donor" "lung_fetal_organoid" \
    "brain" "breast" "heart" "eye" "gut_fetal" "skin" "COVID" "pancreas")

for DATASET in "${DATASET_List[@]}"; do
export PROJECT="_${DATASET}_"
export SavePrefix="${HOME}/Islander/models/${PROJECT}"
export RUNNAME="MODE-${MODE}-ONLY_LEAK-${LEAKAGE}_MLP-${MLPSIZE}"
echo "DATASET-${DATASET}_${RUNNAME}"
mkdir -p $SavePrefix

# === Training ===
python scTrain.py \
    --gpu 3 \
    --lr ${LR} \
    --mode ${MODE} \
    --epoch ${EPOCH} \
    --dataset ${DATASET} \
    --leakage ${LEAKAGE} \
    --project ${PROJECT} \
    --mlp_size ${MLPSIZE} \
    --runname "${RUNNAME}" \
    --savename "${SavePrefix}/${RUNNAME}";

# === Benchmarking ===
python scBenchmarker.py \
    --islander \
    --saveadata \
    --dataset "${DATASET}" \
    --save_path "${SavePrefix}/${RUNNAME}";
done

We have also provided variants of Islander, which make use of different forms of semi-supervised learning loss (triplet and supervised contrastive loss). See scripts/_Islander_SCL.sh and scripts/_Islander_Triplet.sh for details.

Step 3: Run integration methods and benchmark with scIB

export DATASET_List="lung_fetal_donor"
echo -e "\n\n"

echo "DATASET-${DATASET}_HVG"
export CUDA_VISIBLE_DEVICES=2 & python scBenchmarker.py \
    --all \
    --highvar \
    --saveadata \
    --dataset "${DATASET}" \
    --savecsv "${DATASET}_FULL" \
    --save_path "${HOME}/Islander_dev/models/_${DATASET}_/MODE-mixup-ONLY_LEAK-16_MLP-128 128";
   
# === highly variable genes ===
echo "DATASET-${DATASET}_HVG"
export CUDA_VISIBLE_DEVICES=2 & python scBenchmarker.py \
	--all \
	--highvar \
	--saveadata \
	--dataset "${DATASET}" \
	--savecsv "${DATASET}_HVG" \
	--save_path "${HOME}/Islander_dev/models/_${DATASET}_/MODE-mixup-ONLY_LEAK-16_MLP-128 128";

Step 4: Run and benchmark foundation models

Please refer to the authors' original tutorials (scGPT, Geneformer, scFoundation, UCE) for extracting zero-shot and fine-tuned cell embeddings. We provide a minimal example notebook _nb/Geneformer_Skin.ipynb to extract zero-shot cell embeddings for the skin dataset using pre-trained Geneformer. To evaluate such embedding with scIB:

cd ${HOME}/Islander/src

export DATASET="brain"
echo -e "\n\n"
echo "DATASET-${DATASET}_Geneformer"
python scBenchmarker.py \
    --obsm_keys Geneformer \
    --dataset "${DATASET}" \
    --savecsv "${DATASET}_Geneformer" \
    --save_path "${HOME}/Islander/models/_${skin}_/MODE-mixup-ONLY_LEAK-16_MLP-128 128";

Step 5: Benchmark with scGraph (can be replaced on customized AnnData file)

cd ${HOME}/Islander/src

python scGraph.py \
    --adata_path ${HOME}/Islander/data/lung/emb.h5ad \
    --batch_key sample \
    --label_key cell_type \
    --savename ${HOME}/lung_scGraph;

The output file is in the format Corr-Weights, reported as scGraph scores in the paper. It is based on weighted rank correlation, where the weights are inversely proportional to the inter-cluster centroid distances. Corr-PCA represents the rank correlation using equal weights. Rank-PCA represents rank differences.

	Rank-PCA	Corr-PCA	Corr-Weights
Geneformer	0.610	0.799	0.498
Harmony	0.670	0.924	0.678
Harmony_hvg	0.709	0.941	0.724
Islander	0.292	0.847	0.160

Parameter Settings in scGraph:

scGraph uses PCA for each cell type within each batch to represent cluster-cluster relationships. Batches with fewer than 100 cells or cell types with fewer than 10 cells are excluded. PCA is calculated on the 1,000 highly variable genes, after removing 10% of the cells (5% from each extreme).

All the numerical values mentioned above are adjustable. For further details, please refer to scGraph.py.

---

Case study: scGraph vs scIB on fibroblast cells from the human fetal lung

Please see Fibroblast_Case.ipynb to reproduce the results reported in the paper.

File Organization

├── LICENSE.txt
├── README.md
├── data
├── env.yml  # for GPU environments
├── jupyter_nb
│   ├── Fibroblast_Case.ipynb
│   ├── Geneformer_Skin.ipynb
│   └── Process_Breast.ipynb
├── meta
│   ├── COVID
│   │   ├── batch2cat.json
│   │   └── cell2cat.json
│   ├── ...
├── res
│   └── scGraph
│   └── scIB
├── scripts
│   ├── Lung_Mixup.log 
│   ├── Lung_scGraph.log
│   ├── _Islander_MixUp.sh
│   ├── _Islander_SCL.sh
│   ├── _Islander_Triplet.sh
│   ├── _download_data.sh
│   ├── _scBenchmark.sh
│   ├── _scGraph.sh
│   ├── _scIB_Geneformer.sh
│   └── _scIB_Islander.sh  # scib benchmark on cell islands embeddings
├── src
│   ├── ArgParser.py
│   ├── Data_Handler.py
│   ├── Utils_Handler.py
│   ├── Vis_Handler.py
│   ├── __init__.py
│   ├── scBenchmarker.py
│   ├── scDataset.py
│   ├── scFinetuner.py
│   ├── scGraph.py
│   ├── scLoss.py
│   ├── scModel.py
│   └── scTrain.py
└── teaser.png

Stand-alone scGraph package for evaluations

We also provided a standalone scgraph python package, that can be installed via:

# conda create -n scgraph python=3.10 # to create another conda environment if necessary
pip install scgraph-eval

Python

from scgraph import scGraph

# Initialize the graph analyzer
scgraph = scGraph(
    adata_path="path/to/your/data.h5ad",   # Path to AnnData object
    batch_key="batch",                     # Column name for batch information
    label_key="cell_type",                 # Column name for cell type labels
    trim_rate=0.05,                        # Trim rate for robust mean calculation
    thres_batch=100,                       # Minimum number of cells per batch
    thres_celltype=10,                     # Minimum number of cells per cell type
    only_umap=True,                        # Only evaluate 2D embeddings (mostly umaps)
)

# Run the analysis, return a pandas dataframe
results = scgraph.main()

# Save the results
results.to_csv("embedding_evaluation_results.csv")

Command line

scgraph --adata_path path/to/data.h5ad --batch_key batch --label_key cell_type --savename results

Notebook

We provide a notebook on how to install the scgraph on a labtop, and reproduce the results on using scgraph to evaluate cell embeddings of fibroblast family in human fetal lung.

• Data

• Notebook

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.