Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dev/jm2 #298

Closed
wants to merge 5 commits into from
Closed

Dev/jm2 #298

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f8c559b
feat: add agglomerative clustering algorithm
PotatoXi Jan 1, 2024
6921a5c
Revert "feat: add agglomerative clustering algorithm"
PotatoXi Jan 1, 2024
3630f3a
feat: add agglomerative clustering algorithm
PotatoXi Jan 13, 2024
ffcdfc7
feat: add agglomerative clustering algorithm
PotatoXi Jan 13, 2024
df372c5
Revert "feat: add agglomerative clustering algorithm"
PotatoXi Jan 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 46 additions & 28 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,12 @@
</p>

---
**Documentation**: <a href="https://geochemistrypi.readthedocs.io" target="_blank">https://geochemistrypi.readthedocs.io</a>

**Source Code**: <a href="https://github.com/ZJUEarthData/geochemistrypi" target="_blank">https://github.com/ZJUEarthData/geochemistrypi</a>
___
**Documentation**: `<a href="https://geochemistrypi.readthedocs.io" target="_blank">`https://geochemistrypi.readthedocs.io`</a>`

**Source Code**: `<a href="https://github.com/ZJUEarthData/geochemistrypi" target="_blank">`https://github.com/ZJUEarthData/geochemistrypi`</a>`

---

Geochemistry π is an **open-sourced highly automated machine learning Python framework** dedicating to build up MLOps level 1 software product for data-driven geochemistry discovery on tabular data.

Expand All @@ -24,19 +26,20 @@ Core capabilities are:
+ **Model Inference**

Key features are:

+ **Easy to use:** The automation of data mining process provides the users with simple number options to choose.
+ **Extensible:** It allows appending new algorithms through Scikit-learn with automatic hyper parameter searching by FLAML and Ray.
+ **Traceable**: It integrates MLflow to build special storage mechanism to streamline the end-to-end machine learning lifecycle.

Latest Update: follow up by clicking `Starred` and `Watch` on our [GitHub repository](https://github.com/ZJUEarthData/geochemistrypi), then get email notifications of the newest features automatically.

The following figure is the simplified overview of Geochemistry π: <br>
The following figure is the simplified overview of Geochemistry π: `<br>`

<p align="center">
<img src="https://github.com/ZJUEarthData/geochemistrypi/assets/47497750/28e174f0-1f2f-4367-96bd-9526352101bd" alt="Overview of workflow" width="600" />
</p>

The following figure is the frontend-backend separation architecture of Geochemistry: <br>
The following figure is the frontend-backend separation architecture of Geochemistry: `<br>`

<p align="center">
<img src="https://github.com/ZJUEarthData/geochemistrypi/assets/47497750/3b27cbdb-ff50-4fa6-b1d1-4c75b253fdff" alt="Frontend-backend separation architecture of Geochemistry" width="450" />
Expand All @@ -45,63 +48,79 @@ The following figure is the frontend-backend separation architecture of Geochemi
## Quick Installation

One instruction to download on **command line**, such as Terminal on macOS, Power Shell on Windows.

```
pip install geochemistrypi
```

One instruction to download on **Jupyter Notebook** or **Google Colab**.

```
!pip install geochemistrypi
```

Check the latest version of our software:

```
geochemistrypi --version
```
**Note**: For more detail on installation, please refer to our online documentation in **Installation Manual** under the section of **FOR USER**. Over there, we highly recommend to use virtual environment (Conda) to avoid dependency version problems.

**Note**: For more detail on installation, please refer to our online documentation in **Installation Manual** under the section of **FOR USER**. Over there, we highly recommend to use virtual environment (Conda) to avoid dependency version problems.

## Quick Update

One instruction to update the software to the latest version on **command line**, such as Terminal on macOS, Power Shell on Windows.

```
pip install --upgrade geochemistrypi
```

One instruction to download on **Jupyter Notebook** or **Google Colab**.

```
!pip install --upgrade geochemistrypi
```

Check the latest version of our software:

```
geochemistrypi --version
```



## Example

**How to run:** After successfully downloading, run this instruction on **command line / Jupyter Notebook / Google Colab** whatever directory it is.

### Case 1: Run with built-in data set for testing

On command line:

```
geochemistrypi data-mining
```

On Jupyter Notebook / Google Colab:

```
!geochemistrypi data-mining
```

**Note**: There are four built-in data sets corresponding to four kinds of model pattern.

### Case 2: Run with your own data set without model inference
### Case 2: Run with your own data set

On command line:

```
geochemistrypi data-mining --data your_own_data_set.xlsx
```

On Jupyter Notebook / Google Colab:

```
!geochemistrypi data-mining --data your_own_data_set.xlsx
```

**Note**: Currently, `.xlsx` and `.csv` files are supported. Please specify the path your data file exists. For Google Colab, don't forget to upload your dataset first.

### Case 3: Implement model inference on application data
Expand All @@ -117,11 +136,11 @@ On Jupyter Notebook / Google Colab:
```
!geochemistrypi data-mining --training your_own_training_data.xlsx --inference your_own_inference_data.xlsx
```

**Note**: Please make sure the column names (data schema) in both training data file and inference data file are the same. Because the operations you perform via our software on the training data will be record automatically and subsequently applied to the inference data in the same order.

The training data in our pipeline will be divided into the train set and test set used for training the ML model and evaluating the model's performance. The score includes two types. The first type is the scores from the prediction on the test set while the second type is cv scores from the cross validation on the train set.


### Case 4: Activate MLflow web interface

On command line:
Expand All @@ -135,6 +154,7 @@ On Jupyter Notebook / Google Colab:
```
!geochemistrypi data-mining --mlflow
```

**Note**: Once you run our software, there are two folders (`geopi_output` and `geopi_tracking`) generated automatically. Make sure the directory where you execute using the above command should have the genereted file `geopi_tracking`.

Copy the URL shown on the console into any browser to open the MLflow web interface. The URL is normally like this http://127.0.0.1:5000. Search MLflow online to see more operations and usages.
Expand All @@ -146,8 +166,6 @@ For more details: Please refer to:
+ [Geochemistry π - Download and Run the Beta Version (International - Youtube)](https://www.youtube.com/watch?v=EeVaJ3H7_AU&list=PLy8hNsI55lvh1UHjhVhqNUj3xPdV9sEiM&index=9)
+ [Geochemistry π - Download and Run the Beta Version (China - Bilibili)](https://www.bilibili.com/video/BV1UM4y1Q7Ju/?spm_id_from=333.999.0.0&vd_source=27944ab3b73a78970c1a52a5dcbb9140)



## Roadmap

### First Phase
Expand All @@ -165,6 +183,7 @@ Its data section provides feature engineering based on **arithmatic operation**.
Its models section provides both **supervised learning** and **unsupervised learning** methods from **Scikit-learn** framework, including four types of algorithms, regression, classification, clustering, and dimensional reduction. Integrated with **FLAML** and **Ray** framework, it allows the users to run AutoML easily, fastly and cost-effectively on the built-in supervised learning algorithms in our framework.

The following figure is the hierarchical architecture of Geochemistry π:

<p align="center">
<img src="https://github.com/ZJUEarthData/geochemistrypi/assets/47497750/9c3ddc2b-700c-4685-b52f-f5f9a8931849" alt="Hierarchical Architecture" width="450" />
</p>
Expand All @@ -173,39 +192,39 @@ The following figure is the hierarchical architecture of Geochemistry π:

Currently, we are building three access ways to provide more user-friendly service, including **web portal**, **CLI package** and **API**. It allows the user to perform **continuous training** and **model inference** by automating the ML pipeline and **machine learning lifecycle management** by unique storage mechanism in different access layers.

The following figure is the system architecture diagram: <br>
The following figure is the system architecture diagram: `<br>`

![System Architecture Diagram](https://github.com/ZJUEarthData/geochemistrypi/assets/47497750/20b5a2a4-f2de-492d-a2df-9282196d8c4f)


The following figure is the customized automated ML pipeline: <br>
The following figure is the customized automated ML pipeline: `<br>`

<p align="center">
<img src="https://github.com/ZJUEarthData/geochemistrypi/assets/47497750/07078b43-30bd-46cf-abad-2da509fae6aa" alt="Customized automated ML pipeline" width="400" />
</p>

The following figure is the design pattern hierarchical architecture: <br>
The following figure is the design pattern hierarchical architecture: `<br>`

![Design Pattern](https://github.com/ZJUEarthData/geochemistrypi/assets/47497750/aa84ab12-c95e-4282-a60e-64ba2858c437)
![Workflow Object](https://github.com/ZJUEarthData/geochemistrypi/assets/47497750/f08885bf-1bec-4045-bf6b-82c5c18d3f8f)

The following figure is the storage mechanism: <br>
The following figure is the storage mechanism: `<br>`

<p align="center">
<img src="https://github.com/ZJUEarthData/geochemistrypi/assets/47497750/401f3429-c44f-4b76-b085-7a9dcc987cde" alt="Storage Mechanism" width="500" />
</p>

The whole package is under construction and the documentation is progressively evolving.



## Team Info

**Leader:**

+ Can He (Sany, National University of Singapore, Singapore)
Email: [email protected]

**Technical Group:**
+ Jianming Zhao (Jamie, Jilin University, Changchun, China)

+ Jianming Zhao (Jamie, Zhejiang University, China)
+ Jianhao Sun (Jin, China University of Geosciences, Wuhan, China)
+ Kaixin Zheng (Hayne, Sun Yat-sen University, China)
+ Jianing Wang (National University of Singapore, Singapore)
Expand All @@ -215,6 +234,7 @@ The whole package is under construction and the documentation is progressively e
+ Chengtu Li（Trenki, Henan Polytechnic University, Beijing, China）

**Product Group**:

+ Yang Lyu (Daisy, Zhejiang University, China)
+ Wenyu Zhao (Molly, Zhejiang University, China)
+ Keran Li (Kirk, Chengdu University of Technology, China)
Expand All @@ -225,15 +245,14 @@ The whole package is under construction and the documentation is progressively e
+ Zhenglin Xu (Garry, Jilin University, China)
+ Junchi Liao(Roceda, University of Electronic Science and Technology of China, China)



## Join Us :)

**The recruitment of research interns is ongoing !!!**

**Key Point: All things are done online, remote work (\*^▽^\*)**

**What can you learn?**

+ Learning the full cycle of data mining (Scikit-learn, Ray, Mlflow) on tabular data, including the algorithms in regression,classification, clustering, and decomposition.
+ Learning to be a qualified Python developer, including any Python programing contents towards data mining, basic software engineering techniques like frontend (React, Typescript, Ant Design scaffold) and backend (SQL & NoSQL database, RESFful API, FastAPI) development, and cooperation tools like Git.

Expand All @@ -245,6 +264,7 @@ The whole package is under construction and the documentation is progressively e
+ Bonus depending on your performance.

**Current Working Pattern:**

+ Online working and cooperation
+ Three weeks per working cycle -> One online meeting per working cycle
+ One cycle report (see below) per cycle - 5 mins to finish
Expand All @@ -259,11 +279,10 @@ Chinese Page: https://person.zju.edu.cn/zhangzhou#0
**Do you want to contribute to this open-source program?**
Contact with your CV: [email protected]



## In-house Materials

Materials are in both Chinese and English. Others unshown below are internal materials.

1. [Guideline Manual – Geochemistry π (International - Google drive)](https://docs.google.com/document/d/1LjwB5Lazk33E5vbtnFPJio_MyjYQxjEu/edit?usp=sharing&ouid=110717816678586054594&rtpof=true&sd=true)
2. [Guideline Manual – Geochemistry π (China - Tencent Docs)](https://docs.qq.com/doc/DQ21IZUdVQktqRWpm?&u=6868f96d4a384b309036e04e637e367a)
3. [Learning Steps for Newbies – Geochemistry π (International - Google drive)](https://docs.google.com/document/d/1GQO-SXwEx_8midr362pqfxNZtfUf-nA6/edit?usp=sharing&ouid=110717816678586054594&rtpof=true&sd=true)
Expand All @@ -277,15 +296,14 @@ Materials are in both Chinese and English. Others unshown below are internal mat

Technical record videos are on Bilibili and Youtube synchronously while other meeting videos are internal materials.
More Videos will be recorded soon.

1. [ZJU_Earth_Data Introduction (Geochemical Data, Python, Geochemistry π) - Prof. Zhang](https://www.bilibili.com/video/BV1Lf4y1w7EK?spm_id_from=333.999.0.0)
2. [How to Collaborate and Provide Bug Report on Geochemistry π Through GitHub - Can He (Sany)](https://www.youtube.com/watch?v=1DWoEsqsfvQ&list=PLy8hNsI55lvh1UHjhVhqNUj3xPdV9sEiM&index=3)
3. [Geochemistry π - Download and Run the Beta Version](https://www.youtube.com/watch?v=EeVaJ3H7_AU&list=PLy8hNsI55lvh1UHjhVhqNUj3xPdV9sEiM&index=9)
4. [How to Create and Use Virtual Environment on Geochemistry π - Can He (Sany)](https://www.youtube.com/watch?v=4KFi7OXxD-c&list=PLy8hNsI55lvh1UHjhVhqNUj3xPdV9sEiM&index=4)
5. [How to use Github-Desktop in conflict resolution - Qiuhao Zhao (Brad)](https://www.youtube.com/watch?v=KT1g5JpuUVI&list=PLy8hNsI55lvh1UHjhVhqNUj3xPdV9sEiM)
6. [Virtual Environment & Packages On Windows - Jianming Zhao (Jamie)](https://www.youtube.com/watch?v=e4VqSBuNp_o&list=PLy8hNsI55lvh1UHjhVhqNUj3xPdV9sEiM&index=2)
7. [Git Workflow & Coordinating Synchronization - Jianming Zhao (Jamie)](https://www.bilibili.com/video/BV1Sa4y1f74k?spm_id_from=333.999.0.0&vd_source=9adcf2c5fdeffe1d11c89d441ef598ba)


6. [Virtual Environment &amp; Packages On Windows - Jianming Zhao (Jamie)](https://www.youtube.com/watch?v=e4VqSBuNp_o&list=PLy8hNsI55lvh1UHjhVhqNUj3xPdV9sEiM&index=2)
7. [Git Workflow &amp; Coordinating Synchronization - Jianming Zhao (Jamie)](https://www.bilibili.com/video/BV1Sa4y1f74k?spm_id_from=333.999.0.0&vd_source=9adcf2c5fdeffe1d11c89d441ef598ba)

## Contributors

Expand Down
8 changes: 4 additions & 4 deletions geochemistrypi/cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ def main(version: Optional[bool] = typer.Option(None, "--version", "-v", help="S
def data_mining(
data: str = typer.Option("", help="The path of the training data without model inference."),
training: str = typer.Option("", help="The path of the training data."),
application: str = typer.Option("", help="The path of the inference data."),
inference: str = typer.Option("", help="The path of the inference data."),
mlflow: bool = typer.Option(False, help="Start the mlflow server."),
web: bool = False,
) -> None:
Expand Down Expand Up @@ -81,11 +81,11 @@ def start_mlflow():
if data:
cli_pipeline(data)
# If the training data and inference data are provided, start the CLI pipeline with continuous training and inference
elif training and application:
cli_pipeline(training, application)
elif training and inference:
cli_pipeline(training, inference)
# If no data is provided, use built-in data to start the CLI pipeline with continuous training and inference
else:
cli_pipeline(training, application)
cli_pipeline(training, inference)


@app.command()
Expand Down
Loading