Skip to content

Commit

Permalink
chore: Update ydata-synthetic docs
Browse files Browse the repository at this point in the history
Add more information regarding the change to ydata-sdk.
  • Loading branch information
fabclmnt committed Sep 10, 2024
1 parent 2f67331 commit be212f8
Show file tree
Hide file tree
Showing 42 changed files with 193 additions and 402 deletions.
104 changes: 41 additions & 63 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
![](https://img.shields.io/github/workflow/status/ydataai/ydata-synthetic/prerelease)
![](https://img.shields.io/pypi/status/ydata-synthetic)
[![](https://pepy.tech/badge/ydata-synthetic)](https://pypi.org/project/ydata-synthetic/)
![](https://img.shields.io/badge/python-3.9%20%7C%203.10-blue)
![](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue)
[![](https://img.shields.io/pypi/v/ydata-synthetic)](https://pypi.org/project/ydata-synthetic/)
![](https://img.shields.io/github/license/ydataai/ydata-synthetic)
<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=ab07c7a0-c1ee-481e-9368-baf70185cf40" />
Expand All @@ -11,14 +11,15 @@
Join us on [![Discord](https://img.shields.io/badge/Discord-7289DA?style=for-the-badge&logo=discord&logoColor=white)](https://tiny.ydata.ai/dcai-ydata-synthetic)

# YData Synthetic
A package to generate synthetic tabular and time-series data leveraging the state of the art generative models.

## 🎊 The exciting features:
> These are must try features when it comes to synthetic data generation:
> - A new streamlit app that delivers the synthetic data generation experience with a UI interface. A low code experience for the quick generation of synthetic data
> - A new fast synthetic data generation model based on Gaussian Mixture. So you can quickstart in the world of synthetic data generation without the need for a GPU.
> - A conditional architecture for tabular data: CTGAN, which will make the process of synthetic data generation easier and with higher quality!
`YData-Synthetic` is an open-source package developed in 2020 with the primary goal of educating users about generative models for synthetic data generation.
Designed as a collection of models, it was intended for exploratory studies and educational purposes.
However, it was not optimized for the quality, performance, and scalability needs typically required by organizations.

!!! note "Update"
Even though the journey was fun, and we have learned a lot from the community it is now time to upgrade `ydata-synthetic`.
Heading towards the future of synthetic data generation we recommend users to transition to `ydata-sdk`, which provides a superior experience with enhanced performance,
precision, and ease of use, making it the preferred tool for synthetic data generation and a perfect introduction to Generative AI.

## Synthetic data
### What is synthetic data?
Synthetic data is artificially generated data that is not collected from real world events. It replicates the statistical components of real data without containing any identifiable information, ensuring individuals' privacy.
Expand All @@ -32,68 +33,50 @@ Synthetic data can be used for many applications:

> **Looking for an end-to-end solution to Synthetic Data Generation?**<br>
> [YData Fabric](https://ydata.ai/products/synthetic_data) enables the generation of high-quality datasets within a full UI experience, from data preparation to synthetic data generation and evaluation.<br>
> Check out the [Community Version](https://ydata.ai/ydata-fabric-free-trial).

# ydata-synthetic
This repository contains material related with architectures and models for synthetic data, from Generative Adversarial Networks (GANs) to Gaussian Mixtures.
The repo includes a full ecosystem for synthetic data generation, that includes different models for the generation of synthetic structure data and time-series.
All the Deep Learning models are implemented leveraging Tensorflow 2.0.
Several example Jupyter Notebooks and Python scripts are included, to show how to use the different architectures.

Are you ready to learn more about synthetic data and the bext-practices for synthetic data generation?

## Quickstart
The source code is currently hosted on GitHub at: https://github.com/ydataai/ydata-synthetic

Binary installers for the latest released version are available at the [Python Package Index (PyPI).](https://pypi.org/project/ydata-synthetic/)
```commandline
pip install ydata-synthetic
```
> Check out the [Community Version](https://ydata.ai/register).
### The UI guide for synthetic data generation

YData synthetic has now a UI interface to guide you through the steps and inputs to generate structure tabular data.
The streamlit app is available form *v1.0.0* onwards, and supports the following flows:
- Train a synthesizer model
- Generate & profile synthetic data samples
## ydata-synthetic to ydata-sdk
With the upcoming update of `ydata-synthetic`to `ydata-sdk`, users will now have access to a single API that automatically selects and optimizes
the best generative model for their data. This streamlined approach eliminates the need to choose between
various models manually, as the API intelligently identifies the optimal model based on the specific dataset and use case.

#### Installation
Instead of having to manually select from models such as:

```commandline
pip install ydata-synthetic[streamlit]
```
#### Quickstart
Use the code snippet below in a python file (Jupyter Notebooks are not supported):
```python
from ydata_synthetic import streamlit_app
- [GAN](https://arxiv.org/abs/1406.2661)
- [CGAN](https://arxiv.org/abs/1411.1784) (Conditional GAN)
- [WGAN](https://arxiv.org/abs/1701.07875) (Wasserstein GAN)
- [WGAN-GP](https://arxiv.org/abs/1704.00028) (Wassertein GAN with Gradient Penalty)
- [DRAGAN](https://arxiv.org/pdf/1705.07215.pdf) (Deep Regret Analytic GAN)
- [Cramer GAN](https://arxiv.org/abs/1705.10743) (Cramer Distance Solution to Biased Wasserstein Gradients)
- [CWGAN-GP](https://cameronfabbri.github.io/papers/conditionalWGAN.pdf) (Conditional Wassertein GAN with Gradient Penalty)
- [CTGAN](https://arxiv.org/pdf/1907.00503.pdf) (Conditional Tabular GAN)
- [TimeGAN](https://papers.nips.cc/paper/2019/file/c9efe5f26cd17ba6216bbe2a7d26d490-Paper.pdf) (specifically for *time-series* data)
- [DoppelGANger](https://dl.acm.org/doi/pdf/10.1145/3419394.3423643) (specifically for *time-series* data)

streamlit_app.run()
```
The new API handles model selection automatically, optimizing for the best performance in fidelity, utility, and privacy.
This significantly simplifies the synthetic data generation process, ensuring that users get the highest quality output without
the need for manual intervention and tiring hyperparameter tuning.

Or use the file streamlit_app.py that can be found in the [examples folder](https://github.com/ydataai/ydata-synthetic/tree/master/examples/streamlit_app.py).
Are you ready to learn more about synthetic data and the best-practices for synthetic data generation?
For more materials on [synthetic data generation with Python see the documentation](https://docs.fabric.ydata.ai/latest/sdk/).

## Quickstart
Binary installers for the latest released version are available at the [Python Package Index (PyPI).](https://pypi.org/project/ydata-sdk/)
```commandline
python -m streamlit_app
pip install ydata-sdk
```

The below models are supported:
- CGAN
- WGAN
- WGANGP
- DRAGAN
- CRAMER
- CTGAN
### The UI guide for synthetic data generation

[![Watch the video](assets/streamlit_app.png)](https://youtu.be/ep0PhwsFx0A)
YData Fabric offers an UI interface to guide you through the steps and inputs to generate structure data.
You can experiment today with [YData Fabric by registering the Community version](https://ydata.ai/register).

### Examples
Here you can find usage examples of the package and models to synthesize tabular data.
- Fast tabular data synthesis on adult census income dataset [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ydataai/ydata-synthetic/blob/master/examples/regular/models/Fast_Adult_Census_Income_Data.ipynb)
- Tabular synthetic data generation with CTGAN on adult census income dataset [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ydataai/ydata-synthetic/blob/master/examples/regular/models/CTGAN_Adult_Census_Income_Data.ipynb)
- Time Series synthetic data generation with TimeGAN on stock dataset [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ydataai/ydata-synthetic/blob/master/examples/timeseries/TimeGAN_Synthetic_stock_data.ipynb)
- Time Series synthetic data generation with DoppelGANger on FCC MBA dataset [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ydataai/ydata-synthetic/blob/master/examples/timeseries/DoppelGANger_FCC_MBA_Dataset.ipynb)
- More examples are continuously added and can be found in `/examples` directory.
- Tabular [synthetic data generation on Titanic Kaggle dataset](https://github.com/ydataai/ydata-sdk/blob/main/examples/synthesizers/regular_quickstart.py)
- Time Series [synthetic data generation]('https://github.com/ydataai/ydata-sdk/blob/main/examples/synthesizers/time_series_quickstart.py')
- More examples are continuously added and can be found in [examples directory](https://github.com/ydataai/ydata-sdk/tree/main/examples).

### Datasets for you to experiment
Here are some example datasets for you to try with the synthesizers:
Expand All @@ -108,7 +91,7 @@ Here are some example datasets for you to try with the synthesizers:

## Project Resources

In this repository you can find the several GAN architectures that are used to create synthesizers:
Find below useful literature of how to generate synthetic data and available generative models:

### Tabular data
- [GAN](https://arxiv.org/abs/1406.2661)
Expand All @@ -125,11 +108,6 @@ In this repository you can find the several GAN architectures that are used to c
- [TimeGAN](https://papers.nips.cc/paper/2019/file/c9efe5f26cd17ba6216bbe2a7d26d490-Paper.pdf)
- [DoppelGANger](https://dl.acm.org/doi/pdf/10.1145/3419394.3423643)

## Contributing
We are open to collaboration! If you want to start contributing you only need to:
1. Search for an issue in which you would like to work. Issues for newcomers are labeled with good first issue.
2. Create a PR solving the issue.
3. We would review every PRs and either accept or ask for revisions.

## Support
For support in using this library, please join our Discord server. Our Discord community is very friendly and great about quickly answering questions about the use and development of the library. [Click here to join our Discord community!](https://tiny.ydata.ai/dcai-ydata-synthetic)
Expand Down
27 changes: 7 additions & 20 deletions docs/getting-started/installation.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,22 @@

`ydata-synthetic` is available through PyPi, allowing an easy process of installation and integration with the data science programing environments (Google Colab, Jupyter Notebooks, Visual Studio Code, PyCharm) and stack (`pandas`, `numpy`, `scikit-learn`).
`ydata-sdk` is available through PyPi, allowing an easy process of installation and integration with the data science programing environments (Google Colab, Jupyter Notebooks, Visual Studio Code, PyCharm) and stack (`pandas`, `numpy`, `scikit-learn`).

##Installing the package
Currently, the package supports **python versions over 3.9**, and can be installed in Windows, Linux or MacOS operating systems.
Currently, the package supports **python versions over 3.9 and up-to python 3.12**, and can be installed in Windows, Linux or MacOS operating systems.

Prior to the package installation, it is recommended the creation of a virtual or `conda` environment:

=== "conda"
``` commandline
conda create -n synth-env python=3.10
conda create -n synth-env python=3.12
conda activate synth-env
```

The above command creates and activates a new environment called "synth-env" with Python version 3.10.X. In the new environment, you can then install `ydata-synthetic`:
The above command creates and activates a new environment called "synth-env" with Python version 3.12.X. In the new environment, you can then install `ydata-sdk`:

=== "pypi"
``` commandline
pip install ydata-synthetic==1.1.0
pip install ydata-sdk
```

:fontawesome-brands-youtube:{ style="color: #EE0F0F" }
Expand All @@ -27,20 +27,7 @@ The above command creates and activates a new environment called "synth-env" wit
To install inside a Google Colab notebook, you can use the following:

``` commandline
!pip install ydata-synthetic==1.1.0
!pip install ydata-sdk
```

Make sure your Google Colab is running Python versions `>=3.9, <3.11`. Learn how to configure Python versions on Google Colab [here](https://stackoverflow.com/questions/68657341/how-can-i-update-google-colabs-python-version/68658479#68658479).


## Installing the Streamlit App
Since version 1.0.0, the `ydata-synthetic` includes a GUI experience provided by a Streamlit app. The UI supports the data synthesization process from reading the data to profiling the synthetic data generation, and can be installed as follows:

``` commandline
pip install "ydata-synthetic[streamlit]"
```

Note that Jupyter or Colab Notebooks are not yet supported, so use it in your Python environment.

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=dd69a9f9-0901-4cb4-9e56-b1e69877dca1" />

Make sure your Google Colab is running Python versions `>=3.9, <=3.12`. Learn how to configure Python versions on Google Colab [here](https://stackoverflow.com/questions/68657341/how-can-i-update-google-colabs-python-version/68658479#68658479).
78 changes: 0 additions & 78 deletions docs/getting-started/quickstart.md

This file was deleted.

Loading

0 comments on commit be212f8

Please sign in to comment.