Add support for XGBoost and LightGBM

[riley-harper]

Overview

This PR closes #161 and closes #162 by adding support for the XGBoost and LightGBM machine learning models.

We've opted to make both of these models available through optional installation with the Python extras syntax. The XGBoost-PySpark integration is still experimental/unstable, and LightGBM has some significant Scala dependencies, so we thought it best to let users decide whether they wanted to bother with these models and their dependencies or not. This means that this PR should be entirely additive, and current users should not see any changes in hlink unless they start using XGBoost and/or LightGBM.

You can install the Python dependencies of these packages with pip.

pip install hlink[xgboost]
pip install hlink[lightgbm]
# You can install both at once too
pip install hlink[xgboost,lightgbm]

In the code, we detect whether these features are available by trying to import their respective Python packages and setting a flag to indicate whether the import succeeded or failed.

try:
    import synapse.ml.lightgbm
except ModuleNotFoundError:
    _lightgbm_available = False
else:
    _lightgbm_available = True

try:
    import xgboost.spark
except ModuleNotFoundError:
    _xgboost_available = False
else:
    _xgboost_available = True

This has the nice benefit of keeping all of our imports at the top of the file, not in functions or methods. If a user tries to use XGBoost or LightGBM without the required Python package installed, we throw an error with a (hopefully helpful) detailed error message.

ModuleNotFoundError: To use the 'lightgbm' model type, you need to install the synapseml
Python package, which provides LightGBM-Spark integration, and its dependencies. Try installing
hlink with the lightgbm extra:

                   pip install hlink[lightgbm]

Finally, the tests also detect whether the needed packages are installed, and they use pytest.mark.skipif to skip themselves if the packages aren't available. We've updated the CI/CD matrix to run with and without these dependencies. So now there are 6 total test CI/CD runs:

Python version	hlink extras
3.10	dev
3.10	dev,xgboost,lightgbm
3.11	dev
3.11	dev,xgboost,lightgbm
3.12	dev
3.12	dev,xgboost,lightgbm

Code Changes Needed for the New Models

For the most part, the new models integrate cleanly with the existing code. There were a few areas where we had to make some updates.

1. LightGBM does not support feature names with special characters like colons. Spark's Interaction class automatically adds colons to its output vector's attribute names. Eventually these colons end up in the names of the output column of the VectorAssembler. This caused LightGBM to fail with an error. So we've made a new RenameVectorAttributes class which transforms a vector column by replacing strings in its attribute names. We now add a RenameVectorAttributes object to the pipeline each time we add an Interaction, so that we can eliminate the colons from the attribute names.
2. Both XGBoost and LightGBM have multiple feature importance metrics. All of the previous models have only had a single feature importance metric. We've chosen to use weight (number of splits) and gain (total gain across all splits made by the feature) for both XGBoost and LightGBM. This took some refactoring of training step 3 - save model metadata. It's still a little bit clunky, but is hopefully easy enough to read.
3. In hlink.spark.session, we now install LightGBM's Scala dependencies if the synapse.ml Python package is present, using the ADD JAR SQL command. I had trouble using the spark.jars.packages configuration setting, although it really seemed like it should have worked.
4. In the tests, we've made a new hlink.tests.markers module with requires_xgboost and requires_lightgbm markers (decorators). These automatically skip the decorated tests when the required Python packages are not present. This works nicely for testing both locally and CI/CD, and if you run with pytest -ra, it will tell you why it skipped the tests:

=================================================== short test summary info ===================================================
SKIPPED [1] hlink/tests/core/classifier_test.py:10: requires the lightgbm library
SKIPPED [1] hlink/tests/core/classifier_test.py:21: requires the xgboost library
SKIPPED [1] hlink/tests/integration_score_with_trained_models_test.py:497: requires the lightgbm library
SKIPPED [1] hlink/tests/integration_score_with_trained_models_test.py:956: requires the xgboost library
SKIPPED [1] hlink/tests/training_test.py:435: requires the lightgbm library
SKIPPED [1] hlink/tests/training_test.py:498: requires the lightgbm library
SKIPPED [1] hlink/tests/training_test.py:556: requires the xgboost library

To Do Items (Nice to Have)

• See if we can reduce the verbosity of the Spark logs when Spark installs the additional LightGBM Scala dependencies.
• See if we can make the verbosity of XGBoost and LightGBM configurable from the hlink config file.
• Rewrite the introduction to the Models Sphinx docs page to include information pertinent to XGBoost and LightGBM.