20180914a

Changelog:

1. DNN Regressor: The user can choose through a configuration flag (useRegressorDNNTF) which DNN Estimator to use, either the DNNClassifier or DNNRegressor
   • Warning: SpectraLearnPredict2.ini files created with previous version of SLP2 need to be updated by adding useRegressorDNNTF = False under the section [DNNClassifier]
2. Bug fixes on HD5 I/O for plugins. Improved detection of file extensions, bug fixes.
3. HD5 is now the default saving format for Datasets
4. SpectraLearnPredict2 and SpectraKeras can now use either pure Keras APIs or tensorflow.keras APIs through a hardcoded flag (useTFKeras) in slp_config.py
5. New and updated utilities:
   1. RandomCrossValidMaker: new way to do format subsetting for cross validation
   2. RangeToDataMaker: produce a large set of files from a single dataset with specific parameters given in a range
   3. RemoveLimitedDatasets and InfoLimitedDatasets: Remove or identify the number of spectra in a datasets belonging to a specific class that are below a specific threshold.
   4. MergeDatasets: Merge two different training sets. If the axis are not the same, they second will be normalized to the first
6. SpectraKeras_MLP: Massive update
   • hidden layers can be defined via array in parameters (rather than hardcoded)
   • modularization: it will allow for easier implementation of future feature
7. Updated Torque-PBS submission scripts (sub_slp2 and sub_slp2_cv) to accommodate training files not in the main folder. Previously if files were outside the working folder, no log was generated.
8. Updated Slurm submission scripts (sub_ml and sub_ml_cv)
9. Label encoder transformation matrix is now saved into a pkl file: this allow for encoders info for predictions done using software other than SpectraLearnPredict2
10. Bug fixes and improvements:

• Fixed bug prevented correct detection of x-axis with different scale.
• Fixed bug that prevented to use training sets with less than 9 points per training data.
• Fixed label encoding in SVM, NN
• Training output lists more info on the training set

20180703b

Changelog:

1. Extensive support for HDF5 file format. SLP (v1 and v2) now support HDF5 files for both training and test files. Access speed is much much improved.
   1. TxtToHDF5 and HDF5ToTxt: New utilities to manage HDF5 (conversion to/from Txt). Existing Txt files can be easily converted into HDF5. Conversion to HDF5 can also provide built in full spectral normalization (so it doesn't need to be done within SLP2, speeding up the preprocessing).
   2. Added HDF5 support for: PlotData, PlotDataSplit, AddHorizontalOffset, AddLinearBackground, AddNoisyData, AddRelativeHorNoisyData, AddRelativeNoisyData, AddVerticalOffset,MixMakerRruff, XRange`
2. Substantially improved speed and capabilities: RruffDataMaker2, MakeCrossValidSet, RandomCrossValidMaker, AddNoisyData, MixMakerRruff. By using HDF5, it is now extremely efficient and fast.
3. Redesigned training data makers for less I/O usage (older versions available as legacy): RruffDataMaker and XmuDataMaker:
4. New Utility:
   1. PlotSingleDataRruffSpectra: Plot individual data from Rruff.
   2. AddSpectraToLearnFile: Add spectra to existing training files
   3. NormLearnFile: Normalize max spectral intensity to a specific value (if specified) or 1 (if not specified).
5. Binary .npy is no longer actively developed, although it is still supported. New utility TxtHDF5NpyConverter replaces existing ones and allow for interconversion between npy, txt, h5.
6. Simplification of learning matrix handling.
7. Added support for controlling GPU memory allocation.
8. Added profiler and evaluation hooks to dnntf.
9. Improved default parameters in SpectraKeras.
10. UI and bug fixes.
11. 20180703b has several hotfixes for Utilities, and it supersedes 20180623b and 20180703a. SPL itself us unchanged.

20180524a

First Release of SpectralMachine v2. Version 1 is deprecated going forward.

20180130a

First Preview Release of SpectralMachine v2. It includes partial revision of DNN code, and full support for Keras to create fully connected DNN and CNN. Ultimately, v2 will supersede v1, which for now remains the stable and recommended version for production. v1 is release with bug fixes.

20171222c

Stable

20171022a

Major change: DNNTF tf.estimator.inputs.numpy_input_fn uses num_epochs = 1 for testing (used to be None). This is as recommended.

20171020a

Changed shuffle from False (both train and test) to False only for train. This is also now a parameter to retest old sims.

Training until the use of 20171010e (included) has shuffle in input_fn for DNNTF set as False for both train and test.