Update for 2.0.0rc6 (#442)

* Update for 2.0.0rc6

.gitignore

@@ -3,6 +3,15 @@ __pycache__/
 *.py[cod]
 *.exe
 *.zip
+*.txt
+*.lab
+*.dict
+!tests/data/dictionaries/*.dict
+!tests/data/dictionaries/*.txt
+!tests/data/lab/*.txt
+!tests/data/lab/*.lab
+!tests/data/am/*.zip
+!tests/data/lm/*.zip
 report.txt
 
 .idea/
@@ -72,18 +81,14 @@ docs/build/
 # PyBuilder
 target/
 
-*.exe
 
 *.dll
 
-\.pytest_cache/
+.pytest_cache/
 
 docs/source/api/
 
-montreal_forced_aligner/_version.py
-/docs/source/reference/generated/
-<<<<<<< main
-
-docs/source/reference/multiprocessing/generated/
 
 *.lprof
+
+*.pclprof

docs/source/changelog/changelog_2.0.rst

@@ -10,6 +10,20 @@
 Release candidates
 ==================
 
+2.0.0rc6
+--------
+
+- Reverted the default export type to ``long_textgrid``, which can be changed to ``short_textgrid`` or ``json`` via the ``--output_format`` flag for commands that export TextGrids :github_issue:`434`
+- Added more information for when malformed dictionary lines fail to parse (i.e., lines with just tabs on them) :github_issue:`411`
+- Fixed a bug where phones with underscores in them would cause export to crash :github_issue:`432`
+- Changed the overwrite behavior in export to specifically avoid overwriting input files, rather than testing the existence of the overall output directory :github_issue:`431`
+- Added additional initial check to ensure that Kaldi and OpenFst binaries can be successfully invoked, rather than throwing an unintuitive error during feature creation
+- Optimized initial load and TextGrid export :github_issue:`437` and :github_issue:`249`
+- Allow for dictionaries with the same base name in different locations to be used side-by-side :github_issue:`417`
+- Fixed a bug where initial silence was not being properly handled if silence probability training had not been done
+- Removed PronunciationDictionaryMixin and PronunciationDictionary classes and refactored functionality into :class:`~montreal_forced_aligner.dictionary.multispeaker.MultispeakerDictionaryMixin` and :class:`~montreal_forced_aligner.db.Dictionary`
+- Fixed a bug where alignment models would not be adapted during adaptation :github_issue:`421`
+
 2.0.0rc5
 --------
 
@@ -77,7 +91,7 @@ Beta releases
 - Added better progress bars for corpus loading, acoustic modeling, G2P training, transcription and alignment
 - Changed the default behavior of G2P generation to use a threshold system rather than returning a single top pronunciation.  The threshold defaults to 0.99, but can be specified through ``--g2p_threshold``.  Specifying number of pronunciations will override this behavior (use ``--num_pronunciation 1`` for the old behavior).
 - Changed the behavior of G2P evaluation to check whether the generated hypothesis is in the golden pronunciation set, so languages with pronunciation variation will be less penalized in evaluation
-- Added :class:`~montreal_forced_aligner.data.WordData` and :class:`~montreal_forced_aligner.data.Pronunciation` data classes
+- Added :class:`~montreal_forced_aligner.data.WordData` and :class:`~montreal_forced_aligner.db.Pronunciation` data classes
 - Refactored and simplified TextGrid export process
 - Removed the ``multilingual_ipa`` mode in favor of a more general approach to better modeling phones
 - Added functionality to evaluate alignments against golden alignment set

docs/source/reference/acoustic_modeling/helper.rst

@@ -16,15 +16,6 @@ Mixins
 Multiprocessing workers and functions
 -------------------------------------
 
-.. currentmodule:: montreal_forced_aligner.acoustic_modeling.base
-
-.. autosummary::
-   :toctree: generated/
-
-   compute_alignment_improvement_func
-   compare_alignments
-
-
 .. currentmodule:: montreal_forced_aligner.acoustic_modeling.monophone
 
 .. autosummary::
@@ -61,14 +52,6 @@ Multiprocessing workers and functions
 Multiprocessing argument classes
 --------------------------------
 
-.. currentmodule:: montreal_forced_aligner.acoustic_modeling.base
-
-.. autosummary::
-   :toctree: generated/
-
-   AlignmentImprovementArguments
-
-
 .. currentmodule:: montreal_forced_aligner.acoustic_modeling.monophone
 
 .. autosummary::

docs/source/reference/database/index.rst

@@ -6,13 +6,14 @@ Database
 
 MFA uses a SQLite database to cache information during training/alignment runs.  An issue with training larger corpora was running into memory bottlenecks as all the information in the corpus was stored in memory, and fMLLR estimations in later stages would crash.  Additionally, there was always a trade off between storing results for use in other applications like :xref:`anchor` or providing diagnostic information to users, and ensuring that the core MFA workflows were as memory/time efficient as possible.  Offloading to a database frees up some memory, and makes some computations more efficient, and should be optimized enough to not slow down regular processing.
 
-.. currentmodule::  montreal_forced_aligner.corpus.db
+.. currentmodule::  montreal_forced_aligner.db
 
 .. autosummary::
    :toctree: generated/
 
    Dictionary
    Word
+   Pronunciation
    Phone
    File
    TextFile

docs/source/reference/dictionary/helper.rst

@@ -23,13 +23,6 @@ Mixins
    DictionaryMixin
    TemporaryDictionaryMixin
 
-.. currentmodule:: montreal_forced_aligner.dictionary.pronunciation
-
-.. autosummary::
-   :toctree: generated/
-
-   PronunciationDictionaryMixin
-
 .. currentmodule:: montreal_forced_aligner.dictionary.multispeaker
 
 .. autosummary::

docs/source/reference/dictionary/main.rst

@@ -7,5 +7,4 @@ Main classes
 .. autosummary::
    :toctree: generated/
 
-   PronunciationDictionary -- Pronunciation dictionary for Kaldi
    MultispeakerDictionary -- Collection of pronunciation dictionaries that specify speaker-dictionary mappings

docs/source/reference/helper/abc.rst

@@ -8,8 +8,9 @@
        MfaWorker -- Base worker class for MFA
        TopLevelMfaWorker -- MFA workers that have acoustic models
        TrainerMixin -- Trainer type interface
-       TemporaryDirectoryMixin -- Trainer type interface
-       AdapterMixin -- Trainer type interface
+       TemporaryDirectoryMixin -- Mixin for temporary directory functionality
+       DatabaseMixin -- Mixin for database functionality
+       AdapterMixin -- Adaptation type interface
        ExporterMixin -- Abstract exporter type interface
        FileExporterMixin -- File exporter type interface
        ModelExporterMixin -- Model exporter type interface

docs/source/reference/helper/data.rst

@@ -9,5 +9,7 @@
        SoundFileInformation
        PhoneSetType
        WordData
-       Pronunciation
+       WordType
+       PhoneType
+       PronunciationProbabilityCounter
        CtmInterval -- Data class for representing intervals in Kaldi's CTM files

environment.yml

@@ -12,6 +12,7 @@ dependencies:
   - dataclassy
   - kaldi
   - sox
+  - ffmpeg
   - pynini
   - openfst
   - baumwelch

environment_win.yml

@@ -12,6 +12,7 @@ dependencies:
   - dataclassy
   - kaldi
   - sox
+  - ffmpeg
   - praatio
   - biopython
   - sqlalchemy>=1.4

montreal_forced_aligner/abc.py

@@ -25,7 +25,9 @@
     get_type_hints,
 )
 
+import sqlalchemy
 import yaml
+from sqlalchemy.orm import Session
 
 from montreal_forced_aligner.exceptions import KaldiProcessingError
 from montreal_forced_aligner.helper import comma_join, load_configuration
@@ -40,7 +42,6 @@
     "MfaWorker",
     "TopLevelMfaWorker",
     "MetaDict",
-    "CtmErrorDict",
     "FileExporterMixin",
     "ModelExporterMixin",
     "TemporaryDirectoryMixin",
@@ -51,7 +52,6 @@
 
 # Configuration types
 MetaDict = Dict[str, Any]
-CtmErrorDict: Dict[Tuple[str, int], str]
 
 
 class KaldiFunction(metaclass=abc.ABCMeta):
@@ -167,6 +167,84 @@ def language_model_output_directory(self, directory: str) -> None:
         self._language_model_output_directory = directory
 
 
+class DatabaseMixin(TemporaryDirectoryMixin, metaclass=abc.ABCMeta):
+    """
+    Abstract class for mixing in database functionality
+    """
+
+    def __init__(
+        self,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self._db_engine = None
+        self._session = None
+
+    def initialize_database(self) -> None:
+        """
+        Initialize the database with database schema
+        """
+        from montreal_forced_aligner.db import MfaSqlBase
+
+        os.makedirs(self.output_directory, exist_ok=True)
+        MfaSqlBase.metadata.create_all(self.db_engine)
+
+    @property
+    def db_engine(self) -> sqlalchemy.engine.Engine:
+        """Database engine"""
+        if self._db_engine is None:
+            self._db_engine = self.construct_engine()
+        return self._db_engine
+
+    @property
+    def db_path(self) -> str:
+        """Path to SQLite database file"""
+        return os.path.join(self.output_directory, f"{self.identifier}.db")
+
+    def construct_engine(self, same_thread=True, read_only=False) -> sqlalchemy.engine.Engine:
+        """
+        Construct a database engine
+
+        Parameters
+        ----------
+        same_thread: bool, optional
+            Flag for whether to enforce checking access on different threads, defaults to True
+        read_only: bool, optional
+            Flag for whether the database engine should be created as read-only, defaults to False
+
+        Returns
+        -------
+        :class:`~sqlalchemy.engine.Engine`
+            SqlAlchemy engine
+        """
+        connect_args = {}
+        if not same_thread:
+            connect_args["check_same_thread"] = False
+        string = f"sqlite:///{self.db_path}"
+        if read_only:
+            string = f"sqlite:///file:{self.db_path}?mode=ro&nolock=1&uri=true"
+        return sqlalchemy.create_engine(string, connect_args=connect_args)
+
+    def session(self, **kwargs) -> Session:
+        """
+        Construct database session
+
+        Parameters
+        ----------
+        **kwargs
+            Keyword arguments to pass to the Session
+
+        Returns
+        -------
+        :class:`~sqlalchemy.orm.session.Session`
+            SqlAlchemy session
+        """
+        autoflush = kwargs.pop("autoflush", False)
+        if self._session is None:
+            self._session = sqlalchemy.orm.Session(self.db_engine, autoflush=autoflush, **kwargs)
+        return self._session
+
+
 class MfaWorker(metaclass=abc.ABCMeta):
     """
     Abstract class for MFA workers
@@ -501,6 +579,17 @@ def cleanup(self) -> None:
         Clean up loggers and output final message for top-level workers
         """
         try:
+            if getattr(self, "_session", None) is not None:
+                try:
+                    self._session.commit()
+                except Exception:
+                    self._session.rollback()
+                finally:
+                    self._session.close()
+                self._session = None
+            if getattr(self, "_db_engine", None) is not None:
+                self._db_engine.dispose()
+                self._db_engine = None
             if self.dirty:
                 self.log_error("There was an error in the run, please see the log.")
             else:
@@ -754,13 +843,6 @@ def __init__(self, cleanup_textgrids: bool = True, **kwargs):
         self.cleanup_textgrids = cleanup_textgrids
         super().__init__(**kwargs)
 
-    @property
-    def backup_output_directory(self) -> Optional[str]:
-        """Path to store files if overwriting is not allowed"""
-        if self.overwrite:
-            return None
-        return os.path.join(self.working_directory, "backup")
-
     @abc.abstractmethod
     def export_files(self, output_directory: str) -> None:
         """