Skip to content

Commit

Permalink
Deploying to gh-pages from @ browsermt/bergamot-translator@4b0da8d 🚀
Browse files Browse the repository at this point in the history
  • Loading branch information
XapaJIaMnu committed Aug 1, 2023
1 parent 0c4c127 commit 7e4a669
Show file tree
Hide file tree
Showing 214 changed files with 310 additions and 282 deletions.
2 changes: 1 addition & 1 deletion main/.buildinfo
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Sphinx build info version 1
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
config: db95eafa1250a99affece3a7ba5b7280
config: 442fdcf2ba2b4a484439844000dd9759
tags: 645f666f9bcd5a90fca523b33c5a78b7
Binary file not shown.
Binary file modified main/.doctrees/api/file_src_translator_html.cpp.doctree
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file modified main/.doctrees/api/library_index.doctree
Binary file not shown.
Binary file modified main/.doctrees/api/namespace_marian__bergamot.doctree
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file modified main/.doctrees/api/structmarian_1_1bergamot_1_1MemoryBundle.doctree
Binary file not shown.
Binary file modified main/.doctrees/api/unabridged_api.doctree
Binary file not shown.
Binary file modified main/.doctrees/environment.pickle
Binary file not shown.
Binary file modified main/.doctrees/index.doctree
Binary file not shown.
2 changes: 1 addition & 1 deletion main/CI.html
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />

<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Continuous Integration &mdash; Bergamot Translator main 2023-07-31 documentation</title><link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<title>Continuous Integration &mdash; Bergamot Translator main 2023-08-01 documentation</title><link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/css/custom.css" type="text/css" />
<link rel="stylesheet" href="_static/collapsible-lists/css/tree_view.css" type="text/css" />
Expand Down
2 changes: 1 addition & 1 deletion main/Unified_API.html
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />

<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Unified (C++) API of Bergamot Translator &mdash; Bergamot Translator main 2023-07-31 documentation</title><link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<title>Unified (C++) API of Bergamot Translator &mdash; Bergamot Translator main 2023-08-01 documentation</title><link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/css/custom.css" type="text/css" />
<link rel="stylesheet" href="_static/collapsible-lists/css/tree_view.css" type="text/css" />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,12 +59,12 @@ Functions

- :ref:`exhale_function_byte__array__util_8cpp_1a6f199b6686a506deb71095f8b5f9868c`

- :ref:`exhale_function_byte__array__util_8cpp_1ac17b7705d71079521a1e6db2059ada4f`

- :ref:`exhale_function_byte__array__util_8cpp_1a276da62846e557d498b92ee9abc90aa0`
- :ref:`exhale_function_byte__array__util_8cpp_1aa1b8190848999386b1c9f918eb7e32bc`

- :ref:`exhale_function_byte__array__util_8cpp_1a42b492f8a1927d17d3a72774b53eebc2`

- :ref:`exhale_function_byte__array__util_8cpp_1a276da62846e557d498b92ee9abc90aa0`

- :ref:`exhale_function_byte__array__util_8cpp_1a03d0ea07c0548c3c9d9fa3b45e8eccc6`

- :ref:`exhale_function_byte__array__util_8cpp_1a68022e0bdf301426d59e4dc3d89e0412`
Expand Down
4 changes: 2 additions & 2 deletions main/_sources/api/file_src_translator_html.cpp.rst.txt
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ Functions
---------


- :ref:`exhale_function_html_8cpp_1ae52ad7db185889701520ffea1ce74d37`

- :ref:`exhale_function_html_8cpp_1a90be5fddaf86ee1c2f02eb695bb2cb46`

- :ref:`exhale_function_html_8cpp_1ae52ad7db185889701520ffea1ce74d37`

Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.. _exhale_function_byte__array__util_8cpp_1ac17b7705d71079521a1e6db2059ada4f:
.. _exhale_function_byte__array__util_8cpp_1aa1b8190848999386b1c9f918eb7e32bc:

Function marian::bergamot::getModelMemoryFromConfig
===================================================
Expand Down
2 changes: 1 addition & 1 deletion main/_sources/api/namespace_marian__bergamot.rst.txt
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ Functions

- :ref:`exhale_function_byte__array__util_8cpp_1a6f199b6686a506deb71095f8b5f9868c`

- :ref:`exhale_function_byte__array__util_8cpp_1ac17b7705d71079521a1e6db2059ada4f`
- :ref:`exhale_function_byte__array__util_8cpp_1aa1b8190848999386b1c9f918eb7e32bc`

- :ref:`exhale_function_byte__array__util_8cpp_1a42b492f8a1927d17d3a72774b53eebc2`

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -103,21 +103,24 @@ Program Listing for File byte_array_util.cpp
return alignedMemory;
}
AlignedMemory getModelMemoryFromConfig(marian::Ptr<marian::Options> options) {
std::vector<AlignedMemory> getModelMemoryFromConfig(marian::Ptr<marian::Options> options) {
auto models = options->get<std::vector<std::string>>("models");
ABORT_IF(models.size() != 1, "Loading multiple binary models is not supported for now as it is not necessary.");
// If binary model we load into aligned memory. If .npz we leave it be to
// return empty aligned memory, thus allowing traditional file system loads.
if (marian::io::isBin(models[0])) {
AlignedMemory alignedMemory = loadFileToMemory(models[0], 256);
return alignedMemory;
} else if (marian::io::isNpz(models[0])) {
return AlignedMemory();
} else {
ABORT("Unknown extension for model: {}, should be one of `.bin` or `.npz`", models[0]);
std::vector<AlignedMemory> modelMemories(models.size());
for (size_t i = 0; i < models.size(); ++i) {
const auto model = models[i];
if (marian::io::isBin(model)) {
modelMemories[i] = loadFileToMemory(model, 256);
} else if (marian::io::isNpz(model)) {
// if any of the models are npz format, we revert to loading from file for all models.
LOG(debug, "Encountered an npz file {}; will use file loading for {} models", model, models.size());
return {};
} else {
ABORT("Unknown extension for model: {}, should be one of `.bin` or `.npz`", model);
}
}
return AlignedMemory();
return modelMemories;
}
AlignedMemory getShortlistMemoryFromConfig(marian::Ptr<marian::Options> options) {
Expand Down Expand Up @@ -165,7 +168,7 @@ Program Listing for File byte_array_util.cpp
MemoryBundle getMemoryBundleFromConfig(marian::Ptr<marian::Options> options) {
MemoryBundle memoryBundle;
memoryBundle.model = getModelMemoryFromConfig(options);
memoryBundle.models = getModelMemoryFromConfig(options);
memoryBundle.shortlist = getShortlistMemoryFromConfig(options);
getVocabsMemoryFromConfig(options, memoryBundle.vocabs);
memoryBundle.ssplitPrefixFile = getSsplitPrefixFileMemoryFromConfig(options);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Program Listing for File byte_array_util.h
namespace bergamot {
AlignedMemory loadFileToMemory(const std::string& path, size_t alignment);
AlignedMemory getModelMemoryFromConfig(marian::Ptr<marian::Options> options);
std::vector<AlignedMemory> getModelMemoryFromConfig(marian::Ptr<marian::Options> options);
AlignedMemory getQualityEstimatorModel(const marian::Ptr<marian::Options>& options);
AlignedMemory getQualityEstimatorModel(MemoryBundle& memoryBundle, const marian::Ptr<marian::Options>& options);
AlignedMemory getShortlistMemoryFromConfig(marian::Ptr<marian::Options> options);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ Program Listing for File definitions.h
typedef AlignedVector<char> AlignedMemory;
struct MemoryBundle {
AlignedMemory model{};
AlignedMemory shortlist{};
std::vector<AlignedMemory> models{};
AlignedMemory shortlist{};
std::vector<std::shared_ptr<AlignedMemory>> vocabs{};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -73,24 +73,35 @@ Program Listing for File translation_model.cpp
graph->getBackend()->configureDevice(options_);
graph->reserveWorkspaceMB(options_->get<size_t>("workspace"));
// Marian Model: Load from memoryBundle or shortList
if (memory_.model.size() > 0 &&
memory_.model.begin() !=
nullptr) { // If we have provided a byte array that contains the model memory, we can initialise the
// model from there, as opposed to from reading in the config file
ABORT_IF((uintptr_t)memory_.model.begin() % 256 != 0,
"The provided memory is not aligned to 256 bytes and will crash when vector instructions are used on it.");
if (options_->get<bool>("check-bytearray", false)) {
ABORT_IF(!validateBinaryModel(memory_.model, memory_.model.size()),
"The binary file is invalid. Incomplete or corrupted download?");
}
const std::vector<const void *> container = {
memory_.model.begin()}; // Marian supports multiple models initialised in this manner hence std::vector.
// However we will only ever use 1 during decoding.
// if memory_.models is populated, then all models were of binary format
if (memory_.models.size() >= 1) {
const std::vector<const void *> container = std::invoke([&]() {
std::vector<const void *> model_ptrs(memory_.models.size());
for (size_t i = 0; i < memory_.models.size(); ++i) {
const AlignedMemory &model = memory_.models[i];
ABORT_IF(model.size() == 0 || model.begin() == nullptr, "The provided memory is empty. Cannot load the model.");
ABORT_IF(
(uintptr_t)model.begin() % 256 != 0,
"The provided memory is not aligned to 256 bytes and will crash when vector instructions are used on it.");
if (options_->get<bool>("check-bytearray", false)) {
ABORT_IF(!validateBinaryModel(model, model.size()),
"The binary file is invalid. Incomplete or corrupted download?");
}
model_ptrs[i] = model.begin();
LOG(debug, "Loaded model {} of {} from memory", (i + 1), model_ptrs.size());
}
return model_ptrs;
});
scorerEnsemble = createScorers(options_, container);
} else {
// load npz format models, or a mixture of binary/npz formats
scorerEnsemble = createScorers(options_);
LOG(debug, "Loaded {} model(s) from file", scorerEnsemble.size());
}
for (auto scorer : scorerEnsemble) {
scorer->init(graph);
if (shortlistGenerator_) {
Expand Down
2 changes: 1 addition & 1 deletion main/_sources/api/unabridged_api.rst.txt
Original file line number Diff line number Diff line change
Expand Up @@ -301,7 +301,7 @@ Functions
.. toctree::
:maxdepth: 5

function_byte__array__util_8cpp_1ac17b7705d71079521a1e6db2059ada4f.rst
function_byte__array__util_8cpp_1aa1b8190848999386b1c9f918eb7e32bc.rst

.. toctree::
:maxdepth: 5
Expand Down
2 changes: 1 addition & 1 deletion main/_static/documentation_options.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
var DOCUMENTATION_OPTIONS = {
URL_ROOT: document.getElementById("documentation_options").getAttribute('data-url_root'),
VERSION: 'main 2023-07-31',
VERSION: 'main 2023-08-01',
LANGUAGE: 'None',
COLLAPSE_INDEX: false,
BUILDER: 'html',
Expand Down
2 changes: 1 addition & 1 deletion main/api/class_view_hierarchy.html
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />

<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Class Hierarchy &mdash; Bergamot Translator main 2023-07-31 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<title>Class Hierarchy &mdash; Bergamot Translator main 2023-08-01 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
<link rel="stylesheet" href="../_static/collapsible-lists/css/tree_view.css" type="text/css" />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />

<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Class AggregateBatchingPool &mdash; Bergamot Translator main 2023-07-31 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<title>Class AggregateBatchingPool &mdash; Bergamot Translator main 2023-08-01 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
<link rel="stylesheet" href="../_static/collapsible-lists/css/tree_view.css" type="text/css" />
Expand Down
2 changes: 1 addition & 1 deletion main/api/classmarian_1_1bergamot_1_1AlignedVector.html
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />

<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Template Class AlignedVector &mdash; Bergamot Translator main 2023-07-31 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<title>Template Class AlignedVector &mdash; Bergamot Translator main 2023-08-01 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
<link rel="stylesheet" href="../_static/collapsible-lists/css/tree_view.css" type="text/css" />
Expand Down
2 changes: 1 addition & 1 deletion main/api/classmarian_1_1bergamot_1_1Annotation.html
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />

<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Class Annotation &mdash; Bergamot Translator main 2023-07-31 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<title>Class Annotation &mdash; Bergamot Translator main 2023-08-01 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
<link rel="stylesheet" href="../_static/collapsible-lists/css/tree_view.css" type="text/css" />
Expand Down
2 changes: 1 addition & 1 deletion main/api/classmarian_1_1bergamot_1_1AsyncService.html
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />

<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Class AsyncService &mdash; Bergamot Translator main 2023-07-31 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<title>Class AsyncService &mdash; Bergamot Translator main 2023-08-01 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
<link rel="stylesheet" href="../_static/collapsible-lists/css/tree_view.css" type="text/css" />
Expand Down
2 changes: 1 addition & 1 deletion main/api/classmarian_1_1bergamot_1_1AtomicCache.html
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />

<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Template Class AtomicCache &mdash; Bergamot Translator main 2023-07-31 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<title>Template Class AtomicCache &mdash; Bergamot Translator main 2023-08-01 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
<link rel="stylesheet" href="../_static/collapsible-lists/css/tree_view.css" type="text/css" />
Expand Down
2 changes: 1 addition & 1 deletion main/api/classmarian_1_1bergamot_1_1Batch.html
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />

<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Class Batch &mdash; Bergamot Translator main 2023-07-31 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<title>Class Batch &mdash; Bergamot Translator main 2023-08-01 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
<link rel="stylesheet" href="../_static/collapsible-lists/css/tree_view.css" type="text/css" />
Expand Down
2 changes: 1 addition & 1 deletion main/api/classmarian_1_1bergamot_1_1BatchingPool.html
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />

<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Class BatchingPool &mdash; Bergamot Translator main 2023-07-31 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<title>Class BatchingPool &mdash; Bergamot Translator main 2023-08-01 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
<link rel="stylesheet" href="../_static/collapsible-lists/css/tree_view.css" type="text/css" />
Expand Down
2 changes: 1 addition & 1 deletion main/api/classmarian_1_1bergamot_1_1BlockingService.html
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />

<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Class BlockingService &mdash; Bergamot Translator main 2023-07-31 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<title>Class BlockingService &mdash; Bergamot Translator main 2023-08-01 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
<link rel="stylesheet" href="../_static/collapsible-lists/css/tree_view.css" type="text/css" />
Expand Down
2 changes: 1 addition & 1 deletion main/api/classmarian_1_1bergamot_1_1ConfigParser.html
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />

<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Template Class ConfigParser &mdash; Bergamot Translator main 2023-07-31 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<title>Template Class ConfigParser &mdash; Bergamot Translator main 2023-08-01 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
<link rel="stylesheet" href="../_static/collapsible-lists/css/tree_view.css" type="text/css" />
Expand Down
2 changes: 1 addition & 1 deletion main/api/classmarian_1_1bergamot_1_1HTML.html
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />

<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Class HTML &mdash; Bergamot Translator main 2023-07-31 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<title>Class HTML &mdash; Bergamot Translator main 2023-08-01 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
<link rel="stylesheet" href="../_static/collapsible-lists/css/tree_view.css" type="text/css" />
Expand Down
2 changes: 1 addition & 1 deletion main/api/classmarian_1_1bergamot_1_1Logger.html
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" />

<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Class Logger &mdash; Bergamot Translator main 2023-07-31 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<title>Class Logger &mdash; Bergamot Translator main 2023-08-01 documentation</title><link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
<link rel="stylesheet" href="../_static/collapsible-lists/css/tree_view.css" type="text/css" />
Expand Down
Loading

0 comments on commit 7e4a669

Please sign in to comment.