Skip to content

Commit

Permalink
Deploying to gh-pages from @ ccb53fe 🚀
Browse files Browse the repository at this point in the history
  • Loading branch information
claromes committed Jun 24, 2024
1 parent 60520ab commit ceaf7bb
Show file tree
Hide file tree
Showing 13 changed files with 18 additions and 122 deletions.
Binary file removed _images/preview_image.jpg
Binary file not shown.
19 changes: 0 additions & 19 deletions _sources/api.rst.txt
Original file line number Diff line number Diff line change
Expand Up @@ -64,25 +64,6 @@ Utils
.. autofunction:: semicolon_parser
.. autofunction:: timestamp_parser

Exceptions
------------

.. automodule:: waybacktweets.exceptions.exceptions

.. autoclass:: ReadTimeoutError
:members:

.. autoclass:: ConnectionError
:members:

.. autoclass:: HTTPError
:members:

.. autoclass:: EmptyResponseError
:members:

.. autoclass:: GetResponseError
:members:

Config
------------
Expand Down
4 changes: 0 additions & 4 deletions _sources/index.rst.txt
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,6 @@ Pre-release: |release|

Retrieves archived tweets CDX data from the Wayback Machine, performs necessary parsing (see :ref:`field_options`), and saves the data in CSV, JSON, and HTML formats.

.. image:: ../assets/preview_image.jpg
:alt: Preview image
:align: center

.. note::
Intensive queries can lead to rate limiting, resulting in a temporary ban of a few minutes from web.archive.org.

Expand Down
3 changes: 1 addition & 2 deletions _sources/streamlit.rst.txt
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,13 @@ Aplication that displays multiple archived tweets on Wayback Machine to avoid op

Filters
----------

- Filtering by date range: Using the ``from`` and ``to`` filters

- Limit: Query result limits.

- Offset: Allows for a simple way to scroll through the results.

- Only unavailable tweets: Checks if the archived URL still exists on Twitter (see the :ref:`flowchart`)

- Only unique Wayback Machine URLs: Filtering by the collapse option using the ``urlkey`` field and the URL Match Scope ``prefix``


Expand Down
4 changes: 2 additions & 2 deletions _sources/workflow.rst.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
Workflow
================

The tool was written following a proposal not only to Retrieve data from archived tweets, but also to facilitate the reading of these tweets. Therefore, a flow is defined to obtain these results in the best possible way.
The tool was written following a proposal not only to retrieve data from archived tweets, but also to facilitate the reading of these tweets. Therefore, a flow is defined to obtain these results in the best possible way.

Due to limitations of the Wayback CDX Server API, it is not always possible to parse the results with the mimetype ``application/json``, regardless, the data in CDX format are saved.

Expand All @@ -21,5 +21,5 @@ Use the mouse to zoom in and out the flowchart.
C--> |4xx| E[return None]
E--> F{request Archived\nTweet URL}
F--> |4xx| G[return Only CDX data]
F--> |2xx/3xx: application/json| J[return JSON text]
F--> |TODO: 2xx/3xx: application/json| J[return JSON text]
F--> |2xx/3xx: text/html, warc/revisit, unk| K[return HTML iframe tag]
58 changes: 8 additions & 50 deletions api.html
Original file line number Diff line number Diff line change
Expand Up @@ -318,24 +318,24 @@ <h1>API<a class="headerlink" href="#api" title="Link to this heading">¶</a></h1
<p>Generates an HTML file to visualize the parsed data.</p>
<dl class="py class">
<dt class="sig sig-object py" id="waybacktweets.api.visualize.HTMLTweetsVisualizer">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">waybacktweets.api.visualize.</span></span><span class="sig-name descname"><span class="pre">HTMLTweetsVisualizer</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">username</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">json_file_path</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">html_file_path</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#waybacktweets.api.visualize.HTMLTweetsVisualizer" title="Link to this definition"></a></dt>
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">waybacktweets.api.visualize.</span></span><span class="sig-name descname"><span class="pre">HTMLTweetsVisualizer</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">username</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">json_path</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">html_file_path</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#waybacktweets.api.visualize.HTMLTweetsVisualizer" title="Link to this definition"></a></dt>
<dd><p>Class responsible for generating an HTML file to visualize the parsed data.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>username</strong> (<em>str</em>) – The username associated with the tweets.</p></li>
<li><p><strong>json_file_path</strong> (<em>Union</em><em>[</em><em>str</em><em>, </em><em>List</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The path of the JSON file or the JSON data itself.</p></li>
<li><p><strong>json_path</strong> (<em>Union</em><em>[</em><em>str</em><em>, </em><em>List</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The path of the JSON file or the JSON data itself.</p></li>
<li><p><strong>html_file_path</strong> (<em>str</em><em>, </em><em>optional</em>) – The path where the HTML file will be saved.</p></li>
</ul>
</dd>
</dl>
<dl class="py method">
<dt class="sig sig-object py" id="waybacktweets.api.visualize.HTMLTweetsVisualizer._json_loader">
<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_json_loader</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">json_file_path</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#waybacktweets.api.visualize.HTMLTweetsVisualizer._json_loader" title="Link to this definition"></a></dt>
<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_json_loader</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">json_path</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#waybacktweets.api.visualize.HTMLTweetsVisualizer._json_loader" title="Link to this definition"></a></dt>
<dd><p>Reads and loads JSON data from a specified file path or JSON string.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>json_file_path</strong> (<em>Union</em><em>[</em><em>str</em><em>, </em><em>List</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The path of the JSON file or the JSON data itself.</p>
<dd class="field-odd"><p><strong>json_path</strong> (<em>Union</em><em>[</em><em>str</em><em>, </em><em>List</em><em>[</em><em>str</em><em>]</em><em>]</em>) – The path of the JSON file or the JSON data itself.</p>
</dd>
<dt class="field-even">Return type<span class="colon">:</span></dt>
<dd class="field-even"><p><span class="sphinx_autodoc_typehints-type"><code class="xref py py-class docutils literal notranslate"><span class="pre">List</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">Dict</span></code>[<code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code>, <code class="xref py py-data docutils literal notranslate"><span class="pre">Any</span></code>]]</span></p>
Expand Down Expand Up @@ -515,10 +515,10 @@ <h1>API<a class="headerlink" href="#api" title="Link to this heading">¶</a></h1
</dd>
<dt class="field-even">Raises<span class="colon">:</span></dt>
<dd class="field-even"><ul class="simple">
<li><p><a class="reference internal" href="#waybacktweets.exceptions.exceptions.ReadTimeoutError" title="waybacktweets.exceptions.exceptions.ReadTimeoutError"><strong>ReadTimeoutError</strong></a> – If a read timeout occurs.</p></li>
<li><p><a class="reference internal" href="#waybacktweets.exceptions.exceptions.ConnectionError" title="waybacktweets.exceptions.exceptions.ConnectionError"><strong>ConnectionError</strong></a> – If a connection error occurs.</p></li>
<li><p><a class="reference internal" href="#waybacktweets.exceptions.exceptions.HTTPError" title="waybacktweets.exceptions.exceptions.HTTPError"><strong>HTTPError</strong></a> – If an HTTP error occurs.</p></li>
<li><p><a class="reference internal" href="#waybacktweets.exceptions.exceptions.EmptyResponseError" title="waybacktweets.exceptions.exceptions.EmptyResponseError"><strong>EmptyResponseError</strong></a> – If the response is empty.</p></li>
<li><p><strong>ReadTimeoutError</strong> – If a read timeout occurs.</p></li>
<li><p><strong>ConnectionError</strong> – If a connection error occurs.</p></li>
<li><p><strong>HTTPError</strong> – If an HTTP error occurs.</p></li>
<li><p><strong>EmptyResponseError</strong> – If the response is empty.</p></li>
</ul>
</dd>
</dl>
Expand Down Expand Up @@ -577,40 +577,6 @@ <h1>API<a class="headerlink" href="#api" title="Link to this heading">¶</a></h1
</dl>
</dd></dl>

</section>
<section id="module-waybacktweets.exceptions.exceptions">
<span id="exceptions"></span><h2>Exceptions<a class="headerlink" href="#module-waybacktweets.exceptions.exceptions" title="Link to this heading"></a></h2>
<p>Wayback Tweets Exceptions</p>
<dl class="py class">
<dt class="sig sig-object py" id="waybacktweets.exceptions.exceptions.ReadTimeoutError">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">waybacktweets.exceptions.exceptions.</span></span><span class="sig-name descname"><span class="pre">ReadTimeoutError</span></span><a class="headerlink" href="#waybacktweets.exceptions.exceptions.ReadTimeoutError" title="Link to this definition"></a></dt>
<dd><p>Exception raised for read timeout errors.</p>
</dd></dl>

<dl class="py class">
<dt class="sig sig-object py" id="waybacktweets.exceptions.exceptions.ConnectionError">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">waybacktweets.exceptions.exceptions.</span></span><span class="sig-name descname"><span class="pre">ConnectionError</span></span><a class="headerlink" href="#waybacktweets.exceptions.exceptions.ConnectionError" title="Link to this definition"></a></dt>
<dd><p>Exception raised for connection errors.</p>
</dd></dl>

<dl class="py class">
<dt class="sig sig-object py" id="waybacktweets.exceptions.exceptions.HTTPError">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">waybacktweets.exceptions.exceptions.</span></span><span class="sig-name descname"><span class="pre">HTTPError</span></span><a class="headerlink" href="#waybacktweets.exceptions.exceptions.HTTPError" title="Link to this definition"></a></dt>
<dd><p>Exception raised for HTTP errors.</p>
</dd></dl>

<dl class="py class">
<dt class="sig sig-object py" id="waybacktweets.exceptions.exceptions.EmptyResponseError">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">waybacktweets.exceptions.exceptions.</span></span><span class="sig-name descname"><span class="pre">EmptyResponseError</span></span><a class="headerlink" href="#waybacktweets.exceptions.exceptions.EmptyResponseError" title="Link to this definition"></a></dt>
<dd><p>Exception raised for empty responses.</p>
</dd></dl>

<dl class="py class">
<dt class="sig sig-object py" id="waybacktweets.exceptions.exceptions.GetResponseError">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">waybacktweets.exceptions.exceptions.</span></span><span class="sig-name descname"><span class="pre">GetResponseError</span></span><a class="headerlink" href="#waybacktweets.exceptions.exceptions.GetResponseError" title="Link to this definition"></a></dt>
<dd><p>Base class for exceptions in get_response.</p>
</dd></dl>

</section>
<section id="module-waybacktweets.config.config">
<span id="config"></span><h2>Config<a class="headerlink" href="#module-waybacktweets.config.config" title="Link to this heading"></a></h2>
Expand Down Expand Up @@ -712,14 +678,6 @@ <h3>Contents</h3>
<li><a class="reference internal" href="#waybacktweets.utils.utils.timestamp_parser"><code class="docutils literal notranslate"><span class="pre">timestamp_parser()</span></code></a></li>
</ul>
</li>
<li><a class="reference internal" href="#module-waybacktweets.exceptions.exceptions">Exceptions</a><ul>
<li><a class="reference internal" href="#waybacktweets.exceptions.exceptions.ReadTimeoutError"><code class="docutils literal notranslate"><span class="pre">ReadTimeoutError</span></code></a></li>
<li><a class="reference internal" href="#waybacktweets.exceptions.exceptions.ConnectionError"><code class="docutils literal notranslate"><span class="pre">ConnectionError</span></code></a></li>
<li><a class="reference internal" href="#waybacktweets.exceptions.exceptions.HTTPError"><code class="docutils literal notranslate"><span class="pre">HTTPError</span></code></a></li>
<li><a class="reference internal" href="#waybacktweets.exceptions.exceptions.EmptyResponseError"><code class="docutils literal notranslate"><span class="pre">EmptyResponseError</span></code></a></li>
<li><a class="reference internal" href="#waybacktweets.exceptions.exceptions.GetResponseError"><code class="docutils literal notranslate"><span class="pre">GetResponseError</span></code></a></li>
</ul>
</li>
<li><a class="reference internal" href="#module-waybacktweets.config.config">Config</a><ul>
<li><a class="reference internal" href="#waybacktweets.config.config.config"><code class="docutils literal notranslate"><span class="pre">config</span></code></a></li>
<li><a class="reference internal" href="#waybacktweets.config.config.verbose"><code class="docutils literal notranslate"><span class="pre">verbose</span></code></a></li>
Expand Down
34 changes: 2 additions & 32 deletions genindex.html
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,6 @@ <h1 id="index">Index</h1>
| <a href="#J"><strong>J</strong></a>
| <a href="#M"><strong>M</strong></a>
| <a href="#P"><strong>P</strong></a>
| <a href="#R"><strong>R</strong></a>
| <a href="#S"><strong>S</strong></a>
| <a href="#T"><strong>T</strong></a>
| <a href="#U"><strong>U</strong></a>
Expand Down Expand Up @@ -202,8 +201,6 @@ <h2 id="C">C</h2>
<li><a href="api.html#waybacktweets.utils.utils.clean_wayback_machine_url">clean_wayback_machine_url() (in module waybacktweets.utils.utils)</a>
</li>
<li><a href="api.html#waybacktweets.config.config.config">config (in module waybacktweets.config.config)</a>
</li>
<li><a href="api.html#waybacktweets.exceptions.exceptions.ConnectionError">ConnectionError (class in waybacktweets.exceptions.exceptions)</a>
</li>
</ul></td>
</tr></table>
Expand All @@ -220,10 +217,6 @@ <h2 id="E">E</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="api.html#waybacktweets.api.parse.TwitterEmbed.embed">embed() (waybacktweets.api.parse.TwitterEmbed method)</a>
</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="api.html#waybacktweets.exceptions.exceptions.EmptyResponseError">EmptyResponseError (class in waybacktweets.exceptions.exceptions)</a>
</li>
</ul></td>
</tr></table>
Expand All @@ -232,14 +225,12 @@ <h2 id="G">G</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="api.html#waybacktweets.api.visualize.HTMLTweetsVisualizer.generate">generate() (waybacktweets.api.visualize.HTMLTweetsVisualizer method)</a>
</li>
<li><a href="api.html#waybacktweets.api.request.WaybackTweets.get">get() (waybacktweets.api.request.WaybackTweets method)</a>
</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="api.html#waybacktweets.utils.utils.get_response">get_response() (in module waybacktweets.utils.utils)</a>
<li><a href="api.html#waybacktweets.api.request.WaybackTweets.get">get() (waybacktweets.api.request.WaybackTweets method)</a>
</li>
<li><a href="api.html#waybacktweets.exceptions.exceptions.GetResponseError">GetResponseError (class in waybacktweets.exceptions.exceptions)</a>
<li><a href="api.html#waybacktweets.utils.utils.get_response">get_response() (in module waybacktweets.utils.utils)</a>
</li>
</ul></td>
</tr></table>
Expand All @@ -248,10 +239,6 @@ <h2 id="H">H</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="api.html#waybacktweets.api.visualize.HTMLTweetsVisualizer">HTMLTweetsVisualizer (class in waybacktweets.api.visualize)</a>
</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="api.html#waybacktweets.exceptions.exceptions.HTTPError">HTTPError (class in waybacktweets.exceptions.exceptions)</a>
</li>
</ul></td>
</tr></table>
Expand Down Expand Up @@ -288,8 +275,6 @@ <h2 id="M">M</h2>
<li><a href="api.html#module-waybacktweets.api.visualize">waybacktweets.api.visualize</a>
</li>
<li><a href="api.html#module-waybacktweets.config.config">waybacktweets.config.config</a>
</li>
<li><a href="api.html#module-waybacktweets.exceptions.exceptions">waybacktweets.exceptions.exceptions</a>
</li>
<li><a href="api.html#module-waybacktweets.utils.utils">waybacktweets.utils.utils</a>
</li>
Expand All @@ -309,14 +294,6 @@ <h2 id="P">P</h2>
</ul></td>
</tr></table>

<h2 id="R">R</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="api.html#waybacktweets.exceptions.exceptions.ReadTimeoutError">ReadTimeoutError (class in waybacktweets.exceptions.exceptions)</a>
</li>
</ul></td>
</tr></table>

<h2 id="S">S</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
Expand Down Expand Up @@ -447,13 +424,6 @@ <h2 id="W">W</h2>

<ul>
<li><a href="api.html#module-waybacktweets.config.config">module</a>
</li>
</ul></li>
<li>
waybacktweets.exceptions.exceptions

<ul>
<li><a href="api.html#module-waybacktweets.exceptions.exceptions">module</a>
</li>
</ul></li>
<li>
Expand Down
2 changes: 0 additions & 2 deletions index.html
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,6 @@ <h1>Wayback Tweets<a class="headerlink" href="#wayback-tweets" title="Link to th
<img alt="Wayback Tweets" class="align-center" src="_images/waybacktweets_title.png" />
<p>Pre-release: v1.0a3</p>
<p>Retrieves archived tweets CDX data from the Wayback Machine, performs necessary parsing (see <a class="reference internal" href="field_options.html#field-options"><span class="std std-ref">Field Options</span></a>), and saves the data in CSV, JSON, and HTML formats.</p>
<img alt="Preview image" class="align-center" src="_images/preview_image.jpg" />
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Intensive queries can lead to rate limiting, resulting in a temporary ban of a few minutes from web.archive.org.</p>
Expand Down Expand Up @@ -130,7 +129,6 @@ <h2>API Reference<a class="headerlink" href="#api-reference" title="Link to this
<li class="toctree-l2"><a class="reference internal" href="api.html#module-waybacktweets.api.export">Export</a></li>
<li class="toctree-l2"><a class="reference internal" href="api.html#module-waybacktweets.api.visualize">Visualize</a></li>
<li class="toctree-l2"><a class="reference internal" href="api.html#utils">Utils</a></li>
<li class="toctree-l2"><a class="reference internal" href="api.html#module-waybacktweets.exceptions.exceptions">Exceptions</a></li>
<li class="toctree-l2"><a class="reference internal" href="api.html#module-waybacktweets.config.config">Config</a></li>
</ul>
</li>
Expand Down
Binary file modified objects.inv
Binary file not shown.
Loading

0 comments on commit ceaf7bb

Please sign in to comment.