Skip to content

Commit

Permalink
v1.1.0, "sleep away the afternoon"
Browse files Browse the repository at this point in the history
- much more efficient processing of schema updates storage, especially when dealing with large schemas.
- @lileeyao added --exclude-columns and the --jdbc_options features
- @lileeyao added --jdbc_options
- can now blacklist entire databases
- new kafka key format available, using a JSON array instead of an object
- bugfix: unsigned integer columns were captured incorrectly.  1.1 will
  recapture the schema and attempt to correct the error.
  • Loading branch information
Ben Osheroff committed May 20, 2016
1 parent 0633ab0 commit 9cecfe6
Show file tree
Hide file tree
Showing 5 changed files with 37 additions and 25 deletions.
17 changes: 12 additions & 5 deletions docs/site/config/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,9 @@
<li class="toctree-l3"><a href="#properties-file">Properties file</a></li>


<li class="toctree-l3"><a href="#running-against-rds">Running against RDS</a></li>


<li class="toctree-l3"><a href="#filters">Filters</a></li>


Expand Down Expand Up @@ -333,11 +336,6 @@ <h3 id="command-line-options">Command line options</h3>
<td>sync</td>
</tr>
<tr>
<td>--bootstrapper_fetch_size</td>
<td>number of rows fetched at a time during bootstrapping.</td>
<td>64000</td>
</tr>
<tr>
<td>&nbsp;</td>
<td></td>
<td></td>
Expand Down Expand Up @@ -366,6 +364,15 @@ <h3 id="properties-file">Properties file</h3>
</code></pre>

<p>then Maxwell will send <code>batch.size=16384</code> to the kafka producer library.</p>
<h3 id="running-against-rds">Running against RDS</h3>
<hr />
<p>To run Maxwell against RDS, (either Aurora or Mysql) you will need to do the following:</p>
<ul>
<li>set binlog_format to "ROW". Do this in the "parameter groups" section. For a Mysql-RDS instance this parameter will be
in a "DB Parameter Group", for Aurora it will be in a "DB Cluster Parameter Group".</li>
<li>setup RDS binlog retention as described <a href="../here">http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.Concepts.MySQL.html</a>.
The tl;dr is to execute <code>call mysql.rds_set_configuration('binlog retention hours', 24)</code> on the server.</li>
</ul>
<h3 id="filters">Filters</h3>
<hr />
<p>The options <code>include_dbs</code>, <code>exclude_dbs</code>, <code>include_tables</code>, and <code>exclude_tables</code> control whether
Expand Down
6 changes: 3 additions & 3 deletions docs/site/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -182,15 +182,15 @@ <h2>Maxwell = Mysql + Kafka</h2>
<li>Source: <a href="https://github.com/zendesk/maxwell">https://github.com/zendesk/maxwell</a></li>
</ul>
<p><br style="clear:both"/></p>
<pre><code> mysql&gt; insert into `test`.`maxwell` set id = 1, daemon = 'Stanlislaw Lem';
<pre><code> mysql&gt; insert into `test`.`maxwell` set id = 1, daemon = 'Stanislaw Lem';
maxwell: {
&quot;database&quot;: &quot;test&quot;,
&quot;table&quot;: &quot;maxwell&quot;,
&quot;type&quot;: &quot;insert&quot;,
&quot;ts&quot;: 1449786310,
&quot;xid&quot;: 940753,
&quot;commit&quot;: true,
&quot;data&quot;: { &quot;id&quot;:1, &quot;daemon&quot;: &quot;Stanlislaw Lem&quot; }
&quot;data&quot;: { &quot;id&quot;:1, &quot;daemon&quot;: &quot;Stanislaw Lem&quot; }
}
</code></pre>

Expand All @@ -203,7 +203,7 @@ <h2>Maxwell = Mysql + Kafka</h2>
&quot;xid&quot;: 940786,
&quot;commit&quot;: true,
&quot;data&quot;: {&quot;id&quot;:1, &quot;daemon&quot;: &quot;Firebus! Firebus!&quot;},
&quot;old&quot;: {&quot;daemon&quot;: &quot;Stanlislaw Lem&quot;}
&quot;old&quot;: {&quot;daemon&quot;: &quot;Stanislaw Lem&quot;}
}
</code></pre>

Expand Down
Loading

0 comments on commit 9cecfe6

Please sign in to comment.