feat: update all fixtures and custom parsers to match (#713)

* feat: Refactor and update fixtures This patch changes how fixtures are stored. Previously, a fixture's folder identified its domain and its filename identified when it was fetched. This has been changed so that the filename indicates the domain and the modified time of the file indicates how recently it was fetched. A fixture's filename can optionally include a modifier to distinguish between two different page types on the same domain, for example. Also included here are changes to the update-fixture script, both to accomodate the new filename scheme as well as to actually update all fixtures. The functionality for running automatically and opening PRs has been removed but will likely be reintroduced. Finally, all fixtures have been updated. * Remove reference to deleted extractor * feat: first batch of test and parser updates due to new fixtures * feat: update more custom parsers and unit tests * feat: update more custom parsers and unit tests and remove unnecessary parser * feat: update more custom parsers and unit tests * feat: update more parsers and add correct bloomberg html files * fix: remove console statement * feat: all parsers updated and tests passing * fix: update date_published tests to account for test server time difference * fix: cleanup remaining fixtures in folders * feat: move fixtures for newest custom parsers * feat: remove script changes * fix: update dist files to account for reverting script changes * adding .DS_Store to .gitignore * adding .DS_Store to .gitignore -- 2 * adding .DS_Store to .gitignore -- 3 lol * cleaning up some tests * fix: ran build:generator command to update generate-custom-parser dist file * fix: update rollup configs to generate source maps and update source maps * fix: use underscore in place of unused error variable * fix: remove unused fixture Co-authored-by: Postlight Bot <[email protected]> Co-authored-by: flbn <[email protected]>
jocmp · Dec 13, 2022 · c0364ec · c0364ec
1 parent 7b68bcd
commit c0364ec
Show file tree

Hide file tree

Showing 547 changed files with 248,724 additions and 41,727 deletions.
diff --git a/.gitignore b/.gitignore
@@ -13,3 +13,4 @@ tmp/artifacts
 test-output.json
 .tool-versions
 .yarnrc.yml
+**/.DS_Store
diff --git a/dist/generate-custom-parser.js b/dist/generate-custom-parser.js
diff --git a/dist/generate-custom-parser.js.map b/dist/generate-custom-parser.js.map
diff --git a/dist/mercury.esm.js b/dist/mercury.esm.js
diff --git a/dist/mercury.esm.js.map b/dist/mercury.esm.js.map
diff --git a/dist/mercury.js b/dist/mercury.js
diff --git a/dist/mercury.js.map b/dist/mercury.js.map
diff --git a/dist/mercury.web.js b/dist/mercury.web.js
diff --git a/dist/mercury.web.js.map b/dist/mercury.web.js.map
diff --git a/fixtures/247sports.com.html b/fixtures/247sports.com.html
diff --git a/fixtures/247sports.com/1481309665090.html b/fixtures/247sports.com/1481309665090.html
diff --git a/fixtures/abcnews.go.com.html b/fixtures/abcnews.go.com.html
diff --git a/fixtures/abcnews.go.com/1481922563840.html b/fixtures/abcnews.go.com/1481922563840.html
diff --git a/fixtures/ars.html b/fixtures/ars.html
diff --git a/fixtures/arstechnica.com/1587927767738.html → fixtures/arstechnica.com.html b/fixtures/arstechnica.com/1587927767738.html → fixtures/arstechnica.com.html
diff --git a/fixtures/arstechnica.com/1587929444000.html b/fixtures/arstechnica.com/1587929444000.html
diff --git a/fixtures/biorxiv.org/1565356858754.html → fixtures/biorxiv.org.html b/fixtures/biorxiv.org/1565356858754.html → fixtures/biorxiv.org.html
diff --git a/fixtures/blisterreview.com.html b/fixtures/blisterreview.com.html
diff --git a/fixtures/blisterreview.com/1551128228532.html b/fixtures/blisterreview.com/1551128228532.html
diff --git a/fixtures/bookwalker.jp.html b/fixtures/bookwalker.jp.html
diff --git a/fixtures/bookwalker.jp/1554821842273.html b/fixtures/bookwalker.jp/1554821842273.html
diff --git a/fixtures/buzzap.jp/1551497136221.html → fixtures/buzzap.jp.html b/fixtures/buzzap.jp/1551497136221.html → fixtures/buzzap.jp.html
diff --git a/fixtures/clinicaltrials.gov.html b/fixtures/clinicaltrials.gov.html
diff --git a/fixtures/clinicaltrials.gov/1551708504719.html b/fixtures/clinicaltrials.gov/1551708504719.html
diff --git a/fixtures/deadline.com/1556104756617.html → fixtures/deadline.com.html b/fixtures/deadline.com/1556104756617.html → fixtures/deadline.com.html
diff --git a/fixtures/deadspin.com--video.html b/fixtures/deadspin.com--video.html
diff --git a/fixtures/deadspin.com.html b/fixtures/deadspin.com.html
diff --git a/fixtures/deadspin.com/1476389931786.html b/fixtures/deadspin.com/1476389931786.html
diff --git a/fixtures/deadspin.com/1477505848605.html b/fixtures/deadspin.com/1477505848605.html
diff --git a/fixtures/epaper.zeit.de/1566927390034.html → fixtures/epaper.zeit.de.html b/fixtures/epaper.zeit.de/1566927390034.html → fixtures/epaper.zeit.de.html
@@ -11,7 +11,7 @@
     <div class="container">
         <div class="row">
             <div class="col-xs-12">
-                
+
 <div class="article">
     <div class="article__author">
         Politik &#xB7; Jochen Bittner, Tina Hildebrandt    </div>
@@ -28,4 +28,4 @@
         </div>
     </div>
 </body>
-</html>
+</html>
diff --git a/fixtures/fandom.wikia.com/1475595373938.html → fixtures/fandom.wikia.com.html b/fixtures/fandom.wikia.com/1475595373938.html → fixtures/fandom.wikia.com.html
diff --git a/fixtures/fortune.com/1485216994169.html → fixtures/fortune.com.html b/fixtures/fortune.com/1485216994169.html → fixtures/fortune.com.html
diff --git a/fixtures/forward.com.html b/fixtures/forward.com.html
diff --git a/fixtures/forward.com/1488392273490.html b/fixtures/forward.com/1488392273490.html
diff --git a/fixtures/fusion.net/1482529202024.html b/fixtures/fusion.net/1482529202024.html
diff --git a/fixtures/genius.com/1550609084053.html → fixtures/genius.com.html b/fixtures/genius.com/1550609084053.html → fixtures/genius.com.html
diff --git a/fixtures/getnews.jp.html b/fixtures/getnews.jp.html
diff --git a/fixtures/getnews.jp/1555846693544.html b/fixtures/getnews.jp/1555846693544.html
diff --git a/fixtures/github.com.html b/fixtures/github.com.html
diff --git a/fixtures/github.com/1551742565097.html b/fixtures/github.com/1551742565097.html
diff --git a/fixtures/gothamist.com.html b/fixtures/gothamist.com.html
diff --git a/fixtures/gothamist.com/1489073770258.html b/fixtures/gothamist.com/1489073770258.html
diff --git a/fixtures/hellogiggles.com.html b/fixtures/hellogiggles.com.html
diff --git a/fixtures/hellogiggles.com/1482437663500.html b/fixtures/hellogiggles.com/1482437663500.html
diff --git a/fixtures/ici.radio-canada.ca.html b/fixtures/ici.radio-canada.ca.html
diff --git a/fixtures/ici.radio-canada.ca/1489433621634.html b/fixtures/ici.radio-canada.ca/1489433621634.html
diff --git a/fixtures/japan.cnet.com.html b/fixtures/japan.cnet.com.html
diff --git a/fixtures/japan.cnet.com/1555118717655.html b/fixtures/japan.cnet.com/1555118717655.html
diff --git a/fixtures/japan.zdnet.com/1556591023244.html → fixtures/japan.zdnet.com.html b/fixtures/japan.zdnet.com/1556591023244.html → fixtures/japan.zdnet.com.html
diff --git a/fixtures/jvndb.jvn.jp.html b/fixtures/jvndb.jvn.jp.html
@@ -0,0 +1,236 @@
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="ja" lang="ja" style="overflow-x:hidden" class=""><head>
+  <meta http-equiv="Content-Type" content="text/html; charset=Shift_JIS">
+  <title>JVNDB-2018-013542 - JVN iPedia - 脆弱性対策情報データベース</title>
+
+  <meta name="Keywords" content="脆弱性対策情報データベース">
+  <meta name="Description" content="脆弱性対策情報データベース">
+  <meta http-equiv="X-UA-Compatible" content="IE=edge, requiresActiveX=true">
+  <meta http-equiv="Content-Script-Type" content="text/javascript">
+  <meta http-equiv="Content-Style-Type" content="text/css">
+
+  <link href="/common/css/master.css" rel="stylesheet" type="text/css" media="screen,print">
+  <link media="all" href="/common/css/provide.css" type="text/css" rel="stylesheet">
+  <link href="/common/css/font.css" rel="stylesheet" type="text/css" media="screen,print">
+  <link href="/common/css/print.css" rel="stylesheet" type="text/css" media="print">
+  <link href="/common/css/rightmenu.css" rel="stylesheet" type="text/css" media="screen,print">
+
+<link type="text/css" rel="stylesheet" charset="UTF-8" href="https://translate.googleapis.com/translate_static/css/translateelement.css"></head>
+<body>
+<a name="top" id="top"></a>
+<!-- ↓ ヘッダ表示 ↓ -->
+<div id="wrapper">
+
+  <!-- header begin -->
+  <div id="header">
+      <div class="logo"><a href="/"><img height="60" alt="JVN iPedia - 脆弱性対策情報データベース" src="/common/img/logo.gif" width="490" border="0"></a></div>
+      <div class="modifytxt">最終更新日：2019/02/22</div>
+      <div class="modifytxt"></div>
+  </div>
+  <div id="head-bar">
+      <a href="/nav/guide_sysadm.html">【活用ガイド】</a>
+  </div>
+  <!-- header end -->
+
+  <!-- mainarea begin -->
+  <div id="mainarea">
+      <!-- leftarea begin -->
+      <div id="leftarea">
+          <div class="contents">
+              <div id="news-list">
+<br>
+<table class="vuln_table_clase">
+
+<tbody><tr><td colspan="2">
+<h2><font face="arial, geneva, helvetica">JVNDB-2018-013542</font></h2>
+</td></tr>
+
+<tr><td colspan="2">
+<h2>NETWAVE&nbsp;MNG6200&nbsp;デバイスにおける証明書・パスワードの管理に関する脆弱性</h2>
+</td></tr>
+
+<tr><td class="vuln_table_clase_td_header" colspan="2">
+<a name="overview">概要</a>
+</td></tr>
+<tr><td colspan="2">
+<br>
+<blockquote>
+NETWAVE MNG6200 デバイスには、証明書・パスワードの管理に関する脆弱性が存在します。
+</blockquote>
+</td></tr>
+<tr><td class="vuln_table_clase_td_header" colspan="2">
+<a name="cvss">CVSS による深刻度</a> (<a href="/nav/jvndbhelp.html#jvndb3" target="_blank">CVSS とは?</a>)
+</td></tr>
+<tr><td colspan="2">
+<br>
+<div style="margin: 20px 40px;">
+<div class="clearfix">
+<div class="float_left">
+<b>CVSS v3 による深刻度<br>基本値: <a href="/cvss/ja/v3.html#CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H" target="_blank">9.8</a> (緊急) [NVD値]</b>
+<ul>
+  <li>攻撃元区分: ネットワーク</li>
+  <li>攻撃条件の複雑さ: 低</li>
+  <li>攻撃に必要な特権レベル: 不要</li>
+  <li>利用者の関与: 不要</li>
+  <li>影響の想定範囲: 変更なし</li>
+  <li>機密性への影響(C): 高</li>
+  <li>完全性への影響(I): 高</li>
+  <li>可用性への影響(A): 高</li>
+</ul>
+</div>
+<div class="float_left">
+<b>CVSS v2 による深刻度<br>基本値: <a href="/cvss/ja/v2.html#AV:N/AC:L/Au:N/C:P/I:N/A:N" target="_blank">5.0</a> (警告) [NVD値]</b>
+<ul>
+  <li>攻撃元区分: ネットワーク</li>
+  <li>攻撃条件の複雑さ: 低</li>
+  <li>攻撃前の認証要否: 不要</li>
+  <li>機密性への影響(C): 部分的</li>
+  <li>完全性への影響(I): なし</li>
+  <li>可用性への影響(A): なし</li>
+</ul>
+</div>
+</div>
+<div style="margin-top:20px">
+
+</div>
+</div>
+</td></tr>
+
+<tr><td class="vuln_table_clase_td_header" colspan="2">
+<a name="affect">影響を受けるシステム</a>
+</td></tr>
+
+<tr><td colspan="2">
+<br>
+<blockquote>
+
+</blockquote>
+</td></tr>
+
+<tr><td colspan="2">
+<br>
+<blockquote>
+NETWAVE&nbsp;Networks,&nbsp;Inc.
+<ul>
+  <li>MNG6200&nbsp;ファームウェア C4835805jrc12FU121413.cpr</li>
+  </ul>
+</blockquote>
+</td></tr>
+
+<tr><td colspan="2">
+<br>
+<blockquote>
+
+</blockquote>
+</td></tr>
+
+<tr><td class="vuln_table_clase_td_header" colspan="2">
+<a name="impact">想定される影響</a>
+</td></tr>
+
+<tr><td colspan="2">
+<br>
+<blockquote>
+情報を取得される、情報を改ざんされる、およびサービス運用妨害 (DoS) 状態にされる可能性があります。
+</blockquote>
+</td></tr>
+
+<tr><td class="vuln_table_clase_td_header" colspan="2">
+<a name="solution">対策</a>
+</td></tr>
+
+<tr><td colspan="2">
+<br>
+<blockquote>
+参考情報を参照して適切な対策を実施してください。
+</blockquote>
+</td></tr>
+
+<tr><td class="vuln_table_clase_td_header" colspan="2">
+<a name="venderinf">ベンダ情報</a>
+</td></tr>
+
+<tr><td colspan="2">
+<br>
+<blockquote>
+</blockquote>
+</td></tr>
+
+
+<tr><td class="vuln_table_clase_td_header" colspan="2">
+<a name="cwe">CWEによる脆弱性タイプ一覧</a>&nbsp;&nbsp;<a href="http://www.ipa.go.jp/security/vuln/CWE.html" target="_blank">CWEとは?</a>
+</td></tr>
+
+
+<tr><td colspan="2">
+<br>
+<blockquote>
+<ol>
+<li><a href="https://jvndb.jvn.jp/ja/cwe/CWE-255.html" target="_blank">証明書・パスワードの管理(CWE-255)</a> [NVD評価]</li>
+</ol>
+</blockquote>
+</td></tr>
+
+
+<tr><td class="vuln_table_clase_td_header" colspan="2">
+<a name="cve">共通脆弱性識別子(CVE)</a>&nbsp;&nbsp;<a href="http://www.ipa.go.jp/security/vuln/CVE.html" target="_blank">CVEとは?</a>
+</td></tr>
+
+<tr><td colspan="2">
+<br>
+<blockquote>
+<ol>
+<li><a href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-20395" target="_blank">CVE-2018-20395</a></li>
+</ol>
+</blockquote>
+</td></tr>
+
+
+<tr><td class="vuln_table_clase_td_header" colspan="2">
+<a name="reference">参考情報</a>
+</td></tr>
+
+<tr><td colspan="2">
+<br>
+<blockquote>
+<ol>
+<li>National&nbsp;Vulnerability&nbsp;Database&nbsp;(NVD) : <a href="https://nvd.nist.gov/vuln/detail/CVE-2018-20395" target="_blank">CVE-2018-20395</a> </li>
+<li>関連文書 : <a href="https://github.com/ezelf/sensitivesOids/blob/master/oidpassswordleaks.csv" target="_blank">sensitivesOids/oidpassswordleaks.csv</a> </li>
+</ol>
+</blockquote>
+</td></tr>
+
+<tr><td class="vuln_table_clase_td_header" colspan="2">
+<a name="history">更新履歴</a>
+</td></tr>
+<tr><td colspan="2">
+<br>
+<blockquote>
+<ul style="list-style-type: none">
+<li>[2019年02月22日]<br>
+&nbsp;&nbsp;掲載</li>
+</ul>
+</blockquote>
+<br>
+</td></tr>
+</tbody></table>
+<table class="vuln_table_clase_footer">
+<tbody><tr><td colspan="2"><hr></td></tr>
+<tr><td class="vuln_table_clase_date_header_td">公表日</td><td>2018/12/23</td></tr>
+<tr><td class="vuln_table_clase_date_header_td">登録日</td><td>2019/02/22</td></tr>
+<tr><td class="vuln_table_clase_date_header_td">最終更新日</td><td>2019/02/22</td></tr>
+<tr><td colspan="2"><hr></td></tr>
+</tbody></table>
+<!-- 
+</td></tr></table>
+</body>
+</html>
+-->                </div>
+          </div>
+      </div>
+  </div>
+  <div id="footer">
+      <div class="footertext"><img src="/common/img/copyright.png" alt="Copyright © 2007- IPA. All rights reserved."></div>
+  </div>
+</div><div id="goog-gt-tt" class="skiptranslate" dir="ltr"><div style="padding: 8px;"><div><div class="logo"><img src="https://www.gstatic.com/images/branding/product/1x/translate_24dp.png" width="20" height="20" alt="Google Translate"></div></div></div><div class="top" style="padding: 8px; float: left; width: 100%;"><h1 class="title gray">Original text</h1></div><div class="middle" style="padding: 8px;"><div class="original-text"></div></div><div class="bottom" style="padding: 8px;"><div class="activity-links"><span class="activity-link">Contribute a better translation</span><span class="activity-link"></span></div><div class="started-activity-container"><hr style="color: #CCC; background-color: #CCC; height: 1px; border: none;"><div class="activity-root"></div></div></div><div class="status-message" style="display: none;"></div></div>
+
+<div class="goog-te-spinner-pos"><div class="goog-te-spinner-animation"><svg xmlns="http://www.w3.org/2000/svg" class="goog-te-spinner" width="96px" height="96px" viewBox="0 0 66 66"><circle class="goog-te-spinner-path" fill="none" stroke-width="6" stroke-linecap="round" cx="33" cy="33" r="30"></circle></svg></div></div></body></html>