Merge pull request #31 from coreruleset/add-broken-links-check

feat: add broken link checker

.github/workflows/gh-pages.yml

@@ -62,11 +62,11 @@ jobs:
           HUGO_ENVIRONMENT: production
           HUGO_ENV: production
         run: |
-          hugo --gc --minify \
-            --baseURL "${{ steps.pages.outputs.base_url }}/"
+          hugo --gc --minify --baseURL "${{ steps.pages.outputs.base_url }}/"
           hugo -s subsite/docs --baseURL "${{ steps.pages.outputs.base_url }}/docs/"
           mv subsite/docs/public public/docs
           npx -y pagefind --site public
+
       - name: Upload artifact
         uses: actions/upload-pages-artifact@v3
         with:

.github/workflows/test.yml

@@ -27,11 +27,16 @@ jobs:
           && sudo dpkg -i ${{ runner.temp }}/hugo.deb
       - name: Install Dart Sass
         run: sudo snap install dart-sass
+
       - name: Checkout
         uses: actions/checkout@v4
         with:
           submodules: recursive
 
+      - name: Setup Pages
+        id: pages
+        uses: actions/configure-pages@v5
+
       - name: Install Node.js dependencies
         run: "[[ -f package-lock.json || -f npm-shrinkwrap.json ]] && npm ci || true"
 
@@ -41,7 +46,25 @@ jobs:
           HUGO_ENVIRONMENT: testing
           HUGO_ENV: testing
         run: |
-          hugo \
-            --minify \
-            --baseURL "${{ steps.pages.outputs.base_url }}/"
+          hugo --gc --minify --baseURL "${{ steps.pages.outputs.base_url }}/"
+          hugo -s subsite/docs --baseURL "${{ steps.pages.outputs.base_url }}/docs/"
+          mv subsite/docs/public public/docs
+          npx -y pagefind --site public
+
+      - name: Link Checker
+        id: lychee
+        uses: lycheeverse/lychee-action@22134d37a1fff6c2974df9c92a7c7e1e86a08f9c # for v1.9.0
+        with:
+          args: --no-progress -u "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0" -E './content/**/*.md'
+          jobSummary: true
+
+              #      - name: Create Issue From File
+              #if: env.lychee_exit_code != 0
+              #uses: peter-evans/create-issue-from-file@v5
+              #permissions:
+              #issues: write
+              #with:
+              #title: Link Checker Report
+              #content-filepath: ./lychee/out.md
+              #labels: report, automated issue
 

.lycheeignore

@@ -0,0 +1,3 @@
+piwik.netnea.com
+github.com/*
+trustwave.com

config/_default/params.yaml

@@ -12,7 +12,7 @@ custom_css: []
 images: ["/img/social-share.png"]
 
 github:
-  repo_url: "https://github.com/coreruleset/website/blob/master"
+  repo_url: "https://github.com/coreruleset/website/blob/main"
   show_edit_link: true
 
 social_links:
@@ -23,7 +23,7 @@ social_links:
   github: "https://github.com/coreruleset"
   instagram: ""
   landscape: ""
-  linkedin: "https://www.linkedin.com/company/owasp-coreruleset"
+  linkedin: "https://www.linkedin.com/company/owasp-crs"
   mastodon: ""
   pinterest: ""
   rss: ""

content/blog/2017-08-10-testing-wafs-ftw-version-1-0-released.md

@@ -8,11 +8,11 @@ url: /2017/08/10/testing-wafs-ftw-version-1-0-released/
 ---
 
 
-<span style="font-weight: 400;">The OWASP Project maintains an open source set of rules known as the the OWASP Core Rule Set (CRS). The CRS implements protections for the well known, broad classes of web application vulnerabilities identified by OWASP. Over time, this set of rules has become the most popular ruleset for ModSecurity and also found its way into many other popular WAFs.</span>
+The OWASP Project maintains an open source set of rules known as the the OWASP Core Rule Set (CRS). The CRS implements protections for the well known, broad classes of web application vulnerabilities identified by OWASP. Over time, this set of rules has become the most popular ruleset for ModSecurity and also found its way into many other popular WAFs.
 
-<span style="font-weight: 400;">During this same timeframe we have seen Quality Assurance (QA)/DevOps techniques adjust to new Agile development methodologies. To a large extent this Agile pattern matches the historical development practices of CRS. As a result, during the development of the latest CRS version 3.0, the development team decided that a serious overhaul of the regression/unit tests was overdue. While some existing Perl regression tests existed, these were incomplete and considered difficult for the average user to run. The CRS development team also concluded that a more refined testing methodology commits to a higher quality product and allows for a demonstration of the effectiveness of OWASP CRS compared to many other rule sets and WAFs.</span>
+During this same timeframe we have seen Quality Assurance (QA)/DevOps techniques adjust to new Agile development methodologies. To a large extent this Agile pattern matches the historical development practices of CRS. As a result, during the development of the latest CRS version 3.0, the development team decided that a serious overhaul of the regression/unit tests was overdue. While some existing Perl regression tests existed, these were incomplete and considered difficult for the average user to run. The CRS development team also concluded that a more refined testing methodology commits to a higher quality product and allows for a demonstration of the effectiveness of OWASP CRS compared to many other rule sets and WAFs.
 
-![](/images/2017/08/FTW1-282x300.png)
+{{< figure src="images/2017/08/FTW1.png" >}}
 
 As a result of extensive regression test development we are hoping to address a frequent user request to provide a capability to compare the effectiveness of various WAFs. Such comparisons can be tricky as they often attempt to compare varying features. In many situations OWASP CRS comes out favorably as can be seen in the latest Gartner report ([https://www.gartner.com/doc/reprints?id=1-3C4V1AS&amp;ct=160721&amp;st=sb](https://www.gartner.com/doc/reprints?id=1-3C4V1AS&ct=160721&st=sb)), where ModSecurity with CRS effectively acts as a baseline. However comparisons such as these have historically put very little work on testing overall WAF effectiveness. Our hope is to provide a set of tests that will act as a minimum benchmark between WAFs.
 
@@ -30,11 +30,11 @@ To accomplish this goal we started work on a project dubbed the Framework for Te
 - Provide modular support for logging endpoints
 - Build a continuous integration platform to insert security and regression testing behind deploying a WAF
 
-As the first milestone in this project, we are proud to announce the release of FTW version 1.0. This project is available at <https://github.com/CRS-support/ftw> or via pypi. The framework is written in Python, leveraging the existing pytest framework which many developers will be familiar with. It will accept tests in either YAML format or via its programmatic interface and is designed to be modular enough to support multiple WAFs and complicated multi-stage requests.
+As the first milestone in this project, we are proud to announce the release of FTW version 1.0. This project is available at <https://github.com/coreruleset/ftw> or via pypi. The framework is written in Python, leveraging the existing pytest framework which many developers will be familiar with. It will accept tests in either YAML format or via its programmatic interface and is designed to be modular enough to support multiple WAFs and complicated multi-stage requests.
 
-To aid the developer, tests are designed to only require minimal effort to design, where many defaults are provided automatically in a similar manner to how the Scapy project generates packets. Below we provide an example of a simple test file. Of course, more complicated examples involving aspects such as storing cookies and submitting multipart forms are also possible as can be seen in the documentation (<https://github.com/CRS-support/ftw/tree/master/docs>).
+To aid the developer, tests are designed to only require minimal effort to design, where many defaults are provided automatically in a similar manner to how the Scapy project generates packets. Below we provide an example of a simple test file. Of course, more complicated examples involving aspects such as storing cookies and submitting multipart forms are also possible as can be seen in the documentation (<https://github.com/coreruleset/ftw/tree/master/docs>).
 
-```
+```yaml
 ---
   meta:
     author: "csanders-git"
@@ -59,6 +59,6 @@ To aid the developer, tests are designed to only require minimal effort to desig
 
 FTW follows the same extendible concept as ModSecurity. While the project provides the capability to develop extremely flexible web based testing, the core project is provided with only limited tests files itself. The CRS team provides a continuously expanding corpus of tests designed for OWASP CRS 3 within their repo at: <https://github.com/coreruleset/coreruleset/tree/v3.0/dev/util/regression-tests>. While these tests were designed with CRS in mind, they provide a set of web-based attacks to test security features of any WAF against the OWASP CRS Top 10 web attacks. Such testing has already uncovered several underperforming rules such as can be seen in Github issue #480 (<https://github.com/coreruleset/coreruleset/pull/480>), as well as providing a methodical way to develop and test more complex functionality, such as the revamped RCE rules in CRS 3 (<https://github.com/coreruleset/coreruleset/pull/430>).
 
-![](/images/2017/08/FTW2-287x300.png)
+{{< figure src="images/2017/08/FTW2.png" >}}
 
-<span style="font-weight: 400;">At this point the CRS regressions has over 1500 test cases designed for it and this number is growing daily. To utilize such extensive tests we plan to enforce the use of Travis-CI (</span>[<span style="font-weight: 400;">https://travis-ci.org/</span>](https://travis-ci.org/)<span style="font-weight: 400;">) starting with the promotion of OWASP CRS 3.0 to the master branch. It is our sincere hope that an increased reliance on testing and automation will vastly increase the quality of both the CRS ruleset and WAFs as a whole.</span>
+At this point the CRS regressions has over 1500 test cases designed for it and this number is growing daily. To utilize such extensive tests we plan to enforce the use of [Travis-CI](https://travis-ci.org/) starting with the promotion of OWASP CRS 3.0 to the master branch. It is our sincere hope that an increased reliance on testing and automation will vastly increase the quality of both the CRS ruleset and WAFs as a whole.

content/blog/2017-08-15-crs-project-news-august-2017.md

@@ -42,5 +42,3 @@ This is the CRS newsletter covering the period from July until today.
 - There is now a plan to run a real poll where CRS users can vote on feature requests. There are a ton of feature requests recorded on github, but we really are a bit at a loss on what people are really interested in. Stay tuned to learn more about this.
 
 I have been on a holiday for two weeks and it is likely, I overlooked things on the mailinglists and on github. Feel free to speak up and respond to this message highlighting the omissions.
-
-{{< figure src="images/2017/08/christian-folini-2017-450x450.png" width="100px" caption="Christian Folini / [@ChrFolini](https://twitter.com/ChrFolini)" >}}

content/blog/2017-09-13-how-you-can-help-the-crs-project.md

@@ -41,5 +41,3 @@ An important thing for our community is the [monthly project chat on IRC](https:
 So that is an overview where we could use some help. I am sure there will be something that fits your interest.
 
 **\[EDIT\]: Very useful comment below from Walter Hop. In fact I thought the same thing when I read through several github comments on the weekend.**
-
-{{< figure src="images/2017/08/christian-folini-2017-450x450.png" width="100px" caption="Christian Folini / [@ChrFolini](https://twitter.com/ChrFolini)" >}}

content/blog/2017-10-03-crs-project-nominated-for-swiss-dinacon-award.md

@@ -21,22 +21,20 @@ The Swiss Open Source Awards have played a key role in this development. For 201
 
 This is an abbreviation of "Digitale Nachhaltigkeit Conference", which can be translated as the conference on digital sustainability. This means, the Open Source idea has been opened up to a wider group of projects and initiatives that go behind pure software: open data, open access and sustainability of digital projects in general are now also covered.
 
-{{< figure src="/images/2017/10/dinacon-nomination.png" caption="The Nomination of the CRS Project">}}
+{{< figure src="images/2017/10/dinacon-nomination.png" caption="The Nomination of the CRS Project">}}
 
 But how does this apply to the [Core Rule Set](https://coreruleset.org) project?
 
 The point is that the DINACon represents the same qualities that the OWASP ModSecurity Core Rule Set project covers: Quality, transparency and a sustainable code base that is maintained by a group of specialists passionate about the project and security in general. In our particular case, two of these specialists, Fränzi Bühler / [@bufrasch](https://twitter.com/bufrasch) and I happen to live in Switzerland and we hope to attract more Swiss developers into the project.
 
-CRS has been a one-man show for many years. In early 2016, [Chaim Sanders](http://chaimsanders.com/) took over the project and invited [Walter Hop](https://lifeforms.nl/) and I to join him. Right now, we have 8 to 10 regular committers on the project and I feel that we have real momentum now. Following all the [discussions on Github](https://github.com/coreruleset/coreruleset) is seriously challenging and there are so many ideas floating around. It's true, the development of the project is very interesting.
+CRS has been a one-man show for many years. In early 2016, [Chaim Sanders](https://medium.com/@chaim_sanders) took over the project and invited [Walter Hop](https://lifeforms.nl/) and I to join him. Right now, we have 8 to 10 regular committers on the project and I feel that we have real momentum now. Following all the [discussions on Github](https://github.com/coreruleset/coreruleset) is seriously challenging and there are so many ideas floating around. It's true, the development of the project is very interesting.
 
 This nomination comes at exactly the right moment and means an additional push for our project. It comes at a moment when we are actively working on the sustainability and the transparency of our rule base. Traditionally, the rules of the Core Rule Set have been very hard to read. Yet, new initiatives like the rules cleanup project are changing the situation and lately, Franziska Bühler has committed the [pull request](https://github.com/coreruleset/coreruleset/pull/907) that disassembles all the incomprehensible regular expressions and makes them reproducible and understandable. That work is key and if you have looked at the regular expressions that we leverage in the rule set, you understand why we are in awe of her work.
 
-{{< figure src="/images/2017/10/crs-pr-907.png" caption="The PR by Fränzi Bühler passes all the tests." >}}
+{{< figure src="images/2017/10/crs-pr-907.png" caption="The PR by Fränzi Bühler passes all the tests." >}}
 
 As I said, it is a tough market in Switzerland for Open Source projects and especially when it comes to webserver security. The commercial products all have a very high TCO, either via high license costs or integration and support contracts. Yet the commercial players are all well established and Open Source alternatives like ModSecurity and the Core Rule Set have a hard time finding their way on webservers around here and probably worldwide. But we need to spread the word that there is a transparent and highly secure open source alternative to commercial black boxes. Smaller companies, public administrations and organisations on a tight budget need to know they can get the best tools on the market without spending big money: ModSecurity and the Core Rule Set are at their disposal serving as the 1st line of defense against web application attacks like those covered by the [OWASP Top 10](https://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project).
 
 Awards like the DINACon can help us get this message across.
 
 It's great to be nominated for DINACon and I really hope we can get this trophy!
-
-{{< figure src="images/2017/08/christian-folini-2017-450x450.png" width="100px" caption="Christian Folini / [@ChrFolini](https://twitter.com/ChrFolini)" >}}

content/blog/2017-11-07-crs-project-news-november.md

@@ -39,7 +39,7 @@ During the chat we discussed the following
     - Need to check balance but Dune73 will finance privately changes.
     - Shooting for by AppSecEU
     - Idea to start to sell the release poster via a printing service like [Redbubble](https://www.redbubble.com/)
-- Info: CRS nominated for the German Open Source Business award ([http://osbar.it](http://osbar.it/)) 
+- Info: CRS nominated for the German Open Source Business award ([https://osbar.it](http://osbar.it/)) 
     - Everyone is excited thank you to Dune73 for nominating us
 - Plans for new blog posts 
     - Franbuehler writing up about SQL disassembly
@@ -78,7 +78,7 @@ The next community chats will be held on the following dates:
 
 Upcoming talks and talks that were just posted
 
-- ModSec 3.0 talk on B-sides Toronto next week: <http://www.bsidesto.ca/abstracts.html#6>
+- ModSec 3.0 talk on B-sides Toronto next week: <https://github.com/bsidesto/bsidesto.github.io/blob/5fc5ef3169e72b0ec42e959ff2caed331acd5ac2/2017/Presentations2017/BSidesTO2017-libModSecurity.pdf>
 - BlackHat Europe Arsenal demo for libModSecurity in December: <https://www.blackhat.com/eu-17/arsenal/schedule/#modsecurity-300-9079>
 - WAFs FTW! A modern devops approach to security testing: [https://www.youtube.com/watch?v=05Uy0R7UdFw ](https://www.youtube.com/watch?v=05Uy0R7UdFw)
 - Introducing the OWASP ModSEcurity Core Rule Set (CRS) 3.0: <https://www.youtube.com/watch?v=oCxW966128A>