Merge pull request #8 from anjackson/master

Shift to Jupyter Book

.dockerignore

@@ -0,0 +1,2 @@
+*.jsonl
+*.csv

.github/workflows/push-to-docker-hub.yml

@@ -0,0 +1,19 @@
+name: Build, scan and push to Docker Hub
+
+on:
+  push:
+    tags:
+      - '*'
+    branches:
+      - master
+
+
+jobs:
+  run_docker_build_workflow:
+    uses: ukwa/ukwa-services/.github/workflows/push-to-docker-hub.yml@master
+    secrets:
+      DOCKER_HUB_USERNAME: ${{ secrets.DOCKER_HUB_USERNAME }}
+      DOCKER_HUB_ACCESS_TOKEN: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }}
+
+
+

.gitignore

@@ -1,3 +1,7 @@
-public
-content/crawls
-data/crawls
+_build
+.ipynb_checkpoints
+__pycache__
+content/storage/*.jsonl
+content/storage/*.csv
+.Trash-*
+.python-version

.gitmodules

@@ -1,3 +0,0 @@
-[submodule "themes/minimal"]
-	path = themes/minimal
-	url = https://github.com/calintat/minimal.git

Dockerfile

@@ -1,13 +1,17 @@
-FROM klakegg/hugo:0.65.3 AS hugo
+FROM python:3.11
 
-COPY . /src
+RUN apt-get install -y libffi-dev
 
-WORKDIR /src
+WORKDIR /ukwa-reports
 
-ENV HUGO_DESTINATION=/onbuild
-
-RUN hugo
-
-FROM nginx
-COPY --from=hugo /onbuild /usr/share/nginx/html/intranet
+# Python dependencies and shared code:
+COPY setup.py .
+COPY ukwa_reports ./ukwa_reports
+RUN pip install --no-cache -v .
 
+# Jupyter Book work:
+COPY content .
+COPY build.sh .
+# Default action is to run the full build script to generate output at ./_build
+# Use volumes to map input (content) and/or output (_build)
+CMD ./build.sh

README.md

@@ -0,0 +1,9 @@
+UKWA Intranet
+=============
+
+[![Build Status](https://travis-ci.org/ukwa/ukwa-reports.svg?branch=master)](https://travis-ci.org/ukwa/ukwa-reports)
+[![Docker Hub](https://img.shields.io/badge/docker-ready-blue.svg)](https://registry.hub.docker.com/r/ukwa/ukwa-intranet/)
+
+This static website acts as a the gateway for our 'intranet' services. It's a static site built using [Hugo](https://gohugo.io/), and is deployed by being embedded in the [`ukwa-services/manage/intranet`](https://github.com/ukwa/ukwa-services/tree/master/manage/intranet) stack.
+
+See that project for more details.

build.sh

@@ -0,0 +1,13 @@
+#!/bin/sh
+
+# https://discourse.jupyter.org/t/debugger-warning-it-seems-that-frozen-modules-are-being-used-python-3-11-0/16544
+export PYDEVD_DISABLE_FILE_VALIDATION=1
+
+# Build the book part:
+jb build --path-output . content/
+
+# Copy over CSV files, retaining the full paths:
+echo Copying CSV files from the content folder to the _build: 
+cd content 
+find . -name "*.csv" -exec cp -v {} ../_build/html/{} \;
+cd -

content/_config.yml

@@ -0,0 +1,38 @@
+# Book settings
+# Learn more at https://jupyterbook.org/customize/config.html
+
+title: UKWA Technical Documentation
+author: The UK Web Archive
+logo: "assets/logos/ukwa-2018-onwhite-close.svg"
+
+# Don't include these:
+exclude_patterns: ['_build', '**.ipynb_checkpoints']
+
+# Auto-exclude files not in the toc
+only_build_toc_files: true
+
+# Force re-execution of notebooks on each build?
+# See https://jupyterbook.org/content/execute.html
+execute:
+  execute_notebooks: auto
+  # Long timeout because some analyses take a while:
+  timeout: 1000
+
+# Add a bibtex file so that we can create citations
+bibtex_bibfiles:
+  - references.bib
+
+# Information about where the book exists on the web
+repository:
+  url: https://github.com/ukwa/ukwa-reports  # Online location of your book
+  path_to_book: content  # Optional path to your book, relative to the repository root
+  branch: master  # Which branch of the repository should be used when creating links (optional)
+
+# Add GitHub buttons to your book
+# See https://jupyterbook.org/customize/config.html#add-a-link-to-your-repository
+html:
+  use_issues_button: true
+  use_repository_button: true
+  use_edit_page_button: true
+  home_page_in_navbar: false
+

content/_toc.yml

@@ -0,0 +1,17 @@
+# Table of contents
+# Learn more at https://jupyterbook.org/customize/toc.html
+
+format: jb-book
+root: intro
+parts:
+- caption: Reports
+  chapters:
+  - file: storage/summary
+  - file: storage/timeline
+  - file: storage/indexed
+  - file: storage/dls
+#- caption: Examples
+#  chapters:
+#  - file: markdown
+#  - file: notebooks
+#  - file: markdown-notebooks

content/intro.md

@@ -0,0 +1,6 @@
+# UKWA Reports
+
+This sub-section of the UKWA internal web site contains regularly re-generated reports.
+
+```{tableofcontents}
+```

content/markdown-notebooks.md

@@ -0,0 +1,53 @@
+---
+jupytext:
+  formats: md:myst
+  text_representation:
+    extension: .md
+    format_name: myst
+    format_version: 0.13
+    jupytext_version: 1.11.5
+kernelspec:
+  display_name: Python 3
+  language: python
+  name: python3
+---
+
+# Notebooks with MyST Markdown
+
+Jupyter Book also lets you write text-based notebooks using MyST Markdown.
+See [the Notebooks with MyST Markdown documentation](https://jupyterbook.org/file-types/myst-notebooks.html) for more detailed instructions.
+This page shows off a notebook written in MyST Markdown.
+
+## An example cell
+
+With MyST Markdown, you can define code cells with a directive like so:
+
+```{code-cell}
+print(2 + 2)
+```
+
+When your book is built, the contents of any `{code-cell}` blocks will be
+executed with your default Jupyter kernel, and their outputs will be displayed
+in-line with the rest of your content.
+
+```{seealso}
+Jupyter Book uses [Jupytext](https://jupytext.readthedocs.io/en/latest/) to convert text-based files to notebooks, and can support [many other text-based notebook files](https://jupyterbook.org/file-types/jupytext.html).
+```
+
+## Create a notebook with MyST Markdown
+
+MyST Markdown notebooks are defined by two things:
+
+1. YAML metadata that is needed to understand if / how it should convert text files to notebooks (including information about the kernel needed).
+   See the YAML at the top of this page for example.
+2. The presence of `{code-cell}` directives, which will be executed with your book.
+
+That's all that is needed to get started!
+
+## Quickly add YAML metadata for MyST Notebooks
+
+If you have a markdown file and you'd like to quickly add YAML metadata to it, so that Jupyter Book will treat it as a MyST Markdown Notebook, run the following command:
+
+```
+jupyter-book myst init path/to/markdownfile.md
+```

content/markdown.md

@@ -0,0 +1,55 @@
+# Markdown Files
+
+Whether you write your book's content in Jupyter Notebooks (`.ipynb`) or
+in regular markdown files (`.md`), you'll write in the same flavor of markdown
+called **MyST Markdown**.
+This is a simple file to help you get started and show off some syntax.
+
+## What is MyST?
+
+MyST stands for "Markedly Structured Text". It
+is a slight variation on a flavor of markdown called "CommonMark" markdown,
+with small syntax extensions to allow you to write **roles** and **directives**
+in the Sphinx ecosystem.
+
+For more about MyST, see [the MyST Markdown Overview](https://jupyterbook.org/content/myst.html).
+
+## Sample Roles and Directives
+
+Roles and directives are two of the most powerful tools in Jupyter Book. They
+are kind of like functions, but written in a markup language. They both
+serve a similar purpose, but **roles are written in one line**, whereas
+**directives span many lines**. They both accept different kinds of inputs,
+and what they do with those inputs depends on the specific role or directive
+that is being called.
+
+Here is a "note" directive:
+
+```{note}
+Here is a note
+```
+
+It will be rendered in a special box when you build your book.
+
+Here is an inline directive to refer to a document: {doc}`markdown-notebooks`.
+
+
+## Citations
+
+You can also cite references that are stored in a `bibtex` file. For example,
+the following syntax: `` {cite}`holdgraf_evidence_2014` `` will render like
+this: {cite}`holdgraf_evidence_2014`.
+
+Moreover, you can insert a bibliography into your page with this syntax:
+The `{bibliography}` directive must be used for all the `{cite}` roles to
+render properly.
+For example, if the references for your book are stored in `references.bib`,
+then the bibliography is inserted with:
+
+```{bibliography}
+```
+
+## Learn more
+
+This is just a simple starter to get you started.
+You can learn a lot more at [jupyterbook.org](https://jupyterbook.org).