pipeline-notebooks/pipeline-section.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9db9a678",
   "metadata": {
    "tags": [
     "execution_cell"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<!-- \n",
       "If you can see this code, this cell's output is not trusted.\n",
       "Please execute this cell and save the notebook, or click File -> Trust Notebook\n",
       "-->\n",
       "<script>\n",
       "var shown = true;\n",
       "\n",
       "function filter_cells_by_tag(tag) {\n",
       "    out = Array();\n",
       "    all_cells = Jupyter.notebook.get_cells()\n",
       "    for (var i=0; i<all_cells.length; i++) {\n",
       "        var curr_cell = all_cells[i];\n",
       "        var tags = curr_cell._metadata.tags;\n",
       "        if (tags != undefined) {\n",
       "            for (var j=0; j<tags.length; j++) {\n",
       "                var curr_tag = tags[j];\n",
       "                if (curr_tag == tag) {\n",
       "                    out.push(curr_cell);\n",
       "                    break;\n",
       "                }\n",
       "            }\n",
       "        }\n",
       "    }\n",
       "    return out;\n",
       "}\n",
       "\n",
       "function set_cell_visibility(tag, show, input_only) {\n",
       "    var cells = Jupyter.notebook.get_cells();\n",
       "    var marked_cells = filter_cells_by_tag(tag);\n",
       "    for (var i=0; i<marked_cells.length; i++) {\n",
       "        var curr_cell = marked_cells[i];\n",
       "        if (input_only) {\n",
       "            obj = curr_cell.input\n",
       "        } else {\n",
       "            obj = curr_cell.element\n",
       "        }\n",
       "        if (show) {\n",
       "            obj.show();\n",
       "        } else {\n",
       "            obj.hide();\n",
       "        }\n",
       "    }\n",
       "}\n",
       "\n",
       "function toggle_cell_visibility(tag) {\n",
       "    set_cell_visibility(tag, shown, false)\n",
       "    shown = ! shown;\n",
       "}\n",
       "\n",
       "set_cell_visibility('execution_cell', false, true);\n",
       "</script>\n",
       "To toggle visibility of explanation cells click <a href=\"javascript:toggle_cell_visibility('explanatory_cell')\">here</a>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<!-- \n",
    "If you can see this code, this cell's output is not trusted.\n",
    "Please execute this cell and save the notebook, or click File -> Trust Notebook\n",
    "-->\n",
    "<script>\n",
    "var shown = true;\n",
    "\n",
    "function filter_cells_by_tag(tag) {\n",
    "    out = Array();\n",
    "    all_cells = Jupyter.notebook.get_cells()\n",
    "    for (var i=0; i<all_cells.length; i++) {\n",
    "        var curr_cell = all_cells[i];\n",
    "        var tags = curr_cell._metadata.tags;\n",
    "        if (tags != undefined) {\n",
    "            for (var j=0; j<tags.length; j++) {\n",
    "                var curr_tag = tags[j];\n",
    "                if (curr_tag == tag) {\n",
    "                    out.push(curr_cell);\n",
    "                    break;\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "    return out;\n",
    "}\n",
    "\n",
    "function set_cell_visibility(tag, show, input_only) {\n",
    "    var cells = Jupyter.notebook.get_cells();\n",
    "    var marked_cells = filter_cells_by_tag(tag);\n",
    "    for (var i=0; i<marked_cells.length; i++) {\n",
    "        var curr_cell = marked_cells[i];\n",
    "        if (input_only) {\n",
    "            obj = curr_cell.input\n",
    "        } else {\n",
    "            obj = curr_cell.element\n",
    "        }\n",
    "        if (show) {\n",
    "            obj.show();\n",
    "        } else {\n",
    "            obj.hide();\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "function toggle_cell_visibility(tag) {\n",
    "    set_cell_visibility(tag, shown, false)\n",
    "    shown = ! shown;\n",
    "}\n",
    "\n",
    "set_cell_visibility('execution_cell', false, true);\n",
    "</script>\n",
    "To toggle visibility of explanation cells click <a href=\"javascript:toggle_cell_visibility('explanatory_cell')\">here</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b20fc36",
   "metadata": {},
   "source": [
    "# SEC Filing Section Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef7df371",
   "metadata": {
    "tags": [
     "explanatory_cell"
    ]
   },
   "source": [
    "This notebook defines the pipeline for extracting the narrative text sections\n",
    "from the 10-K, 10-Q, and S-1 filings. This notebook contains both\n",
    "exploration code and the code for defining the API. Code cells marked\n",
    "with `#pipeline-api` are included in the API definition.\n",
    "\n",
    "To demonstrate how off-the-shelf Unstructured Bricks extract\n",
    "meaningful data from complex source documents, we will apply\n",
    "a series of Bricks with explanations before defining the API.\n",
    "\n",
    "#### Table of Contents\n",
    "\n",
    "1. [Pulling in Raw Documents](#raw)\n",
    "1. [Reading the Document](#reading)\n",
    "1. [Custom Partitioning Bricks](#custom)\n",
    "1. [Cleaning Bricks](#cleaning)\n",
    "1. [Staging Bricks](#staging)\n",
    "1. [Define the Pipeline API](#pipeline)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36dfce19",
   "metadata": {},
   "source": [
    "## Section 1: Pulling in Raw Documents <a id=\"raw\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce437859",
   "metadata": {
    "tags": [
     "explanatory_cell"
    ]
   },
   "source": [
    "First, let's pull in a filing from the SEC EDGAR database.\n",
    "In this case, we'll pull in the most recent 10-K for Royal Gold (RGLD),\n",
    "a publicly traded precious metals company."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b5350cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from prepline_sec_filings.fetch import (\n",
    "    get_form_by_ticker, open_form_by_ticker, get_filing\n",
    ")\n",
    "\n",
    "# Fetch the 2021 10-K instead of the most recent one\n",
    "rgld_cik = 85535\n",
    "accession_number = \"000155837021011343\"\n",
    "text = get_filing(rgld_cik,\n",
    "                  accession_number,\n",
    "                  company='Unstructured Technologies', \n",
    "                  email='support@unstructured.io')\n",
    "\n",
    "# Alternatively, use the following to fetch the most recent filing:\n",
    "# text = get_form_by_ticker(\n",
    "#    'rgld', \n",
    "#    '10-K', \n",
    "#    company='Unstructured Technologies', \n",
    "#    email='support@unstructured.io'\n",
    "#)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be77e966",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<XBRL>\n",
      "<?xml version='1.0' encoding='UTF-8'?>\n",
      "\n",
      "      <!-- iXBRL document created with: Toppan Merrill Bridge iXBRL 9.6.7811.37134 -->\n",
      "      <!-- Based on: iXBRL 1.1 -->\n",
      "      <!-- Created on: 8/11/2021 10:45:07 PM -->\n",
      "      <!-- iXBRL Library version: 1.0.7811.37150 -->\n",
      "      <!-- iXBRL Service Job ID: 19f8db26-9ac2-4427-9c71-ed8734c57db7 -->\n",
      "\n",
      "  <html xmlns:us-gaap=\"http://fasb.org/us-gaap/2020-01-31\" xmlns:link=\"http://www.xbrl.org/2003/linkbase\" xmlns:country=\"http://xbrl.sec.gov/country/2020-01-31\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:rgld=\"http://www.royalgold.com/20210630\" xmlns:xbrldt=\"http://xbrl.org/2005/xbrldt\" xmlns:ixt-sec=\"http://www.sec.gov/inlineXBRL/transformation/2015-08-31\" xmlns:srt=\"http://fasb.org/srt/2020-01-31\" xmlns:ix=\"http://www.xbrl.org/2013/inlineXBRL\" xmlns:ref=\"http://www.xbrl.org/2006/ref\" xmlns:utr=\"http://www.xbrl.org/2009/utr\" xmlns:xbrli=\"http://www.xbrl.org/2003/instance\" xmlns:dei=\"http://xbrl.sec.gov/dei/2020-01-31\" xmlns:iso4217=\"http://www.xbrl.org/2003/iso4217\" xmlns=\"http://www.w3.org/1999/xhtml\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xmlns:xs=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:ixt=\"http://www.xbrl.org/inlineXBRL/transformation/2015-02-26\" xmlns:xbrldi=\"http://xbrl.org/2006/xbrldi\"><head><meta content=\"text/html\" http-equiv=\"content-type\" /><title></title></head><body><div style=\"margin-top:30pt;\"></div><div style=\"display:none;\"><ix:header><ix:hidden><ix:nonNumeric contextRef=\"Duration_7_1_2020_To_6_30_2021_srt_RangeAxis_srt_MinimumMember_76I-qBLrdESpASXnsjZqeg\" name=\"us-gaap:ShareBasedCompensationArrangementByShareBasedPaymentAwardAwardRequisiteServicePeriod1\" id=\"Hidden_MwI9vniubUaVqtDplWzBDw\">P1Y</ix:nonNumeric><ix:nonFraction unitRef=\"Unit_Standard_pure_C_bj9fXCb0qRMr5YMSJzdQ\" contextRef=\"Duration_7_1_2020_To_6_30_2021_srt_TitleOfIndividualAxis_rgld_OfficersAndCertainEmployeesMember_us-gaap\n"
     ]
    }
   ],
   "source": [
    "print(text[1375:3284])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "179f274a",
   "metadata": {
    "tags": [
     "explanatory_cell"
    ]
   },
   "source": [
    "Wow! We're able to pull in the document, but it's really messy. \n",
    "To help, we'll apply Unstructured Bricks to extract the information we're most interested in. Ultimately, we want to be able to ask the API\n",
    "for a section and get back the narrative text within that section. Once\n",
    "we extract the narrative text, we can spin up a labeling task or send\n",
    "it to a downstream ML service for inference."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fad407f5",
   "metadata": {},
   "source": [
    "```json\n",
    "[\n",
    "  {\n",
    "    \"text\": \"You should carefully consider the risks described in this section. Our future performance is subject to risks and uncertainties that could have a material adverse effect on our business, results of operations, and financial condition and the trading price of our common stock. We may be subject to other risks and uncertainties not presently known to us. In addition, please see our note about forward-looking statements included in the MD&A.\",\n",
    "    \"type\": \"NarrativeText\"\n",
    "  },\n",
    "  {\n",
    "    \"text\": \"Our revenue is subject to volatility in metal prices, which could negatively affect our results of operations or cash flow.\",\n",
    "    \"type\": \"NarrativeText\"\n",
    "  },\n",
    "  {\n",
    "    \"text\": \"Market prices for gold, silver, copper, nickel, and other metals may fluctuate widely over time and are affected by numerous factors beyond our control. These factors include metal supply and demand, industrial and jewelry fabrication, investment demand, central banking actions, inflation expectations, currency values, interest rates, forward sales by metal producers, and political, trade, economic, or banking conditions.\",\n",
    "    \"type\": \"NarrativeText\"\n",
    "  },\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30db6f5f",
   "metadata": {},
   "source": [
    "## Section 2: Reading the Document <a id=\"reading\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb2b1aa0",
   "metadata": {
    "tags": [
     "explanatory_cell"
    ]
   },
   "source": [
    "The first step is to identify and categorize text elements within the\n",
    "document. When we get the SEC document from EDGAR, it is in `.xml` format. Like most text documents, XML documents contains text data we're\n",
    "interested along with other information that we'd like to discard. In\n",
    "an XML or HTML document, that could be styling or formatting tags.\n",
    "Other document types, like PDFs, might have headers, footers, and page\n",
    "numbers we're not interested in.\n",
    "\n",
    "To help with this, Unstructured has created a library of \n",
    "***partitioning bricks*** to break down a document into coherent chunks.\n",
    "Different document types have different partitioning methods. For HTML\n",
    "or XML documents, we identify sections based on text tags and use\n",
    "NLP to differentiate between titles, narrative text, and other section\n",
    "types. For PDF documents, we use visual clues such as the layout of the\n",
    "document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d1cbf28",
   "metadata": {},
   "outputs": [],
   "source": [
    "from unstructured.documents.html import HTMLDocument\n",
    "\n",
    "html_document = HTMLDocument.from_string(text).doc_after_cleaners(skip_headers_and_footers=True, skip_table_text=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4415da63",
   "metadata": {
    "tags": [
     "explanatory_cell"
    ]
   },
   "source": [
    "Here, we see that the `HTMLDocument` Brick was able to extract\n",
    "text from the raw source XML document. This is a generic\n",
    "`HTMLDocument` class that does not have any special knowledge about\n",
    "the structure of EDGAR documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7352e9fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Competition\n",
      "\n",
      "\n",
      "The mining industry in general, and stream and royalty segments in particular, are very competitive. We compete with other stream and royalty companies, mine operators, and financial buyers in efforts to acquire existing stream and royalty interests. We also compete with the lenders, investors, and streaming and royalty companies providing financing to operators of mineral properties in our efforts to create new stream and royalty interests. Our competitors may be larger than we are and may have greater resources and access to capital than we have. Key competitive factors in the stream and royalty acquisition and financing business include the ability to identify and evaluate potential opportunities, transaction structure and consideration, and access to capital.\n",
      "\n",
      "\n",
      "Regulation\n",
      "\n",
      "\n",
      "Operators of the mines that are subject to our stream and royalty interests must comply with numerous environmental, mine safety, land use, waste disposal, remediation and public health laws and regulations promulgated by federal, state, provincial and local governments in the United States, Canada, Chile, the Dominican Republic, Ghana, Mexico, Botswana, Australia and other countries where we hold interests. Although we, as a stream or royalty interest owner, are not\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for element in html_document.pages[0].elements[71:75]:\n",
    "    print(element)\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b9105dc",
   "metadata": {
    "tags": [
     "explanatory_cell"
    ]
   },
   "source": [
    "Internally, the sections of text are represented using `Title` and\n",
    "`NarrativeText` classes, as shown below. Bricks that are executed while\n",
    "processing the document distinguish titles from narrative text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28461ad4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<unstructured.documents.html.HTMLTitle>,\n",
       " <unstructured.documents.html.HTMLNarrativeText>,\n",
       " <unstructured.documents.html.HTMLTitle>,\n",
       " <unstructured.documents.html.HTMLNarrativeText>]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "html_document.pages[0].elements[71:75]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67d14cb5",
   "metadata": {
    "tags": [
     "explanatory_cell"
    ]
   },
   "source": [
    "Below, we show Bricks identifying possible titles and narrative text\n",
    "sections. `\"Regulation\"` is identified as a title and not narrative text\n",
    "because it is short and does not contain any verbs. Conversely, longer\n",
    "sections with multiple, complex sentences are identified as narrative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b33d8a6b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from unstructured.nlp.partition import is_possible_title\n",
    "\n",
    "is_possible_title(\"Regulation\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c8f59e4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "is_possible_title(\"\"\"Operators of the mines that are subject to our \n",
    "stream and royalty interests must comply with numerous environmental, \n",
    "mine safety, land use, waste disposal, remediation and public health \n",
    "laws and regulations promulgated by federal, state, provincial and \n",
    "local governments in the United States, Canada, Chile, the Dominican \n",
    "Republic, Ghana, Mexico, Botswana, Australia and other countries where \n",
    "we hold interests. Although we, as a stream or royalty interest owner, \n",
    "are not\"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b67a1722",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from unstructured.nlp.partition import is_possible_narrative_text\n",
    "\n",
    "is_possible_narrative_text(\"Regulation\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3003af93",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "is_possible_narrative_text(\"\"\"Operators of the mines that are subject to our \n",
    "stream and royalty interests must comply with numerous environmental, \n",
    "mine safety, land use, waste disposal, remediation and public health \n",
    "laws and regulations promulgated by federal, state, provincial and \n",
    "local governments in the United States, Canada, Chile, the Dominican \n",
    "Republic, Ghana, Mexico, Botswana, Australia and other countries where \n",
    "we hold interests. Although we, as a stream or royalty interest owner, \n",
    "are not\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8702f29",
   "metadata": {},
   "source": [
    "## Section 3: Custom Partitioning Bricks <a id=\"custom\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32d7513f",
   "metadata": {
    "tags": [
     "explanatory_cell"
    ]
   },
   "source": [
    "In addition to the partitioning bricks Unstructured provides out of the\n",
    "box, a given recipe may require custom partitioning bricks. In this case,\n",
    "we're interested in identifying sections within the SEC filing. To \n",
    "support that, we'll create a custom partitioning brick to identify \n",
    "section titles. This will help distinguish section headings from sub-headings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "407222da",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "from unstructured.documents.elements import Title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55571113",
   "metadata": {},
   "outputs": [],
   "source": [
    "ITEM_TITLE_RE = re.compile(\n",
    "    r\"(?i)item \\d{1,3}(?:[a-z]|\\([a-z]\\))?(?:\\.)?(?::)?\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6d07983",
   "metadata": {},
   "outputs": [],
   "source": [
    "def is_10k_item_title(title: str) -> bool:\n",
    "    \"\"\"Determines if a title corresponds to a 10-K item heading.\"\"\"\n",
    "    return ITEM_TITLE_RE.match(title) is not None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "418770aa",
   "metadata": {
    "tags": [
     "explanatory_cell"
    ]
   },
   "source": [
    "This is looking pretty good! Those all look like the section headings\n",
    "we're looking for. But wait, it seems like a few might be missing? It\n",
    "turns out some of the titles contain extra whitespace characters. As a\n",
    "result, they are not captured by the regex. Fortunately, Unstructured\n",
    "has tools to help with that, which we'll cover in the next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bebffdaf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ITEM 1A. RISK FACTORS\n",
      "ITEM 16.    FORM 10-K SUMMARY\n"
     ]
    }
   ],
   "source": [
    "for element in html_document.elements:\n",
    "    if isinstance(element, Title) and is_10k_item_title(element.text):\n",
    "        print(element)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b3e3774",
   "metadata": {},
   "source": [
    "## Section 4: Cleaning Bricks <a id=\"cleaning\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f64d54c3",
   "metadata": {
    "tags": [
     "explanatory_cell"
    ]
   },
   "source": [
    "In addition to partitioning bricks, the Unstructured library has\n",
    "***cleaning*** bricks for removing unwanted content from text. In this\n",
    "case, we'll solve our whitespace problem by using the \n",
    "`clean_extra_whitespace` brick. Other uses for cleaning bricks include\n",
    "cleaning out boilerplate, sentence fragments, and other segments\n",
    "of text that could impact labeling tasks or the accuracy of\n",
    "machine learning models. As with partitioning bricks, users can\n",
    "include custom cleaning bricks in a pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04638330",
   "metadata": {},
   "outputs": [],
   "source": [
    "from unstructured.cleaners.core import clean_extra_whitespace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a47bbb09-4745-4aaf-9c64-849f54bded6d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ITEM 1. BUSINESS\n",
      "ITEM 1A. RISK FACTORS\n",
      "ITEM 1B. UNRESOLVED STAFF COMMENTS\n",
      "ITEM 2. PROPERTIES\n",
      "ITEM 3. LEGAL PROCEEDINGS\n",
      "ITEM 4. MINE SAFETY DISCLOSURE\n",
      "ITEM 5. MARKET FOR REGISTRANT’S COMMON EQUITY, RELATED STOCKHOLDER MATTERS AND ISSUER PURCHASES OF EQUITY SECURITIES\n",
      "ITEM 6. RESERVED\n",
      "ITEM 7. MANAGEMENT’S DISCUSSION AND ANALYSIS OF FINANCIAL CONDITION AND RESULTS OF OPERATIONS\n",
      "ITEM 7A. QUANTITATIVE AND QUALITATIVE DISCLOSURE ABOUT MARKET RISK\n",
      "ITEM 8. FINANCIAL STATEMENTS AND SUPPLEMENTARY DATA\n",
      "ITEM 9. CHANGES IN AND DISAGREEMENTS WITH ACCOUNTANTS ON ACCOUNTING AND FINANCIAL DISCLOSURE\n",
      "ITEM 9A. CONTROLS AND PROCEDURES\n",
      "ITEM 9B. OTHER INFORMATION\n",
      "ITEM 10. DIRECTORS, EXECUTIVE OFFICERS AND CORPORATE GOVERNANCE\n",
      "ITEM 11. EXECUTIVE COMPENSATION\n",
      "ITEM 12. SECURITY OWNERSHIP OF CERTAIN BENEFICIAL OWNERS AND MANAGEMENT AND RELATED STOCKHOLDER MATTERS\n",
      "ITEM 13. CERTAIN RELATIONSHIPS AND RELATED TRANSACTIONS AND DIRECTOR INDEPENDENCE\n",
      "ITEM 14. PRINCIPAL ACCOUNTANT FEES AND SERVICES\n",
      "ITEM 15. EXHIBITS AND FINANCIAL STATEMENT SCHEDULES\n",
      "ITEM 16. FORM 10-K SUMMARY\n"
     ]
    }
   ],
   "source": [
    "for element in html_document.elements:\n",
    "    element.text = clean_extra_whitespace(element.text)\n",
    "    if isinstance(element, Title) and is_10k_item_title(element.text):\n",
    "        print(element)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aaee90ba",
   "metadata": {
    "tags": [
     "explanatory_cell"
    ]
   },
   "source": [
    "After applying the cleaning brick, we have a good looking list of \n",
    "section titles! With our set of bricks in place, we have what we\n",
    "need to identify the risk section. For convenience, we've packaged them\n",
    "up in the `SECDocument` class in the Python module for the pipeline. Notice\n",
    "we include the `# pipeline-api` comment to include the `SECDocument`\n",
    "import in the pipeline definition. Applying the full pipeline, we're\n",
    "able to cleanly extract the risk narrative portion of the SEC filing.\n",
    "The function is parameterized so we're easily able to get other sections\n",
    "as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "394561e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pipeline-api\n",
    "from prepline_sec_filings.sections import section_string_to_enum, validate_section_names, SECSection\n",
    "from prepline_sec_filings.sec_document import SECDocument, REPORT_TYPES, VALID_FILING_TYPES"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "247f3d20",
   "metadata": {},
   "outputs": [],
   "source": [
    "sec_document = SECDocument.from_string(text)\n",
    "risk_narrative = sec_document.get_section_narrative(SECSection.RISK_FACTORS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5afb5b54",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You should carefully consider the risks described in this section. Our future performance is subject to risks and uncertainties that could have a material adverse effect on our business, results of operations, and financial condition and the trading price of our common stock. We may be subject to other risks and uncertainties not presently known to us. In addition, please see our note about forward-looking statements included in the MD&A.\n",
      "\n",
      "\n",
      "Our revenue is subject to volatility in metal prices, which could negatively affect our results of operations or cash flow.\n",
      "\n",
      "\n",
      "Market prices for gold, silver, copper, nickel, and other metals may fluctuate widely over time and are affected by numerous factors beyond our control. These factors include metal supply and demand, industrial and jewelry fabrication, investment demand, central banking actions, inflation expectations, currency values, interest rates, forward sales by metal producers, and political, trade, economic, or banking conditions.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for element in risk_narrative[:3]:\n",
    "    print(element)\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ae254ed",
   "metadata": {},
   "source": [
    "## Section 5: Staging Bricks <a id=\"staging\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a9a3d90",
   "metadata": {
    "tags": [
     "explanatory_cell"
    ]
   },
   "source": [
    "After we've extracted the information we want, the next step is to\n",
    "prepare the data for downstream tasks. This can include preparation\n",
    "to:\n",
    "\n",
    "- Feed the data into a downstream ML model\n",
    "- Convert the data into a labeling task\n",
    "- Upload the data into a database\n",
    "\n",
    "To help with this, Unstructured has created ***staging bricks***\n",
    "that preparing extracted data for downstream tasks. In this case,\n",
    "we'll show how to format the data so that it can be easily\n",
    "uploaded into LabelStudio as a labeling task. Other future uses cases include:\n",
    "\n",
    "- Preparing data for downstream inference by Primer, Co:here, or Vannevar\n",
    "- Chunking text to fit into attention windows for Huggingface\n",
    "- Uploading data to Palantir Foundry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55f28884",
   "metadata": {},
   "outputs": [],
   "source": [
    "from unstructured.staging.label_studio import stage_for_label_studio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aaf2400e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'data': {'text': 'You should carefully consider the risks described in this section. Our future performance is subject to risks and uncertainties that could have a material adverse effect on our business, results of operations, and financial condition and the trading price of our common stock. We may be subject to other risks and uncertainties not presently known to us. In addition, please see our note about forward-looking statements included in the MD&A.',\n",
       "   'ref_id': '7a912bb639b547404be4ceaf5d9083a9'}},\n",
       " {'data': {'text': 'Our revenue is subject to volatility in metal prices, which could negatively affect our results of operations or cash flow.',\n",
       "   'ref_id': 'd4cc8e0e0c2b68ef69282c5250b721c9'}},\n",
       " {'data': {'text': 'Market prices for gold, silver, copper, nickel, and other metals may fluctuate widely over time and are affected by numerous factors beyond our control. These factors include metal supply and demand, industrial and jewelry fabrication, investment demand, central banking actions, inflation expectations, currency values, interest rates, forward sales by metal producers, and political, trade, economic, or banking conditions.',\n",
       "   'ref_id': '96515fdbe9e0c547a557c2fe30137176'}},\n",
       " {'data': {'text': 'Our revenue is directly tied to metal prices and is particularly sensitive to changes in the price of gold, as we derive a majority of our revenue from gold stream and royalty interests. Under our stream agreements, we purchase metal at a fixed price or a stated percentage of the market price and then sell the metal in the open market. If market prices decline, our revenue and cash flow from metal sales decline. A price decline can also impact our revenue under certain sliding-scale royalty agreements as we may receive a lower royalty rate when prices fall below specified thresholds. In addition, some',\n",
       "   'ref_id': '29f82037f1e3bf53caf85af1a87aa539'}},\n",
       " {'data': {'text': 'of our royalty agreements are based on the operator’s concentrate sales to smelters and allow for price adjustments between the operator and the smelter based on metals prices on a future date, typically three to five months after shipment of concentrate. These price adjustments can decrease our revenue in future periods if metal prices decline following shipment.',\n",
       "   'ref_id': 'f80f60dd8411ce334de241dc3088f331'}}]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label_studio_data = stage_for_label_studio(risk_narrative)\n",
    "label_studio_data[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4588632",
   "metadata": {},
   "source": [
    "## Section 6: Define the Pipeline API <a id=\"pipeline\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce62f651",
   "metadata": {
    "tags": [
     "explanatory_cell"
    ]
   },
   "source": [
    "Once we have everything ready to go, we can package up the pipeline API\n",
    "using the `# pipeline-api` comment. Unstructured has tooling that will\n",
    "convert this notebook directly into a REST API. By convention, the REST\n",
    "API calls the `pipeline_api` function when the API is invoked.\n",
    "Once the REST API is up and running, users can make an HTTP request\n",
    "to parse new documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6cb712f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pipeline-api\n",
    "from enum import Enum\n",
    "import re\n",
    "import signal\n",
    "\n",
    "from unstructured.staging.base import convert_to_isd\n",
    "from prepline_sec_filings.sections import (\n",
    "    ALL_SECTIONS,\n",
    "    SECTIONS_10K,\n",
    "    SECTIONS_10Q,\n",
    "    SECTIONS_S1,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6a2bfd8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pipeline-api\n",
    "class timeout:\n",
    "    def __init__(self, seconds=1, error_message='Timeout'):\n",
    "        self.seconds = seconds\n",
    "        self.error_message = error_message\n",
    "    def handle_timeout(self, signum, frame):\n",
    "        raise TimeoutError(self.error_message)\n",
    "    def __enter__(self):\n",
    "        try:\n",
    "            signal.signal(signal.SIGALRM, self.handle_timeout)\n",
    "            signal.alarm(self.seconds)\n",
    "        except ValueError:\n",
    "            pass\n",
    "    def __exit__(self, type, value, traceback):\n",
    "        try:\n",
    "            signal.alarm(0)\n",
    "        except ValueError:\n",
    "            pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79987cb3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pipeline-api\n",
    "def get_regex_enum(section_regex):\n",
    "    class CustomSECSection(Enum):\n",
    "        CUSTOM = re.compile(section_regex)\n",
    "        \n",
    "        @property\n",
    "        def pattern(self):\n",
    "            return self.value\n",
    "            \n",
    "    return CustomSECSection.CUSTOM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db6ae22f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import io"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4206b20c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pipeline-api\n",
    "import csv\n",
    "from typing import Dict\n",
    "from unstructured.documents.elements import Text, NarrativeText, Title, ListItem\n",
    "def convert_to_isd_csv(results:dict) -> str:\n",
    "    \"\"\"\n",
    "    Returns the representation of document elements as an Initial Structured Document (ISD)\n",
    "    in CSV Format.\n",
    "    \"\"\"\n",
    "    csv_fieldnames: List[str] = [\"section\", \"element_type\", \"text\"]\n",
    "    new_rows = []\n",
    "    for section, section_narrative in results.items():\n",
    "        rows: List[Dict[str, str]] = convert_to_isd(section_narrative)\n",
    "        for row in rows:\n",
    "            new_row_item = dict()\n",
    "            new_row_item[\"section\"] = section\n",
    "            new_row_item[\"element_type\"] = row[\"type\"]\n",
    "            new_row_item[\"text\"] = row[\"text\"]\n",
    "            new_rows.append(new_row_item)\n",
    "        \n",
    "    with io.StringIO() as buffer:\n",
    "        csv_writer = csv.DictWriter(buffer, fieldnames=csv_fieldnames)\n",
    "        csv_writer.writeheader()\n",
    "        csv_writer.writerows(new_rows)\n",
    "        return buffer.getvalue()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8e0cad6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pipeline-api\n",
    "from unstructured.staging.label_studio import stage_for_label_studio\n",
    "\n",
    "# List of valid response schemas\n",
    "LABELSTUDIO = \"labelstudio\"\n",
    "ISD = \"isd\"\n",
    "\n",
    "def pipeline_api(text, response_type=\"application/json\", response_schema=\"isd\", m_section=[], m_section_regex=[]):\n",
    "    \"\"\"Many supported sections including: RISK_FACTORS, MANAGEMENT_DISCUSSION, and many more\"\"\"\n",
    "    validate_section_names(m_section)\n",
    "    \n",
    "    sec_document = SECDocument.from_string(text)\n",
    "    if sec_document.filing_type not in VALID_FILING_TYPES:\n",
    "        raise ValueError(\n",
    "            f\"SEC document filing type {sec_document.filing_type} is not supported, \"\n",
    "            f\"must be one of {','.join(VALID_FILING_TYPES)}\"\n",
    "        )\n",
    "    results = {}\n",
    "    if m_section == [ALL_SECTIONS]:\n",
    "        filing_type = sec_document.filing_type\n",
    "        if filing_type in REPORT_TYPES:\n",
    "            if filing_type.startswith(\"10-K\"):\n",
    "                m_section = [enum.name for enum in SECTIONS_10K]\n",
    "            elif filing_type.startswith(\"10-Q\"):\n",
    "                m_section = [enum.name for enum in SECTIONS_10Q]\n",
    "            else:\n",
    "                raise ValueError(f\"Invalid report type: {filing_type}\")\n",
    "\n",
    "        else:\n",
    "            m_section = [enum.name for enum in SECTIONS_S1]\n",
    "    for section in m_section:\n",
    "        results[section] = sec_document.get_section_narrative(\n",
    "            section_string_to_enum[section]\n",
    "        )\n",
    "    for i, section_regex in enumerate(m_section_regex):\n",
    "        regex_enum = get_regex_enum(section_regex)\n",
    "        with timeout(seconds=5):\n",
    "            section_elements = sec_document.get_section_narrative(regex_enum)\n",
    "            results[f\"REGEX_{i}\"] = section_elements\n",
    "    if response_type == \"application/json\":\n",
    "        if response_schema == LABELSTUDIO:\n",
    "            return {section:stage_for_label_studio(section_narrative) for section, section_narrative in results.items()}\n",
    "        elif response_schema == ISD:\n",
    "            return {section:convert_to_isd(section_narrative) for section, section_narrative in results.items()}\n",
    "        else:\n",
    "            raise ValueError(f\"output_schema '{response_schema}' is not supported for {response_type}\")\n",
    "    elif response_type == \"text/csv\":\n",
    "        if response_schema != ISD:\n",
    "            raise ValueError(f\"output_schema '{response_schema}' is not supported for {response_type}\")\n",
    "        return convert_to_isd_csv(results)\n",
    "    else:\n",
    "        raise ValueError(f\"response_type '{response_type}' is not supported\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0c2949b",
   "metadata": {
    "tags": [
     "explanatory_cell"
    ]
   },
   "source": [
    "As we can see, the output now matches what we wanted at the beginning\n",
    "of the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6403cb1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'text': 'You should carefully consider the risks described in this section. Our future performance is subject to risks and uncertainties that could have a material adverse effect on our business, results of operations, and financial condition and the trading price of our common stock. We may be subject to other risks and uncertainties not presently known to us. In addition, please see our note about forward-looking statements included in the MD&A.',\n",
       "  'type': 'NarrativeText'},\n",
       " {'text': 'Our revenue is subject to volatility in metal prices, which could negatively affect our results of operations or cash flow.',\n",
       "  'type': 'NarrativeText'},\n",
       " {'text': 'Market prices for gold, silver, copper, nickel, and other metals may fluctuate widely over time and are affected by numerous factors beyond our control. These factors include metal supply and demand, industrial and jewelry fabrication, investment demand, central banking actions, inflation expectations, currency values, interest rates, forward sales by metal producers, and political, trade, economic, or banking conditions.',\n",
       "  'type': 'NarrativeText'},\n",
       " {'text': 'Our revenue is directly tied to metal prices and is particularly sensitive to changes in the price of gold, as we derive a majority of our revenue from gold stream and royalty interests. Under our stream agreements, we purchase metal at a fixed price or a stated percentage of the market price and then sell the metal in the open market. If market prices decline, our revenue and cash flow from metal sales decline. A price decline can also impact our revenue under certain sliding-scale royalty agreements as we may receive a lower royalty rate when prices fall below specified thresholds. In addition, some',\n",
       "  'type': 'NarrativeText'},\n",
       " {'text': 'of our royalty agreements are based on the operator’s concentrate sales to smelters and allow for price adjustments between the operator and the smelter based on metals prices on a future date, typically three to five months after shipment of concentrate. These price adjustments can decrease our revenue in future periods if metal prices decline following shipment.',\n",
       "  'type': 'NarrativeText'}]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "risk_narrative = pipeline_api(text, m_section=[\"RISK_FACTORS\"])[\"RISK_FACTORS\"]\n",
    "risk_narrative[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2516c2d0-55e2-4d4c-9be9-cababa6a452c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BUSINESS\n",
      "[]\n",
      "---------------\n",
      "RISK_FACTORS\n",
      "[{'text': 'You should carefully consider the risks described in this section. Our future performance is subject to risks and uncertainties that could have a material adverse effect on our business, results of operations, and financial condition and the trading price of our common stock. We may be subject to other risks and uncertainties not presently known to us. In addition, please see our note about forward-looking statements included in the MD&A.', 'type': 'NarrativeText'}, {'text': 'Our revenue is subject to volatility in metal prices, which could negatively affect our results of operations or cash flow.', 'type': 'NarrativeText'}, {'text': 'Market prices for gold, silver, copper, nickel, and other metals may fluctuate widely over time and are affected by numerous factors beyond our control. These factors include metal supply and demand, industrial and jewelry fabrication, investment demand, central banking actions, inflation expectations, currency values, interest rates, forward sales by metal producers, and political, trade, economic, or banking conditions.', 'type': 'NarrativeText'}, {'text': 'Our revenue is directly tied to metal prices and is particularly sensitive to changes in the price of gold, as we derive a majority of our revenue from gold stream and royalty interests. Under our stream agreements, we purchase metal at a fixed price or a stated percentage of the market price and then sell the metal in the open market. If market prices decline, our revenue and cash flow from metal sales decline. A price decline can also impact our revenue under certain sliding-scale royalty agreements as we may receive a lower royalty rate when prices fall below specified thresholds. In addition, some', 'type': 'NarrativeText'}]\n",
      "---------------\n",
      "UNRESOLVED_STAFF_COMMENTS\n",
      "[]\n",
      "---------------\n",
      "PROPERTIES\n",
      "[{'text': 'We do not own or operate the properties on which we hold stream or royalty interests and therefore much of the information disclosed in this Form\\xa010-K regarding these properties is provided to us by the operators. For example, the operators of certain properties provide us information regarding metals production, estimates of mineral reserves and additional mineralized material and production estimates. A list of our producing and development stage streams and royalties, as well their respective reserves, are summarized in Table\\xa01 “Operators’ Estimated Proven and Probable Gold Reserves” below within this Item\\xa02. More information is available to the public regarding certain properties in which we have stream or royalty interests, including reports filed with the SEC or with the Canadian securities regulatory agencies available at www.sec.gov or www.sedar.com, respectively.', 'type': 'NarrativeText'}, {'text': 'We manage our business under two reportable segments, consisting of the acquisition and management of stream interests and the acquisition and management of royalty interests. The description of our principal stream and royalty interests set forth below includes the location, operator, stream or royalty rate, access and any material current developments at the property. For any reported production amounts discussed below, we consider reported production to relate to the amount of metal sales subject to our stream and royalty interests. Please refer to Item\\xa07, MD&A, for discussion on production estimates, historical production and revenue for our principal properties. The map below illustrates the location of our principal producing stage properties.', 'type': 'NarrativeText'}, {'text': 'In 2018, the SEC adopted amendments to the disclosure requirements for mining properties. Effective for fiscal years beginning on or after January 1, 2021, the disclosure requirements under SEC Industry Guide 7 have been replaced with new disclosure requirements under Regulation S-K Subpart 1300. The property and reserve disclosures as of June 30, 2021, contained in this report are presented in accordance with SEC Industry Guide 7. We expect that the property and reserve disclosures as of December 31, 2021, contained in our Transitional Report on Form 10-KT to be filed in early 2022 will be presented in accordance with Regulation S-K Subpart 1300.', 'type': 'NarrativeText'}, {'text': 'We consider both historical and future potential revenues to determine which stream and royalty interests in our portfolio are principal to our business. Estimated future potential revenues from producing properties are based on a number of factors, including reserves subject to our stream and royalty interests, production estimates, feasibility studies, technical reports, metal price assumptions, mine life, legal status and other factors and assumptions, any of which could change and could cause us to conclude that one or more of such stream and royalty interests are no longer principal to our business.', 'type': 'NarrativeText'}]\n",
      "---------------\n",
      "LEGAL_PROCEEDINGS\n",
      "[]\n",
      "---------------\n",
      "MINE_SAFETY\n",
      "[]\n",
      "---------------\n",
      "MARKET_FOR_REGISTRANT_COMMON_EQUITY\n",
      "[{'text': 'Our common stock is listed and traded on the Nasdaq Global Select Market under the symbol “RGLD.” As of August 5, 2021, we had 779 holders of record of our common stock. This figure does not reflect the beneficial ownership of shares held in nominee name.', 'type': 'NarrativeText'}, {'text': 'On November 17, 2020, we announced an increase in our annual dividend for calendar year 2021 from $1.12 to $1.20, payable on a quarterly basis of $0.30 per share. The newly declared dividend is 7% higher than the dividend paid during calendar year 2020. We have steadily increased our annual dividend for 20 years, or since calendar year 2001. We expect to pay our annual dividend using cash on hand.', 'type': 'NarrativeText'}]\n",
      "---------------\n",
      "MANAGEMENT_DISCUSSION\n",
      "[{'text': 'A discussion of the changes in our financial condition and results of operations for the fiscal years ended June 30, 2020, and 2019, has been omitted from this report, but may be found in Item 7, MD&A, of our Annual Report on Form 10-K for the year ended June 30, 2020, filed with the SEC on August 6, 2020, which is available free of charge on the SEC’s website at www.sec.gov and our website at www.royalgold.com.', 'type': 'NarrativeText'}, {'text': 'We acquire and manage precious metal streams, royalties, and similar interests. We seek to acquire existing stream and royalty interests or finance projects that are in production or in the development stage in exchange for stream or royalty interests.', 'type': 'NarrativeText'}, {'text': 'We manage our business under two segments:', 'type': 'NarrativeText'}, {'text': 'Acquisition and Management of Stream Interests — A metal stream is a purchase agreement that provides, in exchange for an upfront deposit payment, the right to purchase all or a portion of one or more metals produced from a mine, at a price determined for the life of the transaction by the purchase agreement. As of June 30,\\xa02021, we owned eight stream interests, which are on six producing properties and two development stage properties. Stream interests accounted for 69% and 72% of our total revenue for the fiscal years ended June 30 2021, and 2020, respectively. We expect stream interests to continue representing a significant portion of our total revenue.', 'type': 'ListItem'}]\n",
      "---------------\n",
      "MARKET_RISK_DISCLOSURES\n",
      "[]\n",
      "---------------\n",
      "FINANCIAL_STATEMENTS\n",
      "[{'text': 'We have audited the accompanying consolidated balance sheets of Royal Gold, Inc. (the Company) as of June 30, 2021 and 2020, the related consolidated statements of operations and comprehensive income,\\xa0changes in equity and cash flows for each of the three years in the period ended June 30, 2021, and the related notes\\xa0(collectively referred to as the “consolidated financial statements”). In our opinion, the consolidated financial statements present fairly, in all material respects, the financial position of the Company at June 30, 2021 and 2020, and the results of its operations and its cash flows for each of the three years in the period ended June 30, 2021, in conformity with U.S. generally accepted accounting principles.', 'type': 'NarrativeText'}, {'text': \"We also have audited, in accordance with the standards of the Public Company Accounting Oversight Board (United States) (PCAOB), the Company's internal control over financial reporting as of June 30, 2021, based on criteria established in Internal Control-Integrated Framework issued by the Committee of Sponsoring Organizations of the Treadway Commission (2013 framework), and our report dated August 12, 2021 expressed an unqualified opinion thereon.\", 'type': 'NarrativeText'}, {'text': \"These financial statements are the responsibility of the Company's management. Our responsibility is to express an opinion on the Company’s financial statements based on our audits. We are a public accounting firm registered with the PCAOB and are required to be independent with respect to the Company in accordance with the U.S. federal securities laws and the applicable rules and regulations of the Securities and Exchange Commission and the PCAOB.\", 'type': 'NarrativeText'}, {'text': 'We conducted our audits in accordance with the standards of the PCAOB. Those standards require that we plan and perform the audit to obtain reasonable assurance about whether the financial statements are free of material misstatement, whether due to error or fraud. Our audits included performing procedures to assess the risks of material misstatement of the financial statements, whether due to error or fraud, and performing procedures that respond to those risks. Such procedures included examining, on a test basis, evidence regarding the amounts and disclosures in the financial statements. Our audits also included evaluating the accounting principles used and significant estimates made by management, as well as evaluating the overall presentation of the financial statements. We believe that our audits provide a reasonable basis for our opinion.', 'type': 'NarrativeText'}]\n",
      "---------------\n",
      "ACCOUNTING_DISAGREEMENTS\n",
      "[]\n",
      "---------------\n",
      "CONTROLS_AND_PROCEDURES\n",
      "[{'text': 'Under the supervision and with the participation of our management, including our Chief Executive Officer and Chief Financial Officer, we evaluated the effectiveness of the design and operation of our disclosure controls and procedures as of June 30, 2021. Based on this evaluation, our Chief Executive Officer and Chief Financial Officer have concluded that our disclosure controls and procedures were effective as of June 30, 2021, at the reasonable assurance level.', 'type': 'NarrativeText'}, {'text': 'Our management is responsible for establishing and maintaining adequate internal control over financial reporting. Our internal control over financial reporting is designed to provide reasonable assurance regarding the reliability of financial reporting and the preparation of financial statements for external purposes in accordance with generally accepted accounting principles.', 'type': 'NarrativeText'}, {'text': 'Management assessed the effectiveness of our internal control over financial reporting as of June\\xa030,\\xa02021. In making this assessment, management used the criteria set forth by the Committee of Sponsoring Organizations of the Treadway Commission (COSO) in Internal Control—Integrated Framework (2013 Framework). Based on management’s assessment and those criteria, management concluded that our internal control over financial reporting was effective as of June 30, 2021.', 'type': 'NarrativeText'}, {'text': 'Our independent registered public accounting firm, Ernst\\xa0& Young\\xa0LLP, has issued an attestation report on our internal control over financial reporting as of June\\xa030,\\xa02021.', 'type': 'NarrativeText'}]\n",
      "---------------\n",
      "FOREIGN_JURISDICTIONS\n",
      "[]\n",
      "---------------\n",
      "MANAGEMENT\n",
      "[{'text': 'A discussion of the changes in our financial condition and results of operations for the fiscal years ended June 30, 2020, and 2019, has been omitted from this report, but may be found in Item 7, MD&A, of our Annual Report on Form 10-K for the year ended June 30, 2020, filed with the SEC on August 6, 2020, which is available free of charge on the SEC’s website at www.sec.gov and our website at www.royalgold.com.', 'type': 'NarrativeText'}, {'text': 'We acquire and manage precious metal streams, royalties, and similar interests. We seek to acquire existing stream and royalty interests or finance projects that are in production or in the development stage in exchange for stream or royalty interests.', 'type': 'NarrativeText'}, {'text': 'We manage our business under two segments:', 'type': 'NarrativeText'}, {'text': 'Acquisition and Management of Stream Interests — A metal stream is a purchase agreement that provides, in exchange for an upfront deposit payment, the right to purchase all or a portion of one or more metals produced from a mine, at a price determined for the life of the transaction by the purchase agreement. As of June 30,\\xa02021, we owned eight stream interests, which are on six producing properties and two development stage properties. Stream interests accounted for 69% and 72% of our total revenue for the fiscal years ended June 30 2021, and 2020, respectively. We expect stream interests to continue representing a significant portion of our total revenue.', 'type': 'ListItem'}]\n",
      "---------------\n",
      "COMPENSATION\n",
      "[{'text': 'The information required by this item will be included in our proxy statement for our 2021 annual stockholders’ meeting to be filed with the SEC within 120\\xa0days after June\\xa030,\\xa02021, and is incorporated by reference into this report.', 'type': 'NarrativeText'}]\n",
      "---------------\n",
      "PRINCIPAL_STOCKHOLDERS\n",
      "[{'text': 'The information required by this item will be included in our proxy statement for our 2021 annual stockholders’ meeting to be filed with the SEC within 120\\xa0days after June\\xa030,\\xa02021, and is incorporated by reference into this report.', 'type': 'NarrativeText'}]\n",
      "---------------\n",
      "RELATED_PARTY_TRANSACTIONS\n",
      "[]\n",
      "---------------\n",
      "ACCOUNTING_FEES\n",
      "[]\n",
      "---------------\n",
      "EXHIBITS\n",
      "[{'text': 'Description of capital stock (filed as Exhibit 4.2 to Royal Gold’s Quarterly Report on Form 10-Q on November 7, 2019, and incorporated herein by reference)', 'type': 'NarrativeText'}, {'text': 'Form of Employment Agreement by and between Royal Gold, Inc. and William Heissenbuttel, dated January 2, 2020 (filed as Exhibit 10.1 to Royal Gold’s Amendment No. 1 to Current Report on Form 8-K/A filed on January 3, 2020, and incorporated herein by reference).', 'type': 'NarrativeText'}, {'text': 'Employment Agreement by and between Royal Gold Corporation and Mark Isto effective January 2, 2020 (filed as Exhibit 10.2 to Royal Gold’s Amendment No. 1 to Current Report on Form 8-K/A on January 3, 2020, and incorporated herein by reference).', 'type': 'NarrativeText'}, {'text': 'Form of Employment Agreement by and between Royal Gold, Inc. and each of Paul Libner and Randy Shefman (filed as Exhibit 10.1 to Royal Gold’s Amendment No. 1 to Current Report on Form 8-K/A on January 3, 2020, and incorporated herein by reference).', 'type': 'NarrativeText'}]\n",
      "---------------\n",
      "FORM_SUMMARY\n",
      "[{'text': 'Registrants may voluntarily include a summary of information required by Form 10-K under this Item 16. We have elected not to include this summary information.', 'type': 'NarrativeText'}]\n",
      "---------------\n"
     ]
    }
   ],
   "source": [
    "all_narratives = pipeline_api(text, m_section=[\"_ALL\"])\n",
    "for section, elems in all_narratives.items():\n",
    "    print(section)\n",
    "    print(elems[:4])\n",
    "    print(\"---------------\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de063b9d-a987-4396-86b7-5ba3bb74bc92",
   "metadata": {},
   "outputs": [],
   "source": [
    "risk_narrative = pipeline_api(text, response_type=\"text/csv\", m_section=[\"RISK_FACTORS\"])\n",
    "\n",
    "with open('risk_narrative.csv', 'w') as out:\n",
    "    out.write(risk_narrative)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06ed2f2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_narratives = pipeline_api(text, response_type=\"text/csv\", m_section=[\"_ALL\"])\n",
    "\n",
    "with open('all_narratives.csv', 'w') as out:\n",
    "    out.write(all_narratives)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f8bc380",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}