scispaCy.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "V-vtgorDGlaL"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "!pip install scispacy"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%capture\n",
        "!pip install 'https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.0/en_ner_bionlp13cg_md-0.5.0.tar.gz'\n",
        "#Trained on the BIONLP13CG corpus for about 16 types of entities like Organ, Organism,Cell,Cancer,Cellular Component,Pathological_formation"
      ],
      "metadata": {
        "id": "xN2M9K-XHKok"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%%capture\n",
        "!pip install 'https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.0/en_ner_bc5cdr_md-0.5.0.tar.gz'\n",
        "#Trained on BC5CDR corpus for Disease and Chemical entities"
      ],
      "metadata": {
        "id": "1CypY3PMI9KD"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Importing Libraries"
      ],
      "metadata": {
        "id": "llN4nM9MJK3L"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import scispacy\n",
        "import spacy\n",
        "import en_ner_bionlp13cg_md\n",
        "import en_ner_bc5cdr_md\n",
        "from scispacy.abbreviation import AbbreviationDetector\n",
        "from pprint import pprint\n",
        "import pandas as pd"
      ],
      "metadata": {
        "id": "G8Fp0D2_JHeC"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "case_report1 = \"\"\"Background: Hypoglycemia is uncommon in people who are not being treated for diabetes mellitus and, when present, the differential diagnosis is broad. Artifactual hypoglycemia describes discrepancy between low capillary and normal plasma glucose levels regardless of symptoms and should be considered in patients with Raynaud’s phenomenon.\n",
        "\n",
        "Case Presentation: A 46-year-old female patient with a history of a sleeve gastrectomy started complaining about episodes of lipothymias preceded by sweating, nausea, and dizziness. During one of these episodes, a capillary blood glucose was obtained with a value of 24 mg/dl. She had multiple emergency admissions with low-capillary glycemia. An exhaustive investigation for possible causes of hypoglycemia was made for 18 months. The 72h fasting test was negative for hypoglycemia. A Raynaud’s phenomenon was identified during one appointment.\n",
        "\n",
        "Conclusion: Artifactual hypoglycemia has been described in various conditions including Raynaud’s phenomenon, peripheral arterial disease, Eisenmenger syndrome, acrocyanosis, or hypothermia. With this case report, we want to reinforce the importance of being aware of this diagnosis to prevent anxiety, unnecessary treatment, and diagnostic tests.\"\"\""
      ],
      "metadata": {
        "id": "mLr6a2BJJTLe"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "case_report2 = \"\"\" ntroduction: This study aimed at assessing the patterns of care and glycemic control of patients with diabetes (DM) in real life during a follow-up of 2 years in the public and private health sectors in Brazil.\n",
        "Methods: BINDER was an observational study of patients >18 years old, with type-1 (T1DM) and type-2 DM (T2DM), followed at 250 sites from 40 cities across the five regions of Brazil. The results for the 1,266 participants who were followed for 2 years are presented.\n",
        "Main results: Most patients were Caucasians (75%), male (56.7%) and from the private health sector (71%). Of the 1,266 patients who entered the analysis, 104 (8.2%) had T1DM and 1162 (91.8%) had T2DM. Patients followed in the private sector represented 48% of the patients with T1DM and 73% of those with T2DM. For T1DM, in addition to insulins (NPH in 24%, regular in 11%, long-acting analogues in 58%, fast-acting analogues in 53%, and others in 12%), the patients received biguanide (20%), SGLT2-I (4%), and GLP-1Ra (<1%). After 2 years, 13% of T1DM patients were using biguanide, 9% SGLT2-I, 1% GLP-1Ra, and 1% pioglitazone; the use of NPH and regular insulins decreased to 13% and 8%, respectively, while 72% were receiving long-acting insulin analogues, and 78% fast-acting insulin analogues. Treatment for T2DM consisted of biguanide (77%), sulfonylureas (33%), DPP4 inhibitors (24%), SGLT2-I (13%), GLP-1Ra (2.5%), and insulin (27%), with percentages not changing during follow-up. Regarding glucose control, mean HbA1c at baseline and after 2 years of follow-up was 8.2 (1.6)% and 7.5 (1.6)% for T1DM, and 8.4 (1.9)% and 7.2 (1.3)% for T2DM, respectively. After 2 years, HbA1c<7% was reached in 25% of T1DM and 55% of T2DM patients from private institutions and in 20.5% of T1DM and 47% of T2DM from public institutions.\n",
        "Conclusion: Most patients did not reach the HbA1c target in private or public health systems. At the 2-year follow-up, there were no significant improvements in HbA1c in either T1DM or T2DM, which suggests an important clinical inertia.\n",
        "Introduction\n",
        "According to the International Diabetes Federation, 463 million people are currently living with diabetes (DM) worldwide (1, 2). In 2019, it was estimated that there were about 16.8 million people aged from 20 to 79 years with DM in Brazil, with a projected increase of 55% by the year 2045 (1, 2). Type 2 diabetes (T2DM) comprises approximately 90% of all DM diagnoses (3). Estimates related to the number of existing cases of type 1 diabetes (T1DM) in children and adolescents from 0 to 14 years show that Brazil occupies the third place in the global panorama, with 55,500 cases, behind India (95,600) and the United States (94,200) (1).\n",
        "Chronic non-communicable diseases (NCDs) are responsible for nearly two thirds of deaths in Brazil, 5.3% of which due to DM (4). In addition, DM is known to be an important risk factor for chronic cardiovascular disease (CVD), which accounts for 31.3% of deaths in our country (5).\n",
        "Over the last decades, age-standardized rates have shown a tendency to reduced mortality caused by CVD and DM in Brazil (6, 7), in agreement with the aging of the population and the extension of life with the disease. The considerable burden of these diseases was highlighted in the Project on the Global Burden of Disease in Brazil (Burden of disease in Brazil, 1990–2016), in which DM was identified to be responsible for 4.7% of disability-adjusted life-years (DALY) in total and 6.1% of DALY originated by NCDs (8).\n",
        "One of the great current challenges is, therefore, to deal with this increase in morbidity, which requires controlling the disease and preventing complications. These data are even more worrisome when considering the number of affected people in Brazil. Brazilian data on the prevalence of DM representative of the population of nine capitals date from the 1980s (9). At that time, it was estimated that approximately 7.6% of the Brazilian population aged between 30 and 69 years had DM, with both genders being equally affected, and with the prevalence of the disease increasing with age and body fat. A more recent estimate of the prevalence of self-reported DM in Brazil was performed by the Surveillance System of Risk and Protective Factors for Chronic Diseases by Telephone Survey (VIGITEL, Vigilância de Fatores de Risco por Inquérito Telefônico), implemented in 27 state capitals since 2006 (10). In the VIGITEL 2018, 8.1% of women and 7.1% of men ≥18 years old in Brazil reported having DM; the numbers increased with age, reaching 23.1% in individuals over 64 years of age, and decreased with higher the level of education, affecting 15.2% of the participants with from 0 to 8 years of schooling and 3.7% in the group with higher education (10).\n",
        "The high prevalence of DM exerts a negative impact on health not only due to mortality, but also through complications and disabilities resulting from the prolonged time living with the disease and poor metabolic control. In addition to the health-related effects, diabetes is associated with an unwanted economic impact on both individual and society levels. Studies show that associated costs increase according to the duration of DM and the presence of micro- and macrovascular complications (11, 12). Inadequate glycemic control can aggravate these medical conditions and has been reported in studies including patients with T1DM and T2DM treated in the Brazilian Public Unified Health System (SUS, Sistema Único de Saúde) (11, 13, 14). Data related to the management of diabetes in the private sector in Brazil are still scarce.\n",
        "To understand this scenario, there is a lack of data on the prevalence of chronic complications and comorbidities, including cardiovascular risk factors, associated with DM in the Brazilian population. In this regard, public and private health services represent opportunities to access professional care and different medications, providing information to guide better strategies for secondary and tertiary prevention of DM. The disease burden of DM is a relevant concern that requires secondary and tertiary prevention strategies. To develop these actions, it is necessary to understand the epidemiological and current management landscape of patients with diabetes in Brazil.\n",
        "The BrazIliaN Type 1 & 2 DiabetEs Disease Registry (BINDER) study was an observational study, with both a cross-sectional and a longitudinal phase, designed to assess the demographic and clinical characteristics, patterns of care and glycemic control of patients with DM in real life during a follow-up of 2 years in the public and private health sectors in Brazil. In this paper, we present the results of the longitudinal analysis which included the patients followed for 2 years.\n",
        "Patients and methods\n",
        "Study design and population\n",
        "This was a observational study of individuals with DM followed for 2 years in the BINDER study. BINDER included patients with T1DM and T2DM followed by 250 physicians from different public and private healthcare services, geographically distributed in 40 cities across the five regions of Brazil. The study had both cross-sectional and longitudinal phases (for a total duration of 2 years). Five waves of data collection were performed; for each wave, information from the last 6 months was obtained. To be enrolled in the study, patients had to be 18 years or older, have T1DM or T2DM, and had to have attended at least one medical visit at the study site in the 6 months prior to study entry. Pregnancy, gestational diabetes and other types of DM except T1DM or T2DM were excluded.\n",
        "Each medical specialist (endocrinologists, cardiologists, or general practitioners) was responsible for recruiting about ten patients. To minimize patient selection bias, investigators were instructed to recruit patients in a retrospective consecutive manner starting from the patients that were last seen in the service according to medical charts. The initial sample of the study comprised 2,488 patients who entered the first wave of data collection (baseline visit). In the longitudinal phase, four subsequent follow-up visits were planned to occur every 6 months until the completion of the 2-year follow-up period. In this paper, we present the results obtained for the 1,266 participants who completed the final visit scheduled to occur after 2 years of follow-up and comprised the population of the longitudinal analysis.\n",
        "Participating study centers were selected by the Associação Brasileira de Organizações Representativas de Pesquisa Clínica according to a proprietary database. A total of 250 sites/medical specialists of 40 Brazilian cities of the five country regions were chosen: 124 in the Southeast; 48 in the Northeast; 38 in the South Region; 30 in the Central-West Region; and 10 in the North Region.\n",
        "The participant physicians collected data from patient medical charts covering the medical appointments that occurred from 07-Apr-2016, the date of study initiation, to 13-Dec-2019, the date of the final visit for the study.\n",
        "The study was conducted after the approval by the ethics committee of the Universidade Federal de São Paulo (São Paulo, Brazil), and the study was conducted in accordance with the Declaration of Helsinki and the International Conference on Harmonization guidelines for Good Clinical Practice. Informed consent was obtained from all patients.\n",
        "Data collection, variables and evaluation criteria\n",
        "Data were collected from medical charts using an electronic CRF (e-CRF), and data management was performed according to the Data Validation Plan with data review processes in order to clarify data issues.\n",
        "Variables of interest in the cross-sectional (baseline) phase were age, gender, ethnicity, educational level, body mass index (BMI), age at diagnosis, DM duration (time since diagnosis), abdominal circumference, blood pressure and laboratory results, risk factors for CVD, comorbidities, DM complications, glycemic control, medical specialties involved in patient care, and type of treatment. For the subsequent waves and longitudinal phase, collected data included glycemic control (HbA1c), weight, BMI, use of insulins and other medications, number of medications, and comorbidities and complications.\n",
        "The achievement of individual HbA1c target (<7.0% or defined individual target) in patients with T1DM and T2DM at the study baseline (cross-sectional phase) and after 2 years of follow-up was the primary objective of the study and was described by the proportion of patients who reached the target in the overall study population and per DM type. The proportion was complemented by the respective 95% confidence interval (CI). Secondary objectives included the description of patients regarding their demographic and clinical characteristics, presence of comorbidities, complications, patterns of treatment and hospitalizations at baseline and during the follow-up period.\n",
        "As this is a disease registry, non-interventional study, no data were collected beyond those required for routine clinical practice. However, Adverse Drug Reactions to any Sanofi product that occurred during the course of the study was to be reported to the Sponsor within 24 hours from the moment the investigator was notified about the case, in compliance with pharmacovigilance practice.\n",
        "Statistical considerations and analysis\n",
        "Statistical analysis was based on pooled data from all patients. Given the observational nature of the study, the statistical analysis was mainly descriptive, using appropriate summary statistics according to the type of variable. Descriptive statistics as number of non-missing data, range (minimum and maximum values), mean, standard deviation (SD), median and interquartile range (IQR) were calculated for summarizing numerical variables. Frequencies and proportions were calculated for summarizing categorical variables. There was no data imputation for missing/not available data in the calculations. The number of participants with available information for each variable are displayed in the tables, when considered relevant.\n",
        "For the longitudinal phase, statistical analysis was based on pooled data of all patients who had available data at baseline and also at the end of follow-up, after 2 years. Descriptive analyses were performed according to the DM type and health care system (private and public sectors).\n",
        "For the cross-sectional phase, nearly 2,500 patients were planned to be enrolled. Considering a planned sample size of 2,500 patients for the cross-sectional phase and assuming that T2DM comprise 90% of DM cases, the study expected to recruit about 2,250 patients with T2DM and 250 patients with T1DM. The sample size of 2,250 T2DM patients would ensure 95% CIs with a maximum width of 2.1% below and above point estimate. On the other hand, with a sample size of 250 T1DM patients, the maximum expected width was 6.2% below and above the point estimate.\n",
        "Sample size calculation was performed based on published data from population studies conducted in Brazil that estimated the proportion of patients with HbA1c values within the target. Considering an expected proportion of 27% of patients with T2DM within the HbA1c target (3), the sample of 2,250 patients with T2DM would allow assessing this proportion with 95% CIs with a maximum width of 1.8% below and above the point estimate; and for an expected proportion of 10% of T1DM patients within the HbA1c target (14), the sample of 250 patients with T1DM would allow assessing this proportion with 95% CIs with a maximum width of 3.7% below and above the point estimate.\n",
        "Results\n",
        "Baseline characteristics and comorbidities of the subset of patients who entered the longitudinal analysis were similar to those of the patients comprising the total study sample (data not shown). The baseline sample comprised 91.9% of patients with T2DM, the mean age was 63 years, and 52.2% were from the Southeast Region, while the sample at the end of the follow-up period had 91.8% of patients with T2DM, mean age of 62 years, and 51.8% from the Southeast Region.\n",
        "Patient characteristics\n",
        "The study sample for the longitudinal analysis comprised a total of 1,266 patients of the BINDER study who had completed the 2 years of follow-up with data collection in all five waves. As shown in Table 1, 56.7% of patients were male, 74.7% were Caucasian, and 33.5% had a college or higher degree of education. One hundred and four patients had T1DM (8.2%), and 1162 (91.8%) had T2DM. At the time of the initial study visit, the mean age of T1DM and T2DM patients were 35.0 and 63.7 years, respectively; patients aged 18 to 30 years comprised 38.5% of the T1DM group and 0.5% of the T2DM patients. T1DM patients were under treatment for a longer time (mean treatment duration: 15.8years for T1DM vs 9.8 years for T2DM), although the mean time since DM diagnosis was similar between T1DM (16.5 years) and T2DM (17.8 years). Of the assessed patients, 48% of those with T1DM and 73.2% of those with T2DM were followed in the private health sector (Table 1). A family history of DM was reported by 12.2% and 25.3% of patients with T1DM and T2DM, respectively.\n",
        "Table 1\"\"\""
      ],
      "metadata": {
        "id": "V_eBXio7JWQm"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def show_medical_abbreviation(model,document):\n",
        "    \"\"\"\n",
        "    This function detects and resolves medical abbreviations in word entities\n",
        "\n",
        "    Parameters:\n",
        "         model(module): A pretrained biomedical model from ScispaCy(https://allenai.github.io/scispacy/)\n",
        "         document(str): Document to be processed\n",
        "\n",
        "    Returns: List of unique abbreviations and their resolution\n",
        "     \"\"\"\n",
        "    nlp = model.load()\n",
        "    nlp.add_pipe('abbreviation_detector')\n",
        "    doc = nlp(document)\n",
        "    abbreviated = list(set([f\"{abrv}  {abrv._.long_form}\" for abrv in doc._.abbreviations]))             #list is set to ensure only unique values are returned\n",
        "    return abbreviated"
      ],
      "metadata": {
        "id": "M12NIXQQJZ2b"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "show_medical_abbreviation(en_ner_bionlp13cg_md,case_report1)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "snlzkJlhKPX7",
        "outputId": "2849868c-6892-4b79-b82e-cc1422e9d448"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/scispacy/abbreviation.py:248: UserWarning: [W036] The component 'matcher' does not have any patterns defined.\n",
            "  global_matches = self.global_matcher(doc)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[]"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "show_medical_abbreviation(en_ner_bc5cdr_md,case_report1)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Px3x3FNKKXSv",
        "outputId": "13a862d4-79d9-4b41-9065-7579e7a82e52"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[]"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "show_medical_abbreviation(en_ner_bionlp13cg_md,case_report2)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "t_hR_2STKfKs",
        "outputId": "97c2fdef-5ef4-45a1-a5fc-b15c77b06a23"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['BINDER  BrazIliaN Type 1 & 2 DiabetEs Disease Registry',\n",
              " 'DALY  disability-adjusted life-years',\n",
              " 'IQR  interquartile range',\n",
              " 'Paulo  Paulo, Brazil',\n",
              " 'CVD  cardiovascular disease',\n",
              " 'BMI  body mass index',\n",
              " 'T2DM  type-2 DM',\n",
              " 'target  target',\n",
              " 'CI  confidence interval',\n",
              " 'NCDs  non-communicable diseases',\n",
              " 'SD  standard deviation',\n",
              " 'Brazil  Brazil, 1990–2016',\n",
              " 'e-CRF  electronic CRF']"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "show_medical_abbreviation(en_ner_bc5cdr_md,case_report2)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "weFfQp3cKmaX",
        "outputId": "31caab27-e140-4559-cb01-1c357217eb3f"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['BINDER  BrazIliaN Type 1 & 2 DiabetEs Disease Registry',\n",
              " 'DALY  disability-adjusted life-years',\n",
              " 'IQR  interquartile range',\n",
              " 'Paulo  Paulo, Brazil',\n",
              " 'CVD  cardiovascular disease',\n",
              " 'BMI  body mass index',\n",
              " 'T2DM  type-2 DM',\n",
              " 'target  target',\n",
              " 'CI  confidence interval',\n",
              " 'NCDs  non-communicable diseases',\n",
              " 'SD  standard deviation',\n",
              " 'Brazil  Brazil, 1990–2016',\n",
              " 'e-CRF  electronic CRF']"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    }
  ]
}