diff --git a/docs/source/AdministratorGuide/Resources/storage.rst b/docs/source/AdministratorGuide/Resources/storage.rst index f1c3724e465..5e352402a8d 100644 --- a/docs/source/AdministratorGuide/Resources/storage.rst +++ b/docs/source/AdministratorGuide/Resources/storage.rst @@ -354,9 +354,7 @@ Multi Protocol with FTS External services like FTS requires pair of URLs to perform third party copy. This is implemented using the same logic as described above. There is however an extra step: once the common protocols between 2 SEs have been filtered, an extra loop filter is done to make sure that the selected protocol can be used as read from the source and as write to the destination. Finally, the URLs which are returned are not necessarily the url of the common protocol, but are the native urls of the plugin that can accept/generate the common protocol. For example, if the common protocol is gsiftp but one of the SE has only an SRM plugin, then you will get an srm URL (which is compatible with gsiftp). - -.. versionadded:: v7r1p37 - The FTS3Agent can now use plugins to influence the list of TPC protocols used. See :ref:`fts3` +The FTS3Agent can use plugins to influence the list of TPC protocols used. See :ref:`fts3` MultiHop support diff --git a/docs/source/AdministratorGuide/ServerInstallations/environment_variable_configuration.rst b/docs/source/AdministratorGuide/ServerInstallations/environment_variable_configuration.rst index 9c13dbd8f6a..2576f7e5216 100644 --- a/docs/source/AdministratorGuide/ServerInstallations/environment_variable_configuration.rst +++ b/docs/source/AdministratorGuide/ServerInstallations/environment_variable_configuration.rst @@ -58,7 +58,7 @@ DIRAC_NO_CFG If set to anything, cfg files on the command line must be passed to the command using the --cfg option. DIRAC_USE_JSON_ENCODE - Controls the transition to JSON serialization. See the information in :ref:`jsonSerialization` page (default=Yes since 8.1) + Controls the transition to JSON serialization (default=Yes since 9.0) DIRAC_ROOT_PATH If set, overwrites the value of DIRAC.rootPath. diff --git a/docs/source/AdministratorGuide/Systems/DataManagement/fts3.rst b/docs/source/AdministratorGuide/Systems/DataManagement/fts3.rst index 57d5c4d2a35..7822ce7c1a1 100644 --- a/docs/source/AdministratorGuide/Systems/DataManagement/fts3.rst +++ b/docs/source/AdministratorGuide/Systems/DataManagement/fts3.rst @@ -4,8 +4,6 @@ FTS3 support in DIRAC --------------------- -.. versionadded:: v6r20 - .. contents:: Table of contents :depth: 2 @@ -148,10 +146,6 @@ The FTS3Operation goes to ``Processed`` when all the files are in a final state, FTS3 Plugins ------------ -.. versionadded:: v7r1p37 - The ``FTS3Plugin`` option - - The ``FTS3Plugin`` option allows one to specify a plugin to alter some default choices made by the FTS3 system. These choices concern: * the list of third party protocols used @@ -165,8 +159,6 @@ This can be useful if you want to implement a matrix-like selection of protocols MultiHop support ---------------- -.. versionadded:: v7r3p21 - .. |trade| unicode:: U+2122 .. warning:: diff --git a/docs/source/AdministratorGuide/Systems/DataManagement/s3.rst b/docs/source/AdministratorGuide/Systems/DataManagement/s3.rst index 5766f4dfa1f..aa20311aa70 100644 --- a/docs/source/AdministratorGuide/Systems/DataManagement/s3.rst +++ b/docs/source/AdministratorGuide/Systems/DataManagement/s3.rst @@ -4,8 +4,6 @@ S3 support in DIRAC ------------------- -.. versionadded:: v7r0p19 - .. contents:: Table of contents :depth: 2 diff --git a/docs/source/AdministratorGuide/Systems/Transformation/index.rst b/docs/source/AdministratorGuide/Systems/Transformation/index.rst index a5373836c68..1fefb4646db 100644 --- a/docs/source/AdministratorGuide/Systems/Transformation/index.rst +++ b/docs/source/AdministratorGuide/Systems/Transformation/index.rst @@ -46,10 +46,7 @@ Within the TS a user can (for example): Disadvantages: -- For very large installations, the submission may be perceived as slow, since there is no use (not yet) of Parametric jobs. - - .. versionadded:: v6r20p3 - Bulk submission of jobs is working for the transformations, so job submission can be sped up considerably. +- For very large installations, the submission may be perceived as slow, since there is no use (not yet) of Parametric jobs. Bulk submission of jobs is working for the transformations, so job submission can be sped up considerably. Several improvements have been made in the TS to handle scalability, and extensibility issues. While the system structure remains intact, "tricks" like threading and caching have been extensively applied. @@ -691,9 +688,6 @@ Multi VO Configuration ---------------------- - -.. versionadded:: v6r20p5 - There are two possibilities to configure the agents of the transformation system for the use in a multi VO installation. - Use the same WorkflowTaskAgent and RequestTaskAgents for multiple VOs, no diff --git a/docs/source/AdministratorGuide/Tutorials/basicTutoSetup.rst b/docs/source/AdministratorGuide/Tutorials/basicTutoSetup.rst deleted file mode 100644 index df6ae742400..00000000000 --- a/docs/source/AdministratorGuide/Tutorials/basicTutoSetup.rst +++ /dev/null @@ -1,519 +0,0 @@ -.. _tuto_basic_setup: - -==================== -Basic Tutorial setup -==================== - -.. set highlighting to console input/output -.. highlight:: console - -Tutorial goal -============= - -The aim of the tutorial is to have a self contained DIRAC setup. You will be guided through the whole installation process both of the server part and the client part. -By the end of the tutorial, you will have: - -* a Configuration service, to serve other servers and clients -* a ComponentMonitoring service to keep track of other services and agents installed -* a SystemAdministrator service to manage the DIRAC installation in the future -* the WebApp, to allow for web interface access - -The setup you will have at the end is the base for all the other tutorials. - - -More links -========== - -* :ref:`server_installation` - -Basic requirements -================== - -This section is to be executed as ``root`` user. - -In this tutorial, we will use a freshly installed CC7 x86_64 virtual machine, with all the default options, except the hostname being ``dirac-tuto``. Make sure that the hostname of the machine is set to ``dirac-tuto``. Modify the ``HOSTNAME`` variable in the ``/etc/sysconfig/network`` file as such:: - - HOSTNAME=dirac-tuto - -Then reboot the machine and check the hostname. You should get the following output:: - - [root@dirac-tuto ~]# hostname - dirac-tuto - - -Machine setup -============= - -This section is to be executed as ``root`` user. - -Make sure that the machine can address itself using the ``dirac-tuto`` alias. Modify the ``/etc/hosts`` file as such:: - - 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 dirac-tuto - ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 dirac-tuto - - -------------------------- -Create the ``dirac`` user -------------------------- - -The user that will run the server will be ``dirac``. Set the password for that user to ``password``, -and ensure that files below ``/opt/dirac/`` belong to this user: - -.. literalinclude:: basicTutoSetup.sh - :language: bash - :start-after: # START add_dirac - :end-before: # END add_dirac - -------------- -Install runit -------------- - -The next step is to install ``runit``, which is responsible for supervising DIRAC processes - -First, install the `RPM `_: - -.. literalinclude:: basicTutoSetup.sh - :language: bash - :start-after: # START runit - :end-before: # END runit - - -Create the file ``/opt/dirac/sbin/runsvdir-start``, which is responsible for starting runit, with the following content: - -.. literalinclude:: basicTutoSetup.sh - :language: bash - :start-after: # START runsvdir-start - :end-before: # END runsvdir-start - :caption: /opt/dirac/sbin/runsvdir-start - -Then, edit the systemd ``runsvdir-start`` service to match the following: - -.. literalinclude:: basicTutoSetup.sh - :language: bash - :start-after: # START systemd-runsvdir - :end-before: # END systemd-runsvdir - :caption: /usr/lib/systemd/systemd/runsvdir-start.service - -make ``runsvdir-start`` executable and (re)start ``runsvdir``: - -.. literalinclude:: basicTutoSetup.sh - :language: bash - :start-after: # START restartrunsv - :end-before: # END restartrunsv - - -------------- -Install MySQL -------------- - -First of all, remove the existing (outdated) installation, and install all the necessary RPMs for MySQL 8.0: - -.. literalinclude:: basicTutoSetup.sh - :language: bash - :start-after: # START mysqlInstall - :end-before: # END mysqlInstall - -Start the mysql service, which will then initialize itself, and, among other things, create temporary password for the -mysql ``root`` account, which needs to be changed during the first login: - -.. literalinclude:: basicTutoSetup.sh - :language: bash - :start-after: # START mysqlStart - :end-before: # END mysqlStart - -To change the root password, create a ``mysqlSetup.sql`` file, which changes the password to a strong password, removes -a plugin to enforce the strong password (only for tutorial purposes, of course), and then sets the password to -``password``, which is easier to remember: - -.. literalinclude:: basicTutoSetup.sh - :language: mysql - :start-after: # START mysqlSetup - :end-before: # END mysqlSetup - :caption: mysqlSetup.sql - -Now get the temporary password from the ``/var/log/mysqld.log``, and change it using the ``mysqlSetup.sql`` file: - -.. literalinclude:: basicTutoSetup.sh - :language: bash - :start-after: # START mysqlInit - :end-before: # END mysqlInit - -Server installation -=================== - -This section is to be executed as ``dirac`` user - ------------------- -CA and certificate ------------------- - -DIRAC relies on TLS for securing its connections and for authorization and authentication. Since we are using a self contained installation, we will be using our own CA. There are a bunch of utilities that we will be using to generate the necessary files. - -We create a script ``setupCA`` to download utilities from the DIRAC repository and source ``utilities.sh``, and then -create the CA and certificates both for the server and the client: - -.. literalinclude:: basicTutoSetup.sh - :language: bash - :start-after: # START setupCA - :end-before: # END setupCA - :caption: setupCA - -Execute the script:: - - bash setupCA - -At this point, you should find: - -* The CA in ``/opt/dirac/etc/grid-security/certificates``:: - - [dirac@dirac-tuto caUtilities]$ ls /opt/dirac/etc/grid-security/certificates/ - 855f710d.0 ca.cert.pem - -* The host certificate (``hostcert.pem``) and key (``hostkey.pem``) in ``/opt/dirac/etc/grid-security``:: - - [dirac@dirac-tuto caUtilities]$ ls /opt/dirac/etc/grid-security/ - ca certificates hostcert.pem hostkey.pem openssl_config_host.cnf request.csr.pem - -* The user credentials for later in ``/opt/dirac/user/``:: - - [dirac@dirac-tuto caUtilities]$ ls /opt/dirac/user/ - client.key client.pem client.req openssl_config_user.cnf - --------------------- -Install DIRAC Server --------------------- - -This section is to be run as ``dirac`` user in its home folder:: - - sudo su dirac - cd ~ - -First we create the ``install.cfg`` file, which is used to tell the installation script we obtain in a moment what to -install and how to configure the server with the following content: - -.. literalinclude:: basicTutoSetup.sh - :language: bash - :start-after: # START install.cfg - :end-before: # END install.cfg - :caption: install.cfg - -Then we download the installer, make it executable, and run it with the ``install.cfg`` file (assuming the file is in -the user's home folder): - -.. literalinclude:: basicTutoSetup.sh - :language: bash - :start-after: # START installDirac - :end-before: # END installDirac - - -The output should look something like this:: - - Status of installed components: - - Name Runit Uptime PID - ================================================= - 1 Web_WebApp Run 4 24338 - 2 Configuration_Server Run 53 24142 - 3 Framework_ComponentMonitoring Run 36 24207 - 4 Framework_SystemAdministrator Run 20 24247 - - -You can verify that the components are running:: - - [dirac@dirac-tuto DIRAC]$ runsvstat /opt/dirac/startup/* - /opt/dirac/startup/Configuration_Server: run (pid 24142) 288 seconds - /opt/dirac/startup/Framework_ComponentMonitoring: run (pid 24207) 271 seconds - /opt/dirac/startup/Framework_SystemAdministrator: run (pid 24247) 255 seconds - /opt/dirac/startup/Web_WebApp: run (pid 24338) 239 seconds - - -The logs are to be found in ``/opt/dirac/runit/``, grouped by component. - -The installation created the file ``/opt/dirac/etc/dirac.cfg``. The content is the same as the ``install.cfg``, with the addition of the following:: - - DIRAC - { - Setup = MyDIRAC-Production - VirtualOrganization = tutoVO - Extensions = WebApp - Security - { - } - Configuration - { - Master = yes - Name = MyDIRAC-Production - Servers = dips://dirac-tuto:9135/Configuration/Server - } - } - LocalSite - { - Site = dirac-tuto - } - Systems - { - Databases - { - User = Dirac - Password = Dirac - Host = localhost - Port = 3306 - } - NoSQLDatabases - { - Host = dirac-tuto - Port = 9200 - } - } - -This part is used as configuration for all your services and agents that you will run. It contains two important information: - -* The database credentials -* The address of the configuration server: ``Servers = dips://dirac-tuto:9135/Configuration/Server`` - -The Configuration service will serve the content of the file ``/opt/dirac/etc/MyDIRAC-Production.cfg`` to every client, be it a service, an agent, a job, or an interactive client. The content looks like such:: - - DIRAC - { - Extensions = WebApp - VirtualOrganization = tutoVO - Configuration - { - Name = MyDIRAC-Production - Version = 2019-04-11 06:52:18.414086 - MasterServer = dips://dirac-tuto:9135/Configuration/Server - } - Setups - { - MyDIRAC-Production - { - Configuration = Production - Framework = Production - } - } - } - Registry - { - Users - { - ciuser - { - DN = /C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser - Email = adminUser@cern.ch - } - } - Groups - { - dirac_user - { - Users = ciuser - Properties = NormalUser - } - dirac_admin - { - Users = ciuser - Properties = ServiceAdministrator - Properties += CSAdministrator - Properties += JobAdministrator - Properties += FullDelegation - Properties += ProxyManagement - Properties += Operator - } - } - Hosts - { - dirac-tuto - { - DN = /C=ch/O=DIRAC/OU=DIRAC CI/CN=dirac-tuto - Properties = TrustedHost - Properties += CSAdministrator - Properties += JobAdministrator - Properties += FullDelegation - Properties += ProxyManagement - Properties += Operator - } - } - DefaultGroup = dirac_user - } - Operations - { - Defaults - { - EMail - { - Production = adminUser@cern.ch - Logging = adminUser@cern.ch - } - } - } - WebApp - { - Access - { - upload = TrustedHost - } - } - Systems - { - Framework - { - Production - { - Services - { - ComponentMonitoring - { - Port = 9190 - Authorization - { - Default = ServiceAdministrator - componentExists = authenticated - getComponents = authenticated - hostExists = authenticated - getHosts = authenticated - installationExists = authenticated - getInstallations = authenticated - updateLog = Operator - } - } - SystemAdministrator - { - Port = 9162 - Authorization - { - Default = ServiceAdministrator - storeHostInfo = Operator - } - } - } - URLs - { - ComponentMonitoring = dips://dirac-tuto:9190/Framework/ComponentMonitoring - SystemAdministrator = dips://dirac-tuto:9162/Framework/SystemAdministrator - } - FailoverURLs - { - } - Databases - { - InstalledComponentsDB - { - DBName = InstalledComponentsDB - Host = localhost - Port = 3306 - } - } - } - } - } - - -This configuration will be used for example by Services in order to: - -* know their configuration (for example the ``ComponentMonitoring`` Service will use everything under ``Systems/Framework/Production/Services/ComponentMonitoring`` ) -* Identify host and persons (``Registry`` section) - -Or by clients to get the URLs of given services (for example ``ComponentMonitoring = dips://dirac-tuto:9190/Framework/ComponentMonitoring``) - -Since this configuration is given as a whole to every client, you understand why no database credentials are in this file. Services and Agents running on the machine will have their configuration as a merge of what is served by the Configuration service and the ``/opt/dirac/etc/dirac.cfg``, and thus have access to these private information. - -The file ``/opt/dirac/bashrc`` is to be sourced whenever you want to use the server installation. - -Client installation -=================== - -Now we will create another linux account ``diracuser`` and another installation to be used as client - --------------------- -Setup client session --------------------- - -This section has to be ran as ``root`` - -Create an account ``diracuser`` with password ``password``, and add in its ``~/.globus/`` directory the user -certificate you created earlier: - -.. literalinclude:: basicTutoSetup.sh - :language: bash - :start-after: # START user_diracuser - :end-before: # END user_diracuser - - --------------------- -Install DIRAC client --------------------- - -This section has to be ran as ``diracuser`` in its home directory:: - - sudo su diracuser - cd - -We will do the installation in the ``~/DiracInstallation`` directory. For a client, the configuration is really minimal, -so we will just install the code and its dependencies. Create the structure, download the installer, and then install -the same version as for the server: - -.. literalinclude:: basicTutoSetup.sh - :language: bash - :start-after: # START installClient1 - :end-before: # END installClient1 - -In principle, your system administrator will have managed the CA for you. In this specific case, since we have our own CA, we will just link the client installation CA with the server one: - -.. literalinclude:: basicTutoSetup.sh - :language: bash - :start-after: # START installClient2 - :end-before: # END installClient2 - -The last step is to configure the client to talk to the proper configuration service. This is easily done by creating a ``~/DiracInstallation/etc/dirac.cfg`` file with the following content: - -.. literalinclude:: basicTutoSetup.sh - :language: bash - :caption: ~/DiracInstallation/etc/dirac.cfg - :start-after: # START dirac.cfg - :end-before: # END dirac.cfg - -You should now be able to get a proxy:: - - [diracuser@dirac-tuto DIRAC]$ source ~/DiracInstallation/bashrc - [diracuser@dirac-tuto DIRAC]$ dirac-proxy-init - Generating proxy... - Proxy generated: - subject : /C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser/CN=460648814 - issuer : /C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser - identity : /C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser - timeleft : 23:59:59 - DIRAC group : dirac_user - rfc : True - path : /tmp/x509up_u501 - username : ciuser - properties : NormalUser - - -And you can observe that the Configuration Service has served the client:: - - [diracuser@dirac-tuto DIRAC]$ grep ciuser /opt/dirac/runit/Configuration/Server/log/current - 2019-04-11 14:54:10 UTC Configuration/Server NOTICE: Executing action ([::1]:33394)[dirac_user:ciuser] RPC/getCompressedDataIfNewer() - 2019-04-11 14:54:10 UTC Configuration/Server NOTICE: Returning response ([::1]:33394)[dirac_user:ciuser] (0.00 secs) OK - --------------- -Use the WebApp --------------- - -This section is to be executed as ``diracuser``. - -First you need to convert your user certificate into a ``p12`` format (you will be prompt for a password, you can leave it empty):: - - cd ~/.globus/ - openssl pkcs12 -export -out certificate.p12 -inkey userkey.pem -in usercert.pem - -This will create the file ``~/.globus/certificate.p12``. - -Use your favorite browser, and add this certificate. - -You should be able to access the WebApp using the following address ``https://localhost:8443/DIRAC/`` - - -Conclusion -========== - -We have seen how to install a DIRAC server and client using a personal CA, and how to access the WebApp. Starting from here, you will be able to extend on further tutorials. diff --git a/docs/source/AdministratorGuide/Tutorials/basicTutoSetup.sh b/docs/source/AdministratorGuide/Tutorials/basicTutoSetup.sh deleted file mode 100644 index 5eec514e3ff..00000000000 --- a/docs/source/AdministratorGuide/Tutorials/basicTutoSetup.sh +++ /dev/null @@ -1,257 +0,0 @@ -#!/bin/bash - -set -euv - -# this doesn't work on cern SL6 machines in openstack, there we still have `hostname` -# Sed the /etc/hosts file to append dirac-tuto -sed -i "s/\(127.*\)/\1 dirac-tuto/" /etc/hosts -sed -i "s/\(::1.*\)/\1 dirac-tuto/" /etc/hosts - -# on cern SL6 image on openstack just use HOSTNAME -HOSTNAME=`hostname` - - -# avoid calling passwd, which might be the cern passwd in /usr/sue/bin -# START add_dirac -adduser -s /bin/bash -d /home/dirac dirac || echo "User dirac already exists." -echo password | /usr/bin/passwd --stdin dirac -mkdir -p /opt/dirac/sbin -chown -R dirac:dirac /opt/dirac/ -# END add_dirac - - -#localinstall does not error when rpm is already installed -# START runit -yum localinstall -y http://diracproject.web.cern.ch/diracproject/rpm/runit-2.1.2-1.el7.cern.x86_64.rpm -# END runit - -cat > /opt/dirac/sbin/runsvdir-start <<'EOF' -# START runsvdir-start -#!/bin/bash -cd /opt/dirac -RUNSVCTRL='/sbin/runsvctrl' -chpst -u dirac $RUNSVCTRL d /opt/dirac/startup/* -killall runsv svlogd -RUNSVDIR='/sbin/runsvdir' -exec chpst -u dirac $RUNSVDIR -P /opt/dirac/startup 'log: DIRAC runsv' -# END runsvdir-start -EOF - -# runsvdir-start can fail to start/restart if it does not contain the shebang at the top of the file -# we remove the first line of the script -sed -i '1d' /opt/dirac/sbin/runsvdir-start - -cat > /lib/systemd/system/runsvdir-start.service < mysqlSetup.sql < install.cfg < setupCA < installDirac < InstallDiracClient < ~diracuser/DiracInstallation/etc/dirac.cfg < /tmp/dummy.txt - -Now create a file called ``/tmp/testSE.py``, with the following content - -.. code-block:: python - - import DIRAC - - DIRAC.initialize() # Initialize configuration - - localFile = '/tmp/dummy.txt' - lfn = '/tutoVO/myFirstFile.txt' - - from DIRAC.Resources.Storage.StorageElement import StorageElement - - - se = StorageElement('StorageElementOne') - - print "Putting file" - print se.putFile({lfn: localFile}) - - print "Listing directory" - print se.listDirectory('/tutoVO') - - print "Getting file" - print se.getFile(lfn, '/tmp/') - - print "Removing file" - print se.removeFile(lfn) - - print "Listing directory" - print se.listDirectory('/tutoVO') - - - - -This script uploads ``/tmp/dummy.txt`` on the StorageElement as ``myFirstFile.txt``, list the directory, downloads the uploaded file and removes it from the StorageElement. The output should be something like that:: - - [diracuser@dirac-tuto ~]$ python /tmp/testSE.py - Putting file - {'OK': True, 'Value': {'Successful': {'/tutoVO/myFirstFile.txt': 10}, 'Failed': {}}} - Listing directory - {'OK': True, 'Value': {'Successful': {'/tutoVO': {'Files': {'myFirstFile.txt': {'Accessible': True, 'Migrated': 0, 'Unavailable': 0, 'Lost': 0, 'Exists': True, 'Cached': 1, 'Checksum': '166203b7', 'Mode': 420, 'File': True, 'Directory': True, 'TimeStamps': (1555342476, 1555342476, 1555342476), 'Type': 'File', 'Size': 10}}, 'SubDirs': {}}}, 'Failed': {}}} - Getting file - {'OK': True, 'Value': {'Successful': {'/tutoVO/myFirstFile.txt': 10}, 'Failed': {}}} - Removing file - {'OK': True, 'Value': {'Successful': {'/tutoVO/myFirstFile.txt': True}, 'Failed': {}}} - Listing directory - {'OK': True, 'Value': {'Successful': {'/tutoVO': {'Files': {}, 'SubDirs': {}}}, 'Failed': {}}} - -The list of files within ``tmp`` should also contain ``dummy.txt`` as well as ``myFirstFile.txt``. - -Adding a second DIRAC SE -======================== - -It is often interesting to have a second SE. - -As ``dirac`` user, create a new directory:: - - [dirac@dirac-tuto ~]$ mkdir /opt/dirac/storageElementTwo/ - -Now the rest is to be installed with ``diracuser`` and a proxy with ``dirac_admin`` group. - -We need another StorageElement service. However, it has to have a different *name*, *Port* and *BasePath* than the first one, so we will just call this service ``StorageElementTwo``:: - - [diracuser@dirac-tuto ~]$ dirac-admin-sysadmin-cli --host dirac-tuto - Pinging dirac-tuto... - [dirac-tuto]$ install service DataManagement StorageElementTwo -m StorageElement -p Port=9147 -p BasePath=/opt/dirac/storageElementTwo/ - Loading configuration template /home/diracuser/DIRAC/DIRAC/DataManagementSystem/ConfigTemplate.cfg - Adding to CS service DataManagement/StorageElementTwo - service DataManagement_StorageElementTwo is installed, runit status: Run - - -Using the WebApp, add the new StorageElement definition in the ``/Resources/StorageElements`` section:: - - StorageElementTwo - { - BackendType = DISET - DIP - { - Host = dirac-tuto - Port = 9147 - Protocol = dips - Path = /DataManagement/StorageElementTwo - Access = remote - } - } - - -In order to test it, just re-use ``/tmp/testSE.py``, replacing ``StorageElementOne`` with ``StorageElementTwo`` diff --git a/docs/source/AdministratorGuide/Tutorials/dmsWithTs.rst b/docs/source/AdministratorGuide/Tutorials/dmsWithTs.rst deleted file mode 100644 index 2ce51e9f109..00000000000 --- a/docs/source/AdministratorGuide/Tutorials/dmsWithTs.rst +++ /dev/null @@ -1,281 +0,0 @@ -========================================================= -Large Scale DataManagement with the Transformation System -========================================================= - -Pre-Requisite -============= - -You should: - -* have a machine setup as described in :ref:`tuto_basic_setup` -* have installed two DIRAC SE using the tutorial (:ref:`tuto_install_dirac_se`). -* have installed the DFC using the tutorial (:ref:`tuto_install_dfc`). -* have followed the tutorial on identity management (:ref:`tuto_managing_identities`) -* have installed the RMS using the tutorial (:ref:`tuto_install_rms`) -* have installed the TS using the tutorial (:ref:`tuto_install_ts`) - - -Tutorial Goal -============= - -The aim of the tutorial is to demonstrate how large scale data management operations (removals, replications, etc.) can -be achieved using the Transformation System. By the end of the tutorial, you will be able to: - -* Submit simple transformation for manipulating a given list of files -* Have transformations automatically fed thanks to metadata -* Write your own plugin for the TransformationSystem - -The transformations can be monitored and controlled with the ``Transformation Monitor`` in the ``WebApp`` when you use -the ``dirac_prod`` group. - - -More Links -========== - -* :ref:`adminTS` - - -Creating a Transformation with a DIRAC Command -============================================== - -.. highlight:: console - -This section is to be performed as ``diracuser`` with a proxy in ``dirac_prod`` group. - -First we need to create some files and upload them to ``StorageElementOne``:: - - [diracuser@dirac-tuto ~]$ for ID in {1..10}; do echo "MyContent $ID" > File_${ID} ; dirac-dms-add-file /tutoVO/data/Trans_01/File_${ID} File_${ID} StorageElementOne ; done - -Then we create the list of LFNs we just uploaded:: - - [diracuser@dirac-tuto ~]$ dirac-dms-find-lfns Path=/tutoVO/data/Trans_01 > trans01.lfns - -The easiest way to create a transformation to replicate files is by using the :ref:`dirac-transformation-replication` command:: - - [diracuser@dirac-tuto ~]$ dirac-transformation-replication 0 StorageElementTwo --Plugin Broadcast --Enable - Created transformation NNN - Successfully created replication transformation - -This created transformation with the unique transformation ID *NNN* (e.g., 1). - -By default this transformation uses *Metadata* information to obtain the input files using the -``InputDataAgent``. Instead we can also just add files manually with the :ref:`dirac-transformation-add-files` command and using the list we created previously, -replace NNN by the ID of the transformation that was just created:: - - [diracuser@dirac-tuto ~]$ dirac-transformation-add-files NNN trans01.lfns - Successfully added 10 files - - -Now we have to wait until the ``TransformationAgent`` runs again and creates a *Task* for each of the files. Once the -tasks are created, the ``RequestTaskAgent`` creates a request out of each task, which is then processed in the -``RequestExecutingAgent`` of the RMS. - - -Creating a Transformation with a Script -======================================= - - -In this step we want to remove the replicas of our files from ``StorageElementOne``, for this purpose we have to write a -script that creates a removal transformation: - -.. code-block:: python - :caption: createRemoval.py - :linenos: - - #!/bin/env python - - # set up the DIRAC configuration, parse command line arguments - from DIRAC import gLogger, S_OK, S_ERROR - from DIRAC.Core.Base.Script import Script - Script.parseCommandLine() - - from DIRAC.TransformationSystem.Client.Transformation import Transformation - - # create a Transformation instance - myTrans = Transformation() - - # transformation names need to be unique - uniqueIdentifier = "Trans1" - transformationName = "RemoveReplicas_%s" % uniqueIdentifier - myTrans.setTransformationName(transformationName) - - # describe what the transformation will do - description = "Remove replicas from StorageElementOne" - myTrans.setDescription(description) - myTrans.setLongDescription(description) - - # 'Replication' type means we do data management - myTrans.setType('Removal') - - # group transformations that belong together, these can be selected in the WebApp - transGroup = "myRemovals" - myTrans.setTransformationGroup(transGroup) - - # groupSize defines the number of files each request will treat - groupSize = 1 - myTrans.setGroupSize(groupSize) - - # the transformation plugin defines which input files are treated, and how they are grouped, for example - plugin = 'Broadcast' - myTrans.setPlugin(plugin) - - # the 'body' of the transformation, defines a list of Request Operations - # that are executed in order for each file added to the transformation - targetSE = 'StorageElementOne' - transBody = [("RemoveReplica", {"TargetSE": targetSE})] - - myTrans.setBody(transBody) - - res = myTrans.setTargetSE(targetSE) - if not res['OK']: - gLogger.error("TargetSE not valid: %s" % res['Message']) - exit(1) - - res = myTrans.addTransformation() - if not res['OK']: - gLogger.error("Failed to add the transformation: %s" % res['Message']) - exit(1) - - # now activate the transformation - myTrans.setStatus('Active') - myTrans.setAgentType('Automatic') - transID = myTrans.getTransformationID()['Value'] - gLogger.notice('Created RemoveReplica transformation: %r' % transID) - exit(0) - -When we execute the script, the transformation is created with the ID MMM (e.g. 2):: - - [diracuser@dirac-tuto ~]$ python createRemoval.py - Created transformation MMM - Created RemoveReplica transformation: MMML - -To remove a replica from StorageElementOne, we just have to add files to this transformation:: - - [diracuser@dirac-tuto ~]$ dirac-transformation-add-files MMM /tutoVO/data/Trans_01/File_10 - Successfully added 1 files - -And then wait again for the ``TransformationAgent``, ``RequestTaskAgent``, ``RequestExecutingAgent`` chain to complete. - -After a short while, you should see that the folder ``/opt/dirac/storageElementOne/tutoVO/data/Trans_01/``, no longer -contains ``File_10``. - - -Using Metadata Queries to Add Files to Transformations -====================================================== - -Adding files manually to transformations can be useful, but if we want to automatically add files to transformations we -can make use of metadata queries in combination with the ``InputDataAgent``, which executes the queries and adds new -files to the corresponding transformation. - -To benefit from metadata query, we first have to create a metadata key, and add the key to a directory. These -operations can be done with the ``dirac-dms-filecatalog-cli``:: - - [diracuser@dirac-tuto ~]$ dirac-dms-filecatalog-cli - Starting FileCatalog client - - File Catalog Client $Revision: 1.17 $Date: - - FC:/$ ls -l - drwxrwxr-x 0 ciuser dirac_user 0 2019-05-06 14:30:36 tutoVO - -In the ``dirac-dms-filecatalog-cli``, like in the other DIRAC CLIs you can use ``help`` and ``help `` to see -information about the available commands. - -Initially there are no metadata keys defined:: - - FC:/$ meta show - FileMetaFields : {} - DirectoryMetaFields : {} - -We now create in integer directory metadata called ``TransformationID``:: - - FC:/$ meta index -d TransformationID int - Added metadata field TransformationID of type int - FC:/$ meta show - FileMetaFields : {} - DirectoryMetaFields : {'TransformationID': 'INT'} - -Let's add the ``TransformationID=1`` to the files we uploaded earlier:: - - FC:/$ meta set /tutoVO/data/Trans_01/ TransformationID 1 - /tutoVO/data/Trans_01 {'TransformationID': '1'} - -You can see the metadata set for a given diretory with the ``meta get`` command, and you can use the ``find`` command -inside the ``dirac-dms-filecatalog-cli`` to search for files with metadata:: - - FC:/$ meta get /tutoVO/data/Trans_01/ - !TransformationID : 1 - FC:/$ find / TransformationID=1 - Query: {'TransformationID': 1} - /tutoVO/data/Trans_01/File_1 - [..snip..] - /tutoVO/data/Trans_01/File_9 - QueryTime 0.00 sec - -Now let us create another directory, and set a different metadata value, before we create another transformation -including an inputdata query:: - - FC:/$ mkdir /tutoVO/data/Trans_02/ - Successfully created directory: /tutoVO/data/Trans_02 - FC:/$ meta set /tutoVO/data/Trans_02/ TransformationID 2 - /tutoVO/data/Trans_02 {'TransformationID': '2'} - FC:/$ meta get /tutoVO/data/Trans_02/ - !TransformationID : 2 - -Now upload some files to this folder:: - - [diracuser@dirac-tuto ~]$ for ID in {1..10}; do echo "MyContent $ID" > File_${ID} ; dirac-dms-add-file /tutoVO/data/Trans_02/File_${ID} File_${ID} StorageElementOne ; done - -We can also use the command ``dirac-dms-find-lfns`` to search for files with given metadata:: - - [diracuser@dirac-tuto ~]$ dirac-dms-find-lfns Path=/ TransformationID=2 - - -Now we create a transformation, which uses the metadata to pick up the files:: - - [diracuser@dirac-tuto ~]$ dirac-transformation-replication 2 StorageElementTwo --Plugin=Broadcast --Enable - Created transformation LLL - Successfully created replication transformation - -In fact the command ``dirac-transformation-replication`` already uses metadata, the first argument is the value for the -``TransformationID`` metadata. Now we have to wait for the ``InputDataAgent``, ``TransformationAgent``, -``RequestTaskAgent``, ``RequestExecutingAgent`` chain to run its course. - -In the log file of the ``InputDataAgent`` in ``/opt/dirac/pro/runit/Transformation/InputDataAgent/log/current`` -eventually this line should appear:: - - Transformation/InputDataAgent INFO: 10 files returned for transformation LLL from the metadata catalog - - -You may add some more files to ``/tutoVO/data/Trans_02/`` and see them appearing in your transformation:: - - [diracuser@dirac-tuto ~]$ for ID in {11..20}; do echo "MyContent $ID" > File_${ID} ; dirac-dms-add-file /tutoVO/data/Trans_02/File_${ID} File_${ID} StorageElementOne ; done - - -InputDataQuery in the Script ----------------------------- - -To add the metadata query functionality to our ``createRemoval.py`` script from above, we just need to insert a couple -of lines - -.. code-block:: python - :lineno-start: 44 - - metaQuery = {'TransformationID': 2} - myTrans.setInputMetaQuery(metaQuery) - - ... - -Adapt the script by inserting the lines and changing the ``uniqueIdentifier`` and execute it:: - - [diracuser@dirac-tuto ~]$ python createRemoval.py - Created transformation JJJ - Created RemoveReplica transformation: JJJL - -Conclusion -========== - -You now have all the knowledge to perform DataManagement in DIRAC with the TransformationSystem. - -To learn how to extend the system by creating new transformation plugins, please see how to -:ref:`dev-ts-transformationagent-plugins` and :ref:`dev-ts-body-plugins` diff --git a/docs/source/AdministratorGuide/Tutorials/index.rst b/docs/source/AdministratorGuide/Tutorials/index.rst deleted file mode 100644 index 2e905d6157d..00000000000 --- a/docs/source/AdministratorGuide/Tutorials/index.rst +++ /dev/null @@ -1,20 +0,0 @@ -.. _dirac-admin-tutorials: - -============================= -DIRAC Administrator tutorials -============================= - -Each of this tutorial is a step by step guide. - -.. toctree:: - :maxdepth: 1 - :numbered: - - basicTutoSetup - managingIdentities - diracSE - installDFC - installRMS - installTS - dmsWithTs - installWMS diff --git a/docs/source/AdministratorGuide/Tutorials/installDFC.rst b/docs/source/AdministratorGuide/Tutorials/installDFC.rst deleted file mode 100644 index bcf24170bd8..00000000000 --- a/docs/source/AdministratorGuide/Tutorials/installDFC.rst +++ /dev/null @@ -1,133 +0,0 @@ -.. _tuto_install_dfc: - -================================= -Installing the DIRAC File Catalog -================================= - -.. set highlighting to console input/output -.. highlight:: console - -Pre-requisite -============= - -You should: - -* have a machine setup as described in :ref:`tuto_basic_setup` -* be able to install dirac components -* have installed a DIRAC SE using the tutorial (:ref:`tuto_install_dirac_se`). - -Tutorial goal -============= - -The aim of the tutorial is to install the DIRAC FileCatalog (DFC) -By the end of the tutorial, you will be able to do all sort of simple Data Management operations. - -More links -========== - -More information can be found at the following places: - -* Introduction to DataManagement: :ref:`data-management-system` -* Catalog resource definition :ref:`resourcesCatalog` -* How-to datamanagement for user :ref:`howto_user_dms` - -Installing the DFC -================== - -This section is to be executed as ``diracuser`` with a proxy with ``dirac_admin`` group. - -The DFC is no different than any other DIRAC service with a database. The installation step are thus very simple:: - - [diracuser@dirac-tuto ~]$ dirac-admin-sysadmin-cli --host dirac-tuto - Pinging dirac-tuto... - [dirac-tuto]$ install db FileCatalogDB - Adding to CS DataManagement/FileCatalogDB - Database FileCatalogDB from DIRAC/DataManagementSystem installed successfully - [dirac-tuto]$ install service DataManagement FileCatalog - Loading configuration template /home/diracuser/DIRAC/DIRAC/DataManagementSystem/ConfigTemplate.cfg - Adding to CS service DataManagement/FileCatalog - service DataManagement_FileCatalog is installed, runit status: Run - - -Adding the FileCatalog resource -=============================== - -In order to be used as a FileCatalog by clients, the DFC needs to be declared. This happens in two places: - -* ``/Resources/FileCatalogs/``: in this section, you define how to access the catalog -* ``/Operations/Defaults/Services/Catalogs/``: in this section, you define how to use the catalog (for example read/write) - - -Since we have only one catalog, we will use it as ``Read-Write`` and as ``Master``. - -Using the WebApp (group ``dirac_admin``), add the following in ``/Resources/FileCatalogs/`` (all options to defaults):: - - FileCatalog - { - } - - -Using the WebApp, add the following in ``/Operations/Defaults/Services/Catalogs``:: - - FileCatalog - { - AccessType = Read-Write - Status = Active - Master = True - } - -From this moment onward, the catalog is totally usable. - -Test the catalog -================ - -Since we have a StorageElement at our disposal, we can use the standard ``dirac-dms-*`` script. - -First, let us create a file and then "put it on the grid":: - - - [diracuser@dirac-tuto ~]$ echo "Hello" > /tmp/world.txt - [diracuser@dirac-tuto ~]$ dirac-dms-add-file /tutoVO/user/c/ciuser/world.txt /tmp/world.txt StorageElementOne - - Uploading /tutoVO/user/c/ciuser/world.txt - Successfully uploaded file to StorageElementOne - - -Now, let's check its replicas and metadata:: - - [diracuser@dirac-tuto ~]$ dirac-dms-lfn-replicas /tutoVO/user/c/ciuser/world.txt - LFN StorageElement URL - ===================================================== - /tutoVO/user/c/ciuser/world.txt StorageElementOne dips://dirac-tuto:9148/DataManagement/StorageElement/tutoVO/user/c/ciuser/world.txt - - [diracuser@dirac-tuto ~]$ dirac-dms-lfn-metadata /tutoVO/user/c/ciuser/world.txt - {'Failed': {}, - 'Successful': {'/tutoVO/user/c/ciuser/world.txt': {'Checksum': '078b01ff', - 'ChecksumType': 'Adler32', - 'CreationDate': datetime.datetime(2019, 4, 16, 9, 5, 58), - 'FileID': 1L, - 'GID': 1, - 'GUID': '09F7E02F-1290-BE21-1DA7-07A266F153B3', - 'Mode': 509, - 'ModificationDate': datetime.datetime(2019, 4, 16, 9, 5, 58), - 'Owner': 'ciuser', - 'OwnerGroup': 'dirac_admin', - 'Size': 6L, - 'Status': 'AprioriGood', - 'UID': 1}}} - -Note that these metadata are those registered in the catalog (which hopefully should match the physical one !) - -We can also check all the user files that belong to us on the grid:: - - [diracuser@dirac-tuto ~]$ dirac-dms-user-lfns - Will search for files in /tutoVO/user/c/ciuser - /tutoVO/user/c/ciuser: 1 files, 0 sub-directories - 1 matched files have been put in tutoVO-user-c-ciuser.lfns - [diracuser@dirac-tuto ~]$ cat tutoVO-user-c-ciuser.lfns - /tutoVO/user/c/ciuser/world.txt - -Finally, let's remove the file:: - - [diracuser@dirac-tuto ~]$ dirac-dms-remove-files /tutoVO/user/c/ciuser/world.txt - Successfully removed 1 files diff --git a/docs/source/AdministratorGuide/Tutorials/installRMS.rst b/docs/source/AdministratorGuide/Tutorials/installRMS.rst deleted file mode 100644 index 946b138fca2..00000000000 --- a/docs/source/AdministratorGuide/Tutorials/installRMS.rst +++ /dev/null @@ -1,138 +0,0 @@ -.. _tuto_install_rms: - -======================================= -Installing the RequestManagement System -======================================= - -.. set highlighting to console input/output -.. highlight:: console - -Pre-requisite -============= - -You should: - -* have a machine setup as described in :ref:`tuto_basic_setup` -* be able to install dirac components -* have installed two DIRAC SE using the tutorial (:ref:`tuto_install_dirac_se`). -* have installed the DFC (:ref:`tuto_install_dfc`) -* have followed the tutorial on identity management (:ref:`tuto_managing_identities`) - -Tutorial goal -============= - -The aim of the tutorial is to install the RequestManagement system components and to use it to perform a simple replication of file. - -More links -========== - -More information can be found at the following places: - -* :ref:`data-management-system` -* :ref:`requestManagementSystem` - -Installing the RMS -================== - -This section is to be executed as ``diracuser`` with a proxy with ``dirac_admin`` group. - -The RMS needs the ``ReqManager`` service and the ``RequestExecutingAgent`` to work (you may want to add the ``CleanReqDBAgent`` if you scale...). - -The RMS is no different than any other DIRAC system. The installation step are thus very simple:: - - [diracuser@dirac-tuto ~]$ dirac-admin-sysadmin-cli --host dirac-tuto - Pinging dirac-tuto... - [dirac-tuto]$ add instance RequestManagement Production - Adding RequestManagement system as Production self.instance for MyDIRAC-Production self.setup to dirac.cfg and CS - RequestManagement system instance Production added successfully - [dirac-tuto]$ restart * - All systems are restarted, connection to SystemAdministrator is lost - [dirac-tuto]$ install db ReqDB - MySQL root password: - Adding to CS RequestManagement/ReqDB - Database ReqDB from DIRAC/RequestManagementSystem installed successfully - [dirac-tuto]$ install service RequestManagement ReqManager - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/RequestManagementSystem/ConfigTemplate.cfg - Adding to CS service RequestManagement/ReqManager - service RequestManagement_ReqManager is installed, runit status: Run - [dirac-tuto]$ install agent RequestManagement RequestExecutingAgent - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/RequestManagementSystem/ConfigTemplate.cfg - Adding to CS agent RequestManagement/RequestExecutingAgent - agent RequestManagement_RequestExecutingAgent is installed, runit status: Run - [dirac-tuto]$ quit - - -By default, the installation of the ``RequestExecutingAgent`` will configure it with a whole bunch of default Operations possible. You can see that in the Agent configuration in ``/Systems/RequestManagement/Production/Agents/RequestExecutingAgent/OperationHandlers`` - - -Testing the RMS -=============== - -This section is to be executed with a proxy with `dirac_data` group. - -The test we are going to do consists in transferring a file from one storage element to another, using the RequestExecutingAgent. - -First, let's add a file:: - - [diracuser@dirac-tuto ~]$ echo "My Test File" > /tmp/myTestFile.txt - [diracuser@dirac-tuto ~]$ dirac-dms-add-file /tutoVO/user/c/ciuser/myTestFile.txt /tmp/myTestFile.txt StorageElementOne - - Uploading /tutoVO/user/c/ciuser/myTestFile.txt - Successfully uploaded file to StorageElementOne - - -We can see that our file is indeed in the ``StorageElementOne``:: - - [diracuser@dirac-tuto ~]$ dirac-dms-lfn-replicas /tutoVO/user/c/ciuser/myTestFile.txt - LFN StorageElement URL - ========================================================== - /tutoVO/user/c/ciuser/myTestFile.txt StorageElementOne dips://dirac-tuto:9148/DataManagement/StorageElement/tutoVO/user/c/ciuser/myTestFile.txt - -Let's replicate it to ``StorageElementTwo`` using the RMS:: - - [diracuser@dirac-tuto ~]$ dirac-dms-replicate-and-register-request myFirstRequest /tutoVO/user/c/ciuser/myTestFile.txt StorageElementTwo - Request 'myFirstRequest' has been put to ReqDB for execution. - RequestID(s): 8 - You can monitor requests' status using command: 'dirac-rms-request ' - - -The Request has a name (``myFirstRequest``) that we chose, but also an ID, returned by the system (here ``8``). The ID is guaranteed to be unique, while the name is not, so it is recommended to use the ID when you interact with the RMS. You can see the status of your Request, using its name or ID:: - - [diracuser@dirac-tuto ~]$ dirac-rms-request myFirstRequest - Request name='myFirstRequest' ID=8 Status='Waiting' - Created 2019-04-23 14:37:05, Updated 2019-04-23 14:37:05, NotBefore 2019-04-23 14:37:05 - Owner: '/C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser', Group: dirac_data - [0] Operation Type='ReplicateAndRegister' ID=8 Order=0 Status='Waiting' - TargetSE: StorageElementTwo - Created 2019-04-23 14:37:05, Updated 2019-04-23 14:37:05 - [01] ID=2 LFN='/tutoVO/user/c/ciuser/myTestFile.txt' Status='Waiting' Checksum='1e750431' - - [diracuser@dirac-tuto ~]$ dirac-rms-request 8 - Request name='myFirstRequest' ID=8 Status='Waiting' - Created 2019-04-23 14:37:05, Updated 2019-04-23 14:37:05, NotBefore 2019-04-23 14:37:05 - Owner: '/C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser', Group: dirac_data - [0] Operation Type='ReplicateAndRegister' ID=8 Order=0 Status='Waiting' - TargetSE: StorageElementTwo - Created 2019-04-23 14:37:05, Updated 2019-04-23 14:37:05 - [01] ID=2 LFN='/tutoVO/user/c/ciuser/myTestFile.txt' Status='Waiting' Checksum='1e750431' - - -You can here clearly see that the Request consists of one ``ReplicateAndRegister`` operation (which does what it says) targeting the LFN ``/tutoVO/user/c/ciuser/myTestFile.txt``. The RequestExecutingAgent will pick up the request and execute it. And shortly you should be able to see it done:: - - [diracuser@dirac-tuto ~]$ dirac-rms-request 8 - Request name='myFirstRequest' ID=8 Status='Done' - Created 2019-04-23 14:37:05, Updated 2019-04-23 14:37:29, NotBefore 2019-04-23 14:37:05 - Owner: '/C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser', Group: dirac_data - [0] Operation Type='ReplicateAndRegister' ID=8 Order=0 Status='Done' - TargetSE: StorageElementTwo - Created 2019-04-23 14:37:05, Updated 2019-04-23 14:37:29 - [01] ID=2 LFN='/tutoVO/user/c/ciuser/myTestFile.txt' Status='Done' Checksum='1e750431' - - [diracuser@dirac-tuto ~]$ dirac-dms-lfn-replicas /tutoVO/user/c/ciuser/myTestFile.txt - LFN StorageElement URL - ========================================================== - /tutoVO/user/c/ciuser/myTestFile.txt StorageElementTwo dips://dirac-tuto:9147/DataManagement/StorageElementTwo/tutoVO/user/c/ciuser/myTestFile.txt - StorageElementOne dips://dirac-tuto:9148/DataManagement/StorageElement/tutoVO/user/c/ciuser/myTestFile.txt - - -Conclusion -========== - -You now have an RMS in place, which is the base for all the asynchronous operations in DIRAC. This is used for big scale operations, failover, or even more ! diff --git a/docs/source/AdministratorGuide/Tutorials/installTS.rst b/docs/source/AdministratorGuide/Tutorials/installTS.rst deleted file mode 100644 index 3379d6c4b44..00000000000 --- a/docs/source/AdministratorGuide/Tutorials/installTS.rst +++ /dev/null @@ -1,138 +0,0 @@ -.. _tuto_install_ts: - -=================================== -Installing the TransformationSystem -=================================== - -.. set highlighting to console input/output -.. highlight:: console - -Pre-Requisite -============= - -You should: - -* have a machine setup as described in :ref:`tuto_basic_setup` -* have installed two DIRAC SE using the tutorial (:ref:`tuto_install_dirac_se`). -* have installed the DFC using the tutorial (:ref:`tuto_install_dfc`). -* have followed the tutorial on identity management (:ref:`tuto_managing_identities`) -* have installed the RMS using the tutorial (:ref:`tuto_install_rms`) - - -Tutorial Goal -============= - -The aim of the tutorial is to install the Transformation system components and to use it to perform an automatic replication. - - -More Links -========== - -* :ref:`adminTS` -* See the options for services and agents in code documentation of the :mod:`~DIRAC.TransformationSystem` - -Installing the TransformationSystem -=================================== - -.. highlight:: console - -This section is to be executed as ``diracuser`` with a proxy int the ``dirac_admin`` group. - -The Transformation System(TS) needs the ``TransformationManager`` service and the ``TransformationAgent``, ``InputDataAgent``, -``RequestTaskAgent`` to work for data management purposes. The ``WorkflowTaskAgent`` is needed to submit jobs. -Finally the ``TransformationCleaning`` cleans up if transformations are finished. As the agents need to run one after -the other we set the PollingTime to 30 seconds to reduce the waiting time once we create transformations. - -The TS is no different than any other DIRAC system. The installation steps are thus very simple:: - - [diracuser@dirac-tuto ~]$ dirac-admin-sysadmin-cli --host dirac-tuto - Pinging dirac-tuto... - [dirac-tuto]$ add instance Transformation Production - Adding Transformation system as Production self.instance for MyDIRAC-Production self.setup to dirac.cfg and CS - Transformation system instance Production added successfully - [dirac-tuto]$ restart * - All systems are restarted, connection to SystemAdministrator is lost - [dirac-tuto]$ install db TransformationDB - MySQL root password: - Adding to CS Transformation/TransformationDB - Database TransformationDB from DIRAC/TransformationSystem installed successfully - [dirac-tuto]$ install service Transformation TransformationManager - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/TransformationSystem/ConfigTemplate.cfg - Adding to CS service Transformation/TransformationManager - service Transformation_TransformationManager is installed, runit status: Run - [dirac-tuto]$ install agent Transformation TransformationAgent -p PollingTime=30 - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/TransformationSystem/ConfigTemplate.cfg - Adding to CS agent Transformation/TransformationAgent - agent Transformation_TransformationAgent is installed, runit status: Run - [dirac-tuto]$ install agent Transformation InputDataAgent -p PollingTime=30 - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/TransformationSystem/ConfigTemplate.cfg - Adding to CS agent Transformation/InputDataAgent - agent Transformation_InputDataAgent is installed, runit status: Run - [dirac-tuto]$ install agent Transformation WorkflowTaskAgent -p PollingTime=30 -p MonitorTasks=True -p MonitorFiles=True - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/TransformationSystem/ConfigTemplate.cfg - Adding to CS agent Transformation/WorkflowTaskAgent - agent Transformation_WorkflowTaskAgent is installed, runit status: Run - [dirac-tuto]$ install agent Transformation RequestTaskAgent -p PollingTime=30 -p MonitorTasks=True -p MonitorFiles=True - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/TransformationSystem/ConfigTemplate.cfg - Adding to CS agent Transformation/RequestTaskAgent - agent Transformation_RequestTaskAgent is installed, runit status: Run - -Add a ProductionManagement Group -================================ - -We create a new group ``dirac_prod``, which will be used to manage transformations - -Using the ``Configuration Manager`` application in the WebApp, create a new section ``dirac_prod`` in ``/Registry/Groups``:: - - Users = ciuser - Properties = ProductionManagement, NormalUser - AutoUploadProxy = True - - -After restarting the ``ProxyManager``, you should now be able to get a proxy belonging to the ``dirac_prod`` group that -will be automatically uploaded. - -The ``ProductionManagement`` property allows users in the group to access and change all transformations. There is also -a ``ProductionSharing`` property to only allow access to transformations in the same group and ``ProductionUser`` to -only allow users to access their own transformations. - -Add a ProdManager Shifter -========================= - -Using the ``Configuration Manager`` application in the WebApp, create a new shifter ``ProdManager`` in the -``/Operations/Defaults/Shifter`` section:: - - ProdManager - { - User = ciuser - Group = dirac_prod - } - - - -Add a Sites which the StorageElements belong to -=============================================== - -Using the ``Configuration Manager`` application in the WebApp, create a new section ``Sites`` in ``/Resources``, which -contains a *Grid* with two *Sites*, to which the two SEs are associated:: - - Sites - { - MyGrid - { - MyGrid.Site1.uk - { - SE = StorageElementOne - } - MyGrid.Site2.de - { - SE = StorageElementTwo - } - } - } - - -Conclusion -========== - -You now have a Transformation System in place, which is the base for all automatic operations in DIRAC. diff --git a/docs/source/AdministratorGuide/Tutorials/installWMS.rst b/docs/source/AdministratorGuide/Tutorials/installWMS.rst deleted file mode 100644 index d4fb341d9ae..00000000000 --- a/docs/source/AdministratorGuide/Tutorials/installWMS.rst +++ /dev/null @@ -1,335 +0,0 @@ -.. _tuto_install_wms: - - -======================================== -Installing the WorkloadManagement System -======================================== - -.. set highlighting to console input/output -.. highlight:: console - -Pre-Requisite -============= - -You should: - -* have a machine setup as described in :ref:`tuto_basic_setup` -* have installed two DIRAC SE using the tutorial (:ref:`tuto_install_dirac_se`). -* have followed the tutorial on identity management (:ref:`tuto_managing_identities`) -* have installed the TS using the tutorial (:ref:`tuto_install_ts`) - -Tutorial Goal -============= - -The aim of the tutorial is to install the WorkloadManagement system components and to use them to generate and submit a simple job. - - -More Links -========== - -* :ref:`WMS` -* Information about the types and options of the :ref:`Computing Elements` -* Information about the user jobs using the DIRAC API: :ref:`user jobs` - -Installing the WorkloadManagementSystem -======================================= - -.. highlight:: console - -This section is to be executed as ``diracuser`` with the ``dirac_admin`` proxy (reminder: ``dirac-proxy-init -g dirac_admin``). - -Basically, the WorkloadManagement System (WMS) needs the ``SiteDirector`` agent to install pilots on Computing Elements (CEs) as well as -different services and agents such as the ``JobManager``, the ``JobMonitoring`` and the ``Matcher`` to manage the jobs and their status. -The executors are used to check the jobs and schedule them on Task Queues. - -The WMS is no different than any other DIRAC system. The installation steps are thus very simple:: - - [diracuser@dirac-tuto ~]$ dirac-proxy-init -g dirac_admin - [diracuser@dirac-tuto ~]$ dirac-admin-sysadmin-cli --host dirac-tuto - Pinging dirac-tuto... - [dirac-tuto]> add instance WorkloadManagement Production - Adding WorkloadManagement system as Production self.instance for MyDIRAC-Production self.setup to dirac.cfg and CS WorkloadManagement system instance Production added successfully - [dirac-tuto]> restart * - All systems are restarted, connection to SystemAdministrator is lost - [dirac-tuto]> install db JobDB - MySQL root password: - Adding to CS WorkloadManagement/JobDB - Database JobDB from DIRAC/WorkloadManagementSystem installed successfully - [dirac-tuto]> install db JobLoggingDB - MySQL root password: - Adding to CS WorkloadManagement/JobLoggingDB - Database JobLoggingDB from DIRAC/WorkloadManagementSystem installed successfully - [dirac-tuto]> install db PilotAgentsDB - MySQL root password: - Adding to CS WorkloadManagement/PilotAgentsDB - Database PilotAgentsDB from DIRAC/WorkloadManagementSystem installed successfully - [dirac-tuto]> install db SandboxMetadataDB - MySQL root password: - Adding to CS WorkloadManagement/SandboxMetadataDB - Database SandboxMetadataDB from DIRAC/WorkloadManagementSystem installed successfully - [dirac-tuto]> install db TaskQueueDB - MySQL root password: - Adding to CS WorkloadManagement/TaskQueueDB - Database TaskQueueDB from DIRAC/WorkloadManagementSystem installed successfully - [dirac-tuto]> install service WorkloadManagement PilotManager - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg - Adding to CS service WorkloadManagement/PilotManager - service WorkloadManagement_PilotManager is installed, runit status: Run - [dirac-tuto]> install service WorkloadManagement JobManager - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg - Adding to CS service WorkloadManagement/JobManager - service WorkloadManagement_JobManager is installed, runit status: Run - [dirac-tuto]> install service WorkloadManagement JobMonitoring - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg - Adding to CS service WorkloadManagement/JobMonitoring - service WorkloadManagement_JobMonitoring is installed, runit status: Run - [dirac-tuto]> install service WorkloadManagement JobStateUpdate - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg - Adding to CS service WorkloadManagement/JobStateUpdate - service WorkloadManagement_JobStateUpdate is installed, runit status: Run - [dirac-tuto]> install service WorkloadManagement Matcher - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg - Adding to CS service WorkloadManagement/Matcher - service WorkloadManagement_Matcher is installed, runit status: Run - [dirac-tuto]> install service WorkloadManagement OptimizationMind - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg - Adding to CS service WorkloadManagement/OptimizationMind - service WorkloadManagement_OptimizationMind is installed, runit status: Run - [dirac-tuto]> install service WorkloadManagement SandboxStore - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg - Adding to CS service WorkloadManagement/SandboxStore - service WorkloadManagement_SandboxStore is installed, runit status: Run - [dirac-tuto]> install service WorkloadManagement WMSAdministrator - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg - Adding to CS service WorkloadManagement/WMSAdministrator - service WorkloadManagement_WMSAdministrator is installed, runit status: Run - [dirac-tuto]> install service Framework BundleDelivery - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/Framework/ConfigTemplate.cfg - Adding to CS service WorkloadManagement/BundleDelivery - service WorkloadManagement_BundleDelivery is installed, runit status: Run - [dirac-tuto]> install service Framework Monitoring - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/Framework/ConfigTemplate.cfg - Adding to CS service WorkloadManagement/Monitoring - service WorkloadManagement_BundleDelivery is installed, runit status: Run - [dirac-tuto]> install agent WorkloadManagement SiteDirector - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg - Adding to CS service WorkloadManagement/SiteDirector - agent WorkloadManagement_SiteDirector is installed, runit status: Run - [dirac-tuto]> install agent WorkloadManagement JobCleaningAgent - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg - Adding to CS service WorkloadManagement/JobCleaningAgent - agent WorkloadManagement_JobCleaningAgent is installed, runit status: Run - [dirac-tuto]> install agent WorkloadManagement PilotStatusAgent - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg - Adding to CS service WorkloadManagement/PilotStatusAgent - agent WorkloadManagement_PilotStatusAgent is installed, runit status: Run - [dirac-tuto]> install agent WorkloadManagement StalledJobAgent - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg - Adding to CS service WorkloadManagement/StalledJobAgent - agent WorkloadManagement_StalledJobAgent is installed, runit status: Run - [dirac-tuto]> install executor WorkloadManagement Optimizers - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg - Adding to CS service WorkloadManagement/Optimizers - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg - Adding to CS service WorkloadManagement/JobPath - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg - Adding to CS service WorkloadManagement/JobSanity - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg - Adding to CS service WorkloadManagement/InputData - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/WorkloadManagementSystem/ConfigTemplate.cfg - Adding to CS service WorkloadManagement/JobScheduling - executor WorkloadManagement_Optimizers is installed, runit status: Run - [dirac-tuto]> restart WorkloadManagement * - -Create and submit a job -======================= - -This section is to be executed as ``diracuser`` with the ``dirac_user`` proxy (reminder: ``dirac-proxy-init``). - -Create a Python script to generate and submit a simple job. Copy paste the following lines into a new file called ``job.py`` - -.. code-block:: python - - #!/bin/env python - # Magic lines necessary to activate the DIRAC Configuration System - # to discover all the required services - from DIRAC.Core.Base.Script import Script - Script.parseCommandLine(ignoreErrors=True) - from DIRAC.Interfaces.API.Job import Job - from DIRAC.Interfaces.API.Dirac import Dirac - - j = Job() - dirac = Dirac() - - j.setName('MyFirstJob') - j.setJobGroup('MyJobs') - - # Specify CPU requirements - j.setCPUTime(21600) - - # Specify the log level of the job execution: INFO (default), DEBUG, VERBOSE - j.setLogLevel('DEBUG') - - # Executabe and arguments can be given in one call - j.setExecutable('echo', arguments='Hello world!') - - result = dirac.submitJob(j) - if not result['OK']: - print("ERROR:", result['Message']) - else: - print(result['Value']) - -This script creates a new job called ``MyFirstJob`` and aims at executing ``echo "Hello World!"``. The output should be something like that:: - - [diracuser@dirac-tuto ~]$ python job.py - - [diracuser@dirac-tuto ~]$ dirac-wms-job-status - JobID= Status=Waiting; MinorStatus=Pilot Agent Submission; Site=ANY; - -As we have not defined any CE yet, the job cannot run and remains ``Waiting``. - -Adding a CE -=========== - -First, as ``root``, we create a new user ``diracpilot`` that is going to simulate an SSH Computing Element on ``dirac-tuto``:: - - adduser -s /bin/bash -d /home/diracpilot diracpilot - echo password | /usr/bin/passwd --stdin diracpilot - -As ``diracuser``, connect to ``diracpilot`` through SSH a first time to initialize the connection and make sure everything works:: - - ssh diracpilot@dirac-tuto - -Then, as ``diracuser`` with the ``dirac_admin`` proxy, we need to define a CE in a ``/Resources/Sites//`` section of the configuration file using the WebApp (create the sections if necessary):: - - Resources - { - Sites - { - MyGrid - { - MyGrid.Site1.uk - { - CE = dirac-tuto - CEs - { - dirac-tuto - { - CEType = SSH - SSHHost = dirac-tuto - SSHUser = diracpilot - SSHPassword = password - SSHType = ssh - Queues - { - queue - { - CPUTime = 40000 - MaxTotalJobs = 5 - MaxWaitingJobs = 10 - BatchError = /home/diracpilot/localsite/error - ExecutableArea = /home/diracpilot/localsite/submission - RemoveOutput = True - } - } - } - } - } - } - } - } - -We set the type of the CE, ``SSH`` in our case, as well as the required parameters to access the Element. -Then we configure the queue that is going to receive the jobs. A queue corresponds to a set of Worker Nodes in practice. - -Note: make sure the ``CPUTime`` of the queue is above the ``CPUTime`` of the job, else the job will not be scheduled to run on this Worker Node. - -Configuring the pilots -====================== - -A job is not able to run directly on a Worker Node and needs to be executed by a pilot that has the knowledge of its environment and knows how to run jobs within it. -The pilot is the first job to be deployed on a Worker Node and it installs and configures DIRAC and asks for pending jobs in Task Queues that would match the environment of the Worker Node. Add the following lines in the ``/Operations/MyDIRAC-Production`` section using the WebApp:: - - Pilot - { - Version = v7r0p36 - CheckVersion = False - Command - { - Test = GetPilotVersion - Test += CheckWorkerNode - Test += InstallDIRAC - Test += ConfigureBasics - Test += ConfigureCPURequirements - Test += ConfigureArchitecture - Test += CheckCECapabilities - Test += LaunchAgent - } - GenericPilotGroup = dirac_user - GenericPilotUser = ciuser - pilotFileServer = dirac-tuto:8443 - } - -We pass our credentials information to the pilot so that it can interact with DIRAC as it needs to execute the commands defined in ``Commands``. -Only a small script called ``pilotWrapper`` is directly passed to the CE, most of the files used by the pilot will be downloaded from ``pilotFileServer`` during the script execution. -These files can be uploaded and updated at each commit done to the configuration, we just need to create the directory that is going to contain the files required by the pilot and add the information within the configuration. First, add the option below to the configuration, in the ``/WebApp`` section:: - - StaticDirs = pilot - -As ``dirac``, create the pilot repository that will contain all the pilot files that will be updated whenever a CS update is triggered:: - - mkdir -p /opt/dirac/webRoot/www/pilot - - -..warning:: Do not put the Pilot configuration in ``Operations/Defaults``, DIRAC would not be able to get it. - -Configuring the Sandbox -======================= - -We need to define a Sandbox to pass input files related to the job to the Worker Node and then to get the results of the execution. -A Sandbox is represented as a StorageElement and can be installed in this way. As ``diracuser`` with the ``dirac_admin`` proxy, executes :: - - [diracuser@dirac-tuto ~]$ dirac-admin-sysadmin-cli --host dirac-tuto - Pinging dirac-tuto... - [dirac-tuto]> install service DataManagement ProductionSandboxSE -m StorageElement -p Port=9146 -p BasePath=/opt/dirac/storage/sandboxes - -Then the following lines have to be added to the configuration in the ``/Resources/StorageElements`` section using the WebApp:: - - ProductionSandboxSE - { - BackendType = DISET - DIP - { - Host = dirac-tuto - Port = 9146 - Protocol = dips - Path = /DataManagement/ProductionSandboxSE - Access = remote - } - } - -The Storage Element is then used by the ``SandboxStore`` service. -If it is not defined (it should in practice), add the following option in ``Systems/WorkloadManagement/Production/Services/SandboxStore``:: - - LocalSE = ProductionSandboxSE - -Make the Site available for receiving jobs -========================================== - -By default, the Site previously created is not allowed to receive any job from DIRAC. Execute the following command to add it to the list of available Sites:: - - [diracuser@dirac-tuto ~]$ dirac-admin-allow-site MyGrid.Site1.uk "test" -E False - Site MyGrid.Site1.uk status is set to Active - -Finally restart the WorkloadManagement system to apply the configuration changes to the components:: - - [diracuser@dirac-tuto ~]$ dirac-admin-sysadmin-cli --host dirac-tuto - Pinging dirac-tuto... - [dirac-tuto]> restart WorkloadManagement * - -After a moment we should get a result performing these commands:: - - [diracuser@dirac-tuto ~]$ dirac-wms-job-status - JobID= Status=Done; MinorStatus=Execution Complete; Site=MyGrid.Site1.uk; - [diracuser@dirac-tuto ~]$ dirac-wms-job-get-output - Job output sandbox retrieved in /home/diracuser// diff --git a/docs/source/AdministratorGuide/Tutorials/managingIdentities.rst b/docs/source/AdministratorGuide/Tutorials/managingIdentities.rst deleted file mode 100644 index 2ed8ef68302..00000000000 --- a/docs/source/AdministratorGuide/Tutorials/managingIdentities.rst +++ /dev/null @@ -1,153 +0,0 @@ -.. _tuto_managing_identities: - -=================== -Managing identities -=================== - -.. set highlighting to console input/output -.. highlight:: console - -Pre-requisite -============= - -You should: - -* have a machine setup as described in :ref:`tuto_basic_setup` -* be able to install dirac components - - -Tutorial goal -============= - -Very quickly when using DIRAC, you will need to manage identities of people and their proxies. This is done with the ``ProxyManager`` service and with several configuration options. -In this tutorial, we will install the ``ProxyManager``, create a new group, and define some ``Shifter``. - - -Further reading -=============== - -* :ref:`compAuthNAndAutZ` -* :ref:`manageAuthNAndAuthZ` - -Installing the ``ProxyManager`` -=============================== - -This section is to be performed as ``diracuser`` with ``dirac_admin`` group proxy:: - - [diracuser@dirac-tuto ~]$ source ~/DiracInstallation/bashrc - [diracuser@dirac-tuto ~]$ dirac-proxy-init -g dirac_admin - - -The ``ProxyManager`` will host delegated proxies of the users. As any other service, it is very easy to install with the ``dirac-admin-sysadmin-cli``:: - - [diracuser@dirac-tuto ~]$ dirac-admin-sysadmin-cli -H dirac-tuto - -And then in the CLI:: - - [dirac-tuto]$ install db ProxyDB - MySQL root password: - Adding to CS Framework/ProxyDB - Database ProxyDB from DIRAC/FrameworkSystem installed successfully - [dirac-tuto]$ install service Framework ProxyManager - Loading configuration template /home/diracuser/DiracInstallation/DIRAC/FrameworkSystem/ConfigTemplate.cfg - Adding to CS service Framework/ProxyManager - service Framework_ProxyManager is installed, runit status: Run - - - -.. note:: The ProxyDB contains sensitive information. For production environment, it is recommended that you keep this in a separate database with different credentials and strict access control. - - -Testing the ``ProxyManager`` -============================ - -The simplest way to test it is to upload your user proxy:: - - [diracuser@dirac-tuto ~]$ dirac-proxy-init - Generating proxy... - Uploading proxy for dirac_user... - Proxy generated: - subject : /C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser/CN=6045995638 - issuer : /C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser - identity : /C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser - timeleft : 23:59:59 - DIRAC group : dirac_user - rfc : True - path : /tmp/x509up_u501 - username : ciuser - properties : NormalUser - - Proxies uploaded: - DN | Group | Until (GMT) - /C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser | dirac_user | 2020/04/09 14:43 - -As you can see, the ProxyDB now contains a delegated proxy for the ``ciuser`` with the group ``dirac_user``. - -If you use a proxy with the ``ProxyManagement`` permission, like the ``dirac_admin`` group has, you can retrieve proxies stored in the DB:: - - [diracuser@dirac-tuto ~]$ dirac-proxy-init -g dirac_admin - Generating proxy... - Proxy generated: - subject : /C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser/CN=5472309786 - issuer : /C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser - identity : /C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser - timeleft : 23:59:59 - DIRAC group : dirac_admin - rfc : True - path : /tmp/x509up_u501 - username : ciuser - properties : ServiceAdministrator, CSAdministrator, JobAdministrator, FullDelegation, ProxyManagement, Operator - [diracuser@dirac-tuto ~]$ dirac-admin-get-proxy ciuser dirac_user - Proxy downloaded to /home/diracuser/proxy.ciuser.dirac_user - - -Adding a new group -================== - -Groups are useful to manage permissions and separate activities. For example, we will create a new group ``dirac_data``, and decide to use that group for all the data centrally managed. - -Using the ``Configuration Manager`` application in the WebApp using the ``dirac_admin`` group, create a new section ``dirac_data`` in ``/Registry/Groups``:: - - Users = ciuser - Properties = NormalUser - AutoUploadProxy = True - -You should now be able to get a proxy belonging to the `dirac_data` group that will be automatically uploaded:: - - [diracuser@dirac-tuto ~]$ dirac-proxy-init -g dirac_data - Generating proxy... - Uploading proxy for dirac_data... - Proxy generated: - subject : /C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser/CN=6009266000 - issuer : /C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser - identity : /C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser - timeleft : 23:59:59 - DIRAC group : dirac_data - rfc : True - path : /tmp/x509up_u501 - username : ciuser - properties : NormalUser - - Proxies uploaded: - DN | Group | Until (GMT) - /C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser | dirac_data | 2020/04/09 14:43 - /C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser | dirac_user | 2020/04/09 14:43 - - -.. note:: if you get ``Unauthorized query ( 1111 : Unauthorized query)``, it means the ProxyManager has not yet updated its internal configuration. Just restart it to save time, or wait. - - -Adding a Shifter -================ - -``Shifter`` is basically a role, to which you associate a given proxy, for example ``DataManager`` (it could be anything). You can then tell your Components to use the ``DataManager`` identity to perform certain operations (at random: data management operations ? :-) ). - -Using the ``Configuration Manager`` application in the WebApp, create a new section ``Shifter`` in ``/Operations/Defaults``:: - - DataManager - { - User = ciuser - Group = dirac_data - } - -You can now force any agent (don't, unless you know what you are doing) to use a proxy instead of the host certificate by specifying the ``shifterProxy`` option. diff --git a/docs/source/AdministratorGuide/index.rst b/docs/source/AdministratorGuide/index.rst index b9d1ef861c2..6bfeb791c3a 100644 --- a/docs/source/AdministratorGuide/index.rst +++ b/docs/source/AdministratorGuide/index.rst @@ -22,8 +22,6 @@ This administration documentation refers to the "Core" DIRAC project. HowTo/index Configuration/index Resources/index - Tutorials/index - technologyPreviews Systems/index .. toctree:: diff --git a/docs/source/AdministratorGuide/technologyPreviews.rst b/docs/source/AdministratorGuide/technologyPreviews.rst deleted file mode 100644 index d025322f720..00000000000 --- a/docs/source/AdministratorGuide/technologyPreviews.rst +++ /dev/null @@ -1,37 +0,0 @@ -=================== -Technology Previews -=================== - - -When new technologies are introduced within DIRAC, there are cases when we allow this technology not to be used. -The reason might be that it is not completely mature yet, or that it can be disturbing. These technologies are toggled by either CS flags or environment variables. -They are meant to stay optional for a couple of releases, and then they become the default. -This page keeps a list of such technologies. - -.. _jsonSerialization: - -JSON Serialization -================== - -We aim at replacing the DEncode serialization over the network with json serialization. In order to be smooth, the transition needs to happen in several steps: - -* DISET -> DISET: as it is now. We encode DISET, and decode DISET -* DISET -> DISET, JSON: we send DISET, try to decode with DISET, fallback to JSON if it fails. This step is to make sure all clients will be ready to understand JSON whenever it comes. -* JSON -> DISET, JSON: we send JSON, attempt to decode with DISET, fallback to JSON if it fails. This step is to make sure that if we still have some old clients lying around, we are still able to understand them. -* JSON -> JSON: final step, goodbye DISET. - -The changes from one stage to the next is controlled by environment variables, and it can go to your own pace: - -* ``DIRAC_USE_JSON_ENCODE``: Sends JSON instead of DISET. - -The last stage (JSON only) will be the default of the following release, so before upgrading you will have to go through the previous steps. - -HTTPS Services -============== - -The aim is to replace the DISET services with HTTPS services. The changes should be almost transparent for users/admins. However, because it is still very much in the state of preview, we do not yet describe how/what to change. If you really want to play around, please check :ref:`httpsTornado`. - -OAuth2 authorization -===================== - -The main idea is to start using access tokens to communicate with DIRAC systems and third-party services. Tokens can be delivered to DIRAC using Identity Providers such as Indigo IaM (WLCG) and EGI CheckIn. As a result of this integration, a new component type, named :ref:`apis` was developed. diff --git a/docs/source/DeveloperGuide/Overview/index.rst b/docs/source/DeveloperGuide/Overview/index.rst index 89be55b3fb9..c93b5856728 100644 --- a/docs/source/DeveloperGuide/Overview/index.rst +++ b/docs/source/DeveloperGuide/Overview/index.rst @@ -112,7 +112,6 @@ Interfaces control the ongoing tasks. The Web interfaces are based on the DIRAC Web Portal framework which ensures secure access to the system service using X509 certificates loaded into the user browsers. -.. versionadded:: 8.0 DIRAC also has the ability to implement additional interfaces based on the http protocol, see :ref:`apis`. DIRAC Framework diff --git a/docs/source/UserGuide/GettingStarted/GettingUserIdentity/index.rst b/docs/source/UserGuide/GettingStarted/GettingUserIdentity/index.rst index 2cb043e3ec6..76603499119 100644 --- a/docs/source/UserGuide/GettingStarted/GettingUserIdentity/index.rst +++ b/docs/source/UserGuide/GettingStarted/GettingUserIdentity/index.rst @@ -54,9 +54,6 @@ If another non-default user group is needed, the command becomes:: where ``user_group`` is the desired DIRAC group name for which the user is entitled. -.. versionadded:: 8.0 - added the possibility to generate proxy with new `dirac-login` command, use *--help* switch for more information. E.g.: dirac-login - Token authorization ------------------- diff --git a/docs/source/UserGuide/Tutorials/JobManagementAdvanced/index.rst b/docs/source/UserGuide/Tutorials/JobManagementAdvanced/index.rst index 0b70d79c050..a939114b72a 100644 --- a/docs/source/UserGuide/Tutorials/JobManagementAdvanced/index.rst +++ b/docs/source/UserGuide/Tutorials/JobManagementAdvanced/index.rst @@ -345,8 +345,6 @@ using the "setNumberOfProcessors" method of the API:: Calling ``Job().setNumberOfProcessors()``, with a value bigger than 1, will translate into adding also the "MultiProcessor" tag to the job description. -.. versionadded:: v6r20p5 - Users can specify in the job descriptions NumberOfProcessors and WholeNode parameters, e.g.:: NumberOfProcessors = 16; diff --git a/docs/source/UserGuide/Tutorials/ManagingUserCredentials/index.rst b/docs/source/UserGuide/Tutorials/ManagingUserCredentials/index.rst index 47e2d61e417..14c1bd8528f 100644 --- a/docs/source/UserGuide/Tutorials/ManagingUserCredentials/index.rst +++ b/docs/source/UserGuide/Tutorials/ManagingUserCredentials/index.rst @@ -86,13 +86,13 @@ Creating a user proxy The switches below will create a proxy of group "dirac_user" (if defined) and will securely upload such proxy to the DIRAC proxy store (ProxyManager), from where it could later be downloaded:: - dirac-proxy-init --group dirac_user --upload + dirac-proxy-init --group dirac_user The additional "--debug" switch (alias of "-ddd") can be used for debugging purposes, and its output would end up being similar to the following:: - $ dirac-proxy-init --group dirac_user --upload --debug + $ dirac-proxy-init --group dirac_user --debug Generating proxy... Enter Certificate password: Contacting CS... @@ -130,9 +130,6 @@ Creating a user proxy VOMS extension corresponding to the DIRAC group if the gLite UI environment is available. This proxy is stored in the local "/tmp/" directory, as shown. - If the gLite UI environment is not available, the VOMS extensions will not be loaded into the proxy. - This is not a serious problem, still most of the operations will be possible. - Getting the proxy information @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ diff --git a/src/DIRAC/DataManagementSystem/Agent/FTS3Agent.py b/src/DIRAC/DataManagementSystem/Agent/FTS3Agent.py index 70e95042e90..2fef08f18cf 100644 --- a/src/DIRAC/DataManagementSystem/Agent/FTS3Agent.py +++ b/src/DIRAC/DataManagementSystem/Agent/FTS3Agent.py @@ -1,6 +1,4 @@ """ -.. versionadded:: v6r20 - FTS3Agent implementation. It is in charge of submitting and monitoring all the transfers. It can be duplicated. diff --git a/src/DIRAC/RequestManagementSystem/private/RequestValidator.py b/src/DIRAC/RequestManagementSystem/private/RequestValidator.py index 17cdaf9e102..ca0fce41ad2 100644 --- a/src/DIRAC/RequestManagementSystem/private/RequestValidator.py +++ b/src/DIRAC/RequestManagementSystem/private/RequestValidator.py @@ -282,11 +282,9 @@ def setAndCheckRequestOwner(request, remoteCredentials): ownershipCheck = request.OwnerDN if not hasattr(request, "OwnerDN") and hasattr( request, "Owner" - ): # Requests created by v8.1 client for v8.1 servers + ): # Requests created by v9 client for v9 servers ownershipCheck = request.Owner - if hasattr(request, "OwnerDN") and hasattr( - request, "Owner" - ): # Requests created by v8.0 client for v8.1 servers + if hasattr(request, "OwnerDN") and hasattr(request, "Owner"): # Requests created by v8.0 client for v9 servers ownershipCheck = request.Owner # ## diff --git a/src/DIRAC/TransformationSystem/Agent/RequestTaskAgent.py b/src/DIRAC/TransformationSystem/Agent/RequestTaskAgent.py index 713f5d89535..45e1a34785b 100644 --- a/src/DIRAC/TransformationSystem/Agent/RequestTaskAgent.py +++ b/src/DIRAC/TransformationSystem/Agent/RequestTaskAgent.py @@ -12,8 +12,6 @@ * The options *SubmitTasks*, *MonitorTasks*, *MonitorFiles*, and *CheckReserved* need to be assigned any non-empty value to be activated -* .. versionadded:: v6r20p5 - It is possible to run the RequestTaskAgent without a *shifterProxy* or *ShifterCredentials*, in this case the credentials of the authors of the transformations are used to submit the jobs to the RMS. This enables the use of diff --git a/src/DIRAC/TransformationSystem/Agent/WorkflowTaskAgent.py b/src/DIRAC/TransformationSystem/Agent/WorkflowTaskAgent.py index 9853469820a..ba44240b647 100755 --- a/src/DIRAC/TransformationSystem/Agent/WorkflowTaskAgent.py +++ b/src/DIRAC/TransformationSystem/Agent/WorkflowTaskAgent.py @@ -11,8 +11,6 @@ The options *SubmitTasks*, *MonitorTasks*, *MonitorFiles*, and *CheckReserved* need to be assigned any non-empty value to be activated -* .. versionadded:: v6r20p5 - It is possible to run the RequestTaskAgent without a *shifterProxy* or *ShifterCredentials*, in this case the credentials of the authors of the transformations are used to submit the jobs to the RMS. This enables the use of