k8s-delphin-parsing

It’s a high-performance architecture for ‘deep’ linguistic processing of documents with DELPHI-IN tools pyDelphin and ACE combined with a parallel processing using kubernetes and RQ.

Running the k8s-delphin-parsing on IBM Cloud

Prerequisites

It is important to ensure that you have IBM Cloud account, if you do not sign up here. You also need to instanciate IBM Cloud Kubernetes Service and IBM Cloud Object Storage. Create a IBM Cloud Object Storage service credential. Ensure that you have installed:

• kubectl
• ibmcloud cli
• helm

Deploying

First step is log in your IBM Cloud account and then set the cluster you created as the context for this session:

ibmcloud login -a cloud.ibm.com -r us-south -g Default --sso
ibmcloud ks cluster-config --cluster $cluster_name

Installing Tiller, the Helm server, in your cluster:

kubectl create serviceaccount tiller -n kube-system
kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller -n kube-system
helm init --service-account tiller

Installing the IBM Cloud Object Storage plug-in:

helm repo add iks-charts https://icr.io/helm/iks-charts
helm repo update
helm fetch --untar iks-charts/ibm-object-storage-plugin
helm plugin install ./ibm-object-storage-plugin/helm-ibmc
chmod 755 ~/.helm/plugins/helm-ibmc/ibmc.sh
helm ibmc install iks-charts/ibm-object-storage-plugin --name ibm-object-storage-plugin

Now fill the delphin-parsing-chart/values.yaml file. Values decscription:

• ace.options ACE Command-line Options
• ace.replicas number of ace instances
• credentials.apikey the API key that you can retrieve from your IBM Cloud Object Storage service credentials earlier
• credentials.guid the GUID of your IBM Cloud Object Storage service instance you can get that with ibmcloud resource service-instance <service_name> | grep GUI
• cos.inbucket the name of the existing bucket with the data that you want to process
• cos.outbucket the name of the existing bucket where the results will be save
• cos.endpoint see https://cloud.ibm.com/docs/services/cloud-object-storage/basics?topic=cloud-object-storage-endpoints#select-regions-and-endpoints
• cos.storageClassName find your service instance in the IBM Cloud dashboard and review the Class and Location of your existing bucket. Then, use the appropriate storage class.

Finally, deploy:

helm install ./delphin-parsing-chart/

Monitoring the parse

Get the public IP address for the worker node in the cluster.

ibmcloud ks workers --cluster $cluster_name

Open a browser and check out the app with the following URL: http://<IP_address>:30001.

Architecture

Authors

• Alexandre Rademaker
• Henrique Muniz

Lincese

TODO