Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Zentral ansible role? #25

Open
juju4 opened this issue Mar 9, 2017 · 15 comments
Open

Zentral ansible role? #25

juju4 opened this issue Mar 9, 2017 · 15 comments

Comments

@juju4
Copy link

juju4 commented Mar 9, 2017

Hello

Thanks a lot for your work.
I look at the repo, the other ones from same user and ansible galaxy but I didn't find the role(s) to setup zentral.
I'm interested in the tool but I would go more to other cloud like digitalocean or Azure.
Roles should make it easy but where?

Thanks a lot!
@juju4
Copy link
Author

juju4 commented Mar 31, 2017

Just say if busy, in the plan, not coming or whatever.
I might work on one but prefer to avoid duplicate.

Thanks

@np5
Copy link
Member

np5 commented Apr 1, 2017

Sorry for the late reply.

We don't have a Zentral ansible role.

@juju4
Copy link
Author

juju4 commented Apr 9, 2017

Pushed a first review https://github.com/juju4/ansible-zentral
but architecture is a bit more complicated than I expected with image. lot of dependencies :)

I would appreciate a bit of guidance, especially as it seems, I need to setup a specific daemon for simplesamlphp. right?
for now, all services fails on configuration but I think it's because zentral.conf module is not available.

Thanks

@np5
Copy link
Member

np5 commented Apr 10, 2017

Quick tips and review.

  • set the ZENTRAL_CONF_DIR variable to point to the dir containing base.json.

  • Try to start with a simplier configuration.

    • Postgres
    • Rabbitmq
    • Elasticsearch
    • Nginx
    • Zentral gunicorn
    • Zentral workers

  • Spin-up a zentral-all-in-one image on GCE or AWS. This is a good example of a simple deployment on Ubuntu 16.04, with some management scripts to update zentral, get the certificates, configure the admin user…

@juju4
Copy link
Author

juju4 commented Apr 16, 2017

It's not that simple having many dependencies and daemons.
Was already playing a lot with the vagrant all in one but as image, they hide complexity and partly the understanding of the innerwork of zentral.
Documented post-install is done properly I think. That's more what is in image and less documented which is harder

On the setup, I installed all daemons except promotheus but at this point zentral gunicorn or workers are not launching correctly. I also added redis because I thought it was needed but doesn't seem so.

$ /usr/bin/python server/manage.py runserver runworkers
Traceback (most recent call last):
  File "server/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 316, in execute
    settings.INSTALLED_APPS
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 53, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 41, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 97, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/var/_zentral/zentral/server/server/settings.py", line 19, in <module>
    from zentral.conf import saml2_idp_metadata_file, settings as zentral_settings
  File "/var/_zentral/zentral/zentral/conf/__init__.py", line 3, in <module>
    from .utils import find_conf_file, load_config_file
  File "/var/_zentral/zentral/zentral/conf/utils.py", line 37
    raise ImproperlyConfigured("{} error in file {}".format(filetype, filepath)) from None
                                                                                    ^
SyntaxError: invalid syntax

$ python -mjson.tool ../conf/start/zentral/base.json
[OK]
$ /usr/local/bin/gunicorn --chdir /var/_zentral/zentral/server -b 0.0.0.0:8000 -w 4 --access-logfile /var/log/zentral/gunicorn.log --error-logfile /var/log/zentral/gunicorn.err server.wsgi
Traceback (most recent call last):
  File "/usr/local/bin/gunicorn", line 11, in <module>
    sys.exit(run())
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 74, in run
    WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 203, in run
    super(Application, self).run()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 72, in run
    Arbiter(self).run()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 231, in run
    self.halt(reason=inst.reason, exit_status=inst.exit_status)
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 344, in halt
    self.stop()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 393, in stop
    time.sleep(0.1)
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 244, in handle_chld
    self.reap_workers()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 524, in reap_workers
    raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

If I launch manually from zentral all-in-one vagrant the zentral_worker

vagrant@vagrant:/home/zentral/app/releases/current$ sudo -H -u zentral /home/zentral/app/venv/bin/python server/manage.py runworkers
Traceback (most recent call last):
  File "server/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/zentral/app/venv/lib/python3.5/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/home/zentral/app/venv/lib/python3.5/site-packages/django/core/management/__init__.py", line 316, in execute
    settings.INSTALLED_APPS
  File "/home/zentral/app/venv/lib/python3.5/site-packages/django/conf/__init__.py", line 53, in __getattr__
    self._setup(name)
  File "/home/zentral/app/venv/lib/python3.5/site-packages/django/conf/__init__.py", line 41, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/zentral/app/venv/lib/python3.5/site-packages/django/conf/__init__.py", line 97, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 665, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/home/zentral/app/releases/2017.03.02-19.36.11-master-b98066/server/server/settings.py", line 19, in <module>
    from zentral.conf import saml2_idp_metadata_file, settings as zentral_settings
  File "/home/zentral/app/releases/2017.03.02-19.36.11-master-b98066/zentral/conf/__init__.py", line 38, in <module>
    settings = load_config_file(find_conf_file(conf_dir, "base"))
  File "/home/zentral/app/releases/2017.03.02-19.36.11-master-b98066/zentral/conf/utils.py", line 12, in find_conf_file
    raise ImproperlyConfigured('{} is required'.format(' or '.join(filepaths)))
zentral.core.exceptions.ImproperlyConfigured: ('/home/zentral/app/releases/2017.03.02-19.36.11-master-b98066/conf/base.json or /home/zentral/app/releases/2017.03.02-19.36.11-master-b98066/conf/base.yml is required', None)

@np5
Copy link
Member

np5 commented Apr 16, 2017 via email

@np5
Copy link
Member

np5 commented Apr 16, 2017 via email

@juju4
Copy link
Author

juju4 commented Apr 19, 2017

I added ZENTRAL_CONF_DIR in systemd conf files and in /etc/profile.
Python3 was probably the main issue. would be a good addition to the doc. don't remember seeing it.

vagrant@vmyrole:~$ more /lib/systemd/system/zentral-workers.service
[Unit]
Description=Zentral workers 
#Requires=redis.service
#Wants=mysqld.service postgresql.service
#After=redis.service

[Service]
## no daemon mode
#Type=simple
## daemon mode
#Type=forking
User=_zentral
Group=_zentral
Environment="ZENTRAL_CONF_DIR=/var/_zentral/zentral/conf/start/zentral"
WorkingDirectory=/var/_zentral/zentral
ExecStart=/usr/bin/python3 server/manage.py runserver runworkers
#ExecStart=/usr/bin/python server/manage.py runserver runworkers --prometheus-sd-file /var/_zentral/zentral/conf/prometheus_sd/workers.yml --
promotheus-base-port 8100   
UMask=0022
TimeoutSec=30
RestartSec=15s
#Restart=always
Restart=on-failure
SyslogIdentifier=zentral
PrivateTmp=true

[Install]
WantedBy=default.target
vagrant@vmyrole:~$ sudo -H -u _zentral -s
_zentral@vmyrole:/home/vagrant$ export ZENTRAL_CONF_DIR=/var/_zentral/zentral/conf/start/zentral
_zentral@vmyrole:/home/vagrant$ /usr/bin/python3 server/manage.py runserver runworkers
/usr/bin/python3: can't open file 'server/manage.py': [Errno 2] No such file or directory
_zentral@vmyrole:/home/vagrant$ cd /var/_zentral/zentral
_zentral@vmyrole:~/zentral$ /usr/bin/python3 server/manage.py runserver runworkers
CommandError: "runworkers" is not a valid port number or address:port pair.
_zentral@vmyrole:~/zentral$ /usr/bin/python3 server/manage.py runserver runworkers --prometheus-sd-file /var/_zentral/zentral/conf/prometheus_sd/workers.yml --promotheus-base-port 8100
usage: manage.py runserver [-h] [--version] [-v {0,1,2,3}]
                           [--settings SETTINGS] [--pythonpath PYTHONPATH]
                           [--traceback] [--no-color] [--ipv6] [--nothreading]
                           [--noreload] [--nostatic] [--insecure]
                           [addrport]
manage.py runserver: error: unrecognized arguments: --prometheus-sd-file /var/_zentral/zentral/conf/prometheus_sd/workers.yml --promotheus-base-port 8100
_zentral@vmyrole:~/zentral$ /usr/bin/python3 server/manage.py runserver runworkers 8100
usage: manage.py runserver [-h] [--version] [-v {0,1,2,3}]
                           [--settings SETTINGS] [--pythonpath PYTHONPATH]
                           [--traceback] [--no-color] [--ipv6] [--nothreading]
                           [--noreload] [--nostatic] [--insecure]
                           [addrport]
manage.py runserver: error: unrecognized arguments: 8100

gunicorn seems fine with python3 but web interface gets

FATAL:  Peer authentication failed for user "zentral"

most probably because shell user is _zentral...

@np5
Copy link
Member

np5 commented Apr 19, 2017 •
edited
Loading

manage.py, the management script for Django, accepts only one command at a time. runserver and runworkers are two different commands, and runworkers is interpreted as an argument for runserver in your examples. That's why it doesn't work.

Look in the docker-entrypoint.sh script for examples how to use the different management commands.

@juju4
Copy link
Author

juju4 commented Apr 23, 2017

I looked at docker-entrypoint when starting.
At this point, base is here and building fine in travis https://travis-ci.org/juju4/ansible-zentral

Just need to review those services
ah runserver runworkers is a bad copy/paste most probably... Good catch!

once fixed, just a relationship warning and web interface is up

$ sudo systemctl status zentral-workers
● zentral-workers.service - Zentral workers
   Loaded: loaded (/usr/lib/systemd/system/zentral-workers.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2017-04-23 02:48:42 UTC; 9s ago
 Main PID: 30665 (python3)
   CGroup: /system.slice/zentral-workers.service
           ├─30665 /usr/bin/python3 server/manage.py runworkers
           ├─30670 /usr/bin/python3 server/manage.py runworkers
           ├─30671 /usr/bin/python3 server/manage.py runworkers
           ├─30672 /usr/bin/python3 server/manage.py runworkers
           ├─30673 /usr/bin/python3 server/manage.py runworkers
           └─30674 /usr/bin/python3 server/manage.py runworkers

Apr 23 02:48:44 vmyrole zentral[30665]: File "/usr/lib64/python3.4/site-packages/django/db/utils.py", line 94, in __exit__
Apr 23 02:48:44 vmyrole zentral[30665]: six.reraise(dj_exc_type, dj_exc_value, traceback)
Apr 23 02:48:44 vmyrole zentral[30665]: File "/usr/lib64/python3.4/site-packages/django/utils/six.py", line 685, in reraise
Apr 23 02:48:44 vmyrole zentral[30665]: raise value.with_traceback(tb)
Apr 23 02:48:44 vmyrole zentral[30665]: File "/usr/lib64/python3.4/site-packages/django/db/backends/utils.py", line 64, in execute
Apr 23 02:48:44 vmyrole zentral[30665]: return self.cursor.execute(sql, params)
Apr 23 02:48:44 vmyrole zentral[30665]: django.db.utils.ProgrammingError: relation "inventory_machinesnapshot" does not exist
Apr 23 02:48:44 vmyrole zentral[30665]: LINE 1: ...ventory_machinesnapshot"."public_ip_address" FROM "inventory...
Apr 23 02:48:44 vmyrole zentral[30665]: ^
Apr 23 02:48:44 vmyrole zentral[30665]: 2017-04-23 02:48:44,002 PID30670 workers INFO inventory worker dummy - sleep 30 seconds

@np5
Copy link
Member

np5 commented Apr 23, 2017

You need to run the Django migrate command the first time and after every Zentral code update to synchronize the table structure in Postgres with the models described in the code. This will create update and remove the necessary tables.

This is done too in the docker-entrypoint.py script.

@juju4
Copy link
Author

juju4 commented Apr 23, 2017

ok. I put it aside because I was not sure it mapped manage.py migrate but it is, so good.
Centos7 fun because postgresql < 7.4 ... using postgres repo for 9.5. fine on Xenial.

I also change server/settings.py to match customized pgsql db/user/pass.
Environment storage is not the best from security
https://news.ycombinator.com/item?id=8826024
and more recently https://diogomonica.com/2017/03/27/why-you-shouldnt-use-env-variables-for-secret-data/

Any option to set user password in an unattended way?

One thing to clarify. the gunicorn and manage.py runserver are the same?
because in existing images, there is only zentral_web and zentral_workers while I added at the beginning a zentral systemd conf mapping manage.py runserver.

Thanks

@np5
Copy link
Member

np5 commented Apr 26, 2017

runserver is a standard Django management command. It will start a simple server that is absolutely not recommended for production use. That's why we use gunicorn in the zentral_web_app.service of the zentral-all-in-one images.

@juju4
Copy link
Author

juju4 commented Apr 27, 2017

perfect. removing it. will review the rest this weekend.

@pirxthepilot
Copy link

Following this thread :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@np5 @juju4 @pirxthepilot