FreeNAS 11.2 FreeBSD jail deployment

From Mayan EDMS Wiki
Jump to: navigation, search

These instructions are adapted from and borrow from the Mayan EDMS Basic deployment instructions, and were tested on a FreeNAS 11.2 Beta 3 installation. They should work with little or no modification on a bare metal FreeBSD system.

Being a Django and a Python project, familiarity with these technologies is recommended to better understand why Mayan EDMS does some of the things it does.

Configure FreeNAS[edit]

Add the Mayan user[edit]

  1. Navigate to Account > Users in the FreeNAS GUI
  2. Click the yellow "+" button to add a user
  3. Fill out the form as follows:
Username mayan
Full name Mayan EDMS User
User ID 973
New Primary group Unchecked
Primary group wheel
Home directory /nonexistent
Enable password login no
Notes
Other values can be left at their defaults
The User ID should be an unused number that you note; you will need it later

Optional: Create a Dataset for Mayan[edit]

I chose to create a discrete dataset in my main FreeNAS storage pool to store Mayan's uploaded documents. This is ultimately the storage that's referred to later in configuration as MAYAN_MEDIA_ROOT. You can just let Mayan store its documents in the jail storage (/mnt/iocage/jails/mayan) if you'd prefer.
If you would like to store your Mayan documents in a FreeNAS dataset for simpler storage management, backups, and perhaps security configurations, follow the steps in this subsection then look for the green highlighted text in the following steps for configuration changes to look out for.

Create the dataset[edit]

  1. Navigate to Storage > Pools in the FreeNAS GUI
  2. Choose Add Dataset from the overflow menu of the pool you would like to add to
  3. For this guide, we'll name the dataset mayan but you can call it whatever you'd like as long as you can find it later

Modify the dataset's permissions[edit]

  1. Choose Edit Permissions from the overflow menu of the newly-created mayan dataset. Use the following values:
ACL Type Unix
Apply User Checked
User mayan
Apply Group Checked
Group wheel
Apply Mode Checked
Owner Read, Write, Execute
Group Read, Write, Execute
Other None

Create the Jail[edit]

  1. Navigate to Jails in the FreeNAS GUI
  2. Click the yellow "+" button to add a jail
  3. Fill out the form as follows:
Jail name mayan
Release 11.2-RELEASE
DHCP Autoconfigure IPv4 Checked

Optional: Create Mount Point[edit]

If you're using a FreeNAS dataset for storage, you'll need to share it with the jail. If you skipped the optional steps above, skip ahead to the next section.
  1. Navigate to Shell in the FreeNAS GUI
    • Here, you are accessing your FreeNAS server's shell, not the new Mayan jail (yet)
  2. Execute the following commands as root:
    • mkdir /mnt/iocage/jails/mayan/root/mnt/storage
      • For this guide we've chosen /mnt/storage as the jail's mount point for the shared-in dataset. This can be changed to be whatever you'd like, but be sure to adjust further references in this guide to whatever you choose.
    • chown mayan:wheel /mnt/iocage/jails/mayan/root/mnt/storage
  3. Back in the FreeNAS GUI Jails interface, select Mount Points from the Mayan jail's overflow menu
  4. Click the yellow "+" button to add a mount point
    • In the Source field, navigate to the mayan dataset you created earlier (e.g. /mnt/your_pool/mayan)
    • In the destination field, navigate to the mount point you created in the previous step (e.g. (/mnt/iocage/jails/mayan)/mnt/storage)

Start and Enter the Jail[edit]

  1. Navigate to Shell in the FreeNAS GUI
  2. Enter the following command which will start the Mayan jail and enter its shell:
    • sudo iocage exec mayan

Configure the Jail or FreeBSD Environment[edit]

  1. Install binary dependencies:
    • pkg update
    • The package management tool is not yet installed on your system.
      Do you want to fetch and install it now? [y/N]: y
    • pkg install sudo nano py27-sqlite3 gcc ghostscript9-base gnupg1 graphviz libjpeg-turbo py27-filemagic png libreoffice tiff poppler-utils postgresql10-server python27 py27-pip py27-virtualenv redis sane-backends py27-supervisor tesseract hs-zlib
    • Proceed with this action? [y/N]: y
    • Somewhere in the neighborhood of 240 packages will be installed
  2. Set dependencies to run at boot:
    • sysrc postgresql_enable=YES redis_enable=YES supervisord_enable=YES
  3. Initialize the PostgreSQL server:
    • service postgresql initdb
  4. Start the database server:
    • service postgresql start
  5. Make sure Mayan EDMS can find the binaries it needs:
    • ln -s /usr/local/bin/gpg /usr/bin/gpg
      ln -s /usr/local/bin/libreoffice /usr/bin/libreoffice
      ln -s /usr/local/bin/pdfinfo /usr/bin/pdfinfo
      ln -s /usr/local/bin/pdftoppm /usr/bin/pdftoppm
      ln -s /usr/local/bin/pdftotext /usr/bin/pdftotext
  6. Create a user account for the installation:
    • adduser
    • Username: mayan
       Full name: Mayan EDMS User
       Uid (Leave empty for default): 973
       Login group [mayan]:
       Login group is mayan. Invite mayan into other groups? []: wheel
       Login class [default]:
       Shell (sh csh tcsh zsh nologin) [sh]: nologin
       Home directory [/home/mayan]:
       Home directory permissions (Leave empty for default):
       Use password-based authentication? [yes]:
       Use an empty password? (yes/no) [no]: yes
       Lock out the account after creation? [no]: yes
       Username   : mayan
       Password   : <blank>
       Full Name  : Mayan EDMS User
       Uid        : 973
       Class      : 
       Groups     : wheel 
       Home       : /home/mayan
       Home Mode  : 
       Shell      : /usr/sbin/nologin
       Locked     : yes
       OK? (yes/no): yes
       adduser: INFO: Successfully added (mayan) to the user database.
       adduser: INFO: Account (mayan) is locked.
       Add another user? (yes/no): no
       Goodbye!
    • Use the same User ID (Uid) you used in FreeNAS, above.
  7. Create the parent directory where the project will be deployed:
    mkdir -p /opt
  8. Create the Python virtual environment for the installation:
    virtualenv /opt/mayan-edms
  9. Make the mayan user the owner of the installation directory:
    chown -R mayan:mayan /opt/mayan-edms
  10. Install Mayan EDMS:
    Professional Edition
    sudo -u mayan /opt/mayan-edms/bin/pip install --no-cache-dir -i https://<product key>@pypi.mayan-edms.com mayan-edms-pro --extra-index-url=https://pypi.org/simple
    Community Edition
    sudo -u mayan /opt/mayan-edms/bin/pip install --no-cache-dir mayan-edms
  11. Install the Python client for PostgreSQL and Redis:
    sudo -u mayan /opt/mayan-edms/bin/pip install --no-cache-dir psycopg2==2.7.3.2 redis==2.10.6
  12. Create the database for the installation:
    sudo -u postgres psql -c "CREATE USER mayan WITH password 'mayanuserpass';"
    sudo -u postgres createdb -O mayan mayan
    • Replace mayanuserpass with a password of your choosing, if desired
  13. Initialize the project:
    sudo -u mayan MAYAN_DATABASE_ENGINE=django.db.backends.postgresql MAYAN_DATABASE_NAME=mayan MAYAN_DATABASE_PASSWORD=mayanuserpass MAYAN_DATABASE_USER=mayan MAYAN_DATABASE_HOST=127.0.0.1 MAYAN_MEDIA_ROOT=/opt/mayan-edms/media /opt/mayan-edms/bin/mayan-edms.py initialsetup
    • Enter the same MAYAN_DATABASE_PASSWORD you used in the previous step
    • If you're following the optional FreeNAS dataset steps, replace the MAYAN_MEDIA_ROOT path with /mnt/storage or whatever you assigned as a mount point earlier.
  14. Collect the static files:
    Professional Edition
    sudo -u mayan MAYAN_MEDIA_ROOT=/opt/mayan-edms/media /opt/mayan-edms/bin/mayan-edms.py collectstatic -i tests --noinput
    Community Edition
    sudo -u mayan MAYAN_MEDIA_ROOT=/opt/mayan-edms/media /opt/mayan-edms/bin/mayan-edms.py collectstatic --noinput
    • If you're following the optional FreeNAS dataset steps, replace the MAYAN_MEDIA_ROOT path with /mnt/storage or whatever you assigned as a mount point earlier.
  15. Create the supervisor file at /etc/supervisor/conf.d/mayan.conf:
    • mkdir -p /etc/supervisor/conf.d
    • nano /etc/supervisor/conf.d/mayan.conf
    • [supervisord]
      environment=
          MAYAN_ALLOWED_HOSTS='["*"]',  # Allow access to other network hosts other than localhost
          MAYAN_CELERY_RESULT_BACKEND="redis://127.0.0.1:6379/0",
          MAYAN_BROKER_URL="redis://127.0.0.1:6379/0",
          PYTHONPATH=/opt/mayan-edms/lib/python2.7/site-packages:/opt/mayan-edms/data,
          MAYAN_MEDIA_ROOT=/opt/mayan-edms/media,
          MAYAN_DATABASE_ENGINE=django.db.backends.postgresql,
          MAYAN_DATABASE_HOST=127.0.0.1,
          MAYAN_DATABASE_NAME=mayan,
          MAYAN_DATABASE_PASSWORD=mayanuserpass,
          MAYAN_DATABASE_USER=mayan,
          MAYAN_DATABASE_CONN_MAX_AGE=60,
          DJANGO_SETTINGS_MODULE=mayan.settings.production,
          MAYAN_SIGNATURES_GPG_PATH=/usr/bin/gpg
      
      [program:mayan-gunicorn]
      autorestart = true
      autostart = true
      command = /opt/mayan-edms/bin/gunicorn -w 2 mayan.wsgi --max-requests 500 --max-requests-jitter 50 --worker-class gevent --bind 0.0.0.0:8000 --timeout 120
      user = mayan
      
      [program:mayan-worker-fast]
      autorestart = true
      autostart = true
      command = nice -n 1 /opt/mayan-edms/bin/mayan-edms.py celery worker -Ofair -l ERROR -Q converter -n mayan-worker fast.%%h --concurrency=1
      killasgroup = true
      numprocs = 1
      priority = 998
      startsecs = 10
      stopwaitsecs = 1
      user = mayan
      
      [program:mayan-worker-medium]
      autorestart = true
      autostart = true
      command = nice -n 18 /opt/mayan-edms/bin/mayan-edms.py celery worker -Ofair -l ERROR -Q checkouts_periodic,documents_periodic,indexing,metadata,sources,sources_periodic,uploads,documents -n mayan-worker medium.%%h --concurrency=1
      killasgroup = true
      numprocs = 1
      priority = 998
      startsecs = 10
      stopwaitsecs = 1
      user = mayan
      
      [program:mayan-worker-slow]
      autorestart = true
      autostart = true
      command = nice -n 19 /opt/mayan-edms/bin/mayan-edms.py celery worker -Ofair -l ERROR -Q mailing,tools,statistics,parsing,ocr -n mayan-worker-slow.%%h --concurrency=1
      killasgroup = true
      numprocs = 1
      priority = 998
      startsecs = 10
      stopwaitsecs = 1
      user = mayan
      
      [program:mayan-celery-beat]
      autorestart = true
      autostart = true
      command = nice -n 1 /opt/mayan-edms/bin/mayan-edms.py celery beat --pidfile= -l ERROR
      killasgroup = true
      numprocs = 1
      priority = 998
      startsecs = 10
      stopwaitsecs = 1
      user = mayan
    • Be very careful that:
      • There are exactly four spaces before each line of the environment variables at the top of the config
      • There is a comma after each environment variable except the last
      • Each line of the [program] blocks contains exactly one key = value pair (be sure there aren't any surprise carriage returns/newlines)
      • If you're following the optional FreeNAS dataset steps, replace the MAYAN_MEDIA_ROOT path with /mnt/storage or whatever you assigned as a mount point earlier.
  1. Edit the Supervisor config file to read this new configuration:
    • nano /usr/local/etc/supervisord.conf
  2. Edit the last two lines to read (be sure to remove the commenting semicolons):
    • [include]
      files = /etc/supervisor/conf.d/*.conf
  3. Configure Redis to discard data when it runs out of memory:
    • nano /usr/local/etc/redis.conf
  4. Add these lines to the end:
    • maxmemory-policy allkeys-lru
      save ""
      databases 1
  5. Start the services:
    • service redis start
      service supervisord start

Connect to your Mayan EDMS instance![edit]

  1. In the FreeNAS Jails GUI, note the IP address for your Mayan jail (or run ifconfig in the jail's shell)
  2. Point a web browser to that IP address, port 8000. e.g. 192.168.1.100:8000

Troubleshooting[edit]

Are the workers working?[edit]

Run supervisorctl in the jail. Each job and its status should be listed:
mayan-celery-beat                RUNNING   pid 40698, uptime 1:47:55
mayan-gunicorn                   RUNNING   pid 40699, uptime 1:47:55
mayan-worker-fast                RUNNING   pid 40697, uptime 1:47:55
mayan-worker-medium              RUNNING   pid 40696, uptime 1:47:55
mayan-worker-slow                RUNNING   pid 40695, uptime 1:47:55

Ask Supervisor to log verbosely[edit]

Set loglevel=debug in /usr/local/etc/supervisord.conf.

Check the worker logs[edit]

Navigate to /tmp in the jail, then take a look at the mayan supervisor log files.

Check the Supervisord log[edit]

Navigate to /var/log and take a look at supervisord.log