Stratus Guides - Using SupervisorD

Instructions on how to split consumers to the SupervisorD service on STRATUS.

Supervisor is a client/server system that allows its users to control a number of processes like Magento consumers. Note: this is only possible on PHP 7.4 and above.

Installation

Versions of Python available in your environment will depend on which PHP version you are using. You can see which version is currently running on your system by running python --version Note that installing SupervisorD is only possible on PHP 7.4 and above.

Python 2.7 install

Installing SupervisorD for Python 2.7.x

  1. pip install --target=/srv/.local/lib/python2.7/site-packages/ setuptools supervisor

  2. Generate the initial config file with the following commands:

mkdir -p /srv/supervisord
cd /srv/supervisord;
wget -c https://raw.githubusercontent.com/magemojo/m2-patches/main/supervisord.conf
  1. Binary files can be found under /srv/.local/lib/python2.7/site-packages/bin/ directory, so we can create a symlink and test using this command:
cd ~/.local; ln -s /srv/.local/lib/python2.7/site-packages/bin/ bin;
/srv/.local/bin/supervisord -c /srv/supervisord/supervisord.conf
  1. Create the start.sh file if needed. If this environment is using PWA, you will not need to create start.sh since it already exists, so you may skip to step 5. Otherwise create the file as follows:
mkdir -p /srv/pwa
cd /srv/pwa
touch start.sh ; chmod a+rwx start.sh
  1. Edit start.sh file and update/add the sequence to start supervisord daemon.
pip install setuptools supervisor
/srv/.local/bin/supervisord -c /srv/supervisord/supervisord.conf

Important! If you are running a PWA application in that container, then please update /srv/supervisord/supervisord.conf file line nodaemon=true to nodaemon=false

Python 3.7 install

Installing SupervisorD for Python 3.7.x

  1. pip3 install --target=/srv/.local/lib/python3.7/site-packages/ setuptools supervisor

  2. Generate the initial config file with the following commands:

mkdir -p /srv/supervisord
cd /srv/supervisord;
wget -c https://raw.githubusercontent.com/magemojo/m2-patches/main/supervisord.conf
  1. Binary files can be found under /srv/.local/lib/python3.7/site-packages/bin/ directory, so we can create a symlink and test using this command:
cd ~/.local; ln -s /srv/.local/lib/python3.7/site-packages/bin/ bin;
/srv/.local/bin/supervisord -c /srv/supervisord/supervisord.conf
  1. Create the start.sh file if needed. If this environment is using PWA, you will not need to create start.sh since it already exists, so you may skip to step 5. Otherwise create the file as follows:
mkdir -p /srv/pwa
cd /srv/pwa
touch start.sh ; chmod a+rwx start.sh
  1. Edit start.sh file and update/add the sequence to start supervisord daemon.
pip3 install setuptools supervisor
/srv/.local/bin/supervisord -c /srv/supervisord/supervisord.conf

Important! If you are running a PWA application in that container, then please update /srv/supervisord/supervisord.conf file line nodaemon=true to nodaemon=false

Launching SupervisorD in the NodeJS container

  1. Go to the Stratus panel and start the NodeJS container.

  2. Please place a ticket with support and request that Tier2 “Create a custom PHP Container to run consumers with SupervisorD” and reference this article. When that portion is completed, you may return here to complate the configuration below titled “Configure consumers to run in supervisord”

Configure consumers to run in supervisord

  1. Login back to the SSH container and execute the following:
mkdir -p /srv/supervisord/conf.d
cd /srv/supervisord
echo "LD_LIBRARY_PATH=/srv/php/lib /srv/php/bin/php bin/magento queue:consumers:start async.operations.all" > async.sh; chmod a+rwx async.sh
cd /srv/supervisord/conf.d
cat <<EOT >> /srv/supervisord/conf.d/async.conf
[program:async-queue]
process_name=%(program_name)s
directory=/srv/public_html
command=bash /srv/supervisord/async.sh
autostart=true
autorestart=true
user=www
numprocs=1
redirect_stderr=true
stdout_logfile=/srv/supervisord/%(program_name)s.log
EOT

** Note: Adjust directory= with where Magento 2 docroot is. Example above is for async.operations.all, create one by one for each Consumer that you wish to offload from Cron to Supervisor mechanism.

  1. Start/Restart the NodeJS container; If running ps aux | grep start and kill start.sh process running, so it restarts with the new setting.

  2. To test if supervisor is active:

/srv/.local/bin/supervisorctl status

Correct output:

async-queue                      RUNNING   pid 93608, uptime 0:00:15
  1. In the last step, we have to disable Consumers running as Cron tasks in env.php
'cron_consumers_runner' => [
        'cron_run' => false,
        'max_messages' => 0,
        'consumers' => [
        ]
    ],

The empty place will consider they all will be running from something else and that is Supervisor.

POISON PILL

To keep Consumers in order and prevent failures or memory leaking (known problem), here is instruction on creating an additional Supervisord program called Poison Pill which will restart all consumers at 7200 seconds intervals.

  1. Create a file named poison.sh within /srv/supervisord directory with the following contents:
#!/bin/bash
while :
do
    sleep 7200
    ps aux | grep consumer | awk {'print $2'} | xargs kill -s 9
done
  1. Run the following:
cat <<EOT >> /srv/supervisord/conf.d/poison.conf
[program:poison-pill]
process_name=%(program_name)s
directory=/srv/public_html
command=bash /srv/supervisord/poison.sh
autostart=true
autorestart=true
user=www
numprocs=1
redirect_stderr=true
stdout_logfile=/srv/supervisord/%(program_name)s.log
EOT

Logs can be found in the /srv/supervisord folder


Last modified January 1, 0001