Production Site Infrastructure
Several components make up the ARLO infrastructure.
For the live production site, the Nester web interface is ran through WSGI in the Apache webserver. All actions are made against Apache. You need to sudo to take these actions as root.
Starting / Stopping
The easiest way to restart is to simply restart the Apache service:
sudo service apache2 restart
Alternatively, to have WSGI reload the Python files (for example, after deploying updates), we need to touch it's config file for the project, this time as the arlo user:
sudo -u arlo touch /data/environments/production/nester/django.wsgi
Adapt Java Pool
The Java Pool responds to interactive backend requests, for example generating spectra and importing MediaFiles. Each Adapt process is configured to listen on a unique port, and HAProxy is configured to distribute requests from the Nester interface to these processes.
Note this will stop the Java Pool processes, and may kill other processes associated with the JavaPool, which may include debug processes manually added (generally safe, if there are problems, they are likely related).
# Need to be user 'arlo' sudo -u arlo -i cd /data/environments/production/ # Stop all Tracked Processes ./stop.sh
# Need to be user 'arlo' sudo -u arlo -i cd /data/environments/production/ # Start ONLY the Java Pool ./start-java.sh
The QueueRunner process(es) spawn Adapt processes that process the Job queue. The launch methods for the QueueRunner processes changes often, adapting to current performance issues (load, memory use, etc.). This process may change often, so check back here and if anything seems different that documented, don't be surprised.
Currently, I am running the QueueRunner processes via a 'screen' session using a one-liner to enforce a few features:
- Restart the processes often to minimize memory leaks.
- Pause between some tasks to allow system load to stabilize for other tasks.
Accessing the Screen Session
# Need to be user 'arlo' sudo -u arlo -i # hack to reset terminal permissions so we can use arlo's screen script /dev/null # list the existing sessions screen -ls # if no existing session, we'll have to create a new with 'screen -S queue' # access the existing screen screen -x queue
Screen commands are prefixed by hitting "Ctrl - a", followed by the command:
- Ctrl-a, d - (D)etach from the screen session
- Ctrl-a, c - (C)reate a new terminal in the screen
- Ctrl-a, n/p - Jump to the (N)ext / (P)revious terminal
Be sure you detach from the screen, not 'logout' or 'exit' so that Adapt continues running in the background.
Starting / Stopping QueueRunner
To stop, just send a Ctrl-c to the terminal. It's best to watch the log and wait if there is something going on that someone may care about.
To start, we use this command, which will loop forever running Adapt processes, pausing in between each.
cd /data/environments/production/ while /bin/true; do ./start-java-oneshot.sh; sleep 300; done
|Type of page 'Production Infrastructure':|
|Where do I link to:||Who links here:|