Production Infrastructure

From ARLO
Jump to: navigation, search


Production Site Infrastructure

Several components make up the ARLO infrastructure.

Web Server

For the live production site, the Nester web interface is ran through WSGI in the Apache webserver. All actions are made against Apache. You need to sudo to take these actions as root.

Starting / Stopping

The easiest way to restart is to simply restart the Apache service:

sudo service apache2 restart

Alternatively, to have WSGI reload the Python files (for example, after deploying updates), we need to touch it's config file for the project, this time as the arlo user:

sudo -u arlo touch /data/environments/production/nester/django.wsgi


Adapt Java Pool

The Java Pool responds to interactive backend requests, for example generating spectra and importing MediaFiles. Each Adapt process is configured to listen on a unique port, and HAProxy is configured to distribute requests from the Nester interface to these processes.

Stopping

Note this will stop the Java Pool processes, and may kill other processes associated with the JavaPool, which may include debug processes manually added (generally safe, if there are problems, they are likely related).

# Need to be user 'arlo'
sudo -u arlo -i
cd /data/environments/production/
# Stop all Tracked Processes
./stop.sh

Starting

# Need to be user 'arlo'
sudo -u arlo -i
cd /data/environments/production/
# Start ONLY the Java Pool
./start-java.sh


Adapt QueueRunner

The QueueRunner process(es) spawn Adapt processes that process the Job queue. The launch methods for the QueueRunner processes changes often, adapting to current performance issues (load, memory use, etc.). This process may change often, so check back here and if anything seems different that documented, don't be surprised.

Currently, I am running the QueueRunner processes via a 'screen' session using a one-liner to enforce a few features:

  • Restart the processes often to minimize memory leaks.
  • Pause between some tasks to allow system load to stabilize for other tasks.

Accessing the Screen Session

# Need to be user 'arlo'
sudo -u arlo -i
# hack to reset terminal permissions so we can use arlo's screen
script /dev/null
 
# list the existing sessions
screen -ls
# if no existing session, we'll have to create a new with 'screen -S queue'
# access the existing screen
screen -x queue

Screen commands are prefixed by hitting "Ctrl - a", followed by the command:

  • Ctrl-a, d - (D)etach from the screen session
  • Ctrl-a, c - (C)reate a new terminal in the screen
  • Ctrl-a, n/p - Jump to the (N)ext / (P)revious terminal

Be sure you detach from the screen, not 'logout' or 'exit' so that Adapt continues running in the background.

Starting / Stopping QueueRunner

To stop, just send a Ctrl-c to the terminal. It's best to watch the log and wait if there is something going on that someone may care about.

To start, we use this command, which will loop forever running Adapt processes, pausing in between each.

cd /data/environments/production/
while /bin/true; do ./start-java-oneshot.sh; sleep 300; done





Type of page 'Production Infrastructure':
Where do I link to:
Who links here:

PageName::Production Infrastructure