Installation

From ARLO
Jump to: navigation, search


Migrating from Deployment


Installation for Test / Development

Vagrant / SaltStack Configuration

https://bitbucket.org/arloproject/arlo-vagrant

A video overview of this is available at https://www.youtube.com/watch?v=mFPuYl4_-8U

This repository contains two components; a SaltStack configuration for deploying the ARLO project, and a Vagrant configuration for automating the build in a VirtualBox VM. This enables a convenient method for creating a local development environment independent of the user's local system.

Vagrant manages the provisioning of a new VM (by default in VirtualBox), downloading a base Ubuntu image, and installing SaltStack on the VM. This Vagrant configuration is pretty generic.

The real work is done by SaltStack, which manages the configuration of the VM, downloading, installing, and configuring all necessary services to run ARLO.

The SaltStack configuration will likely be the canonical source for installation requirements. If performing a stand-alone installation without SaltStack, I believe these configurations will still be useful for anyone even with no prior experience with SaltStack.

Manual Installation

These installation steps are appropriate for a test / development installation, but I would recommend not depending on these for any kind of production workload. This configuration is appropriate only for a single or very limited number of users, and has several security implications.

These steps assume Ubuntu 12.04, though any Linux distribution is likely to succeed with minimal adaptation.

As mentioned above, I would consider the Salt configuration the canonical source of knowledge for installation requirements, but these separate steps are maintained below for convenience.


Install System Software

We'll need several additional packages before proceeding with installation.

root@precise64:~# apt-get install python-virtualenv git libmysqlclient-dev msyql-client build-essential python-dev ant openjdk-7-jdk unzip
# If this will also be the MySQL Server
root@precise64:~# apt-get install mysql-server

Add a New User

Here we create a new user, 'arlo', with a home directory in /opt/arlo/ where we will install everything.

root@precise64:~# useradd -m -d /opt/arlo -s /bin/bash arlo
root@precise64:~# sudo -u arlo -i
arlo@precise64:~$


Retrieve Git Repos

arlo@precise64:~$ git clone https://bitbucket.org/arloproject/arlo-nester.git /opt/arlo/nester
arlo@precise64:~$ git clone https://bitbucket.org/arloproject/arlo-adapt.git /opt/arlo/adapt


Aparapi

arlo@precise64:~$ mkdir /opt/arlo/aparapi
arlo@precise64:~$ cd /opt/arlo/aparapi/
arlo@precise64:~/aparapi$ wget "https://github.com/aparapi/aparapi/releases/download/1.0.0-SNAPSHOT/dist_linux_x86_64.zip"
arlo@precise64:~/aparapi$ unzip dist_linux_x86_64.zip

Nester virtualenv

Create a Python virtualenv then install the packages necessary for Nester

arlo@precise64:~$ virtualenv /opt/arlo/nester-virtualenv
arlo@precise64:~$ source /opt/arlo/nester-virtualenv/bin/activate
(nester-virtualenv)arlo@precise64:~$ pip install -r /opt/arlo/nester/requirements.txt
(nester-virtualenv)arlo@precise64:~$ deactivate
arlo@precise64:~$


Database

Create a MySQL / MariaDB database and associated user. You'll use the database username and password in the following sections.

mysql> create database arlo;
Query OK, 1 row affected (0.00 sec)
 
mysql> grant all privileges on arlo.* to 'arlouser'@'localhost' identified by 'arlopass';
Query OK, 0 rows affected (0.00 sec)

Now we need to install the Database from the Django app

arlo@precise64:~$ source /opt/arlo/nester-virtualenv/bin/activate
(nester-virtualenv)arlo@precise64:~$ cd /opt/arlo/nester/
(nester-virtualenv)arlo@precise64:~/nester$ python manage.py migrate
(nester-virtualenv)arlo@precise64:~/nester$ deactivate
arlo@precise64:~/nester$

Storage Paths

For this installation, we'll place the user-files upload directory in the same /opt/arlo/ directory:

arlo@precise64:~$ mkdir /opt/arlo/user-files
# Fix our hacky symlink
arlo@precise64:~$ rm /opt/arlo/nester/media/files/user-files
arlo@precise64:~$ ln -s /opt/arlo/user-files/ /opt/arlo/nester/media/files/user-files

Also make a directory for logs

arlo@precise64:~$ mkdir /opt/arlo/log

Config Files

Paths and Settings

ARLO has several parameters to control deployment. For this Test / Development installation, we'll assume the following for the remainder of this guide (namely used in the Config Files section).

Refer to the following for more information: ADAPT_Configuration_with_ArloSettings

  • Adapt CallMe API
    • Host: localhost
    • Port: 9999
    • Path: /callMe/
  • Database:
    • Name: arlo
    • User: arlouser
    • Password: arlopass
  • Nester Root: /opt/arlo/nester/
  • Media Root: /opt/arlo/nester/media/
  • User Files Root: /opt/arlo/user-files/
  • Upload Temp Dir: /tmp/
  • Apache Root: http://0.0.0.0/ (this will only work for local access, use your IP address / hostname for remote installations)
  • URL Prefix: (Empty)
  • Media URL: http://0.0.0.0/media/
  • Static URL: http://0.0.0.0/static/
  • Allowed Hosts: Replace the entire ALLOWED_HOSTS Jinja templating block with 'ALLOWED_HOSTS = ["*"]' to allow full access
  • Api v0 URL: http://0.0.0.0/api/v0/
  • Cookie Name: development
  • Nester Port: 9000

Nester settings.py

Copy settings.template to settings.py, and edit values that are in the Jinja variable syntax ( "{{ }}" )

arlo@precise64:~$ cd /opt/arlo/nester
arlo@precise64:~/nester$ cp settings.template settings.py
arlo@precise64:~/nester$ vim settings.py

Adapt ArloSettings.properties

arlo@precise64:~$ cd /opt/arlo/adapt/
arlo@precise64:~/adapt$ cp ArloSettings.properties-SAMPLE ArloSettings.properties
arlo@precise64:~/adapt$ vim ArloSettings.properties

For reference, here is the sample I'm using.

startHttpCallMeAPIServer=true
HttpCallMeAPIServerPort=10001
dbUserName=arlouser
dbPassword=arlopass
dbUrl=jdbc\:mysql\://localhost/arlo
dbDriveClassString=com.mysql.jdbc.Driver
numThreads=2
bigDataCacheDirectoryPath=/tmp/
cacheEnabled=false
advancedTagging=true
userFilesDirectoryPath=/opt/arlo/user-files
mediaRootDirectoryPath=/opt/arlo/nester/media
RunQueueDaemonInterval_Seconds=10
QueueRunnerOneShot=false
RunQueueEnableImportAudioFile=true
RunQueueEnableTest=true
RunQueueEnableSupervisedTagDiscoveryParent=true
RunQueueEnableSupervisedTagDiscoveryChild=true
RunQueueEnableUnsupervisedTagDiscovery=true
RunQueueEnableExpertAgreementClassification=true
NesterAPIRootURL=http\://0.0.0.0/api/v0

Convenience Scripts

Now we'll build a number of scripts for convenient operation of ARLO. These will all be placed in /opt/arlo/

Clean Java Build (clean.sh)
#!/bin/bash
 
rm -r adapt/bin/arlo/*.class


Build Java (build.sh)
#!/bin/bash
 
LOGGER_OPTS="-logger org.apache.tools.ant.listener.AnsiColorLogger"
LOGGER_OPTS=""
 
cd /opt/arlo/adapt && ant ${LOGGER_OPTS} -f adapt_ant_build.xml | sed -e "s/\(error\)/`tput setaf 1`\1`tput op`/g"; test ${PIPESTATUS[0]} -eq 0
exit $?


Stop Script (stop.sh)

#!/bin/bash
 
CWD="/opt/arlo"
 
PY_PID="${CWD}/python.pid"
JAVA_PID="${CWD}/java.pid"
 
if [ -f ${PY_PID} ] 
then
	pkill -P `cat ${PY_PID}`
	rm ${PY_PID}
else
	echo "Python PID File does not exist!"
fi
 
if [ -f $JAVA_PID ] 
then
	kill `cat ${JAVA_PID}`
	rm ${JAVA_PID}
else
	echo "JAVA PID File does not exist!"
fi


Start Script (start.sh)

Now we get into the real stuff...

#!/bin/bash
 
source nester-virtualenv/bin/activate
 
HOST="0.0.0.0"
NESTER_PORT=9000
 
# directory contains nester, adapt, log, etc.
CWD="/opt/arlo"
 
# Log Files
NOW=$(date +"%d%b%y.%H%M%S")
ARLO_PY_LOG=${CWD}/log/arlo.python-${NOW}.out
ARLO_JAVA_LOG=${CWD}/log/arlo.java-${NOW}.out
ARLO_PY_LOG_CURRENT=${CWD}/log/arlo.python-current.out
ARLO_JAVA_LOG_CURRENT=${CWD}/log/arlo.java-current.out
touch $ARLO_PY_LOG
touch $ARLO_JAVA_LOG
ln -sf $ARLO_PY_LOG $ARLO_PY_LOG_CURRENT
ln -sf $ARLO_JAVA_LOG $ARLO_JAVA_LOG_CURRENT
 
###
# Start Nester (Python)
cd ${CWD}/nester
export PYTHONPATH="${PYTHONPATH}:${CWD}:${CWD}/nester"
# remove old and re-collect static files
rm -rf ${CWD}/nester/static/*
python manage.py collectstatic --noinput
# start server
nohup python manage.py runserver ${HOST}:${NESTER_PORT} > $ARLO_PY_LOG 2>&1 &
PID=$!
echo $PID > ${CWD}/python.pid
 
###
# Java Settings
 
# JMX Settings
JMX_SETTINGS=" "
JMX_SETTINGS="${JMX_SETTINGS} -Dcom.sun.management.jmxremote.ssl=false "
JMX_SETTINGS="${JMX_SETTINGS} -Dcom.sun.management.jmxremote=true "
JMX_SETTINGS="${JMX_SETTINGS} -Dcom.sun.management.jmxremote.port=62000 "
JMX_SETTINGS="${JMX_SETTINGS} -Dcom.sun.management.jmxremote.authenticate=false "
 
# Garbage Collector Settings
GC_SETTINGS=" "
GC_SETTINGS="${GC_SETTINGS} -XX:MaxHeapFreeRatio=20"
GC_SETTINGS="${GC_SETTINGS} -XX:MinHeapFreeRatio=5"
GC_SETTINGS="${GC_SETTINGS} -Xincgc"
GC_SETTINGS="${GC_SETTINGS} -verbose:gc"
GC_SETTINGS="${GC_SETTINGS} -XX:+PrintGCDetails"
 
# APARAPI Settings
APARAPI_SETTINGS=" "
APARAPI_SETTINGS="${APARAPI_SETTINGS} -Djava.library.path=/opt/arlo/aparapi"
APARAPI_SETTINGS="${APARAPI_SETTINGS} -Dcom.amd.aparapi.executionMode=JTP"
 
# Java Paths
APARAPI_LIB_PATH=/opt/arlo/aparapi
export PATH=$PATH:$JAVA_HOME/bin:$APARAPI_LIB_PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$APARAPI_LIB_PATH
JAVA_BIN=${JAVA_HOME}/bin/java
 
 
# Build Java ClassPaths
export CWD
source ${CWD}/adapt/classpath-defs.sh
 
# Start ADAPT (Java)
nohup ${JAVA_BIN} -server -Xms1G -Xmx1G -DARLO_CONFIG_FILE=ArloSettings.properties ${JMX_SETTINGS} ${APARAPI_SETTINGS} -classpath ${CLASSPATH} arlo.ServiceHead > $ARLO_JAVA_LOG 2>&1 &
PID=$!
echo $PID > ${CWD}/java.pid
 
touch ${CWD}/nester/django.wsgi

Setup File Permissions

arlo@precise64:~$ chmod ug+x stop.sh clean.sh build.sh start.sh

Also, I have an alias I use to Stop, Clean, Build, Start, and tail Log output:

alias scbsl='./stop.sh; ./clean.sh; ./build.sh && ./start.sh && tail -f log/arlo.*current*'


Start ARLO

arlo@precise64:~$ scbsl 
Python PID File does not exist!
JAVA PID File does not exist!
Buildfile: /opt/arlo/adapt/adapt_ant_build.xml
 
build-subprojects:
 
init:
 
build-project:
     [echo] adapt: /opt/arlo/adapt/adapt_ant_build.xml
    [javac] /opt/arlo/adapt/adapt_ant_build.xml:93: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
 
<... SNIP ...>
 
    [javac] 14 warnings
 
build:
 
BUILD SUCCESSFUL
Total time: 6 seconds
2015-06-17 12:04:41,025 INFO executing nester manage.py
2015-06-17 12:04:41,025 INFO sys.argv = ['manage.py', 'collectstatic', '--noinput']
2015-06-17 12:04:41,026 INFO sys.version = 2.7.3 (default, Dec 18 2014, 19:10:20) 
[GCC 4.6.3]
2015-06-17 12:04:41,026 INFO django.get_version() = 1.8
2015-06-17 12:04:41,026 INFO 'runserver' NOT specified
Copying '/opt/arlo/nester-virtualenv/local/lib/python2.7/site-packages/django/contrib/admin/static/admin/img/sorting-icons.gif'
Copying '/opt/arlo/nester-virtualenv/local/lib/python2.7/site-packages/django/contrib/admin/static/admin/img/changelist-bg_rtl.gif'
 
<... SNIP ...>
 
Copying '/opt/arlo/nester-virtualenv/local/lib/python2.7/site-packages/rest_framework/static/rest_framework/js/prettify-min.js'
 
117 static files copied to '/opt/arlo/nester/static'.
 
==> log/arlo.python-current.out <==
2015-06-17 12:14:55,628 INFO executing nester manage.py
2015-06-17 12:14:55,628 INFO sys.argv = ['manage.py', 'runserver', '0.0.0.0:9000']
2015-06-17 12:14:55,628 INFO sys.version = 2.7.3 (default, Dec 18 2014, 19:10:20) 
[GCC 4.6.3]
2015-06-17 12:14:55,629 INFO django.get_version() = 1.8
2015-06-17 12:14:55,629 INFO 'runserver' specified
2015-06-17 07:14:55,936 INFO executing nester manage.py
2015-06-17 07:14:55,937 INFO sys.argv = ['manage.py', 'runserver', '0.0.0.0:9000']
2015-06-17 07:14:55,937 INFO sys.version = 2.7.3 (default, Dec 18 2014, 19:10:20) 
[GCC 4.6.3]
2015-06-17 07:14:55,937 INFO django.get_version() = 1.8
2015-06-17 07:14:55,938 INFO 'runserver' specified
 
==> log/arlo.java-current.out <==
running initialize()
hostname = precise64
Jun 17, 2015 12:14:56 PM arlo.ArloSettings getString
WARNING: Did not find parameter 'processDescription' in config file - using default
 
==> log/arlo.python-current.out <==
2015-06-17 07:14:56,337 DEBUG (0.000) SET SQL_AUTO_IS_NULL = 0; args=None
2015-06-17 07:14:56,434 DEBUG (0.000) SET SQL_AUTO_IS_NULL = 0; args=None
2015-06-17 07:14:56,435 DEBUG (0.001) SHOW FULL TABLES; args=None
2015-06-17 07:14:56,438 DEBUG (0.000) SELECT `django_migrations`.`app`, `django_migrations`.`name` FROM `django_migrations`; args=()
 
==> log/arlo.java-current.out <==
Jun 17, 2015 12:14:56 PM arlo.ServiceHead openConnection
INFO: Database connection established - arlouser@jdbc:mysql://localhost/arlo
Heartbeat restTimeInSeconds =  600
Jun 17, 2015 12:14:56 PM arlo.ServiceHead openConnection
INFO: Database heartbeat established
Jun 17, 2015 12:14:56 PM arlo.ServiceHead openConnection
INFO: connection = com.mysql.jdbc.JDBC4Connection@4f9fe278
taskNameToOutputConnector = {}
database alive at: 2015-06-17 12:14:56.0
2015-06-17 12:14:56.623:INFO:oejs.Server:main: jetty-9.0.0.v20130308
2015-06-17 12:14:56.670:INFO:oejsh.ContextHandler:main: started o.e.j.s.h.ContextHandler@712ff9fa{/callMe,null,AVAILABLE}
2015-06-17 12:14:56.682:INFO:oejs.ServerConnector:main: Started ServerConnector@7fd66f88{HTTP/1.1}{0.0.0.0:10001}
^C
arlo@precise64:~$

Note: If you encounter errors regarding the Java Paths, you may have to explicitly state your Java home, for example, run this in your terminal, or for convenience add it to all of the scripts.

arlo@precise64:~$ export JAVA_HOME="/usr/lib/jvm/java-1.7.0-openjdk-amd64/jre"