Welcome to minoTour.
minoTour is a suite of web based tools that enable real time analysis of read files generated by the Oxford Nanopore minION sequencing device. This page provides additional information on the tools themselves. Full installation instructions for setting up your own minoTour server are contained in the PDF file in the repository itself.
By far the easiest way to use minoTour is with our Nottingham, UK, based server at http://minotour.nottingham.ac.uk/ If you wish to use our server you will need to register at http://minotour.nottingham.ac.uk/register_new.php and then email your user name to me (matt.loose(at)nottingham.ac.uk) to request that we activate your account for data submission).
A video describing some of the features on minoTour is available here: https://www.youtube.com/embed/fZRMMSsw7O4
Installation and Configuration Instructions for your own minoTour server.
We provide an example for configuring an Ubuntu 14.04 server for minoTour. You should obviously modify these instructions as you need for your favour of UNIX. Instructions for configuring minoTour on the same computer as the minION device will be provided at a later date.
Step 1 - LAMP Installation and memcached
If you do not already have LAMP installed then follow the instructions below. You will also need memcached running on your server. This is currently optional but will become required for planned future features in minoTour.
:-$ sudo apt-get update
:-$ sudo apt-get install lamp-server^
Note - you really should set a password for the mySQL root user when prompted to at this point…you will use this later on. Also make sure your local firewalls are configured to allow access for http, https, ssh (for admin) and mySQL and the necessary ports are open on your server.
:-$ sudo apt-get install memcached php5-memcache
:-$ sudo apt-get install php5-curl
Step 2 - Perl Background Processes
You need to install the perl memcached module
sudo apt-get install libcache-memcached-perl
You should also install the following perl module (#idiot check - you may need to install make if it isn't already on your system…
sudo apt-get install make )
To do this - we suggest you use CPAN.
Step 3 - Configure mySQL for minoTour.
If you are using ubuntu, you have to use the following steps to enable mySQL to communicate with the outside world: run the command
Comment out the bind-address line using the # symbol:
#bind-address = 127.0.0.1
Note: you might need to replace with something more appropriate:
bind-address = 0.0.0.0
You also need to modify the maximum packet sie that can be uploaded in a single transaction. We strongly suggest setting this to something big! e.g: change the following line to something like:
max_allowed_packet = 200M
Just check you understand firewalls and mySQL!
restart mysql once.
sudo /etc/init.d/mysql restart
Note: You may find that you need to restart your server at this point. Check you can log in to mySQL. If you can't, restart the server.
Configuring the minoTour package.
The easiest way to install minoTour is via GIT so - first off install git!
sudo apt-get install git
Next you need to clone the minoTour git repository as follows:
git clone https://github.com/minoTour/minoTour.git
The advantage of using GIT is that future updates can be applied very easily.
The database works as follows. Each new run is added to its own unique database. This allows for relatively simple management of run data and shifting of databases to different locations in the future. To manage these databases, a master database is created (‘Gru’ - it controls the minIONs…). Gru allocates databases to users, controls access to the website and is absolutely essential to the operation of minoTour.
In order for this system to work, you create mySQL users that have permission to create and delete databases. This creates risks for your mySQL system and it is unlikely that any centrally managed facility will allow you to do this - you therefore need access to your own mySQL installation. To negate these risks, we create users that can only create and destroy databases which belong to them, defined by their own user names - this means that users cannot delete other databases in your mySQL installation - PHEW! To simplify the creation of these accounts, we have generated a number of scripts to automate this process (found in the folder 'mT_server/db_control/'). These scripts assume that you have access to the mySQL server and can connect to mySQL as root. You should ensure that the scripts have the necessary permissions to run on your server.
Firstly you need to create the basic database Gru and establish an account on mySQL that can be used by the website to show all the data (the ‘webuser'). You need to use the initialiseDB script and have the Gru.sql file (found in the 'mT_server/db_control/setup' folder).
This account has full access to all the databases created by the upload scripts. The website archive features require that the webuser account can drop tables and can access all users data. The datasets that can be seen by individual users is managed by the Gru database itself.
Assuming that you have root access to your mySQL install go to the command line and execute the initialiseDB script. You need to pass the script three variables. The username you wish to use for your webuser, the password and the ip address of the web server which will be connecting to the database. If your web server and mySQL database are on the same machine, use ‘localhost’ for this last option. The script will request your mySQL root password twice whilst it is running.
>./initialiseDB <webusername> <webpassword> <ip>
If this process has been successful you should now be able to log in to mySQL with the following command:
>mysql -u <webusername> -p
and provide the on request. Typing
mysql> show databases;
should list the presence of the Gru database.
The username and password created here will be used later to connect the minoTour website to your mySQL database.
Next you need to create accounts for users to actually upload data to the database. How you manage this process is up to you. We allow users to log in to the website and create an account for accessing the website choosing there own username. We then use this username to enable connection to the mySQL database.
IMPORTANT: Do not use _ in the user name. This will break functionality elsewhere in the package.
To create an account use the ‘createuser’ script (located in 'mT_server/db_control/admin'). This requires 3 arguments. The , and the address from which they are connecting to your mySQL server. This is an important security feature as it limits the possibility of malicious connection to your database. To add additional IP addresses in the future we provide the script addusersource (see below). The simplest option is to use the wildcard symbol %. This will allow the user to connect from any computer anywhere. It will also resolve an issue where the machine connecting to the database does not have a fixed IP address. To create a user account execute the ‘createuser’ script. The username you use here must be associated with an account on the web site. e.g If Joe Bloggs creates a user account on the website called JB123, his account for uploading data in mySQL should also be called JB123.
>./createuser <username> <userpassword> <ip>
You will need to communicate the to the user directly. There is no way that they can find this information out otherwise. The option to change the user password on the website will not alter the mySQL password set here. To add additional connection points for this user run the addusersource script providing the username and the additional IP address.
>./addusersource <username> <new_ip>
If you need to delete a user then run the dropuser script. This requires only the username to prevent any access to the database for that user. This does not remove any data OR prevent the user from seeing the data in the minoTour website.
Having done this, we shall now configure the website.
Important: we will return to mySQL to configure our admin account shortly.
minoTour WebSite installation.
Configuring the website is straightforward. Copy the entire 'minoTour' folder located in the mT_web folder (which contains the web application) into the appropriate root folder for your webserver. In OSX this folder is usually called:
In unix systems it may be
The name of the folder dictates the url for your website. You can change this if you wish.
Upgrade Notes: You may wish to install your new version of minoTour in place of the previous version. We would suggest first setting the site up in an alternative location and checking the configuration before writing over your previous installation. The new version of minoTour includes a 'previous versions' tab which will allow you to go back and use older versions of minoTour. Most importantly you need to keep note of the DB-HOST and DB_PASS values from the db.php file in the config folder and ensure these values are used in the new minoTour install.
You may need to use 'sudo' privileges to copy this folder into the appropriate location.
Before going any further, you need to configure the website to talk to your database.
To do this, navigate to the folder ‘config’ which is located within the minoTour folder. In this folder is a file called ‘db_example.php’. Open this file with a text editor and you will see lines including:
If your database is on the same machine as your webserver, leave DB_HOST set to “127.0.01”. If this doesn’t work, switch it to “localhost”. If your mySQL database is on a remote machine, then enter the IP address of that machine here. Then set DB_USER and DB_PASS to the user name and password that you used when setting up the database above. If you have mySQL configured on a non standard port you can set that here. When you save this file, save it as db.php in the config folder.
Important: Each time a new release of minoTour is made available you should check the db_example.php file to see if new global parameters have been established. For example the twitter features which are discussed at the end of this post!
Now - assuming you are on the same machine as your webServer and have not changed the folder name, open a browser window and navigate to:
And you should see a nice welcoming page… - if you don’t - check that you have php and apache correctly configured on your machine. Google is your friend…
If you do not see the video, then your webServer is unable to connect to the outside world. This isn’t critical but will limit some functionality in our system. In particular it will prevent you submitting bug reports and feature requests via the website to us.
Now return to your web browser - we will make an account for you to access the site - to do so go to register_new.php .
This first account is important as we are going to make it the system administrator. Use your choice of username and password. Providing a valid email address is important as it allows us to contact you in the event of bugs or other problems. Complete the form and register away.
If all has worked well you will be rewarded with a success message… Return to the login page and use your shiny new account to log in.
If your machine is on a network where it can see the outside world, you will have the news items listed. If you do, then the bug reporting and feature requests will work. If you do not, then these buttons won’t function for you.
The last thing we need to do is set this account as an administrator. To do that, we go back to mySQL - log in using your mySQL account - in our case webaccess. Follow these instructions replacing the minoTouruser with the user name of your choice:
mysql> USE Gru;
mysql> UPDATE users SET admin=1 WHERE user_name = 'minoTouruser';
Now return to the website and refresh the page… Assuming all has gone to plan you will have an admin account and you should see a new ‘Admin’ menu has appeared. Currently the Admin processes are limited - you can allocate databases to different users. The second page 'cache administration' allows you to check if the new optimised features of minoTour are working correctly.
Update Notes: This is a new feature of minoTour that dramatically speeds up displaying data. You should follow these instructions.
If you click on 'cache administration' you will see a message that minoTour is currently not using memcached. Memcached is a unix tool that allows a website to store parameters in memory on the server. We use this to dramatically improve performance. You will need to have memcached installed on your host to use this service. We also provide background scripts that allow the server to process data.
To configure memcached in an Ubuntu environment type (you should modify as appropriate for your flavour of unix) Note you should have done this already if you followed the instructions from the start!:
sudo apt-get install memcached php5-memcache
Depending on the size of your minoTour install you can dedicate more or less memory to memCached. We recommend at least 64Mb but larger installations with more users may benefit from more memory. To configure the memory allocated to memcached see the man pages for your particular flavour of unix/memcached.
After configuring memcached correctly you should see the message has changed on your 'cache administration' page to indicate that minoTour can see the memcache.
The perl scripts we are going to run in the background require the memcached-perl library - this can be installed on Ubuntu by typing (again - you should have done this above if all is well!):
sudo apt-get install libcache-memcached-perl
We now need to configure these scripts to work with your installation. As with the web site there is a central configuration file located in the 'nefario' folder within the 'mT_server' folder. The example configuration file is called mT_param.example. Open this in a text editor and complete each line as follows:
directory - is the path to the folder containing your minoTour website. Do not forget the trailing '/' e.g. /var/www/html/minoTour/
memcahce - is the memcache ip and port - this will work as 127.0.0.1:11211 unless you have changed your memcache install.
dbhost - the database host ip
dbuser - the database username you configured on page 2 of this document
dbpass - the database password you configured on page 2 of this document
phploc - the path to php to execute php from the command line
Save this file as mT_param.conf
Now you can set the background processes running by executing the command:
In order to run this in the background you need to use nohup or an equivalent methodology - e.g.
nohup perl mT_control.pl &
If you return to the website and 'cache administration' you should now see a message that your memcache is working and the backend scripts are running.
Great. Nearly there… Now to upload some data…
minupv.0.38W.exe - Data Upload - Windows
We have a version of the minup script which has been compiled to operate on windows. This is available as the file minup.v0.38W.zip and can be downloaded from the website itself from the minUp Scripts menu option. This comes with a windows compatible version of the Last Aligner and is easier to use than the linux mounting system suggested in 3b) and 4 below. However it is unable to take advantage of GNU-PARALLEL for speeding up the last analysis but does seem to perform very well on the Windows machine as specified by Oxford Nanopore.
To install this software, download the folder minup.v0.38W.zip from the website from under the 'minUp Scripts' menu and decompress into a folder in C:\ . Alongside our program you will also need to install CygWin available from https://www.cygwin.com . Once you have installed CygWin you must ensure that the CygWin bin folder containing the file cygwin1.dll is added to your windows path. The command for doing this in command.exe is:
This will allow our minup program to access the linux like commands it requires to function. You should then also add the folder you have downloaded from us to your path. Assuming you decompressed the folder and placed it at the root of the C: you would type the following:
Within this folder are three key programs. minup.v0.38W.exe is the windows data uploader. lastal.exe is the last aligner compatible with windows, and lastdb.exe formats the last database.
To check your installation, try each of these programs.
will generate the following output:
lastal: please give me a database name and sequence file(s) Usage: lastal [options] lastdb-name fasta-sequence-file(s)
will generate the following output:
lastdb: please give me an output name and sequence file(s) Usage: lastdb [options] output-name fasta-sequence-file(s) Prepare sequences for subsequent alignment with lastal. Main Options: -h: show all options and their default settings -p: interpret the sequences as proteins -c: soft-mask lowercase letters
Which will generate the following output: usage: minup.v0.38.py [-h] [-dbh DBHOST] -dbu DBUSERNAME [-dbp DBPORT] -pw DBPASS [-f REF_FASTA] -w WATCHDIR [-n THREADS] -u MINOTOURUSERNAME [-s VIEW_USERS] [-o FLOWCELL_OWNER] [-r RUN_NUM] [-c] [-m] [-a] [-t] [-d] [-v]
minup: A program to analyse minION fast5 files in real-time or post-run. Args that start with '--' (eg. --mysql-host) can also be set in a config file (minup_posix .config ) by using .ini or .yaml-style syntax (eg. mysql-host=value). If an arg is specified in more than one place, then command-line values override config file values which override defaults.
optional arguments: -h, --help show this help message and exit -dbh DBHOST, --mysql-host DBHOST The location of the MySQL database. default is 'localhost'. -dbu DBUSERNAME, --mysql-username DBUSERNAME The MySQL username with create & write privileges on MinoTour. -dbp DBPORT, --mysql-port DBPORT The MySQL port number, else the default port '3306' is used. -pw DBPASS, --mysql-password DBPASS The password for the MySQL username with permission to upload to MinoTour. -f REF_FASTA, --align-ref-fasta REF_FASTA The reference fasta file to align reads against. Using this option enables read alignment provided LastAl and LastDB are in the path. Leaving this entry blank will upload the data without any alignment. To use multiple reference fasta files input them as one text string seperated by commas (no white spaces) -w WATCHDIR, --watch-dir WATCHDIR The path to the folder containing the downloads directory with fast5 reads to analyse - e.g. C:\data\minion\downloads (for windows). -n THREADS, --aligning-threads THREADS The number of threads to use for aligning -u MINOTOURUSERNAME, --minotour-username MINOTOURUSERNAME The MinoTour username with permissions to upload data. -s VIEW_USERS, --minotour-sharing-usernames VIEW_USERS A comma seperated list (with no whitespaces) of other MinoTour users who will also be able to view the data. -o FLOWCELL_OWNER, --flowcell-owner FLOWCELL_OWNER The name of the minion owner. 'minionowner' is the default -r RUN_NUM, --run-number RUN_NUM The run number of the flowcell. The default value is 0. -c, --commment-true Add a comment to the comments field for this run. Follow the prompt once minup starts . -m, --upload-maf-true Upload MAF format alignment data. -a, --upload-align-true Upload long alignment data. -t, --insert-tel-true Store all the telemetry data from the read files online. This feature is currently in development. -d, --drop-db-true Drop existing database if it already exists. -v, --verbose-true Print detailed messages while processing files.
To reduce the complexity of running minUp, a customised config file can be generated by the website. Note that the config file for the Windows version of minup is slightly different to the linux version. The file should be placed in the same folder as the minup.v0.38W.exe file.
You are free to edit this file and change the parameters or add common parameters (such as you're usual shared users as an example).
If you are running minup on the same machine as your mySQL database you may need to modify the config file to set the host name to 'localhost' or equivalent.
To test your installation we have created a small sample of data from the recently released Loman Lab dataset (http://dx.doi.org/10.5524/100102). This sample set consists of just the first 100 or so reads from the run. It is also available from the web site itself. Decompress the demo_data_set.zip folder to a location on your machine. The folder structure is important here. metrichor returns files to a folder called ‘downloads’ and minup looks for this folder in any location you point it at - so keep the folder structure. We also include a copy of the reference genome in this dataset.
To upload this data type the command minup.v0.38W at the command prompt and complete the options as appropriate - for example:
C:\minup.v0.38W -dbh <your_hostname_or_ip> -dbu <minoTouruser> -pw <PASSWORD1> -u <minoTouruser> -f /<PATH>/<TO>/<YOUR>/<COPY>/<OF>/demo_data_set/reference/EcoliMG1655.fasta -w /<PATH>/<TO>/<YOUR>/<COPY>/<OF>/demo_data_set -o Joe_Bloggs -r 1 -m -c -t
Note that you should not include a trailing slash in the path to the reads directory. It is also assumed that the reads directory structure contains a folder called 'downloads'. It is this folder and ALL FOLDERS BELOW from which reads will be uploaded. Thus we upload reads in both the path and fail folders. If you wish to upload PASS reads only then you must include the /PASS option at the end of your path.
You will be asked to enter a comment (due to the -c option set above) describing your run. This can be a brief note on the run as you set it up and will be associated with the run record in minoTour.
If this works you will see the following messages in command.exe…
processing the reference fasta.
finished processing reference fasta.
2014-09-23 10:48:19 CACHED: 101 PROCESSED: 0 ...
2014-09-23 10:49:26 CACHED: 0 PROCESSED: 101
2014-09-23 10:49:36 CACHED: 0 PROCESSED: 101
Now jump over to the website, refresh the page, click on ‘Current Sequencing Run’ and have a look at the data. To end the processing step, just hit CTRL-C in the terminal. This is required to switch your run to a 'previous run' state.
If this hasn’t worked, then give us a shout… @mattloose or email@example.com!
You are now up and running and will be able to upload old run data (just point the —watch-dir to wherever you store your old reads) or perform live read uploads.
OPTIONAL: Custom Twitter Notifications
A standard install of minoTour will tweet via the @minoTour_01 twitter handle to alert you of key events. This is a really useful feature... If you would like to customise the twitter account sending messages you can do so. In the db_example.php file (and so the db.php file) you must add in a twitter consumer key and secret and an access token and secret. Setting these will automatically switch your minoTour installation to sending tweets from that account rather than our central minoTour account. This option is provided for your convenience but we cannot provide advice or guidance on generating your own twitter api keys. This is available from dev.twitter.com .
We also caution you to make sure your keys are kept securely on the server.