Augustus is a very popular tool for gene annotation, however its installation process can be a bit tricky. For example, if we just download and install Augustus like below, changes are it will not work.

wget http://bioinf.uni-greifswald.de/augustus/binaries/augustus-3.2.3.tar.gz
tar xvzf augustus-3.2.3.tar.gz
cd augustus-3.2.3
make

You will probably get these errors:

cd auxprogs && make
make[1]: Entering directory '/home/jxyue/Tools/augustus-3.2.3/auxprogs'
cd bam2hints; make;
make[2]: Entering directory '/home/jxyue/Tools/augustus-3.2.3/auxprogs/bam2hints'
g++ -Wall -O2    -c bam2hints.cc -o bam2hints.o -I/usr/include/bamtools 
bam2hints.cc:16:27: fatal error: api/BamReader.h: No such file or directory
 #include <api/BamReader.h>
                           ^
compilation terminated.
Makefile:29: recipe for target 'bam2hints.o' failed
make[2]: *** [bam2hints.o] Error 1
make[2]: Leaving directory '/home/jxyue/Tools/augustus-3.2.3/auxprogs/bam2hints'
Makefile:7: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/home/jxyue/Tools/augustus-3.2.3/auxprogs'
Makefile:7: recipe for target 'all' failed
make: *** [all] Error 2

This is because the auxiliary tools, bam2hints and filterBam, depend on the pre-installation of bamtools with proper library path configured.

Therefore, let’s install bamtools first. It will be much easier if we just do system-wide installation like:

sudo apt-get install bamtools libbamtools-dev

However, when we are working with a shared computing server and we do not have the root permission, we mostly likely will need install bamtools under our local directory like below.

git clone git://github.com/pezmaster31/bamtools.git
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=/your/path/to/bamtools ..
make
make install

Now bamtools should have been correctly installed. Next, we need to modify the Makefiles of bam2hints and filterBam to adapt them with our manually installed bamtools.

First, go to the “augustus-3.2.3/auxprogs/bam2hints” directory and make the following changes for the Makefile:

Replace:
INCLUDES = /usr/include/bamtools
By:
INCLUDES = $(BAMTOOLS)/include/bamtools

Replace:
LIBS = -lbamtools -lz
By:
LIBS = $(BAMTOOLS)/lib64/libbamtools.a -lz

Then, go to the “augustus-3.2.3/auxprogs/filterBam/src” directory and make the following changes for the Makefile:

Replace:
BAMTOOLS = /usr/include/bamtools
By:
# BAMTOOLS = /usr/include/bamtools

Replace:
INCLUDES = -I$(BAMTOOLS) -Iheaders -I./bamtools
By:
INCLUDES = -I$(BAMTOOLS)/include/bamtools -Iheaders -I./bamtools

Replace:
LIBS = -lbamtools -lz
By:
LIBS = $(BAMTOOLS)/lib64/libbamtools.a -lz

Now, we are finally ready to compile Augustus. Get back to the “augustus-3.2.3” directory and type “make BAMTOOLS=/your/path/to/bamtools”, viola!

References

https://github.com/pezmaster31/bamtools/wiki/Building-and-installing

https://www.biostars.org/p/189048/