Sunday, 6 November 2016

Newznab Pi

A lot of the newsgroup indexing sites are either being shut down or are becoming invite only which is a shame as there's a lot of useful stuff in the groups that isn't only illegal downloads. There's quite a few Linux, freeware and shareware groups out there. 

It's time to convert a Raspberry Pi into a usenet indexer, I tried this with an original Pi and it was far too slow but hopefully the model 2 will do the job. All configuration is done from a Linux desktop, if you haven't got an ssh command then you'll need to get something like Putty.

So what do we need, obviously a Raspberry Pi model 2 in a box, something like this will do.

An external hard drive of some sort, I've used an old 320Gb and got hold of a case from Amazon, it's
nothing exciting but it works, I ordered the case on Saturday evening and it arrived on Sunday lunchtime, good going there Amazon.

A USB hub also makes life easier, nice cheap one from Maplin. You're also going to need an ethernet lead to connect it to your router, £1 in Poundworld.

Some coffee and a sausage inna bun will help.

It's time to make a start, put it all together but don't worry about connecting the external drive yet, we need to get hold of Raspbian Jessie Lite from here. Extract it and then copy it to a micro SD card, easiest way is to use a cross platform app called Etcher

Once it's finished, put the card into the Pi and give it some power. Have a look at the connected devices in the router config and ssh into the pi with ssh pi@ or whatever yours says, password to connect is raspberry. We now run:

sudo raspi-config

You should see this:

Scroll down to advanced, then down to update, once that's completed go back down to advanced options and put in your own hostname, mine is called Orac as the Pi is in a clear case just like Orac in Blakes 7, under advanced we want to select boot to console requiring user to login and enable ssh. Once last thing under advanced is memory split, set this to 16 to give as much RAM to the running of the Pi as possible. Last thing we do is select Expand Filesystem at the top.

Select finish and wait for a reboot, you should now be able to connect with 

ssh pi@orac

 or whatever you've called your pi.

Now we create a normal user with

sudo adduser username

then add them to the superuser group with

sudo adduser username sudo

Reboot with 

sudo shutdown -r now 

Now you should be able to login with ssh orac or ssh username@orac.

Let's remove the default Pi login with

sudo deluser pi

We need some helpful utilities now, so let's enter:

sudo apt-get install rsync mc

And now it's time to connect up the external drive.

Type dmesg at the command line and it should come up as /dev/sda or something similar, we now need to partition it so we enter:

sudo fdisk /dev/sda or whatever yours came up as.

Then D to delete any existing partitions, then N to create a new one, followed by P for primary partition, then enter 3 times followed by w to write the changes. There may be other options here depending on how many partitions there were on the drive when you started.

Next we have to format the drive with:

sudo mkfs.ext4 /dev/sda1

Now create a temporary mount point with 

sudo mkdir /mnt/temp

Then copy the filesystem to the external drive with:

sudo rsync -axv / /mnt/temp

Let's back some config files with:

sudo cp -R /boot /boot.orig

Time to edit some  stuff to allow us to boot from the external drive:

sudo nano /boot/cmdline.txt

The original looks like this:

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait

It's all on one line, we need to change the root=/dev/... bit to read root=/dev/sda1 or whatever your USB stick shows, we also need to add a delay to the end with rootdelay=5, it should look like the one below in the end.

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/sda1 rootfstype=ext4 elevator=deadline rootwait rootdelay=5

Save it and then we need to edit fstab with:

sudo nano /etc/fstab

The following entry needs to be added:

/dev/sda1    /   ext4    defaults,noatime  0       1

And the line referring to the sd card needs to be commented out, mine looks like this now

proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
# /dev/mmcblk0p2  /               ext4    defaults,noatime  0       1

/dev/sda1    /   ext4    defaults,noatime  0       1

We now should be running from the external drive, type df -h to check the space, so now we move onto getting Newznab installed, there's a few utilities we need that don't seem to be in the raspbian repository, I've precompiled these so you can get them with these commands:




Let's get them installed and the system updated now

sudo apt-get update

sudo apt-get upgrade -y

sudo dpkg -i *.deb

Let's make some folders and set permissions:

sudo mkdir /var/www
sudo mkdir /var/www/newznab
sudo chmod 0777 /var/www/newznab

And now install some stuff

sudo apt-get install php5 php5-dev php-pear php5-gd php5-mysql php5-curl mysql-client-5.5 libmysqlclient-dev apache2 mysql-server-5.5 lame mediainfo

Make sure you put in a good strong mysql passqword if you're going to get access to this from the Interwebs.

Now we make some changes to the php configuration file:

sudo nano /etc/php5/cli/php.ini

Under the resource limits section we need to change the maximum execution time to 120:

max_execution_time = 120

Then under Module Settings we need to change the time zone settings, in my case I'm the UK so I'm going to change it to:

date,timezone = Europe/London

Save that and then time to edit another one:

sudo nano /etc/php5/apache2/php.ini

Under the resource limits section we need to change the maximum execution time to 120:

max_execution_time = 120


memory_limit = -1  

Then under Module Settings we need to change the time zone settings, in my case I'm the UK so I'm going to change it to:

date,timezone = Europe/London

Now we create the Apache config file for Newznab

sudo nano /etc/apache2/sites-available/newznab.conf

And paste the following into it:

<VirtualHost *:80>
ServerAdmin webmaster@localhost
ServerName localhost

DocumentRoot /var/www/newznab/www
ErrorLog /var/log/apache2/error.log
LogLevel warn
Change the port if you want it to run on a different port.

One more change to the Apache configuration file before we go on:

sudo nano /etc/apache2/apache2.conf

Find this bit

<Directory /var/www/>
        Options Indexes FollowSymLinks
        AllowOverride None
        Require all granted

And change it to this:

<Directory /var/www/>
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted

We now need to get hold of Newznab itself, you can get it from here. Newznab classic works but there are some limitations, if you are serious just pay the money for a better product and you get free upgrades too.

Get the file downloaded and then we need to extract it and copy it to the right place, 

As I have a paid for version I'm going to use a slightly different approach, for this you need to install subversion with the following command:

sudo apt-get install subversion

Then to get Newznab you enter the following command:

sudo svn co svn:// /var/www/newznab

At this point you will be asked for the root password for your system and the SVN login details that have been emailed to you by the Newznab team.

Now it's time to activate the changes:

sudo a2ensite newznab
sudo a2dissite 000-default
sudo a2enmod rewrite
sudo service apache2 restart

Change some folder permissions with:

sudo chmod 777 /var/www/newznab/www/lib/smarty/templates_c
sudo chmod 777 /var/www/newznab/www/covers/movies
sudo chmod 777 /var/www/newznab/www/covers/music
sudo chmod 777 /var/www/newznab/www/covers/tv
sudo chmod 777 /var/www/newznab/www
sudo chmod 777 /var/www/newznab/www/install
sudo chmod 777 /var/www/newznab/nzbfiles/
sudo chmod 777 /var/www/newznab/www/covers/anime
sudo chmod 777 /var/www/newznab/www/covers/tv

And that's it for the moment, you now need to finish setting up Newznab from a browser with:


Once you've done all that, run

sudo chmod 0777 /var/www/newznab/nzbfiles/tmpunrar

And install phpmyadmin with:

sudo apt-get install phpmyadmin

Go to the site admin options and edit site, it works much better if you put in your own Rotten Tomatoes api, your own Amazon settings and your own Newznab ID, the section that asks for the path to some files should show this:


Once that's all completed, go to the View Groups option and make some groups active.
 Then it's back to the command line with

sudo -i

cd  /var/www/newznab/misc/update_scripts

Then run 

php update_binaries_threaded.php

after that it's

php update_releases.php 

That should start populating the indexer, it's up to you how you want to run this, you can it manually but I run a loop with a 10 minute pause each time it restarts.

I've made a file with 

sudo nano /bin/newznabrun

Put this into it

while :

# Beginning Update
cd /var/www/newznab/misc/update_scripts/
php update_binaries_threaded.php
php update_releases.php

        echo "Press [CTRL+C] to stop.."
        sleep 360


Save the file and then run

sudo chmod +x /bin/newznabrun 

You can then type sudo newznabrun to get it going.

That's it all done, hope it all works for you.

Big thanks to the howtogeek website for help with these instructions, I've added a few bits and pieces to get to this stage.