Table of Contents (TOC)

There are many articles, seemingly mostly based on Ubuntu, about using Linux with usenet. However most of these deal with setting up a graphic client.

This is not intended to be a comprehensive Usenet guide. It just gives a quick overview of using Usenet with Linux and includes some command line tools.

Providers

They are more expensive than most of the other providers. Reviews of the major providers vary--some say that astranews, around $10-$12 USD per month for unlimited access is good, others say that it's slow with SSL. The reader is advised to find recent reviews of the main providers. Many offer a free trial as well. The newsgroupreviews site offers reviews of many providers.

There are also sites, such as blocknews, that offer block accounts. That is, you can buy a block of 200 GB, and then pay again when your limit is used up. I'm not sure about other sites that sell blocks, but with blocknews, at least, you have unlimited time in which to use whatever you purchase.

TOC

Searching Usenet

I would recommend doing a bit of googling to find out how to determine if a file a is good or not. Binsearch, and other search tools, give you the option of creating an nzb file, which you can then download. I've never even tried to do this without a GUI browser though, as usenet is quite old, older than www, I'm sure it's not that difficult.

There are various paid newsgroup search engines. Sometimes, for various reasons, some legit, many less so, they get taken down. The best thing to do is to google for something like usenet search engine. I believe Giganews provides its own search engine.

TOC

Using nzbget

Installing nzbget is simple on most modern distributions. Most have a package for it, if not in their official repositories, in one of the third party ones. The project page on SourceForge has some installation information.

With CentOS 6.x, there is no package. I wound up downloading the subversion

svn co https://nzbget.svn.sourceforge.net/svnroot/nzbget/trunk nzbget-svn

At that point, I had to install a few dependencies. For those unfamiliar with installing from source, you can see my article about it. Running ./configure indicated what was missing. I was able to find most of it with

yum provides */<missing.files>

(See my article for a little more detail on using the very handy yum provides option.)

However, there were some par2 libraries missing. Although the part2cmdline was available, I was getting errors about other files connected to par2. In the end, I wound up finding the Fedora 12 rpms for libpar2 and libpar2-devel, which installed by running rpm -ivh. There are almost certainly more elegant ways to do it, but this worked. Finding the rpms can be a bit tricky at times. I used rpm.pbone.net and found them at ftp://ftp.pbone.net/mirror/download.fedora.redhat.com/pub/fedora/linux/releases/13/Everything/x86_64/os/Packages/

However, the latest (May 2013) subversion still didn't work properly. It was giving complaints about libpar2 not having particular patches. I wound up using information from a howto geek article about using nzbget on a Raspberry Pi. It turned out that I had to both install the libpar2 rpms I found and manually install a separate patched version of libpar2. I'm not sure how this worked without breaking anything, but everything still seems to work.

I repeat that doing something like that does present a risk of breaking your system, and many of us who run CentOS do so because of the wish for stability. So, add non official packages at your own risk, or use a distribution, such as Fedora for those who like the RH based systems, that has nzbget available as a precompiled package.

Once nzbget is installed, one has to configure it. We'll assume that your provider is giganews, your username is john and your password is 1234. If using subversion, in the subversion directory, you'll find a file called nzbget.conf.example or possibly just nzbget.conf. If installing through your distribution, the example conf file will probably be in the /usr/share/doc/nzbget-<version> or /usr/share/nzbget directory.

Copy it to your home directory, renaming it .nzbget--note the period in front of the name, this makes it a hidden file.

cp /usr/share/nzbget/nzbget.conf.example ~/.nzbget

Now edit it. It's a long file, but you're only looking for a few lines to change. By default, it will look for a directory in your home directory, called downloads and use a subdiretory called dst. You can change this if you like. Look for the lines

$MAINDIR=~/download

Note the ~ in there. That means your home diretory. nzbget will create this, and the sub directories below it You'll see several references to ${MAINDIR}/<some_subdirectory> in the file. You don't have to create the directories, the first time you run nzbget it will create all of them, including ~/downloads. If you want the downloaded files kept elsewhere, then specify it by changing the $MAINDIR= line.

Next look for

Server1.Host=my1.newsserver.com

We'll change it to giganews.

Server1.Host=news.giganews.com

Underneath that, you'll see several options for Server1. If you're using the default port 119 leave the port line, alone, if using SSL then change the Server1.Port= line to 563. (On Giganews, as per their web page's instructions--your provider may use different ports.) Also, if using SSL change the default of Encryption=no to Encryption=yes. If not using SSL then leave both the port and the Encryption line alone.

Also change the Server1.Username=user and Server1.Password=pass to your user name and password. So, assuming we kept most of the defaults here, our Server1 section, using SSL would read.

Server1.Host=news.giganews.com Server1.Port=563 Server1.Username=john Server1.Password=1234 Server1.Encryption=yes

This section on postprocessing only applies to versions older than nzbget 10 and is only being left in for reference.

See the nzbget article for more detail, but as of nzbget 11 and up and subversion at least as of May, 2013, you don't need to specify any postprocessing configuration or scripts, unpacking and verifying will be done for you. (I wasn't quite sure from the article, and didn't bother investigating whether it does or doesn't apply to nzbget 10. I would say look in the default configuration file and if there's a reference to postprocess-example.sh listed in there, follow these instructions. If it's not in the example configuration, then nothing needs to be done and you can skip ahead to actually using nzbget.)

However on one or two occasions (and this has happened with older versions of nzbget as well, that did use the scripts), I sometimes found that for whatever reason, a download didn't get properly processed. In some cases, I was able to use SABnzbd instead, which correctly processed the file. This is rare, however, in 99 out of 100 cases, nzbget, both the old with the postprocess scripts and the new, without them, will correctly process files.

Prior to nzbget 10 or so, you next needed to use the postscript conf scripts--they will unrar the files, but more importantly, they automate verifying and fixing files. Using the svn version, there is a postprocess-example.sh and postprocess-example.conf. The two files will probably be in the same place as the sample nzbget.conf.example, i.e., if installing from svn, all three will be in the directory created after the svn download, if installed through a package manager, all three files will probably be in /usr/share/nzbget.

At any rate, there should be a couple of lines in the example nzbget.conf file--one a sample for the PostProcess, commented out, and another, uncommented, that just reads

PostProcess=

I just followed the example (which is shown, commented, earlier in the file.) I put the script into my home directory, and, in my .nzbget file, used the line

PostProcess=~/postprocess-example.sh

I didn't bother renaming it. The postprocess shell script refers to a postprocess.conf file--I put that in my home directory as well. You can actually leave the script at its defaults, copy the postprocess-example.conf into your home directory, and it will all just work. By default, that is where the script looks for the conf file. The line in the script is

SCRIPT_CONFIG_FILE=postprocess-example.conf

The conf file can also just be left at defaults. Put it into your home directory. Although it has a line asking for a destination directory, if you leave the file at its defaults, it will use the ~/download/dst directory.

Lastly, before running nzbget, be sure to make the postprocess-example.sh script executable.

chmod 755 ~/postprocess-example.sh

Now, when you run nzbget, it should check files, check for parity with par2 files, automatically fix errors, and give you a decompressed complete file of whatever you downloaded, e.g., a Linux iso.

At first, I didn't use the postprocessing scripts. I would download a set of rars and par2 files. I would then manually run par2repair, and get errors about missing blocks. In such cases, I wasn't able to unrar the the file--most of the rars would decompress, but I would also get an error that one or two files had CRC errors.

Your distribution also probably pulled in par2cmndline or par2--the name of the package varies between distributions, but you do want to have it. Check that you have the command par2verify.

which par2verify

If it's not there, find it with your distribution's search function. In CentOS' case, even though I had to find the par2 libraries, the actual par2verify command is supplied by the par2cmdline package. In some Debian or Ubuntu distributions, it's provided by the par2 package. It is probably necessary for the postprocessing scripts of nzbget.

The free version of unrar, available with almost all distributions is also necessary for post processing to work. Remember, if you are using subversion or running nzbget 10 or higher (one may need 11, the page is a little unclear about this), there is no need to do anything with postprocessing scripts, and one doesn't have to configure them in the .nzbget file.

Running nzbget is very simple. We have the nzb file we downloaded earlier, we'll call it myfile.nzb

nzbget myfile.nzb

You'll now see a screen showing that it is downloading parts of the file. When done, you will see that you have a ~/download directory with subdirectories named dst, nzb, queue, and tmp. (This seems to have, at least in a subversion install from May 2013, changed from download to downloads.) If you look through your ~/.nzbget file, you'll see that the nzb directory is where one is expected to place nzb files, though you don't have to, the queue file is for download queues and the tmp file for temporary files--if you're downloading a large amount of files, you'll see various files in there while downloading, it's automatically cleaned when the download is complete.

Inside the dst (destination) directory, you'll see nzbget.log and and a directory with the name of the nzb file you started with, so in this case, it would be called myfile. If using the postprocessing scripts, in there, you should see the file you wanted, for example, myfile.iso

If there were no duplicates, and all the post processing scripts ran without error, the iso, and possibly an .nfo (info file, usually a small text file with a bit of information about the download) will be all that's in there. Usually, even if you see a great many rar files still in the directory, if you do a listing of all files, you'll see that the myfile.iso is there, and it was successfully decompressed and assembled. As far as I can tell, the reason the post processing scripts don't completely remove all rars in many cases is because there will be some duplicate rar files. Other times it may be that the download was corrupt and there will be no myfile.iso in the dst directory.

If you didn't put in the postprocessing script and conf files, inside myfile you will usually see several rar files and probably at least one par2 file. The par2 file or files are used to check the integrity of the download. However, as I mentioned, when there were problems, and I ran the commands manually, I got errors that I was missing blocks. However, if you do want to run it manually,

par2verify *par2

This will run through the list of files and hopefully tell you all is OK. Otherwise, it might give a message that there are missing parts and repair is either possible or not possible. If possible, then run par2repair *par2. If not possible, then go back to your search engine and try for another version of the desired file. (Or, as mentioned a few times now, use the postprocessing stuff that is provided with nzbget.)

If, however, you are doing it all manually, and the download has been verified to be good, now unrar it. The unrar command is available in most distributions. I cover it, as well as rar, in my page on multimedia but the syntax is simple. Look for a file with suffix of rar, r00, or r01. If there is no .rar, then look for the lowest numbered file. (In other cases, they may all have a .rar suffix, with names like part1.rar, part2.rar, and so on.) For our example, we'll assume the parts are called myfile.r00, myfile.r01, etc. Extract and combine the parts with

unrar e myfile.r00

You should see various messages and eventually get a message that all is OK. You should now have a complete file of what you downloaded, for example a Linux distribution's iso. If not, perhaps you forgot to run par2, or it didn't work properly. Usenet files should be downloaded with caution--there are often incomplete files that not only waste your time and bandwidth, but also use up a quota if your provider has limitations in the amount you can download per month. That's another reason I recommend, especially if you have monthly download limits, that one lets nzbget do the checking for you.

Sometimes, nzbget's post-processing scripts will fail but the file can be salvaged. First run par2repair on the par2 file, as mentioned above. It might give a response that rar1 is fine, rar2 is fine, rar3 can't be repaired. In many cases, you'll have files like file.rar01, file.rar02, file.rar03, and file rar.03_duplicate (or similar). Sometimes, if one replaces the original with the duplicate, it will work.

mv file.rar03_duplicate file.rar03

Then try rar e to extract the file again, or par2repair on the par2 file again. You can also back up your original rar03 file before replacing it with the duplicate, but in my experience, if par2repair can't fix it, it's usually not worth saving.

Another thing that will sometimes happen is that you'll see, while running rar e, that almost the entire file unars without error, but at 98 or 99 percent it fails. Sometimes, the failure might be an included sample file, or other extra that isn't needed. It's worth a try to add the -kb for keep broken option.

rar e -kb myfile.rar

This time, after unrar has completed, even if it fails at 99 percent, you will have your file. It might or might not work. If the breakage was in the main file, it probably won't, but if the unrar failure is do to some extra file, such as the aforementioned sample, you might be able to successfully use the file.

TOC

Using a VPN

Giganews's VPN is provided by VyprVPN. They have a page on setting it up with Ubuntu and NetworkManager that worked quite well when I tested it on Mint13 and Ubuntu 12.04. That page uses pptp. They have another page for OpenVPN, which did not work for me. It's possible that it's something my provider is doing, but I didn't investigate it very thoroughly.

If using NetworkManager, then one can just follow the instructions given by Giganews. However, pptp can also be set up manually. It can be a bit more time consuming, and as you'll see, I haven't worked out a few details, but this should be enough to get you started.

You will need the pptp-client, and possibly the server. as well. Search for pptp with yum, apt, pacman, or whatever tool your distribution uses. This varies between distribution--some will require both server and client, others, just the client. In Ubuntu, I believe (untested by me) that it's pptp-linux and pptpd.

In CentOS 6.x, one needs ppp and pptp. This will include the pppd daemon and add an /etc/ppp directory. In there, you will see a subdirectory called peers. Assuming we're going to use the Giganews us2.vpn server, located in the Northeast of the US, we will create a file in /etc/ppp/peers, called us2. We can name it whatever we want, but if you do, in each case where I use us2 in the file, use the name you've given the file. Remember, our giganews username is john and our password is 1234. The server's address is us2.vpn.giganews.com

I haven't tested if every line in here is necessary, but regardless, it should work. The reader is invited to read man pppd and improve their own file. Remember, in this case, I am calling the file us2, and so every reference in my file to us2 should be replaced with whatever you name your file. The /etc/ppp/peers/us2 file should read

remotename us2 linkname us2 ipparam us2 pty "pptp us2.vpn.giganews.com --nolaunchpppd" name john lock noauth require-mppe-128 refuse-eap

Once that's done, go up one level to directory /etc/ppp. Now we edit /etc/ppp/chap-secrets. We want it to read

john us2 1234 *

In other words, your user name, the name you gave the /etc/ppp/peers server file, in our case, us2, your password, and an asterisk. In the CentOS version, the default file gives headings of client, server, secret and IP addresses.

Now you can test your setup. In CentOS, manually created a symbolic link from /usr/share/doc/ppp-<version>/scripts/pon to /usr/bin. I did the same for poff which is also in there. Most other distributions will, as part of the install, put the scripts into /usr/bin.

Now you can test it with

pon us2 nodetach

The nodetach option runs it in the foreground, so you can be sure that it's working properly. Hopefully, if there are errors, the output to terminal will make them easy to fix--for example, I once put nolaunchppd, instead of pppd (leaving out a p) in my us2 file, and the output was that the nolaunchppd option wasn't recognized.

If all goes well, you will soon see that CHAP authentication succeeded, and then see a local and remote IP address. You can kill it with ctl+c or use the poff command, mentioned above.

poff us2

You should now be able to run it without the nodetach option and, running ip a or ifconfig, be able to see that your ppp0 interface has an address.

There is one problem. Your traffic isn't being routed through the VPN. If you do use NetworkManager, NM does seem to do this automatically, however, if we're doing it manually, and you use one of the many services to view your IP, for example, ipchicken, you will see that your address appears to be the same as it was before invoking pptp. You have to make ppp0 the default route. Note that as of time of writing, I haven't found a consistantly solution to put your default route back to your eth0 or wlan0, so if you're doing all this before finishing the article be warned.

If you try

ip route add default dev ppp0

You will get an error that the file exists. You have to first delete your current default route.

ip route del default ip route add default dev ppp0

Now, if you once again go to ipchicken.com, you'll see that your IP address is hidden, and shows up as an address provided by the VPN service.

When finished, there are some scripts included in some distributions to try to replace your default route back to where it was, usually eth0 or wlan0. In my experience, they have not always worked, and I haven't yet figured out why. If you try to reverse the process we just did, with ip route del default, ip route add default eth0, you won't be able to get past your router. What does work for me is a bit awkward, but at present it's what I do. So, we're finished doing our vpn stuff and we want to go back to using eth0. First we remove the default route, then turn off ppp0 with poff, and then add the gateway with the ip route command. For those unfamiliar with the ip command, as opposed to ifconfig and route, I have a short page about it. In this example, we're assuming that your router's (or other gateway's) address is 192.168.1.1

ip route del default poff us2 ip route add default via 192.168.1.1

Once this is done, typing

route

should show that once again all your traffic is going through eth0 or whatever card you were using. It will still show the us2.vpn.gigagnews.com address in there as a destination. I haven't figured out how to get rid of that, but haven't been too concerned about it.

TOC