Saturday 29 April 2000

, , , , , , ,

Web media divining - tools of the trade

The advent of the widespread availability of fast internet connections, coupled with an abundance of cheap web space has allowed artists, musicians, movie directors and software programmers to distribute their respective works online rather than via more traditional hard-copy mediums. To take full advantage of all the web has to offer it is necessary to seek out and get to grips with a number of essential tools. Amongst these are decompression programs, a download manager, an FTP client and a firewall.

In the realm of computing you are never short of software options; there are literally thousands of equivalent programs available in every category you could possibly imagine, and they're all a mere click of the mouse away. What I've attempted to do here is to narrow down this vast range of options into a more manageable selection. The ones I've chosen are considered the staple diet of the digerati, but they're not everyone's cup of char. If you find that you don't agree with my choices you don't have to stick with them, just pop along to your favourite shareware site, find the search box, tap in a few relevant keywords and take your pick. The world's your lobster... or something.

Getting X from A to B

The first item on the check list is known as a download manager. These go synonymously with internet foraging and are vital in that they allow you to resume broken downloads of bulky files and can handle large numbers of files unsupervised. Assuming you're not completely new to the internet you've probably become accustomed to downloading files using the built-in download manager (if you can call it that) of your internet browser. This is fine for small, single file downloads, but is about as much use as an inflatable dart board for transferring anything else. As an example, imagine trying to download a series of weekly, or even daily, podcasts. You would begin by clicking on the first file (hands slap foreheads across the globe in astonishment!), your browser would then open a dialog box to allow you to choose a suitable location to store your downloaded file and then it would begin transferring.

If this process is interrupted due to a failed connection you would have to begin downloading the file again from the first byte - unfortunately internet browsers have very short memories so can't remember what they were doing just a few seconds ago! Some servers are so flaky you could find yourself caught up in an aeonian start-error-restart loop. Not much fun to put it lightly!

Once the first file has downloaded successfully you would then have to click on the next one and wait for that to begin transferring and so on and so forth for each of the remaining files. I don't think you need me to tell you how infuriatingly slow this whole task can be. This is where download managers step into the breach to save you from slipping into a download induced state of insanity. What these do is allow you to queue a series of files and transfer them sequentially, freeing you from your former computer baby-sitting purgatory to do something more interesting, safe in the knowledge that when you return, your computer will still be happily chugging away without the need for any further intervention from you.

Instead of getting caught up in a mind-numbing click-wait-click loop you can set your download manager to transfer a specific number of files simultaneously (two is a sensible number if you have a slow connection). Once the first file has safely landed in your downloads directory, the next file in the queue will begin transferring automatically until the entire list has been purged. Best of all though, if the connection with the server you are downloading from is broken you won't have to begin again from square one. Your download manager will simply reconnect, erase the last few bytes which have been downloaded (because they may have become corrupt) and will resume the transfer without even breaking into a sweat.

As you know, dial-up modem connections are very fickle things and can spontaneously be broken for a multitude of different reasons. Not to worry though, this problem has also been anticipated and can be counteracted with the help of your download manager's auto reconnect system. Assuming you have informed your download manager which dial-up account you wish to reconnect to should your connection be lost, it will automatically attempt to dial into the service in the event of any problems. But that's not all they can do. When your download task is complete, providing you have ticked the relevant boxes, your computer will automatically disconnect from the internet and shutdown - now that's what I call hands-free computing!

For a long time, amongst people in the know, Gozilla and Getright were ubiquitously assumed to be the cream of the crop, but now the download manager market is no longer a two horse race. Hot on their heels, Flashget has overnight managed to claw back a substantial proportion of the user base shared by Getright and Gozilla. While Flashget takes all the usual duties of a download manager in its stride, where it really starts to break down the boundaries is in the speed department. Whereas most download managers will open a single connection to a file and keep on plugging away at it until it is complete regardless of the server's transfer rate, Flashget will open multiple connections to various servers allowing a single file to be downloaded much quicker. These multiple connections, known as 'jets' in Flashget Land, are assigned with a particular portion of a file to download. Jet 1 for example, could be downloading the first third of a file while jet 2 downloads the second third and jet 3 downloads the final third.

Flashget attempts to locate the fastest servers, known as mirrors, where the files are stored and will subsequently connect to a selection of them to maximise the bandwidth utilisation of your connection. When all the segments have been transferred they are automatically glued back together to form the whole file. In fact, Flashget is so good at its job that using it leads paradoxically to a situation where trying to surf the web while simultaneously downloading feels like treading treacle, but then it would be ridiculous to complain that the transfer speeds are too fast! Incidentally, your download speeds can be restricted so as to prevent Flashget from interfering with your surfing habits. I'm sure the circularity of this compromise is obvious by now. Nonetheless, the option is always there if you want it.

Another area where Flashget makes new ground is its file tracking logs. These are so comprehensive you are always kept informed of what is happening as it happens. If a particular file is causing problems, by taking a swift glance at the server logs you can diagnose the error instantly and take remedial action. This is especially useful if you have left your computer downloading while you get on with something else or go out for the night. If you were using a lesser download manager you would be left completely in the dark as to where the problem lies, but with Flashget the logs are always available for viewing after the event.

Almost unique to Flashget is the ability to download all the files linked from a specific web page with a few measly clicks of the mouse. Subsequent to installing Flashget a new context sensitive item is added to your right click menu. This 'download all by Flashget' option does exactly what it says on the tin. When selected, a menu pops up in front of the web page you are currently browsing allowing you to deselect the files you don't want to download. These will include images which make up the design of the page, amongst other nonessential elements. In a similar vein, where Flashget really comes into its own is when transferring files from web space accounts that have been opened for the sole purpose of hosting downloads. Under these circumstances these will be the only files stored in this location so there is no need to separate the wheat from the chaff, making Flashget's incredibly efficient 'download all' option an even more impressive time saver!

Like Getright and Gozilla, Flashget has its own FTP client built-in to the program. The difference, however, lies in the intuitive nature of Flashget's FTP browser interface which makes navigating FTP sites a breeze. For instance, if you are browsing through several FTP sites consecutively Flashget keeps a record of where you've been so that you can return to these sites with a single click. This hierarchical treelike structure is implemented throughout the client so that you can always see whereabouts in the FTP site you are currently situated. It's a simple arrangement, yet one which makes all the difference. Moreover, Flashget's reconnect system is the most reliable I've used to date. Nearly all modern download managers support this function, nevertheless, many of them tend to crash if asked to reconnect to the internet more than a handful of times. Luckily for us, Flashget's redial feature works time after time. Ultimately, the one and only thing to let the side down is the spyware which automatically plants itself into your system when you first install Flashget. Fortunately this can be swept clear using a spyware removal utility such as Ad-Aware (see the anonymity tutorial for further info).

So there's my top download manager tip. Whether you choose to use it is up to you of course. Other people will tell you that Flashget is awful and will stick to their guns no matter what. Because you will hear so many contradictory predilections, it is advisable to try all of these programs and then choose the one that suits you best, as using lots of them concurrently is likely to be a source of conflict. If none of these programs appeal, consider trying Forty Software's Download Wonder or Download Accelerator Plus from Speedbit to investigate some of the alternatives.

Alternative pathways

Next in the lineup is the humble FTP (File Transfer Protocol) client. An FTP client is used to gain access to remote computers known as FTP sites - another excellent source of freeware, graphics, movies, music, you name it! You can think of these as segregated chunks of space on the other people's hard drives which have been designated as a kind of 'share zone'. If you want to keep things simple you can use your download manager's built-in FTP browser to see what's on offer. These will allow you to download files from FTP sites, but not return the favour - that is upload files. For this task you need a fully functional, standalone FTP client.

All the ins and outs of FTP clients will be explained in much greater detail in a later tutorial so I won't delve any deeper into the topic here. For now it will suffice to say that an FTP client is an invaluable addition to your software arsenal so go and find one now even if you don't plan to use it immediately. In my humble opinion the best FTP client available is Bullet Proof FTP - it's reliable, intuitive and looks good to boot.

The decompression chamber

Thirdly, several decompression tools will be necessary to 'unzip' your archive files. Two of the best and most widely used of these are Winzip and Winace.

Winzip deals almost exclusively with files that have a zip extension. It can handle many other obscure formats, though the majority of these you are very unlikely to ever encounter during your net traversals. Winace on the other hand is capable of decompressing ace, rar and zip files in addition to all the other common compression formats.

Winrar is another indispensable archiving tool every file seeker should equip him or herself with. Although Winace can handle rar archives as well as its own homegrown format, it cannot create new rar archives; for this task you will need Winrar. Apart from this fairly obvious impasse, Winrar is much better at handling rar archives as you would expect seeing as this is precisely what the program was designed for.

It isn't absolutely imperative to install all three of these programs - you could get by with Winrar alone for example, however, each program has its own strengths and weaknesses and to get the best of both worlds (well three worlds actually) it makes sense to use different programs for different tasks. As you would expect, Winzip's most impressive party trick is opening, extracting and creating zip archives so if you are planning to take advantage of all three programs you should associate zip files with Winzip exclusively. The same goes for Winace and Winrar, so associate ace files with Winace and rar files with Winrar. You will be guided through this very simple process by fool-proof wizards following the installation of each program so don't worry about locating the relevant options yourself.

You may be wondering at this stage why three different programs are necessary in the first place. Well this is due to the fact that there is no single, standard format for compressing computer files; some people swear by rar, while others favour zip - you haven't a snowball's chance in hell of getting everyone to agree on a de facto standard, and so to keep our options open we must equip ourselves with as many programs as it takes to process each kind of file.

All three of the most popular compression formats can be spanned across a designated number of archives. This used to be common practice when it was impossible to shoehorn complete archives onto individual, removable media such as floppy disks, or freebie web space accounts. Nowadays this feature is less useful seeing as DVD writer ownership is rapidly becoming the norm. Nevertheless, for historical purposes if nothing else, I'll explain how the various data chunks comprising a spanned archive can be glued back together and decompressed below.

Baffled by data squishing and de-squishing be you not young Jedi. Read on and the compression pool will become much less murky as we go along. You now have the necessary kit in your itinerary so that's a good start.

The finer details


I take it you've downloaded one of these mysterious 'zip' file creatures and are wondering what on earth do you do with it now. Well first you need to understand a little bit about compression. You can think of a zip file, for instance, as an empty beer can (stay with me on this one, all will become clear). While the can is intact you can store the full capacity of liquid in it, however, it takes up a lot of space and is awkward to carry around (it's a king-size specimen, OK?). Now if you put on your size twelve Doc Martins and crush it under your foot, you can no longer store the same amount of liquid in it, yet it is much smaller and easier to carry around. When you want to pour the liquid back into it again, the can is stretched back to its former shape (I'm well aware that you're likely to shred your hands into a bloody pulp in the attempt, but you get the gist I'm sure).

This, in essence is what a compressed archive does. Zip files can consist of a cornucopia of file formats numbering anywhere between one and thousands. If these files were not all held together in a compressed format you would have to click on each one individually in order to download them - this is inconvenient and time consuming to say the least! Also, because the files would be in their expanded, ready to use state they would inevitably take much longer to download. This is where Winzip, Winrar and Winace come in very handy:

If a file has a zip extension I would normally advise using Winzip to extract it since this is the task the program was primarily designed for, but this is far from convenient when you have a large set of files to decompress sequentially. This is why I'd recommend using Winrar to extract the whole shebang simultaneously instead. To get started, open the folder containing the files you have just downloaded, drag a box around all the zip files (using the control key if you need to single out particular files), and then right-click one of them. Select 'extract here' from the context menu and wait while all the files are decompressed.

If when you installed Winrar you didn't agree to have the 'extract here' et al commands added to your context menu, you can achieve the same goal by opening Winrar and browsing for the files that way instead. Once you've located them, drag a box around the whole set and click on the 'extract' button, choose a place to put them and click 'OK'. It is best to choose a separate folder to store the files contained in each archive set otherwise you will soon find yourself swimming in a chaotic sea of orphan files, not knowing which ones belong to which program (or whatever you happen to have downloaded).

Quite often zip archives contain identically named .diz (short for description) text files designed to be read by FTP server software to establish the contents of compressed archives - the information harvested is used to create easily searchable catalogues. These files are far less useful to the end user at this stage, so if prompted to overwrite them click 'yes' to proceed (if your archives are part of a set they will likely be duplicates in any case).

Spanned archives

To complicate matters these can take any one of the following forms...

1. An ace file followed by a series of c?? files.

2. A rar file followed by a series of r?? files.

3. A series of files with sequentially numbered extensions without an initial ace or rar file i.e. .001 followed by .002, .003 and so on.

4. A rar file followed by a series of sequentially numbered files as in the case above.

5. A series of files all with the identical extension, rar. In this case it is the body of the filename which differentiates the files in the set. For example, the first file will be labelled volname.part001.rar, the second file will be labelled volname.part002.rar and so on.

The question marks in the first two file formats represent a series of sequentially ascending numbers, which form a mutually dependent set of files. C?? files are associated with ace archives, whereas 0?? and r?? files are associated with rar archives. These files cannot be decompressed with Winzip so you can close that down for the time being.

If after unzipping, what you are presented with is an ace file followed by a series of c?? files you will need to use Winace to extract them. On the other hand, if you are faced with scenario 2, 3, 4 or 5 from the list above you are best advised to use Winrar to extract them. Although, in theory, Winace can handle both ace and rar formats, sometimes it reports pseudo CRC errors when processing the latter file type. Very often these CRC errors are actually Winace errors rather than real errors within your archive set and are due to a mishandling of the compression format.

Scenario 5 represents Winrar's latest default volume naming scheme (as of version 3.xx). Archives compressed using versions of Winrar prior to 2.9 will take the format presented in scenarios 1 - 4. This isn't to say that you have to worry about keeping an old version of Winrar handy for processing older archives, as the latest build is capable of decompressing past and present formats.

Somehow I've managed to make all this sound much more complicated than it really is, but trust me, there's no need to panic. After opening a couple of archives you will be able to recognise and extract these two different file formats as though you're flying on autopilot. All you have to keep in mind is to use Winace for opening ace files and Winrar for opening rar, r01 or 001 files - note that if there is no rar file you will have to open either the 001 or r01 file instead. These function in exactly the same way as any other rar archive, minus the initial double-clickable rar file.

Unless you have specifically associated 001 and r01 files with Winrar they will not open automatically, in which case you will have to open the program first, browse for the file and double-click on it yourself. You may even want to modify your file association settings to allow you to simply double-click on these files to open them in future. The procedure goes a lot like this: browse for the file you want to associate with Winrar and right-click on it. Now choose 'open with' and select Winrar from the list in the ensuing dialog box, tick the "always use the selected program to open this kind of file" check box and click 'OK'. The icon representing your newly associated file will change to the default one used by rar archives and you will then be able to double-click it to delve inside. Even I'm not sure if any of that makes sense anymore, but we'll carry on regardless. I'm doing my best, honestly!

The initial rar, 001 or ace file actually represents the first file of a compressed, spanned archive. You have probably already noticed that double-clicking on a c??, r?? or a 0?? file will achieve absolutely nothing unless you have previously modified your file association settings. This is because they are not designed to be opened individually (with the exception of 001 files). Instead they will automatically be processed when you double-click on the rar or ace file.

Similarly, when you choose to extract an ace or a rar archive, all the dependent files will be extracted in one go without any further intervention from you. Whenever you do this, depending on the way in which your file associations have been configured, either Winace or Winrar will spring into action and open the archive set in a new window. All you have to do now is choose 'extract' from the menu and select a folder to store the files in, remembering to tick the 'extract with full path' option to ensure that the files end up in the correct folders.

If when you decompress these files you encounter CRC errors it means that one or more of the files have become corrupt during the transfer process. This is usually because they have been transferred too slowly, because they have been resumed too many times or because you are downloading too many files at once and this is causing bytes to be lost or to be incorrectly allocated. It is also possible, however, that the way in which you downloaded them had absolutely nothing to do with the corruption - the files could have been damaged during the initial upload stage instead, in which case there isn't a great deal you can do about it.

If your files are corrupt they will need to be either repaired or downloaded again. You can attempt to repair a corrupt archive using the built-in repair tools of Winrar or Winace (click on 'archive' and then 'repair'), but be warned, the majority of damaged archives are beyond help and will need to be re-downloaded instead. If on your second download attempt the files still don't work it is likely that they were damaged to begin with, in which case you are best advised to delete them and start afresh downloading from an alternative location.

Digital safety nets

Finally you will need a firewall to protect yourself from malicious net intruders. Having your computer hacked into and trashed by a bored, spotty thirteen year old living on the other side of the world can be a tad frustrating - especially if you discover after the event that it is easily avoided. You don't have to lurk in undesirable areas of the web to fall prey to such attacks. It could happen while you're emailing your granny, searching for a job or chatting to a friend using Yahoo messenger.

Inevitably the more time you spend online, the greater is the likelihood of you becoming a target. Well, 'target' is perhaps too strong a word. If someone hacks into your computer it's unlikely that their motivation stems from a personal vendetta. More likely they're just scanning a range of online computers to see whose 'door' has been left wide open to intruders. If you don't install an efficient firewall this is tantamount to leaving a welcome mat on your porch as an open invitation to whatever malevolent forces may be skulking in the vicinity. To discover more about firewalls try searching my blog for the key phrase 'Zone Alarm'.

Wednesday 19 April 2000

,

How can I download files from servers that make referrer checks before permitting access?

Whenever you initiate a download using Flashget you are given the option to enter a referrer URL into the properties menu associated with the file. The space where you enter this information is located directly below the file location URL box. If you enter the URL here, you can trick the server into thinking that you are downloading the file directly from the web page where you found the link.

Thursday 13 April 2000

, ,

My game backup CD image is 700mb in size, but when burnt to a CD it only occupies 400mb. Where did the extra 300mb go?

When there is a very large discrepancy between the initial image size and the final size of your burnt CD, it is usually because the image file also contains the game's audio sound track. Because the space occupied by audio tracks isn't represented in the same way as ordinary computer data it appears as though there is much less data on the final CD than there really is.

To witness this effect with your own eyes, try inserting an audio CD into your CD drive and see if you can find out how much space is being utilised by the tracks using the properties menu.