Tuesday, 28 October 2003

, ,

eMule and the eDonkey network unwrapped

eMule, more commonly known as "that really difficult to configure file sharing client", is a reworking of its more recognisable older brother eDonkey, and contrary to popular belief it's no more taxing to use than any other piece of peer to peer software. Let's face it, if you're prepared to entrust your file transfers to a ladybird and a chocolate spread and are still not satisfied with the results, what have you got to lose by putting your faith in the humble, patient and hard working donkey? If you pass up the opportunity to discover eMule's hidden depths based on mere folklore without first putting it through its paces then the biggest ass in this story will be you.

Unlike archetypal file sharing tools like Napster, eMule doesn't depend on a single, central server to function making it much trickier for big brother to stomp on. Instead it is the users themselves who operate the hundreds of independent servers on which the eDonkey network is founded. These servers are not designed to host all the files available to the network, but do store indexes of the files which users of the network have chosen to share. Whenever you conduct a search, it is these indexes and the servers they reside on which are called upon to provide the results and negotiate the transfer of files from one computer to another.

Even people who have never used eDonkey or eMule somehow instinctively know that its strength lies in the transfer of bulky files like CD images and movies. This isn't a misconception; although any kind of file can be exchanged, the eDonkey protocol is particularly well suited for handling the king-sized variety due to the way server requests are made. The more files a user decides to share, the more strain is put on the CPU of the server. If the server has to process thousands of smaller files, gigabytes worth of MP3s for example, then it can quickly be reduced to crawling speed and it is for this reason users are strongly encouraged to either only share large files or to compress their collections of smaller files into single archives before adding them to their shared folders.

In short, what you get with either of these pack horses is quality over quantity, something which if you've become accustomed to using Kazaa will undoubtedly come as a breath of fresh air. If used correctly you are far less likely to download falsely named files, viruses or trojans making for a much more pleasurable experience.

Aside from this, what sets eDonkey apart from most of the competition is its use of unique file identifiers, or 128 bit MD4 hash codes if you're into technical jargon. These codes consist of 32 case insensitive digits (of the form 0-9A-F) written in hexadecimal notation. So what? Well each file available to the network is assigned with a unique code. These codes are then appended to filenames and converted to ed2k links which can be listed on web pages, in emails or wherever.

eDonkey, being the canny beast it is, is capable of recognising these files irrespective of their filename or file size ensuring that what you end up with is what you intended to download. An added bonus is that this allows you to download the same file from multiple sources making for much faster downloads. This works in the same way as some of the more advanced download managers; eDonkey creates dummy files full of empty space which is then gradually filled in with real data as and when it becomes available.

To add files referenced using an ed2k link to your transfer queue you simply have to click on them as you would with any other web site link. The Windows registry, having been previously instructed to associate such links with the eDonkey client, coordinates the transfer of the file information to the program. The client takes the reigns from this point and drops the filenames into its transfer window where relevant sources will be sought before the files commence downloading.

Another very welcome feature available to eDonkey and eMule users is the credit system. Each user is allocated a unique hash ID and whenever they allow other users to connect to their computer, whether it is to download or upload files, the number of bytes transferred is recorded in their clients.met file along with the ID of the users doing the sharing. If a user connects to your computer and it is detected that you have previously shared data then you are rewarded with superior access to his or her files. The more you share with a particular user the better your share rating becomes and as a result your queue ranking is lowered allowing you to download and upload sooner than other users who have never shared files with this user. The old aphorism "you reap what you sow" springs to mind - you are rewarded for sharing and punished for selfishly hoarding files away where no-one can get at them - a step in the right direction if you ask me.

While eDonkey offers previously unheard of features like file hashing it also provides all the techniques and gadgetry which you have come to expect from any half decent file sharing client; it can handle the automatic resumption of interrupted transfers, you can search the whole network no matter which server you are connected to, it supports simultaneous uploading and downloading, it lets you preview partially downloaded files (providing you have the first and last segments of the file) and it can operate using any port you specify making it effortless to configure so as to circumvent the restrictions imposed by firewalls and proxies.

You may be wondering why, if eDonkey is so wonderful, is there a need for another client based on the same protocol. Firstly, eMule is open source so anyone with sufficient programming knowledge can contribute to the project by either updating the core client or creating modifications for it. This allows its development to progress much faster than would otherwise be the case and the end product, at least in theory, should be superior because bugs and flaws are exposed to greater scrutiny. In my opinion it has a sleeker, friendlier and more intuitive interface. It hogs less system resources, contains no irritating adverts and tends to locate and transfer files faster because it is able to compress files as they are transferred and acquire further sources from other users (a feature which is especially handy if you happen to be downloading rare files).

The eMule coders recognise that not everyone is content to trust software installation routines to do what they claim to do, which I assume is why they offer their client in two flavours, an executable installation file and a zipped archive. The automatically installed version is aimed at people who don't want to create shortcuts to the program and the folder it lives in themselves or first time users, and the zipped version primarily targets people who already have the client installed and want to upgrade it to the latest release.

This couldn't be more straightforward; open the zip file in your favourite decompression program and extract the eMule executable file to the folder where you previously installed the client overwriting the original file. Doing this allows you to maintain all your preferences, your server list, credits, partially transferred files and so on and also ensures that no junk is left over from uninstalling and reinstalling the application. If you chose not to use the installation routine, you may at this point like to create a shortcut to the client and place it in your start menu or on the desktop for quick access.

Before running the client I would recommend that you edit the addresses.dat file found in your eMule installation directory to provide an URL from which your server list can be automatically updated each time you start the program. This can be done by opening the file in Notepad and simply pasting an URL into the otherwise empty space and saving it (the best place to locate such an URL is The Donkey Network - click on the 'server list' link and then copy the address of one of the server.met files listed). If you choose just "the nice ones" your client will be updated with the most popular servers, those with the most users connected and with the most files indexed.

The purpose of this is to ensure that your client is kept up to date with lists of operational servers to maintain the fastest transfer rates possible. Note that it is advisable, but not imperative that you connect to the most popular servers as each server communicates with every other server to find out which files are available. Connecting to one of the "nice ones" simply allows you to speed up this communication process thereby reducing the time it takes to make source requests etc. For this to work effectively remember that once you have started the client you will need to enter the preferences menu and from within the 'server' tab tick the "auto-update serverlist at startup" option.

With this taken care of you can forget completely about server lists and use eMule as you would any other file sharing client. This aspect of configuring eMule is what causes newbies the most headaches and with this out of the way you can look forward to an easy ride from now on. Not so complicated when you know what you're doing is it? Without the server list the client is totally useless and if you didn't read around the topic before taking the plunge you wouldn't know anything about this necessity. For this reason many people give up on the client before they have chance to see what it's capable of.

Now you are ready to run the client for the first time. To do this double click on eMule's shortcut and wait for a few seconds while the server list is updated. While eMule attempts to make a connection to one of the servers in the server list keep an eye on the variable icon in the bottom right hand corner to give you a rough idea of your success rate. A globe with two green arrows indicates that you have established a strong connection, a globe with two yellow arrows indicates that you have established a weak connection (i.e. you have a low ID rating) and a globe with two red arrows indicates that you are not connected to the network at all.

If you have a low ID rating you may wish to make a note of it (see the 'serverinfo' box) and look up its meaning at the eMule home page. Knowing the source of the problem will aid you in finding a way to resolve it. When you are running the client minimised you can view your connection status by looking at the donkey icon in your task tray; a grey donkey denotes that you are not connected to a server and a brown donkey indicates that you have successfully established a connection to a server.

In the preferences menu the first thing to change is your nickname; the option which allows you to do this can be found under the 'general' tab. By no means is this essential to the operation of the client, but who wants to live life as a number? (or more accurately as the URL of the eMule home page). Before moving on to the next tab it's a good idea to click on the box marked 'ed2k' - this provides the means to associate eDonkey web links with eMule. If the box is greyed out it means that this has already been taken care of by the installation routine.

Next visit the 'connection' tab and click on the 'wizard' button to make sure eMule knows what kind of internet connection you intend to unleash the client upon. It is vital that you provide accurate information here otherwise you will find that you will not be able to utilise your full allocation of bandwidth or the client will attempt to transfer files at a rate which your connection is incapable of reaching - this will cause a jam with the result that you will not have enough free bandwidth to send file request data packets to eDonkey servers.

Also, because eMule automatically regulates your download/upload ratio you could find that people are attempting to download from you at a rate which causes your connection to grind to a halt making it impossible to do anything else with it. The wizard should be able to provide optimum settings based on your connection, but if you find that eMule is using more bandwidth than you would like, you have the option to tweak these settings yourself. Remember though that if you decrease the rate at which people can download from you, you also decrease the rate at which you can download from other users. eMule is firmly founded on a "give and take" philosophy, which I'm sure you'll agree is good news for the network as a whole and is worth the inconvenience of having less control over the client.

You may have noticed that there are two places to enter what seems to be identical information in the 'connection' section of the preferences menu. The values in the 'capacities' boxes are the upper limits of your upload and download connection speed and are used to construct the graphs in the 'statistics' window. The values in the 'limits' boxes, however, allow you to set how much bandwidth you would like eMule to utilise. For hands free operation, tick the "auto-connect on startup" and "reconnect on loss" tick boxes and move onto the next section of the preferences menu.

Here it is wise to tell the client that you would like to update the server.met file each time eMule is started; this is done by ticking the relevant box as alluded to earlier in this tutorial. From the 'servers' window it is also possible to add individual servers manually or by providing the URL of a server list whenever you feel it is necessary, but this I feel merely complicates the issue unnecessarily.

The options accessed via the 'directories' tab allow you to choose which files you would like to share with other users on the network. By default your 'incoming' folder (where completed downloads are moved to) and your 'temp' folder (where partially transferred files are stored) are shared. To share other folders you are required to scroll through the explorer-like directory tree and put ticks in the boxes next to the folders you wish to become accessible to other users.

The remaining options are probably best left alone unless you are experiencing difficulties, so let's close the preferences tab and see what we can find. You can add files to your download queue in a variety of different ways; some are more effective than others. The built-in search engine, surprisingly, is not the best way to locate files; more often than not it will return zero hits even if the files you seek are available to the network. I sense this is one aspect of the client which needs serious development.

But not to worry, the people who see eMule as a goldmine of files are not the ones using the built-in search engine. No doubt they are instead using an independent file database search engine such as the one at Share Provider. What these do is index all the files available to the network allocating each of them with a unique 'hash' code. If you know the hash code of a file you wish to download it can be pasted into eMule to be added to the transfer queue. A common misconception amongst newbies is that these files reside on the same server where they are linked from. If this was the case the service would quickly be shut down.

Actually what they do is provide unique file identifiers which can be obtained irrespective of the availability of the files themselves. For instance, if you add these file identifiers into your download queue, eMule will attempt to download them by scouring the network for relevant sources. If no sources are available, the file will remain queued until someone sharing that file joins the network allowing the transfer to initiate.

The advantage of using such a system is that you can avoid downloading fake files because you know exactly what you are getting before the transfer commences. Was that a sigh of relief from the 56k-ers amongst us? Also far more information pertaining to the files is available than would otherwise be the case. Along with the hash link you get a synopsis of the movie plot or game or reviews of music albums etc, file sizes, popularity information, media length, quality ratings and there's even a place where users can post comments about the files they've just downloaded.

At Share Connector you can either use the search engine to track down files or you can browse through the category listings. Because it's not possible to list every single file in each category you will find that only the latest releases appear here. This is a great option if you don't really know what you are looking for but want to see what's new by window shopping.

Once you've found something you want to download you can either click on the "add all files from this release to eDonkey 2000" link or you can add each link manually, if for whatever reason, your browser isn't capable of communicating with eMule. To do this, click on the link to be taken to the file details page, right click on the link and copy it into your clipboard. Now open eMule, click on the search button and paste the link into the direct download ed2k link box and press start. Again this will add the file to your download queue where you will be able to see it transferring providing sufficient file sources are available.

Share Connector is probably the best search engine of this kind, yet it is not the only one. File Donkey and eMoogle (German) provide eDonkey search engines, but without the accompanying information as you would find at Share Connector. Whenever you run a search query the keywords are matched with filenames and the most frequently occurring ones are chosen to avoid confusion. Share Virus and Share the Files are functionally very similar to Share Provider, as are more specialist search engine sites like Music Donkey and TV Underground.

When you have a selection of files queued to download switch to the 'transfer' window and have a peek at the pretty coloured bars - these are progress meters. These hues aren't random, they actually mean something. Who would have thought it, eh? The green bar at the top represents the total download progress of the file and the different shades of blue denote the current availability of various parts of the file (the darker the shade of blue the greater the number of available sources). Red areas indicate parts of the file which are missing in all available sources, black areas show the parts of the file you already have and yellow segments represent the parts of the file which are currently downloading.

If you click on one of the files a list of sources will drop down with yet more pretty coloured bars. Just to confuse matters further, the colours here have a slightly different meaning. Yellow indicates a segment of the file which is currently being uploaded to you, the green bits represent the segments of the file you already have, silver areas are the parts of the file which the source computer is also still searching for and the black sectors highlight parts of the file which you are still missing. Almost like swapping stickers or baseball cards at school isn't it?

If you look to the right of the progress bars you will see a column labelled 'sources'. In this column you will find a set of three numbers for each file which take the format 23/67(5). The first number represents the number of useful sources i.e. the ones which are available to you, the second value indicates the total number of sources for the currently downloading file and the value in brackets denotes the number of sources you are actually making use of at that moment in time (or the number of parts you are currently transferring if you like).

Click on a currently downloading file and a 'queue ranking' figure will be appended to the bottom of the 'priority' menu (QR: 100 for example). This value indicates how long you will have to wait before you can make use of this source - the lower the number the sooner you will be able to access the source.

So there you have it, you are now a fully qualified donkey rider - where you take your new pet is up to you. Like all fabled donkeys, this one just needed the right kind of carrot on a stick to get it plodding. In this instance, an up to date server list functions as the perfect carrot substitute - such a silly little thing to come between you and the open road, isn't it? Bon voyage!

Friday, 10 October 2003

, , ,

My movie is too large to fit on a CD-R. Is it possible to chop it up and put it on two CD-Rs?

You can do this using Virtual Dub's 'save segmented AVI' option, but this is very slow because you have to recompress the file using the full processing mode for both video and audio streams. If you only want to cut the movie in two then it would be much faster to do it manually using the techniques described in my blog entry entitled, "How do I delete frames from a movie?"

What you need to do is open the file you wish to cut in half and place a marker at the point you would like to make the incision remembering to note down the frame number displayed in the status bar. Now move the slider to the final frame of the movie and place the end marker. Delete this part of the movie and save the remaining frames as name-of-movie part 1.

The second stage requires you to open the original file and repeat the process to highlight and delete the first half of the movie and save the remaining frames. To make the split perfectly precise use the 'go to' option situated in the 'edit' menu and type in the number of the frame at which you cut the file in the first stage described above. You can name this segment name-of-movie part 2 and proceed to burn the AVI files to separate CD-Rs.