OK, here I go! You asked for it! I'll keep it a little practical and leave some technical info to the real techies
The basis of images on a computer is understanding that all images on computers are made up of pixels, little dots. Every pixel can have one color. So, ideally, when you save an image, you actually save the information of every pixel (what color, what effect, tranparency etc...). For larger images, this would result in enormous files. That's why different compression techniques have been designed which result in different file formats. Let's take a look at the most common on the web.
TIFF & BMP
First of all, you have formats that do save the information of pixels seperately. The TIFF and BMP (Windows) are the most common filetypes using this system.
This means that these files can be extremely big because there is no compression whatsoever. But this also entails that there is no quality loss. When compressing an image, there is always some loss.
BMP is rarely used on the internet and should be avoided. Non-Microsoft programs can often not read these files. TIFF-files should also not be used on the web, but this is a very common format for high-quality photographs or images, especially used in printing.
JPEG compresses images. Instead of remembering all information of all pixels, only the exact information of some pixels is saved (I believe every 8 pixels, but I could be mistaken). From the pixels in between, only the variation to this one pixel is saved. That's why you sometimes see little squares of 8 (?) pixels in badly made JPEG's.
This compression has several consequences. First of all, and this is the major reason for inventing this technolgy, you have a significantly smaller file. A JPEG-file is in average 10 times smaller than the same image in TIFF. On the other hand, you always have some loss. In most images, this isn't noticable on a screen by the naked eye. But depending on how much compression you are using (mostly expressed by the 'quality' of the JPEG), you can get some bluring.
JPEG, together with GIF, is probably the mostly used format on the web. The small size is of course the reason for this. But, understanding the compression technique a little, you might understand that a JPEG-compression is not ideal for all kinds of images. JPEG is ideal for images with a great variety in colors and shades. A photgraph is ideal. An image with big spaces in the same color are not so good. You often get bluring. So logo's should be saved in JPEG, but in GIF.
GIF & PNG
GIF and PNG also use a compression technique, but different from JPEG. Where JPEG saves pixels and variations to these pixesl, GIF saves spaces with the same color. So, as we said before, GIF is ideal for saving an image where you have less shades of colors but rather big spaces of the same color, like buttons, logo's or backgrounds. GIF-files are also used very much on the internet because you can make a certain color transparant. This is something that you cannot do with a JPEG.
PNG is a rather new format to compete against GIF because there is still a patent on GIF. There are some small differences between them, but the main advice is not to use PNG because it is not so widely supported.
There is a lot more to say about image formats (animated GIFs, transparancy, printing, colors, ...), but this might get you started in the right direction. A good understanding of these things should be the basis for graphical designers or amateurs on the web. You can easily make you site twice as fast using the correct formats.
There must be numerous sites out there explaining this a lot better and more in detail. I can't remember all the ones I read, but there was a comprehensive article about it on SitePoint not so long ago: http://www.sitepoint.com/article/601
Well, that's it for now. But, as I sais in other posts, I'm not 'all-knowing'. So if someone can correct or add something, please feel free!