Using simultaneous connections entire websites can be downloaded very quickly. Links can be checked and converted to HTTP or local disk file system. Decide which file types to download and directories to crawl. Handle cookie and session login.