FTP stands for File Transfer Protocol and refers to a rather old method of transferring files through a network.
FTP owes its great popularity to anonymous FTP: many large internet servers offer all users access to FTP archives. This access is not blocked by a password (unlike the other FTP).
A major drawback of FTP is that during the login process the username and password are transmitted unencrypted. A secure alternative is SFTP (Secure FTP) based on SSH. HTTP, the protocol used to transfer web pages, is also often used as an alternative to FTP.
This blog post is only about using FTP—that is, from the client point of view. For FTP to work, an FTP server must be running on the remote site.
The ancestor of all FTP clients is the interactive text command ftp. Because it typically transfers files from or to the current directory, you should use cd to change to the relevant working directory before starting ftp. The FTP session can then be initiated using the command ftp user@ftpservername or simply ftp ftpservername. If you want to use anonymous FTP, you need to enter anonymous as the user name.
Once the connection has been established and the password entered, you’re ready to go: You can use the cd, pwd, and ls commands, which have the same meaning as in Linux, to move through the directories of the FTP archive. To transfer a file from the FTP archive to the current directory of your computer, you can run get file. The file name remains unchanged. Conversely, you can use put to transfer a file from your current directory to a directory of the FTP archive. Of course, this is only possible if you have write permission for the directory. With anonymous FTP, this is usually only the case for a directory with a name like /pub/incoming. The FTP session can be terminated using the quit or bye command. For a reference of the main FTP commands, see this table.
The ftp command isn’t very easy to use. Fortunately, there are countless alternatives. Most web browsers and file managers available on Linux can also be used for FTP downloads. If you want to work in text mode, you can use the interactive ncftp command. The wget, curl, and lftp commands help with the automated transfer of files or entire directory trees via FTP.
SFTP (Secure FTP)
The sftp command is part of the openssh package. Internally, sftp uses a completely different protocol than ftp and, like ssh, can only be used if an SSH server is running on the remote site. Anonymous FTP is not possible with sftp. Apart from that, the operation of the program is the same as that of ftp. sftp -b batchfile allows you to automate SFTP downloads.
You can also use the Dolphin (KDE) or Nautilus (GNOME) file managers to initiate an SFTP connection by entering the address “sftp://user@servername”. After the password request, the programs display the FTP directory as a local directory. Both file managers also directly support the SSH protocol, which works even if sftp is not available. For this purpose, you need to specify the address in the following format: fish://user@servername.
wget
The interactive approach of the ftp command is unsuitable for automating downloads—such as in a script. But ftp is also very inflexible in other respects. For example, it’s impossible to resume an interrupted download on your own. This can be avoided by using the wget command, which is specifically designed to carry out large downloads or transfer entire directories. wget supports the FTP, HTTP, and HTTPS protocols in equal measure.
In its basic form, wget simply downloads the specified file:
user$ wget ftp://myftpserver.com/name.abc
If the download is interrupted for any reason, it can be resumed without any trouble by using -c:
user$ wget -c ftp://myftpserver.com/name.abc
Downloads of large files, like ISO images of Linux distributions, can take several hours if the internet connection isn’t very good. That's why it's a good idea to perform the download overnight. The following command almost ensures that the file is actually on the computer the next morning:
user$ wget -t 20 --retry-connrefused \
http://mydownloadserver.com/name.iso
Due to -t 20, the download is retried up to 20 times after a connection failure. --retry-connrefused causes a new attempt to be started even after the connection refused error. This is useful if the download server is known to be unreliable and is repeatedly unavailable for short periods of time.
The following command downloads all the files necessary to later read the specified web page offline in an unchanged state:
user$ wget -p -k -E -H http://mywebsite.com/page.html
Let us briefly explain the meaning of the options: -p also downloads CSS files and images. -k changes the links in the downloaded files so that they refer to local files. -E adds the .html identifier to downloaded script files (ASP, PHP, etc.). -H also tracks links to external websites.
If you want to read an entire website offline, the following recursive download command (-r option) will help. The recursion depth is limited to four levels by –l 4:
user$ wget -r -l 4 -p -E -k http://mywebsite.com
curl
The curl command helps to transfer files from or to FTP, HTTP, or other servers. The man page lists an impressive range of protocols that curl can handle. In this section, however, I will restrict myself to FTP uploads. For script programming, it’s particularly useful that curl can also process data from standard input or write to standard output. So you don't need to create a *.tar.gz file first and then transfer it to the FTP server, but can perform both operations simultaneously using a pipe.
The following command transfers the specified file to the FTP server backupserver and saves it in the dir directory:
user$ curl -T file -u username:password ftp://backupserver/dir
To process data from the standard input channel, you can use -T to specify a hyphen as the file name. The following command saves the result of the tar command directly in the name.tgz file on the FTP server:
user$ tar czf - dir/ | curl -T - -u usern:pw ftp://bserver/name.tgz
curl is also well suited for testing REST APIs. In the following example, a service on the https://ipinfo.io website tries to determine your location based on the IP address of your call:
user$ curl https://ipinfo.io
{
"ip": "91.115.157.28",
"hostname": "91-115-157-28.adsl.highway.telekom.at",
"city": "Graz",
"region": "Styria",
"country": "AT",
"loc": "47.0667,15.4500",
"postal": "8041",
"timezone": "Europe/Vienna", ...
}
lftp
lftp is a convenient interactive FTP client. However, the command is also well-suited to execute FTP uploads or other commands in a script. For this purpose, you can either use -c to pass several FTP commands separated by semicolons to lftp or use -f to specify a file that contains these commands line by line. The first command will always be user username,password servername to connect to the FTP server. The following command demonstrates a file upload:
root# lftp -c "open -u username,password backupserver; put www.tgz"
If you want to give the file a different name on the FTP server, you need to additionally specify the -o <newName> option. lftp displays the current progress during the upload.
To transfer an entire directory to the backup server instead of a file, you can use the mirror -R command. This command usually copies directories from the FTP server to the local computer. -R reverses the transfer direction. Here is another example:
root# lftp -c "open -u usern,passw bserver; mirror -R directory"
Unlike other FTP clients, lftp supports the du command, which you can use to determine how much disk space your backup files already consume. This is important if your storage space on the backup server is strictly limited. The following command shows how you can determine the amount of memory already used without interactive intervention. The -s option indicates that you are only interested in the final sum. -m ensures that MiB is used as the unit of measurement:
user$ lftp -c "open -u username,password bserver; du -s -m"
2378 .
If you want to use the result for a calculation, the second column (i.e., the dot indicating that the numerical value refers to the current directory) will interfere. Simply add cut -f 1 after the command to extract the first column:
user$ lftp -c "open -u usern,passw bserver; du -s -m" | cut -f 1
2378
rsync, mirror, and sitecopy
rsync helps you to copy or synchronize entire directory trees. If neither an SSH nor an rsync server is running on the partner computer, you can use the mirror or sitecopy commands instead of rsync. The Perl script mirror from the package of the same name copies entire directory trees from an FTP server to the local computer. The sitecopy command, on the other hand, is optimized to upload a directory tree to a web server, with the data transfer taking place either via FTP or WebDAV.
Editor’s note: This post has been adapted from a section of the book Linux: The Comprehensive Guide by Michael Kofler.
Comments