Table of Contents

Transferring files securely between machines is a fundamental part of the ETL (Extract, Transform, Load) process, which involves extracting data from one source, transforming it for analysis, and loading it into a data warehouse. The challenge? Ensuring these files are both sent and received without interception by malicious entities.

For years, FTP (File Transfer Protocol) served as the go-to method to transfer files between a client and server on a network. However, for more sensitive data, SFTP has emerged as the more secure alternative. So, let's dive into what SFTP is and how you can use it for safe file transfers.

Here are the 5 key takeaways from the article: 

  • SFTP, which stands for SSH File Transfer Protocol, is the preferred choice over the traditional FTP for securely transferring files due to its encrypted nature, protecting data from unauthorized interception.
  • Unlike FTP which sends data in plaintext, SFTP employs the SSH protocol to encrypt data during transfer, ensuring that malicious entities can't decipher the content.
  • Users can either build their own SFTP server, especially on Linux systems or opt for managed cloud services like AWS Transfer for SFTP for a hassle-free experience.
  • Using the sftp command, users can connect to remote servers and initiate secure file transfers. The process includes using commands like get for downloading and put for uploading.
  • Integrate.io fully supports SFTP, making it easier for businesses to establish secure ETL workflows with various cloud data warehouses and analytics platforms.

In this article, we'll go over the basics of SFTP, its key benefits, and how it can be used for secure data transfers. 

What is SFTP and What Does It Stand For?

SFTP stands for SSH File Transfer Protocol. It's a network protocol designed for securely transferring, accessing, and managing files on a remote computer. Unlike traditional FTP which transmits data in plaintext, SFTP offers an encrypted alternative, thereby enhancing security.

Specifically, SFTP leverages the SSH (Secure Shell) protocol's encryption capabilities. This ensures that any files transferred are shielded from prying eyes during transit. Such robust security makes SFTP the preferred choice over FTP in most cases.

For businesses driven by data and reliant on insights from their analytics, SFTP provides several advantages:

  • Ensures end-to-end encryption and user authentication to safeguard sensitive information.
  • Complies with key industry regulations like HIPAA, Sarbanes-Oxley, PCI DSS, and GDPR.
  • Facilitates data accessibility across the organization.
  • Supports bulk data transfers, boosting speed and efficiency.

For more information about Integrate.io's native SFTP connector, visit our Integration page.

How Does SFTP Work?

SFTP builds on the existing FTP protocol, which is not safe enough to use for many purposes. FTP transfers files using plaintext, which potentially allows bad actors to "eavesdrop" on your communications.

As mentioned above, SFTP depends on the SSH protocol to securely encrypt the files it transfers. Thanks to this protection, anyone trying to spy on an SFTP connection will see nothing but unintelligible bits and bytes, rather than meaningful data.

Part of this increased security includes multiple ways to authenticate users. As we'll discuss in the next section, you can log into an SFTP connection using a username and password, or a username and SSH key.

How to Set Up an SFTP Server

There are two main ways to set up an SFTP server: 

  • Build your own: Any Linux system can function as an SFTP server. You'll need to set up users, determine authentication methods (password or SSH keys), and assign directories and permissions. On Windows servers, third-party software like WinSCP or Filezilla SFTP Server is required, as Windows doesn't natively support SFTP.
  • Use a managed cloud service: For those seeking a hassle-free approach, various cloud-based solutions offer different features and pricing structures. AWS Transfer for SFTP stands out as a popular choice, integrating seamlessly with Amazon S3's robust storage solutions. SFTP To Go, based on AWS's offering, delivers a similar experience with added user-friendliness and cost efficiency.

    How to Connect with SFTP

    Before securely transferring files with SFTP, you need to establish a connection to the remote server. In this tutorial, we’ll assume you’re using a Unix-based operating system such as Linux or macOS. 

    The sftp command connects to a remote server using the SFTP protocol. To get started, you will need the server’s domain name or IP address, as well as your username on the server. For example:

    > sftp username@example.com > sftp username@192.168.1.1

    By default, SFTP uses TCP port 22 for communications. If you want to use a different port, you’ll have to use the -P flag to indicate the alternate port number:

    > sftp -P 2222 username@example.com

    Enter the appropriate sftp command into the terminal. The connection will be initialized and the server will ask you to enter your password. After entering your password correctly, you’ll be greeted by a welcome message and see an SFTP command prompt:

    Connected to example.com. sftp>

    If you would prefer not to enter your password every time you use SFTP, SSH keys provide an efficient alternative. After setting up and sharing your public key with the remote server, it'll recognize and authenticate you automatically.

    How to Transfer Files Using SFTP

    Now that you’ve connected to a remote server, how do you transfer files using SFTP? The steps to follow will depend on whether you’re transferring files with SFTP from a remote server to your machine, or vice versa.

    Regardless of which direction you want to transfer files, it’s a good idea to first check your current directories using the lpwd and pwd commands. The lpwd command tells you the current working directory on your local machine, and the pwd command tells you the current working directory on the remote server:

    sftp> lpwd Local working directory: /home/user sftp> pwd Remote working directory: /

    You can use the lcd command to change the local working directory, and the cd command to change the remote working directory.

    Transferring Files with SFTP From a Remote Server to the Local Machine

    The get command is used to transfer files from a remote server to your local machine. You need to specify the relative path to the file you want to download, based on your current working directory on the server:

    sftp> get Winter.jpg

    The above command will download the file “Winter.jpg” from the current remote working directory.

    sftp> get Downloads/manual.pdf

    The above command will download the file “manual.pdf” from the directory “Downloads,” which is located in the current remote directory.

    Once you execute a get command, you’ll see a progress bar in the terminal window that indicates the status of the download, e.g.:

    Fetching /Downloads/manual.pdf to manual.pdf /Downloads/manual.pdf                                                   100%   12MB   2.9MB/s   00:04

    When the download is complete, you can find the file in your current local working directory.

    To download an entire directory at once, use the -r flag:

    get -r Downloads

    Transferring Files with SFTP From the Local Machine to a Remote Server

    Transferring files from your machine to a remote server is very similar to the reverse operation. The major difference is the command that you use: put instead of get. For example:

    sftp> put Winter.jpg sftp> put Downloads/manual.pdf

    The first command above transfers the file "Winter.jpg," which is located in your current local working directory, to the current working directory on the server. The second command transfers the file "manual.pdf," which is located in the directory "Downloads" in the current local working directory, to the server.

    As before, you should see a progress bar in the terminal window during the upload:

    Uploading Winter.jpg to /Winter.jpg Winter.jpg                                100% 847KB 488.5KB/s  00:01

    List of Useful SFTP Commands

    • bye: This command closes the SFTP connection.
    • cd: This command changes the current remote working directory.
    • chmod: This command changes the permissions of a given file or directory. You can grant or revoke the permissions to read, write, and/or execute the given file or directory.
    • df: This command displays the available disk space on the server, which can help you troubleshoot the reason for a failed transfer.
    • exit: This command closes the SFTP connection.
    • get: This command downloads the given file or directory.
    • lcd: This command changes the current local working directory.
    • lls: This command displays the contents of the current local working directory.
    • ls: This command displays the contents of the current remote working directory.
    • lpwd: This command displays the current local working directory.
    • put: This command uploads the given file or directory.
    • pwd: This command displays the current remote working directory.
    • quit: This command closes the SFTP connection.
    • sftp: This command initiates an SFTP connection.

    Transferring Files with SFTP and Integrate.io

     

    Integrate.io includes full support for the SFTP protocol in ETL workflows. Using Integrate.io and SFTP, you can send data to, and receive data from, dozens of cloud data warehouses and analytics platforms.

    To start using SFTP with Integrate.io, follow the steps below:

    • Click on your avatar, and select Account Settings from the dropdown menu.
    • Click on Connections in the left menu, and select "New Connection"

    thumbnail image

    • Click on "Secure File Transfer Protocol (SFTP)", and enter the relevant connection information.

    thumbnail image

    • Click on "Test Connection" to test the connection. If successful, click on "Create SFTP Connection."

    For more details, check out our full article on creating an SFTP connection in Integrate.io.

    Want to learn more about how the Integrate.io platform can help you build powerful data integration pipelines to the cloud, including with the SFTP protocol? Schedule a call with our team for a chat about your business needs and a trial of the Integrate.io platform.