Demystifying SFTP Data Transfer:
A Step-by-Step Tutorial

In today’s data-rich world where large volumes of data exchange have become frequent, ensuring the data transfer mechanism is safe and secure is essential. Data security is of utmost importance, and the data transfer tool you select should provide data encryption. 

Secure File Transfer Protocol (SFTP) is one such file transfer mechanism that businesses can leverage to securely transfer files between users and locations, regardless of their network or device type. The data encryption offered by SFTP ensures that sensitive data is protected from unauthorized access.

The 5 key takeaways from this article:

• Secure File Transfer Protocol (SFTP) stands out as a robust choice for transferring data securely with its encryption and authentication features. Unlike traditional FTP, it protects your data from unauthorized access.
• SFTP not only excels in security but also offers compatibility with various operating systems, large file-handling capabilities, and real-time access for collaborative work.
• Before diving into SFTP, you need to have the right tools, including reliable SFTP clients and servers, and establish your SFTP credentials, which include username/password setup and SSH key generation.
• SFTP connection can be established and managed from a client interface as well as the command line, easing navigation and handling file and folder permissions.
• SFTP file transfer can be automated using scripts that include multiple commands to be executed. The scripts can be scheduled to execute using tools like Windows Task Scheduler.

In this article, we’ll break down the entire SFTP process. The easy steps discussed in this article can be leveraged by both beginners and experienced to brush up on these SFTP skills.

The Unified Stack for Modern Data Teams

Get a personalized platform demo & 30-minute Q&A session with a Solution Engineer

GET A LIVE DEMO

Introduction

Secure File Transfer Protocol (SFTP) is a secure file transfer protocol that uses the Secure Shell (SSH) protocol to encrypt all data transmissions. This makes it a much more secure option than traditional file transfer protocols like FTP, which do not encrypt data by default.

SFTP is often used to transfer sensitive data like financial records, medical records, and intellectual property. Being more reliable than FTP, it also serves as a good choice for transferring large files.

In today’s digital world, the exponential growth of interactions underscores the urgency to safeguard sensitive information from cyber threats. A secure file transfer protocol, such as SFTP ensures encrypted transmission, shielding critical data from unauthorized access and breaches. With remote work, collaborations, and compliance requirements, secure file transfer not only protects privacy but also improves collaboration and keeps up with regulatory standards.

Let’s walk through SFTP’s features and benefits in detail.

Foundations of SFTP

Businesses and organizations need to constantly transfer sensitive data. This data could include financial records, health-related customer information, and other confidential data. There are various methods to transfer files across systems like email attachments, cloud storage, and other methods, but not all approaches are secure and efficient. If this data is not protected, it could be vulnerable to cyberattacks. Secure file transfer (SFT) is a method that allows transferring data over a network in a secure manner.

In addition to its security features, SFTP also offers several other benefits, including:

• Authentication: SFTP uses SSH authentication, which provides strong authentication for both the client and server. This helps to prevent unauthorized access to files.
• Data Integrity: SFTP encryption ensures data integrity during transfer. This helps to ensure that files are not corrupted during transmission.
• Encryption: SFTP uses strong encryption to protect all data transmissions. This helps to prevent unauthorized access to files.
• Compatibility: SFTP can be used on any platform that supports Secure Shell like Windows, Linux, and macOS.
• Large file sharing: Even when large files are involved, SFTP allows easy transfer, which can be difficult to share with other file-sharing methods. A variety of files including videos, audio files, etc. can be efficiently shared.
• Centralized storage: SFTP allows you to store files in a central location, which makes it easier for team members to find and access the files they need.
• Real-time access: SFTP allows multiple users to access files in real-time. This can help collaborate on documents or projects, as everyone can see the latest changes as they are made in one place.
• Scheduled Transfers: Scheduled transfers allow businesses to program file transfers as and when most suitable. This reduces network resource utilization and optimizes the data flow. This not only prevents congestion but also ensures that critical tasks are not affected by large file transfers.

Let’s compare SFTP to other transfer protocols like FTP and FTPS:

Setting up SFTP: The Preliminaries

Before diving into the world of Secure File Transfer Protocol (SFTP), there are some essential steps and tools you need to consider to ensure a smooth setup process.

Required Tools and Software

Recommended SFTP Clients and Servers:

To facilitate secure file transfers, it is essential to consider reliable client and server software. 

Popular SFTP clients and servers include WinSCP, FileZilla, Cyberduck, SolarWinds, and ExaVault.

System Requirements:

Ensure that your system meets the requirements for running SFTP software. Make sure the software you choose matches your OS (Windows, Linux, macOS). 

Establishing SFTP Credentials

Before you can initiate secure file transfers, you'll need to set up your SFTP credentials. This involves creating the necessary authentication mechanisms to ensure secure access:

Username and Password Setup:

Once you’ve chosen the SFTP client/server, create a strong and unique username and password.

Generating SSH keys:

For setting up SFTP using SSH keys, you need to generate SSH Keys. To do so, follow these steps:

• Open the command line terminal in your Mac/Windows machine
• Set up the RSA Token by using this command: ssh-keygen -t rsa
• A few self-explanatory questions will be presented upon executing the above command. Press enter for default and proceed. If you wish to keep an additional layer of security, you can set a passphrase as it is promoted.
• The output will look similar to this image:

• In the specified directory, you will see two keys: id_rsa, and id_rsa.pub. id_rsa is the private key and Id_rsa.pub is your public key.
• To log in using the SSH key, add the public key ‘id_rsa.pub’ to the SFTP server’s “authorized keys”.

Demystifying SFTP Data Transfer: Step-by-Step Tutorial

Having the SFTP setup done, we’re now good to dive deeper into action.

• Initiating Connection to the Server
  • Using SFTP Client Interface
    1. Launch your preferred SFTP client software: This could be FileZilla, WinSCP, Cyberduck, or any other SFTP client you've chosen (for the tutorial, this article considers WinScp as shown below).
    2. Enter Server Details: In the client interface, you'll see fields to enter connection details. These include the server's hostname or IP address, the port number (usually 22 for SFTP), your username, and your authentication method (password or private key).
    3. Authenticate: Enter the username and password in the designated fields and click ‘Login’ and you should be connected.

• • Using Command Line

1. 1. 1. To initiate a connection to the server using the command line, you can use the command: sftp username@hostname
      2. It will prompt you to enter your password for the account you’re trying to connect to. Enter the password and the SFTP connection will be initiated.

Navigating Directories

Now that you’ve connected to the SFTP server, it’s time to navigate the directories. Let’s discuss some common commands you can use in the terminal to do the same.

• Common commands to navigate directories:
  • ls: List files and directories in the current remote directory.
  • cd directory_name: Change your remote directory.
  • pwd: Display the path of the current remote directory.
  • mkdir new_directory: Create a new directory in the current remote directory.
  • rmdir directory_to_remove: Remove an empty directory from the current remote directory

SFTP also allows you to manage permissions for files and folders, ensuring that the right users have the appropriate access. Permissions are often represented as combinations of read (r), write (w), and execute (x) privileges.

• Handling file and folder permissions:
  • chmod <mode> <file>: The chmod command changes the permissions of the specified file. The chmod command takes two arguments: the mode and the file. The mode is a combination of three octal digits, each of which represents the permissions for the owner, the group, and the others. The following table shows the octal values for the permissions:

    Permission	Octal value
    Read	4
    Write	2
    Execute	1

    For example, the following command would change the permissions of the file file.txt to allow the owner to read and write the file and to allow everyone else to read the file: chmod 644 file.txt
    The 6 in the mode represents the permissions for the owner: read (4) and write (2). The 44 in the mode represents the permissions for the group and the others: read (4) and write (4).
  • chown <username> <file>: The chown command changes the owner of the specified file. The chown command takes two arguments: the username and the file.
    chown username file.txt command changes the owner of the file to the username.
  • chgrp <group> <file>: The chgrp command changes the group of the specified file. The chgrp command takes two arguments: the group name and the file. 
    chgrp groupname file.txt command changes the group of the file.txt to group ‘groupname’.
• Transferring Files
  So far, we’ve established an SFTP connection and can navigate the local and remote directories. Let’s see how we can transfer files from a local machine to a remote machine.
  • Uploading files to the server
    1. Once the SFTP connection is established, navigate to the local directory, which has the file you wish to transfer.
    2. Use the put command to upload the file. The syntax for the put command is: put <local_file> <remote_file>
    3. You can confirm if the file has been copied to the remote directory using the ‘ls’ command.
  • Downloading files from the server
    Use the get command to download the file. The syntax for the get command is: get <remote_file> <local_file>
    If using tools like WinSCP or FileZilla, you can leverage the user interface to establish an SFTP connection and upload/download the files from the UI.
• Troubleshooting Common Issues
  Often while transferring files through SFTP, we’re faced with issues that can commonly be categorized among connection failures, file transfer errors, and permission errors. Let’s walk through each of these, and their possible solutions.
  • Connection failures
    • Incorrect hostname or IP address: If you’re unable to establish the connection, check if the hostname and the IP address are correct. 
    • Incorrect username or password: Make sure the username and password you’re using are correct.
    • Port settings: Make sure the port number is correct, the default being 22.
    • Firewall and network problems: Check if there’s a network problem or if your firewall is blocking the SFTP connection.
    • Make sure the tool you’re using is updated.
    • If the problem persists, try connecting using another tool.
    • Try connecting from another computer or network.
  • File transfer and permission errors
    • File permissions: Make sure that you have the correct permissions to access the file you are trying to transfer.
    • Corrupt File: Make sure the file you’re trying to send/receive is not corrupted.
    • If the problem persists, try transferring a smaller-sized file.
    • Try transferring the file to a different directory to troubleshoot the permissions issue.

Enhancing Your SFTP Skills: Advanced Tips

Automating File Transfers

When file transfers are frequent and repetitive, businesses and individuals can save time and effort by leveraging the automation capabilities supported by SFTP. Instead of downloading or uploading files manually every time, you can automate a set of commands to be executed as and when needed.

One of the ways to automate the execution of these commands is by using scripts. The scripts essentially contain a set of commands that you’d usually do manually. Instead of executing each command one at a time, the script file allows you to execute them together. 

Different SFTP software supports command line connectivity, for the demonstration, let’s consider WinSCP.

To run WinSCP from the command line, open the command line and change the directory to where WinSCP is installed (generally C:\Program Files (x86)\WinSCP). To establish connectivity further, you can refer to the WinSCP documentation. 

Creating script:

Let’s create a simple script that downloads a file from the remote server, and uploads it back again.

open sftp://user1:password@example.com/ -hostkey="ssh-rsa 2048 xxxxx..."

cd /home/user1

get examplefile.txt d:\

close

open sftp://user2:password@example.com/

cd /home/user2

put d:\examplefile.txt 

close

exit

This script connects to the ‘example.com’ server with account ‘user1’, downloads the file, and closes the session. It then opens the connection again with account ‘user2’, uploads the file, and closes the session.

Executing the script:

Save the script created as ‘myscript.txt’, and it can now be executed with the following command: winscp.com /ini=nul /log=myscript.log /script=myscript.txt

Here, the /script command passes the myscript.txt to winscp.exe, and the /log=myscript.log enables session logging. 

The above command can be embedded in a .bat file.

To further schedule the task execution, you can utilize Window’s Task Scheduler.

Here are a few steps you can follow:

Search ‘Task Scheduler’ in the search bar.

1. Go the ‘Action’ > ‘Create Basic Task’
2. Give a name to the task and proceed.
3. Choose when you want to run the task and click Next.
4. Select Start a program and click Next.
5. Browse WinScp.exe file.
6. In the ‘Add arguments’ add the desired WinSCP command-line parameters as mentioned above.
7. Review the task and click ‘Finish’.

Implementing Advanced Security Measures

To add additional layers of security, you can configure two-factor authentication and IP Whitelisting to ensure only authorized users can access the connection. 

Various SFTP tools allow you to configure 2FA and IP Whitelisting through the interface. If using a Linux environment, you can follow these steps:

Two-Factor Authentication

1. To enable 2FA, we’ll be using the Google Authenticator package. Run the following commands to install the package:
   • sudo apt update
   • sudo apt upgrade
   • sudo apt install libpam-google-authenticator
2. Once done, you’ll have to make a few changes to the ‘sshd_config’ file.
   • Open the file using the following command: sudo nano /etc/ssh/sshd_config
   • Find and comment out the following line: # PasswordAuthentication yes
   • Uncomment the following line: ChallengeResponseAuthentication yes
   • Enable the UsePAM with the following line: UsePAM yes
3. Next, you need to create user groups for the SFTP users. You can do so using this command: sudo groupadd sftpusers
4. At the end of the sshd_config file, replace the line Subsystem sftp    /usr/lib/openssh/sftp-server with Subsystem sftp internal-sftp
5. Finally, in the end, enter the following lines:
   • Match Group sftpusers
   •     ForceCommand internal-sftp
   •     ChrootDirectory /var/sftp/%u
   •     PermitTunnel no
   •     AllowAgentForwarding no
   •     AllowTcpForwarding no
   •     X11Forwarding no

Whitelist IP Configurations

Now that we have the 2FA set up, let’s create a config that has a list of IPs that will allow the specific IPs to access the server without two-factor authentication. 

1. Create an access.conf file with the following command: sudo nano /etc/security/access-local.conf

2. And add the following content to it:

• • #2FA Disabled user: IPs
  • #Enable 2FA for all others
  • -: ALL: ALL

Having done that, now you can create two-factor enabled users and IP-whitelisted users, with the file structure and permissions as discussed above. 

Conclusion

In a world where data fuels innovation and collaboration, the importance of secure and reliable data transfer is increasing every day. SSH File Transfer Protocol offers not just security but a wealth of benefits, making it the preferred choice for organizations and developers. Its robust authentication methods and encryption ensure that your data remains private, and shielded from cyber threats.

SFTP’s compatibility across various platforms, support for large file sharing, real-time access, and automation capabilities strengthen its versatility and usability. It streamlines collaboration, improves productivity, and aligns with regulatory standards such as GDPR and HIPAA.

Continuous learning and practice are the keys to mastering SFTP. As an individual or as a part of an organization, learning SFTP benefits when data transfer comes into the picture. 

How Integrate.io Can Help with SFTP

Integrate.io is a cloud-based no-code ETL solution that provides a rich set of features. Integrate.io provides 100+ built-in connectors including SFTP which allows quick, easy, and secure data integration. Its drag-and-drop user interface allows even non-technical users to leverage cloud capabilities.

Leveraging SFTP along with Integrate.io can prove to be a killer combination for your business. Get in touch with our team of data experts to discuss your business requirements or sign up for a 14-day free trial today to experience the magic firsthand.

The Unified Stack for Modern Data Teams

Get a personalized platform demo & 30-minute Q&A session with a Solution Engineer

GET A LIVE DEMO

FAQs

What is the difference between FTP, FTPS, and SFTP? 

FTP transfers data without encryption, FTPS adds encryption using SSL/TLS, while SFTP secures data with SSH encryption and authentication.

How does SFTP ensure data security during transfer?

SFTP ensures data security during transfer via strong encryption and secure authentication mechanisms like hash encryption and SSH.

Can I use SFTP on any operating system?

Yes, SFTP is compatible with various operating systems, including Windows, Linux, and macOS.

Why does my SFTP connection fail?

SFTP connection failures can occur due to incorrect credentials, firewall issues, or network problems.

How can I improve my SFTP transfer speeds?

SFTP transfer speeds can be improved by optimizing your network, using compression, or upgrading your SFTP client/server.

Is there a file size limit with SFTP transfers?

SFTP typically doesn't have strict file size limits, but server configurations may impose limits.

How do I automate SFTP file transfers?

You can automate SFTP file transfers using scripts and scheduling tools like Task Scheduler on Windows.

What are common SFTP commands to know?

Common SFTP commands include ls, cd, pwd, mkdir, rmdir, chmod, chown, and chgrp.

How do I manage permissions on SFTP?

Permissions on SFTP are managed using commands like chmod, chown, and chgrp, ensuring proper access control.