How to Connect Integrate.io to Amazon S3

Many things are more powerful together than apart, and that includes big data. By combining and transforming information from multiple sources, including files, websites, and databases, you can identify valuable insights that would otherwise have gone unnoticed in isolation—a process known as data integration.

Being able to rapidly, automatically perform data integration is essential for data-driven organizations. In this article, we’ll offer this all-in-one guide for connecting Integrate.io to Amazon S3, one of the most popular cloud data storage services.

What is Amazon S3?

Amazon S3 (Simple Storage Service) is an offering from Amazon Web Services for storing objects in the public cloud through a user-friendly web interface. Perhaps the greatest benefit of S3 is that it is highly available and reliable: according to the AWS website, it is designed to provide 99.999999999 percent durability and 99.99 percent availability per year.

S3 can be used to store any data in object format. Each object consists of a file together with its metadata and an ID number. S3 has multiple storage classes depending on how often you access the data: S3 Standard is for the most frequently accessed data, while S3 Glacier is for infrequently accessed data archives, with retrieval times ranging from minutes to hours.

For more information on Integrate.io's native Amazon S3 connector, visit our Integration page.

What is Integrate.io?

Integrate.io is a powerful, feature-rich data integration platform that makes it easy to build robust data pipelines to your cloud data warehouse. With its user-friendly, drag-and-drop interface, Integrate.io makes it easy for even non-technical users to connect your data sources with your data warehouse or data lake in the cloud. Integrate.io comes equipped with more than 100 integrations with a wide variety of data sources, including SaaS applications, databases, and analytics platforms.

Data integration is the first step toward transforming your raw data into refined business intelligence to help you make smarter data-driven decisions. Amazon S3 is a data storage powerhouse, and connecting it with a data integration platform such as Integrate.io will help you get the most out of your enterprise big data. 

The Unified Stack for Modern Data Teams

Get a personalized platform demo & 30-minute Q&A session with a Solution Engineer

GET A LIVE DEMO

How to Connect Integrate.io to Amazon S3

Connecting Amazon S3 and Integrate.io is a snap. To get started, you’ll first need to allow Integrate.io access to data on Amazon S3. The basic steps are as follows:

1. In the AWS IAM (Identity and Access Management) module, create a user account for working with Integrate.io.

2. Click on the “Add User” button.

3. Enter a username (e.g. “Integrate.io”).Make sure that the box “Programmatic access” is checked in order to generate an access key ID and secret access key.

4. Click on the “Create user” button.

5. In the IAM module, save the user account’s security credentials for later use when creating a connection in Integrate.io.

6. Click on the “Download .csv” button to save the security credentials as a .csv file.

7. In the IAM module, attach a user policy (provided by Integrate.io) to the newly created user account.

8. In the list of users, select the user you created in step 1.

9. Scroll down to “Add permissions”, and click on “Attach existing policies directly.”

10. In the search box, search for “AmazonS3ReadOnlyAccess” if you want to use S3 buckets that are read-only.

11. Click on “Create policy” and then “JSON”, and then copy the Integrate.io policy available here.

12. Finally, click on “Review policy” and “Create policy”.

13. Now, in Integrate.io, create a connection using the saved IAM security credentials.

14. Open the Integrate.io dashboard and go to “Manage connections”.

15. Click on “New connection” and select “Amazon S3”.

16. On the next page, enter the connection name, access key ID, and secret access key. The key ID and secret access key have been generated by Amazon in the previous step.

17. Click on the “Test connection” button to make sure that the connection works and that you’ve entered everything correctly.

18. Finally, click on the “Create Amazon S3 connection” button.

Note: In order to be compatible with Integrate.io, your S3 bucket names should never use an underscore or end with a dash character.

Want a step-by-step in-person tutorial? Check out our YouTube video: How to Connect Integrate.io to Amazon S3.

In Conclusion

As we’ve shown in this tutorial, connecting Amazon S3 with Integrate.io is simple even for the least technically inclined among us. Even better, S3 is just one of the many connections and integrations that Integrate.io enables.

Want to try Integrate.io out for yourself? Get in touch with our team of experts today for a chat about your business needs and objectives, or to start your 14-day risk-free trial of the Integrate.io platform.