Table of Contents
  • Writing to file storage can be cumbersome, even overkill, for large amounts of data.
  • The X-Console can’t operate on the data that you use at runtime, only the variables that you define within the console.

With its low-code and no-code features, Integrate.io brings the power of ETL and data integration to the masses. But even with Integrate.io’s tremendously user-friendly interface, it’s possible that the transformations you design don’t work exactly as you intended—which means you need to debug and resolve the issue fast.

Fortunately, there are multiple debugging options in Integrate.io for exactly this reason. Below, we’ll discuss 3 ways that you can debug in Integrate.io—including a new feature, Component Previewer, that lets you visualize your data as it moves through your ETL pipeline.

  1. Writing to File Storage
  2. The X-Console
  3. Integrate.io’s Component Previewer
  4. Debug Your ETL Workflows Faster with Integrate.io

Debugging in Integrate.io: Writing to File Storage

 

One of the simplest ways to debug in Integrate.io is by writing your intermediate data to file storage, where you can then open the file and examine the result for errors.

Integrate.io has a Clone component that allows users to split a single dataflow into two or more dataflows. You can insert a Clone component into your ETL pipeline for debugging in Integrate.io: one dataflow continues the pipeline as before, while the other redirects the intermediate output to a file storage bucket (e.g., in Amazon S3).

Since file storage isn’t picky about the format of your data, it will accept the output (even buggy) without any issues. Another benefit: writing to file storage lets you view the entire output in one piece, which is highly useful if there’s a minor error hidden somewhere within the data.

Below is a simple example of what this process looks like in the Integrate.io dashboard. Previously, the select_1 and select_3 components were directly connected. The clone_2 component now splits the dataflow into two: one dataflow continues to the select_3 component, while the second is redirected to the file_storage_4 component.

thumbnail image

Debugging in Integrate.io: The X-Console

If you need help debugging your use of Integrate.io functions, the X-Console is made for you. Integrate.io’s X-Console is a new environment separate from the rest of your ETL pipeline, acting as a “sandbox” for you to test the functions in your dataflow before you use them in production. This kind of sandbox environment should be familiar for users of the Unix shell or REPL interpreters for programming languages like Python. The X-Console is highly useful for general troubleshooting and quick sanity checks (for example, to make sure that a function returns the type of data you’re expecting).

To use the X-Console, click on any transformation component in the pipeline, and then click on the edit icon (which looks like a pencil and paper) to the left of any field in the component. This launches the Expression Editor, which serves as “scrap paper” for drafting expressions. To switch over to the X-Console and test these expressions, click on the “X-Console” button in the top right corner of the modal window.

thumbnail image

thumbnail image

Debugging in Integrate.io: Integrate.io’s Component Previewer

While writing to file storage and using the X-Console can be very effective for debugging in Integrate.io, they also have limitations:

 

So that you have the right option for every situation, we’ve introduced a third way to do debugging in Integrate.io: the Component Previewer. This is a new feature available in the Integrate.io package designer that gives you a window into what your data looks like as it moves through the ETL pipeline step-by-step. With the Component Previewer, it’s easier than ever to confirm and validate your dataflow logic and hunt down the source of bugs and errors.

To get started with the Component Previewer, click on any transformation component (except for a Clone component), and then click the yellow “Preview” button. Integrate.io will run your job through the pipeline up until, and including, this step in the transformation. In order to cut down the runtime, the Component Previewer takes only a small fraction of the data from the data source.

When the preview job is complete, the results of the transformation are displayed beneath the “Data Preview” header. You can view the output either as a table or in the RAW format.

thumbnail image

For example, Integrate.io’s Component Previewer is ideal for debugging a complex multi-step transformation where you need to isolate the source of the error. Here’s how to use the Component Previewer for a job that has recently failed:

  1. Click on the “Jobs” icon in the sidebar of the Integrate.io dashboard.
  2. In the search bar at the top, enter “status:failed” and click “Search.”
  3. The failed jobs are shown in a table, sorted by newest to oldest. Click on “View details” to the right of the job in question. This will bring up a side sheet containing the job’s error log. Scan the log for the error you want to debug.

Once you’ve found the source of the error, you can correct it directly from within the Expression Editor. In the example below, the target data warehouse can only accept PER_FirstName and PER_LastName values with a maximum of 6 characters, which caused an error when trying to load names that were too long. We’ve applied the SUBSTRING function in the Expression Editor to truncate these names, aligning them with the target schema:

thumbnail image

Debug Your ETL Workflows Faster with Integrate.io

 

Thanks to the Component Previewer, Integrate.io has made it easier than ever to debug your ETL pipeline—just another reason why Integrate.io is the most powerful, user-friendly ETL platform on the market. With a rich expression language, a simple graphical user interface, more than 100 pre-built integrations, and top-notch technical support, Integrate.io allows anyone to become a data integration pro.

Looking for the best ETL solution for your organization? We can help. Get in touch with our team today for a chat about your business needs and objectives, or for a demo or 7-day pilot of the Integrate.io platform.