Lean integration is a form of process engineering that’s used to improve efficiency in data integration. It focuses on creating value for customers, reducing waste, and prioritizing long-term thinking.
The lean integration approach uses the principles of Lean, an efficiency methodology that was initially developed for manufacturing. Lean is now commonly associated with both Agile and Six Sigma.
What are the Principles of Lean Integration?
Lean integration is a hands-on management system that guides day-to-day decision-making, as well as a long-term strategy.
Ultimately, lean is best summarized by this motto: keep striving to deliver better value to the customer. In the context of data projects, the “customer” can be anyone who benefits from the end result of the project, whether that’s an internal user, a project sponsor, or an external customer that will interact directly with the system.
In all instances, lean integration follows seven principles:
1) Eliminate Waste
Waste is anything that doesn’t provide value to the customer. For example, if the goal of the project is to provide clean data for the data analytics team, then the project owner should know exactly what data they need to provide. If the project spends time cleansing redundant data, then that counts as waste.
2) Automate Processes
Manual work and rework is essentially a form of waste. It’s a waste of human time that is better spent on value-adding tasks. Where possible, projects should seek to automate processes, including testing. This may require long-term thinking, as automation projects can require a major restructuring of existing data workflows.
3) Empower the Team
The Lean approach works best when each individual feels empowered to take action as required. To feel empowered, teams need to have full managerial support. If they break something, the team will learn from the mistake and move on quickly. Cross-functional teams with a wide range of skills are also empowered to take swift action.
4) Continuously Improve
Continuous improvement is a core element of DevOps and other fast-moving environments. Data has to drive all improvement, which means that project leaders require access to detailed analytics. Continuous improvement is an ongoing cycle of forming theories, testing them, measuring the results, and forming new theories. The results of these tests help to decide how to move forward.
5) Build Quality In
Rather than relying on quality assurance to catch errors, developers and engineers should aim at a high rate of FTT (First Time Through). To do this, they’ll need proper time and support from their project leaders.
6) Plan for Change
In the long term, everything is likely to change. For example, if the project is to build a data integration process using disparate sources, it’s safe to assume that one of those sources will change over time. Every lean integration project should look ahead and plan for these inevitable changes. A lean integration solution would be to use a data pipeline built on an Extract, Transform, Load (ETL) process. It is then easy to reconfigure the ETL when one of the sources changes.
7) Optimize the Whole
Data projects often get stuck in the nitty-gritty, with constant pressure to deliver today’s requirements. Lean integration strives to look beyond that and to instead consider the big picture. The Lean philosophy is about asking what you can do to create value for the customer while reducing waste. That means staying focused on optimizing the whole, rather than fixing individual problems.
How is Lean Integration Applied to Data?
The principles of Lean emerged from manufacturing. In that environment, managers walked up and down assembly lines and looked for opportunities to apply the principles.
With lean integration, leaders have to walk around the proverbial shop floor of their data infrastructure. When doing so, they must look out for the eight types of waste:
-
Defects: The data integration process is producing errors, or it is generating redundant outputs.
-
Overproduction: Data is being moved or replicated even though there is no apparent customer demand.
-
Waiting: The data integration process is causing a bottleneck elsewhere in the organization. An example of this is when a team can’t start working until a scheduled data export takes place.
-
Unused talent: Everyone has a particular set of skills, but they may not get to use them. This is a form of waste, especially if an individual could be better utilized by assigning them to a different task.
-
Transportation: In manufacturing, this refers to any movement, whether it’s from the storeroom to the assembly line, or from the assembly line to the customer. In data, this refers to things like data ingestion, replication, storage across distributed file systems, or any potentially wasteful activity where data moves from A to B.
-
Inventory: Data expires after a certain amount of time, and then requires from all repositories. Sometimes, this doesn’t happen, especially with structures like data lakes.
-
Motion: Similar to transportation, except that this refers to the movement of people rather than data. If team members have to perform lengthy tasks to access data or log into a system, then that is wasteful.
-
Extra processing: The data integration process may pass through more steps than required. For example, the transformation stage in ETL converts data to the correct schema. The data is then loaded into a repository. At that point, there is no need to check again that the new data is in the correct schema.
Lean integration often complements other project methodologies such as Agile, as both of these approaches are fundamentally about good communication. Managers can map their processes, identify areas of waste, and devise new ways to create value for customers. But for these ideas to become a reality, the people who carry out those processes must understand what’s happening.
Lean integration also benefits significantly from feedback. Managers have a bird’s eye view of processes, but they often miss out on fine details. Team members can help identify the more subtle issues that lead to waste.