Too Many Data Engineers? How to Get the Right Balance

As companies grow and become more data-dependent, data engineers find themselves in huge demand. Employers are snapping up all the best data engineering talent they can find, and some businesses have invested in fast-track professional development paths for DBAs and other more junior data positions.

But here’s the thing — data engineers work best when they’re part of a balanced team, just like every other professional. Some organizations overlook this point. They build teams that are heavy on data engineers, and when the team fails to meet their objectives, they go out and hire even more data engineers.

It’s like trying to build a football team that’s all quarterbacks. You’re going to run into problems. And you won’t be able to solve those problems by recruiting more quarterbacks.

Who Counts as a Data Engineer?

It’s worth defining some terms before looking deeper at this issue. On a fundamental level, we know what a data engineer is: someone who builds and maintains an organization’s data infrastructure. They generally follow plans laid out by the data architect, and they coordinate with the data science and analytics teams.

In practice, job descriptions may not be quite as tidy. Engineers may end up performing some architectural tasks, or oversee analytics projects, or even get their hands dirty with some database administration. In a DevOps environment, things become even more blurred. Each DevOps team member may perform some data engineering duties during their typical working week.

So let’s define our term like this: a data engineer is someone who performs a data engineering task.

You can instantly see how a Too Many Cooks scenario can emerge on many teams. If you have a dedicated data engineer and you have three people who perform engineering tasks, then you have four data engineers in total.

How Do You Know You Have Too Many Data Engineers?

You’ll generally start to notice a surplus of data engineering resources when you spot signs like:

• Those with the job title of Data Engineer don’t have enough work to fill their schedule
• Miscommunication between engineers on shared tasks
• Regular conflicts and disagreements about implementing engineering strategy
• Teams lack other skills, such as communication or project management
• Over-reliance on engineering-oriented solutions
• Confusion over hierarchy when reporting infrastructure issues
• Some engineers report feeling left out of high-level decisions, such as Data Architect meetings

In some ways, this is a good situation to be in. Having a surfeit of data engineers means that you have a huge pool of talent at your disposal.

But if you’re noticing some of the problems above, it’s time to start thinking about reorganizing your team so that you get the best out of everyone.

Balancing Your Data Engineering Squad

To get the most out of your data engineering professionals, you need to have the right structures in place. Here’s how to get the best out of everyone.

Understand Your Requirements

Before you hire a data engineer, make sure that an engineer is what you actually need. What projects do you want them to deliver? How will your team benefit from the additional capacity?

For instance, if your analytics team is complaining about data quality or availability, your first thought might be to bring in an engineer to help optimize your infrastructure. But perhaps you need to rethink your entire data strategy first? If in doubt, it’s good to talk to your data architect before making any decisions — or bringing in a data architect on a consultancy basis.

You may also find that you can fulfill your requirements by shuffling the deck a little. Take a look at all available resources and see if you can reassign someone with engineering skills to fill a gap.

Refresh Your Agile Strategy

Most development teams have already moved away from rigid structures and gone towards a more Agile approach. However, inertia can creep in over time if you’re not constantly huddling and realigning the team with project goals.

If you’re seeing issues with too many engineers, then take a step back and look at it from a project perspective. What kind of engineering capability do you need? Who’s covering those tasks? If you identify excess capacity, then try to move those engineers to areas where they are needed more.

Remember that frequent communication is the heart of the Agile philosophy. If you see disagreements or conflicts about engineering tasks, then take a look at how your project team collaborates. The move to remote working may have impacted the effectiveness of your huddles, and Slack doesn’t always lead to open communication. 

Build Mini-Teams Around Your Engineers

Engineers are great at many things, but not at everything. For example, many engineers aren’t comfortable with delivering presentations or writing reports. You can empower them by ensuring that each engineer is working closely with someone who has the skills they lack.

Engineers also generally tend to see everything as a nail at which they must swing their hammer. For instance, an engineer might create a data pipeline from scratch using SQL and some batch files. This takes up their time, and it produces a solution that nobody can understand, except for other engineers.

When your engineers have other people to bounce ideas off, they are more likely to see alternatives. Instead of a DIY data pipeline, they might adopt a low-code Cloud ETL, which is faster and easier to support. 

Develop and Grow Your Data Engineers

Data Engineer is often a mid-career position. Your people will typically have plans that go beyond working with data warehouses and ETL. They have a lot of options too, including consultancy, architecture, and senior management.

Talk to each one of your data engineers and find out about their ambitions and then help them develop the skills they require. For instance, if someone wants to become a CTO, they’ll need to build their communication and collaboration skills, while architects will need to become project management experts.

Your engineers will also have specific tech skills that they want to develop, like cybersecurity or cloud tech. If you help them earn certifications and gain experience, you’ll build an army of versatile engineers who can help out in other roles when needed. This helps avoid the problem of too many engineers concentrated in one area.

Improve Your Infrastructure

Some data engineers spend a lot of their time in crisis mode rather than focusing on value-adding projects. You can end up with a whole team of engineers but who are working hard but getting nowhere fast.

This could be down to some deep-seated architectural issues. For example, you could be trying to make on-premise data warehouses work, when all of your users and applications are on the cloud. Rather than hire even more data engineers, it may be time to consider moving to AWS, Azure, or another cloud platform.

The same applies to your digital infrastructure tools. Cloud-based data pipelines will take most of the work out of integrating systems and maintaining connections. This frees up your engineers to work on progressive projects, rather than trying to find the root cause of yet another failed data import.

Empower Your Data Engineers with Integrate.io

Rather than wondering if you have too many data engineers, ask yourself if you’re getting the best out of your engineering team. Are they empowered to help you deliver success? Or do they spend their day papering over the cracks in your infrastructure?

Integrate.io is an automated Cloud-based data pipeline with a massive library of built-in integrations. With Integrate.io, you can complete time-consuming data engineering tasks with just a few clicks. Schedule an intro call to find out how Integrate.io can help your team.