Senior Data Engineer - Darwin

DLG is evolving. Across every facet of our business, our teams are embracing new opportunities and putting customers at the heart of everything they do. By joining them, you’ll have the opportunity to not just be recognised for your skills but encouraged to build upon them and empowered to do your absolute best. 

At Darwin Insurance, a brand within DLG, you’ll help build the next generation of our low-latency, cloud-native data platforms — the systems that power real-time pricing, machine learning workflows, and operational intelligence. You’ll own the design of resilient ETL/ELT pipelines, distributed data processing, and scalable AWS infrastructure, collaborating closely with a tight-knit team of engineers, architects, and product leaders who care deeply about doing things the right way.

We engineer like a startup: Clean Code, TDD, CI/CD, Infrastructure-as-Code, full ownership, and a bias for shipping well-designed, well-tested systems. You’ll have a genuine voice in architectural direction, mentor strong engineers, and help raise the technical bar across the team.

This role is ideal for someone who wants meaningful impact — someone who’s excited by the blend of startup autonomy and big-company reach, and who wants to shape the future of Darwin’s data platform as we scale our ambitions.

What you'll be doing: 

As a Senior Data Engineer, based out of our London office and working closely with Platform Owner, you will play a key role in developing and maintaining robust cloud-based data platform solutions within AWS.

• Cross-Functional Collaboration and Stakeholder Communication: Lead collaborations with cross-functional teams to understand business requirements and communicate complex data concepts clearly to non-technical stakeholders, facilitating informed decision-making across the organization.

• Data Solutions Development and Documentation: Design robust data transformation solutions using ETL and ELT methodologies and maintain comprehensive documentation for all data pipelines and integration procedures to ensure clarity and effective management.

• Performance Monitoring and Data Quality Assurance: Proactively monitor data systems for performance optimization and implement data quality checks to maintain high standards of accuracy and reliability in data processing.

• Security and Compliance: Develop and enforce security protocols to protect data assets and ensure all data handling complies with legal and regulatory standards, supporting robust data governance.

• Innovation and Technology Advancement: Stay abreast of advancements in data engineering technologies and practices, incorporating innovative tools and methods to enhance the organization’s data capabilities.

• Mentorship and Team Leadership: Mentor junior data engineers, fostering an environment of growth and continuous learning, while leading by example in technical expertise and basic project management.

• Architectural Collaboration and Optimization: Work closely with data architects and other engineers to refine data storage and architecture, ensuring the infrastructure supports scalable solutions and future technological developments.

Our hybrid model offers a 'best of both worlds' approach. When you'll be in the office depends on your role and team, but colleagues spend at least 50% of their time in the office.

What you'll need: 

• Significant years of experience, gained in professional services setting, in designing and implementing complex data transformations using both ETL and ELT methodologies to support scalable data architecture in high-demand environments.

• Excellent understanding of distributed data processing, ideally working with Apache Spark

• Extensive experience (5+ years) with advanced SQL techniques and database management, including proficiency with database systems like PostgreSQL, MySQL, and Amazon Redshift, optimizing database performance and integrity in large-scale environments.

• In-depth knowledge of AWS cloud services including Glue, S3, KMS, Lambda, and Kinesis, demonstrating capability in deploying and managing scalable, secure cloud infrastructure.

• Advanced proficiency in Python programming, with demonstrated experience in developing, automating, and enhancing data processing tasks and analytical applications.

• Strong command over Unix/Linux operating environments, with the ability to manage and optimize system performance for data applications.

• Solid understanding and practical experience in Agile development processes, skilled in translating strategic goals into executable stories, with a focus on incremental and iterative delivery.

• Strong advocate for Software Craftsmanship and Clean Code, with a commitment to maintaining high standards in code quality and system reliability.

• Demonstrated experience and passion for Test-Driven Development (TDD), ensuring robust and reliable software solutions through comprehensive testing practices.

• Excellent communication skills, capable of effectively articulating technical concepts to both technical and non-technical stakeholders, facilitating clear understanding and cooperative problem-solving.

• Clear understanding of DevOps practices and Infrastructure as Code, with hands-on experience in automating and optimizing infrastructure deployment and management.

• Over 5 years of experience managing clustered systems, ensuring high availability and performance scalability in distributed computing environments.

• Expert knowledge of various database technologies, including SQL and NoSQL, and their appropriate use cases and trade-offs, guiding optimal database selection and use.

Benefits 

We recognise we wouldn't be where we are today without our colleagues, that's why we offer excellent benefits designed to suit your lifestyle: 

• 9% employer contributed pension 

• 50% off home, motor and pet insurance plus Green Flag breakdown cover 

• Additional optional Health and Dental insurance 

• Up to 10% AIP Bonus

• EV car scheme allows all colleagues to lease a brand new electric or plug-in hybrid car in a tax efficient way. 

• Generous holidays 

• Buy as you earn share scheme 

• Employee discounts and cashback 

• Plus, many more 

We want everyone to get the most out of their time at DLG. Which is why we’ve looked beyond the financial rewards and created an offer that takes your whole life into account. Supporting our people to work at their best – whatever that looks like — and offering real choice, flexibility, and a greater work-life balance that means our people have time to focus on the things that matter most to them. Our benefits are about more than just the money you earn. They’re about recognising who you are and the life you live. 

Be yourself

Direct Line Group is an equal opportunity employer, and we think diversity of background and thinking is a big strength in our people. We're delighted to feature as one of the UK's Top 50 Inclusive Employers and are committed to making our business an inclusive place to work, where everyone can be themselves and succeed in their careers. 

We know you're more than a CV, and the things that make you, you, are what bring potential to our business. We recognise and embrace people that work in different ways so if you need any adjustments to our recruitment process, please speak to the recruitment team who will be happy to support you.  

#LI-AW1

#LI-HYBRID