Lead Data Engineer

Job Type:
Contract Duration:
England, London, Canary Wharf
£66,000 to £98,000 per annum
Salary Description:
Outstanding Benefits
Job Ref:

Lead Data Engineer required by our outstanding innovative and forward-thinking Public-Sector client. They are leading the modernisation of data management through the development of an Enterprise data platform.

The team is comprised of product leads, data scientists, software engineers, user researchers, QA testers, business and data analysts. By using Agile approaches to delivery, the team ‘work out loud’ encouraging open and inclusive discussions to drive development forward.

The Lead Data Engineer will lead the technical development work directly reporting to the Head of Data. The Lead Engineer has the autonomy to shape and direct data technology development for Data Services, and therefore influence the wider data architecture across the organisation.

Key responsibilities

* End-to-end data architecture and handling approach in the Data Services platform. This will involve defining and communicating technical needs for data collection, ingestion, transformation and calculation. (Reporting to the Head of Data)
* Design pipelines, create proof of concepts to production, run functional tests, end to end regression testing, quality/automation tests
* Carry out data modelling and work closely with data scientists
* Lead a team of engineers, supported by QA and analyst staff across London and Glasgow
* Be informed about business needs and data journeys
* Deliver continuous improvements to the functionality to the Data Service platform and evolve the platform to meet business needs accordingly
* Assure that your work and the work of colleagues meets governing standards
* Ensure principles of privacy and security are assessed and designed into our data solutions
* Encourage teamwork, communication and follow agile methodologies.
* Promote our team knowledge retention through documentation and providing training where required.

Key outputs and deliverables

* The data architecture of the data platform meets the users’ needs and has a design that is flexible to meet future needs
* The data software engineering meets the requirements of each data product design
* The data engineering solutions comply with Governance, Privacy and Security requirements
* Data Services data engineering work follows agile methodologies
* Colleagues find you to be an approachable advisor for data architecture and data processing
* Organisation data strategy is defined and delivered based on learned user needs
* The Data Service platform functionality is continually reviewed and improved
* Knowledge retention achieved through documentation and training

Key stakeholder relationships

* The Head of Data, whose portfolio of data activities you will deliver with specialist technical expertise
* Product Owners, who you will share your work with and discuss strategy for how the service is best able to support business needs
* The Data Services Analyst community, who will set the scope of your data engineering work through their research of user and business needs and by their designing data pipelines through writing user stories
* Fellow members of the Data Engineering community, who you will mentor and collaborate with to set the direction of the data service technology, data architecture and its functionality
* Data scientists, who will outline their requirements for data architecture and data set processes, for you to design and develop.
* Government Data privacy and information security staff, who will work with you to assure that security and privacy requirements are identified and addressed in solutions

Specific skills and capabilities


* Hands on experience (including QA) in platforms such as Azure and the Hadoop ecosystem (HIVE, HDFS and Apache spark)
* Proven ability in designing pipelines, proof of concept to production, functional tests, end to end regression testing, quality automation testing and CI/CD
* Extensive experience in data Modelling, including dimensional models time series and data mining
* Master Data Management
* Experience working with and integrating into Business Intelligence tools
* Expert competency with Python and SQL
* Good knowledge of Linux
* Experience of effectively engaging and communicating technical concepts to a non-technical audience
* Business minded manager and engineering team leader
* Proven ability to understand business priorities and design, deliver and communicate technical solutions to meet them
* Experience developing engineer’s technical capability
* Experience defining and implementing effective data engineering working practices in agile teams
* Ability to design and provide effective support for knowledge transfer / retention
* Proven experience of innovation and improving best practice


* Experience working with privacy and information accreditation or assurance teams
* Experience in the delivery of data solutions capable of:
* being certified to an information security standard such as ISO27001 or equivalent
* holding UK Government OFFICIAL-sensitive or equivalent information
* meeting legal and regulatory requirements such as DPA or GDPR
* SQL Databases, such as MS SQL and/or Postgress
* Cloud/Big Data: HDInsight, S3, EMR
* NoSQL Databases, such as Neo4j
* Data Integration and ETL approaches
* Competency with Scala and/or Java
* Experience of metadata management

Civil Service Competencies

* Seeing the Big Picture
* Changing and Improving
* Collaborating and Partnering
* Making Effective Decisions
* Managing a Quality Service
* Building Capability for All

You must be eligible to live and work in the UK for this role without sponsorship.

Contact Details:
Tel: 02393382100
Contact: Brigitte McKenna

You may return to your current search results by clicking here.

Latest Job Listings