Lead Data Engineer

Job Type:
Contract Duration:
England, London, Docklands
£66,000 to £98,000 per annum
Salary Description:
Significant Benefits!
Job Ref:

Lead Data Engineer

Lead Data Engineer required by my truly amazing public sector client in Canary Wharf, London. They do significant work protecting the best interests of the public whilst working independently of the Government.

They are a diverse and vibrant organisation that also offer an outstanding benefits package and huge pension whilst also offering supported development opportunities to build a career!

The data and digital landscape is evolving and the organisation is a leading light in modernising the government approach. Data Services’ are not your ordinary government department, it is an innovative team engineering an enterprise data hub and external facing data exchange that will revolutionise the way that the organisation and its stakeholders report, manage and analyse data.

The Data Services Team

Data Services is a collaborative and forward-thinking team comprised of product leads, data scientists, software engineers, user researchers, QA testers, technical and security architects, business and data business analysts. By using Agile approaches to delivery, we ‘work out loud’ encouraging open and inclusive discussions to drive development forward.

The Role

The Lead Data Engineer will lead technical development work and has the autonomy to shape and direct data technology development for the Data Services, and therefore influence the wider data architecture across the organisation.

Key responsibilities

* End-to-end data architecture and handling approach in the Data Services platform. This will involve defining and communicating technical needs for data collection, ingestion, transformation and calculation. (Reporting to the Head of Data)
* Design pipelines, create proof of concepts to production, run functional tests, end to end regression testing, quality/automation tests
* Carry out data modelling and work closely with data scientists
* Lead a team of engineers, supported by QA and analyst staff across London and Glasgow
* Be informed about business needs and data journeys
* Deliver continuous improvements to the functionality to the Data Service platform and evolve the platform to meet business needs accordingly
* Assure that your work and the work of colleagues meets governing standards
* Ensure principles of privacy and security are assessed and designed into our data solutions
* Encourage teamwork, communication and follow agile methodologies. Promote our team knowledge retention through documentation and providing training where required.

Key stakeholder relationships

* Product Owners, who you will share your work with and discuss strategy for how the service is best able to support business needs
* The Data Services Analyst community, who will set the scope of your data engineering work through their research of user and business needs and by their designing data pipelines through writing user stories
* Fellow members of the Data Engineering community, who you will mentor and collaborate with to set the direction of the data service technology, data architecture and its functionality
* Data scientists, who will outline their requirements for data architecture and data set processes, for you to design and develop.
* Government Data privacy and information security staff, who will work with you to assure that security and privacy requirements are identified and addressed in solutions

Specific skills and capabilities


* Hands on experience (including QA) in platforms such as Azure and the Hadoop ecosystem (HIVE, HDFS and Apache spark)
* Proven ability in designing pipelines, proof of concept to production, functional tests, end to end regression testing, quality automation testing and CI/CD
* Extensive experience in data Modelling, including dimensional models time series and data mining

* Master Data Management
* Experience working with and integrating into Business Intelligence tools
* Expert competency with Python and SQL
* Good knowledge of Linux

* Experience of effectively engaging and communicating technical concepts
* to a non-technical audience
* Business minded manager and engineering team leader

* Proven ability to understand business priorities and design, deliver and communicate technical solutions to meet them
* Experience developing engineer’s technical capability
* Experience defining and implementing effective data engineering working practices in agile teams
* Ability to design and provide effective support for knowledge transfer / retention
* Proven experience of innovation and improving best practice


* Experience working with privacy and information accreditation or assurance teams
* Experience in the delivery of data solutions capable of:

* being certified to an information security standard such as ISO27001 or equivalent
* holding UK Government OFFICIAL-sensitive or equivalent information
* meeting legal and regulatory requirements such as DPA or GDPR SQL Databases, such as MS SQL and/or Postgress

* Cloud/Big Data: HDInsight, S3, EMR
* NoSQL Databases, such as Neo4j Data Integration and ETL approaches
* Competency with Scala and/or Java
* Experience of metadata management

This is a great opportunity to make a significant impact during a period of critical development.

Contact Details:
Tel: 02393382100
Contact: Brigitte McKenna

You may return to your current search results by clicking here.

Latest Job Listings