I am partnered with a Real Time Intelligence Company, who have built a platform that uses AI to transform publicly available data into real time operational insights. With a huge part of their business growing, they are now looking to hire a Senior Data Engineer, who will be an integral part of the team, working on new and exciting projects, building new pipelines, migrating from one service to another service and building a data and analytics infrastructure. Responsibilities • Build and maintain data pipelines for large scale data ingestion, real time analytics and Machine Learning Services • Develop and optimize NLP, LLMs and other AI powered applications • Write CI/CD pipelines and have the ability to do code reviews • Design and build a data analytics infrastructure Qualifications • Professional Experience in using Apache Storm, Kafka or Flink for real time data or streaming • 5+ years experience in Data Engineering and building pipelines using Databricks and Spark • Can demonstrate experience in building CI/CD pipelines and DevOps best practises • Experience with real time data streaming