Job Description
What wearelookingfor:Experienceandeducation:Bachelor's degree in Computer Science, InformationTechnology, Electronics or a related engineering discipline.Amaster's (MCA, MSc, MTech) is an advantage.Onetothreeyearsofprofessionalsoftwareengineering,withcodeyouhavewritten runninginproduction.Production pipelinesyouhavebuiltand run, withexperiencethatgoesbeyond notebooksandreports.The fullchangelifecycle,notthebuildphasealone, andcodereviewgivenandreceived ina professionalcodebase.Exposuretostreamingorchangedatacapture,ortodatagovernanceandprivacywork,isanadvantage.Coretechnical skills:SQL: joins,aggregation,windowfunctions,incrementalandmergepatterns, andtheabilitytoreadaqueryplan.Python:clean,testablecodethatlivesoutsideanotebook.Databricks:theworkspace,jobsandSQLwarehouses,withDeltaLaketables,partitioningandcompaction.PySparkandSparkSQL:thefundamentals,withawarenessofpartitioningandshufflebehaviour.dbtandAirflow:transformationandtestingindbt,andrunninganddebuggingjobsinAirfloworDatabricksWorkflows.AWS: S3,Parquet andDeltafiles, Glue,Lambda andIAMbasics.MySQL and DynamoDB:the operational sources you ingest from, including change data capture against MySQLand export patterns from DynamoDB.Unity Catalog:governance, lineage and access control, including how a grant or a row-level filter changes what your pipeline is allowed to read.Power BIorDatabricksSQL:dashboardsontopofwhatyoudeliver,sothenumbers getconsumedandnotjust landed.Nice to have:Kafka, Kinesis or change data capture tooling such as Debezium against MySQL sources.MLflow,or feature engineeringfor machine learningworkloads.Terraformfordatainfrastructure.Databricks Data EngineerAssociate,AWS Data Engineer Associate, or a generative AI credential.AI-assisteddevelopmentcapability:Daily, deliberate use of an approved assistant such as Claude, Copilot, Cursor or DatabricksAssistant, with the habit of checking its output rather than trusting it.Retrieval basics:preparingcontent forretrieval,building anembedding pipeline,andloadingavectorstoresuchas pgvectororOpenSearch.How you work:Deliversreliably,andflagsslippage earlyratherthanlate.Curious and coachable: asks rather than guesses, and carries review feedback into the next change.Quality-minded, and thinks about the person who reads the code next.Team-first. Helps peers, joins incident reviews, shares what they learn.Checks the number. Reconciles against source by instinct.Whatyouwillworkon:Pipelineandplatformengineering:Ingestion pipelines:start with batch and move into streaming over time, handling schema change, late and duplicate data, and reconciliation against source.Transformationlogic:writteninPySpark,SparkSQLanddbtacrossthebronze,silverandgoldLakehouse layers.Opentableformats:whatDeltaLakeorIcebergactually givesyou,including transactions,schemaevolution, timetravelandcompaction.Data qualitytests:completeness,freshness,uniquenessandreferentialchecks,withalertsthatreachanamedowner.Orchestration:DatabricksWorkflows orAirflowjobs,includingdependencyhandling,failurerecovery andbackfill.GovernanceinUnityCatalog:classification,ownership andaccesscontrolontheassetsyoutouch.Privacyandcorrectness:Handlepersonaldataproperly: classificationandmasking,andwhyresidencyconstraintsacrossthemarketsservedshapepipelinedesign.Reconcilebeforeyoudeclare: checkoutputagainstsourcebeforeyoucallapipeline correct.Startfromsyntheticormaskeddata:avoidproductionpersonaldatawhereverasubstitutewilldo.Craft,qualityandproductionsupport:Tests ship withthe change:same branch,same review.Afollow-upticket is not atest plan.Instrumentanddocumentasyoubuild:logs,metricsanderrormessagesthatmakethepipelinesupportableforthenextengineer.Support your consumers:investigate data questions from analysts and other engineering teams, and close the loop with a fix or an explanation.Productiondefectsendtoend:reproduce,diagnose, fixandtest,thenconfirmthefixinproduction.
Job information is provided by WhatJobs. Please check the original listing for the latest details and availability.