Staff Engineer, Machine Learning & Data Infrastructure Staff

About this role

OKX · job-boards.greenhouse.io · checked today

Staff Engineer, Machine Learning & Data Infrastructure

San Jose, California, United StatesStaffData Platformposted today

At OKX, we believe that the future will be reshaped by crypto, and ultimately contribute to every individual's freedom. OKX is a leading crypto exchange, and the developer of OKX Wallet, giving millions access to crypto trading and decentralized crypto applications (dApps). OKX is also a trusted brand by hundreds of large institutions seeking access to crypto markets.

Skills, with evidence

• Data modelling

Design and optimize offline data pipelines on Databricks, Apache Spark, and warehouse/lakehouse technologies, addressing data modeling, partitioning, joins, shuffles, skew, incremental processing, and cost.must have

• Incremental loads

Design and optimize offline data pipelines on Databricks, Apache Spark, and warehouse/lakehouse technologies, addressing data modeling, partitioning, joins, shuffles, skew, incremental processing, and cost.must have

• Python

Strong Python, SQL, and at least one backend or systems language such as Java, Scala, Go, or C++.must have

• SQL

Strong Python, SQL, and at least one backend or systems language such as Java, Scala, Go, or C++.must have

• Spark

Deep understanding of Databricks and Spark, or comparable offline data platforms, including execution plans, distributed query optimization, storage layouts, data correctness, and workload cost management.must have

• AWS

Build reusable infrastructure for model packaging, versioning, deployment, serving, monitoring, and rollback across cloud platforms and regions.must have · not practised here

• Azure

Build reusable infrastructure for model packaging, versioning, deployment, serving, monitoring, and rollback across cloud platforms and regions.must have · not practised here

• GCP

Build reusable infrastructure for model packaging, versioning, deployment, serving, monitoring, and rollback across cloud platforms and regions.must have · not practised here

• Cost & performance

Improve model inference latency, throughput, resource utilization, and cost.

• Streaming

You will help teams move models from experimentation into reliable production services, improve model execution performance, and build scalable data foundations for training, evaluation, and real-time decisions.

• Data quality

Experience with the model lifecycle, including training/serving integration, validation, deployment, observability, and rollback.

• Failure handling

Build reusable infrastructure for model packaging, versioning, deployment, serving, monitoring, and rollback across cloud platforms and regions.

• Idempotency & backfills

Connect offline and online feature pipelines, ensuring point-in-time correctness, consistent feature definitions, freshness, lineage, and reproducible backfills.

• Warehousing

Design and optimize offline data pipelines on Databricks, Apache Spark, and warehouse/lakehouse technologies, addressing data modeling, partitioning, joins, shuffles, skew, incremental processing, and cost.

Your plan

1. The SQL screen: correct, then fast

SQL · Data quality

• Airbnb: converting on a day with no published rateAdvanced
• Amazon: net revenue across two fan-outsAdvanced
• Build a percentile tableAdvanced
• Duplicate snapshot keysAdvanced
• Flag outlier ordersAdvanced

≈ 2 h

1. Python: the data-wrangling round

Python

• Apply a change-data-capture batchAdvanced
• Deduplicate a stream in bounded memoryAdvanced
• Compact segments without resurrecting deletesAdvanced
• Detect gaps in a partitioned offset streamAdvanced
• Commit batches to a sink exactly onceAdvanced

≈ 4 h

1. Data modelling: the round most people fail

Data modelling · Warehousing

• Claims, reserves, and payments that reverseAdvanced
• Customer 360 tradeoffsAdvanced
• Datasets, owners, and access grantsAdvanced
• Care episodes, providers, and overlapping responsibilitiesAdvanced
• Loads, drivers, and backhauls that vanishAdvanced

≈ 4 h

1. Pipeline design: safe to run twice

Incremental loads · Streaming · Failure handling · Idempotency & backfills

• Fourteen months of wrong numbersAdvanced
• Creator payout runAdvanced
• Delete me from everywhereAdvanced
• Realtime events pipelineAdvanced
• You fixed it. Who already used it?Advanced

≈ 4 h

1. Spark: read the plan Spark actually ran

Spark · Cost & performance

• applyInPandas per country: what Spark ships to Python, and the native rewriteAdvanced
• Filters you wrote in the wrong place: where Catalyst moves themAdvanced
• snappy, gzip or zstd: measure the Parquet codec trade-off yourselfAdvanced
• A correlated COUNT(\*) subquery: the join Spark runs, and the count bug it avoidsAdvanced
• Find the four suspects in a slow pipeline (one of them is innocent)Advanced

≈ 2 h

1. Say it out loud

Rehearse: Data Modeling · Model design

≈ 1 h

25 drills · Advanced + Intermediate≈ 16 hours

Not covered by the plan: AWS, Azure, GCP.

Readiness

0 of 25 drills already done elsewhere on D8LooP.

Apply on job-boards.greenhouse.io

leaves in 13 dremoved the moment OKX closes it