Netflix AI Team Cuts Wide-Partition Read Latency from Seconds to Milliseconds by Splitting Cassandra Partitions Per ID
0
0
Netflix’s engineering team published a method for handling wide partitions in Apache Cassandra. The research work targets Netflix’s TimeSeries Abstraction, a platform for temporal event data.
TL;DR
- Dynamic partitioning splits wide Cassandra partitions per TimeSeries ID, asynchronously and transparently, with no application changes.
- Detection runs on the read path via byte counting and a Kafka event; splitting targets immutable partitions first.
- Bloom filters (single-digit microseconds) plus a cached
wide_rowmetadata lookup route reads to the smaller child partitions. - Checksums, retained original partitions, Data Bridge Spark checks, and shadow comparison guard correctness.
- Reads improved from seconds to low double-digit milliseconds; tail latency fell to ~200 ms; 500MB+ partitions stayed available.
What is Dynamic Repartitioning?
Netflix’s TimeSeries Abstraction ingests and queries petabytes of temporal event data with millisecond latency. It uses Apache Cassandra 4.x as the underlying storage. Cassandra was chosen for throughput, latency, cost, and operational maturity.
Time-series data is organized into partitions that group events by identifier and time range. These partitions can grow ‘wide’ as events accumulate over time. Dynamic repartitioning splits an oversized partition into smaller child partitions asynchronously. Applications keep querying the same logical partition while the storage layout evolves transparently.
Why Wide Partitions Hurt Reads
For most datasets, average read latency stays in single-digit milliseconds. When partitions grow too wide, tail read latencies rise into seconds. These slow reads can produce read timeouts. In extreme cases, clusters see Garbage Collection pauses, high CPU utilization, and thread queueing.
TimeSeries servers also handle very high read throughput, which compounds the problem. Scaling up the Cassandra cluster is always an option. But Netflix team wanted smarter alternatives than just throwing more money at the problem.
The Partitioning Strategy Behind TimeSeries
TimeSeries breaks datasets into discrete time chunks: Time Slices, time buckets, and event buckets. This lets Netflix team query and drop data by time without creating tombstones. When a namespace (dataset) is created, users specify anticipated workload characteristics. A provisioning pipeline runs Monte Carlo simulations to pick infrastructure and partition configuration.
This up-front approach falls short in three situations. Workload can be unknown or inaccurately estimated early in a project. Workload can also evolve as traffic and product requirements change. Finally, data outliers exist, where a few IDs receive far more events.
Discrete Time Slices provide a natural escape hatch for the first two cases. Each new Time Slice can use a different partitioning strategy. But manually tuning thousands of datasets is not sustainable, so automation is required.
Solution 1: Time Slice Re-Partitioning
Cassandra exposes introspection APIs such as nodetool tablehistograms for partition-size percentiles. A background worker monitors these histograms and exposes them through a Cassandra virtual table. It computes an adjustment factor when partition sizes miss a configured density. That density is often set between 2 MiB and 10 MiB, depending on workload.
For example, provisioning once selected 60-second time buckets, producing partitions under 10 KB. That over-partitioning caused high read amplification and thread queueing. The worker updates future Time Slices with a corrected strategy:
DynamicTimeSliceConfigWorker:
namespace: my_dataset_1
Observed: TimeSlices have p99 partitions below configured target of 10MB.
Proposed: time_bucket interval: 60s -> 604800sThis reduced read latencies and timeouts caused by thread queueing. But it only helps when most of the table warrants re-partitioning. It does not help when only a percentage of IDs are wide. For those partial cases, Netflix team offers three options:
- Do Nothing: the right approach when top-level metrics show no impact.
- Partial Returns: aborts an in-flight request breaching a latency SLO, returning data collected so far.
- Block IDs: an extreme step for test or spam IDs that destabilize the system.
Block IDs is applied through configuration, listing the offending TimeSeries IDs:
dgwts.config.<dataset>.block.Ids: "<tsid-1>, <tsid-2>, <tsid-3>"These options do not help when valid, important IDs grow wide. Those callers still need every event, which is where Solution 2 applies.
Solution 2: Dynamic Partitioning per ID
Dynamic partitioning is an asynchronous pipeline that splits wide partitions per TimeSeries ID. It operates at the ID level, not the table level. It has three stages: Detection, Planning & Splitting, and Serving Reads.
Detection happens on the read path. Every read tracks bytes read for a partition. When bytes exceed a configured threshold, the server emits an event to Kafka:
{
"time_slice": "data_20260328",
"time_series_id": "profileId:123",
"time_bucket": 7,
"event_bucket": 2,
"immutable": true,
"version": "0"
}Here, immutable marks a partition no longer receiving writes. The version field is reserved for future use, such as invalidation. Netflix team detects on reads, not writes, because most data never needs splitting. Some reads on wide partitions stay slow for seconds until the pipeline catches up. The initial implementation targets immutable partitions to reduce complexity.
Planning reads the entire partition once to compute an accurate split plan. Checkpointing lets failed planning reads resume from the last saved point. The wide_row metadata table stores split states, checkpoints, and routing information.
Splitting delegates to a strategy such as EventBucketPartitionSplitStrategy. It assigns more event buckets to the same time bucket. For ultra-wide partitions, it caps event buckets to control read amplification. Spreading across partitions still distributes reads over Cassandra replicas.
Validating compares a pre-split checksum against a post-split checksum. A split is marked COMPLETED only when both checksums match. Netflix also tracks pre- and post-split partition sizes to confirm splits are sized well.
The Read Path: Bloom Filters and Metadata Routing
TimeSeries servers periodically load completed split partition-keys into in-memory Bloom filters. Every read checks the Bloom filter, which responds in single-digit microseconds. That check is small enough to be practically invisible to callers. On a hit, the server reads wide_row metadata to route the query:
{
"pre_split_data": {
"time_slice": "data_20260328",
"time_series_id": "6313825",
"time_bucket": 0,
"event_bucket": 2
},
"post_split_data": {
"time_slice": "wide_data_20260328_0",
"event_bucket_partition_strategy": {
"target_event_buckets": 2,
"start_event_bucket": 32
}
}
}That metadata read is backed by a read-through cache. The existing PartitionReader then serves reads from the smaller split partitions. Reusing the same schema minimizes code changes, and results are merged before returning. The original wide partition is never deleted, providing a safe fallback for partial failures and eventual consistency.
Netflix team also verified splits offline using Data Bridge Spark jobs. A phased rollout advanced through read modes as confidence grew. Its Comparison phase compared bytes served by the old and new read paths in shadow mode.
Solution 1 vs Solution 2: Comparison
| Dimension | Time Slice Re-Partitioning (Solution 1) | Dynamic Partitioning per ID (Solution 2) |
|---|---|---|
| Granularity | Table / Time Slice level | Individual TimeSeries ID level |
| Trigger | Background worker on partition histograms | Bytes read exceed a threshold on the read path |
| Scope of effect | Future Time Slices only | Existing immutable wide partitions |
| Core mechanism | Adjust time_bucket interval / target density |
Split into child event-bucket partitions |
| Detection signal | nodetool tablehistograms percentiles |
Per-read byte counting, Kafka event |
| Best when | Most of the table is mis-partitioned | A few IDs are wide outliers |
| Data movement | None; applies to new writes | Async split; original retained as fallback |
| Target / validation | Configured density (2–10 MiB) | Pre/post-split checksum + shadow comparison |
Use Cases With Examples
- Long-lived user activity logs: A
profileId:123accumulating years of playback events becomes wide. Solution 2 spreads it across event buckets, so pagination stays available. Netflix paginated 500MB+ partitions while remaining available using this path. - Device or IoT telemetry: A small set of chatty devices dominate event volume. Solution 2 isolates those hot IDs without repartitioning the whole table.
- Over-provisioned new dataset: A team guesses 60-second buckets and gets sub-10 KB partitions. Solution 1 widens the interval for future Time Slices, cutting read amplification.
- Latency-sensitive dashboards: A caller needs a fast response more than complete data. Partial Returns caps latency at the SLO and returns what was gathered.
Try It: Interactive Splitter Demo
The embedded simulator below models the read path end to end. Set a partition size, a detection threshold, and a target event-bucket count. Running a read walks through Detection, Planning, Splitting, Validation, and Serving. It shows the Kafka detection event and a modeled latency drop; the numbers are illustrative, not measured.
<button class=”btn sec” id=”reset”>Reset</button>
</div>
<div>
<div class=”pipe” id=”pipe”>
<div class=”step” data-i=”0″><span class=”n”>01</span>Detection</div>
<div class=”step” data-i=”1″><span class=”n”>02</span>Planning</div>
<div class=”step” data-i=”2″><span class=”n”>03</span>Splitting</div>
<div class=”step” data-i=”3″><span class=”n”>04</span>Validate</div>
<div class=”step” data-i=”4″><span class=”n”>05</span>Serve reads</div>
</div>
<div class=”stage”>
<p class=”rowlbl”>Original partition (retained as fallback)</p>
<div class=”wide” id=”wide”>profileId:123 — 520 MB</div>
<div class=”children” id=”kids”></div>
<div class=”note” id=”status”>Idle. Press <b>Run read</b> to start the pipeline.</div>
</div>
<div class=”mrow”>
<div class=”metric slow”>
<div class=”k”>Single wide read</div>
<div class=”v” id=”mSlow”>—</div>
<div class=”s”>one large partition, sequential</div>
</div>
<div class=”metric fast”>
<div class=”k”>Split parallel read</div>
<div class=”v” id=”mFast”>—</div>
<div class=”s”>N smaller partitions, merged</div>
</div>
</div>
<pre id=”json”>// Kafka detection event appears here on a wide-partition hit</pre>
</div>
</div>
<div class=”foot”>
<span>Illustrative simulation of the read path — latency values are modeled for teaching, not measured production numbers.</span>
<span><b>Marktechpost</b></span>
</div>
</div>
<script>
(function(){
var d=document.getElementById(‘dwp’);
var $=function(id){return d.querySelector(‘#’+id);};
var sSize=$(‘sSize’),sThr=$(‘sThr’),sBuk=$(‘sBuk’);
var vSize=$(‘vSize’),vThr=$(‘vThr’),vBuk=$(‘vBuk’);
var wide=$(‘wide’),kids=$(‘kids’),status=$(‘status’),json=$(‘json’);
var mSlow=$(‘mSlow’),mFast=$(‘mFast’),run=$(‘run’);
var steps=d.querySelectorAll(‘.step’);
var running=false;
function sync(){
vSize.textContent=sSize.value+’ MB’;
vThr.textContent=sThr.value+’ MB’;
vBuk.textContent=sBuk.value;
wide.textContent=’profileId:123 — ‘+sSize.value+’ MB’;
}
[sSize,sThr,sBuk].forEach(function(s){s.addEventListener(‘input’,function(){if(!running){sync();}});});
// illustrative latency model
function slowMs(mb){ return Math.round(30 + mb*mb*0.012); } // grows ~quadratically
function fastMs(mb,n){ var child=mb/n; return Math.round(8 + child*1.1 + 0.05); } // dominated by largest child + bloom(µs)+cache
function clearPipe(){ steps.forEach(function(s){s.className=’step’;}); }
function setStep(i,cls){ steps[i].className=’step ‘+cls; }
function wait(ms){ return new Promise(function(r){setTimeout(r,ms);}); }
function esc(s){return s;}
async function go(){
if(running) return; running=true; run.disabled=true;
var size=+sSize.value, thr=+sThr.value, n=+sBuk.value;
clearPipe(); kids.innerHTML=”; wide.className=’wide’;
mSlow.textContent=’—’; mFast.textContent=’—’;
var isWide = size > thr;
// 01 Detection
setStep(0,’on’);
status.innerHTML=’Read path tracks bytes read for <b>profileId:123</b>…’;
await wait(650);
if(!isWide){
setStep(0,’done’);
status.innerHTML=’Bytes read (<b>’+size+’ MB</b>) below threshold (<b>’+thr+’ MB</b>). No split needed — served directly.’;
mSlow.textContent=slowMs(size)+’ ms’;
mFast.textContent=slowMs(size)+’ ms’;
json.innerHTML=’// Partition under threshold — no detection event emitted’;
running=false; run.disabled=false; return;
}
json.innerHTML=
‘{n’+
‘ <span class=”k”>”time_slice”</span>: <span class=”g”>”data_20260328″</span>,n’+
‘ <span class=”k”>”time_series_id”</span>: <span class=”g”>”profileId:123″</span>,n’+
‘ <span class=”k”>”time_bucket”</span>: <span class=”b”>7</span>,n’+
‘ <span class=”k”>”event_bucket”</span>: <span class=”b”>2</span>,n’+
‘ <span class=”k”>”immutable”</span>: <span class=”b”>true</span>,n’+
‘ <span class=”k”>”version”</span>: <span class=”g”>”0″</span>n’+
‘}’;
setStep(0,’done’);
status.innerHTML=’Bytes read exceeded threshold → detection event emitted to <b>Kafka</b>.’;
await wait(750);
// 02 Planning
setStep(1,’on’);
status.innerHTML=’Planner reads the full partition <b>once</b> and checkpoints to <b>wide_row</b> metadata.’;
await wait(850); setStep(1,’done’);
// 03 Splitting
setStep(2,’on’);
status.innerHTML=’EventBucketPartitionSplitStrategy assigns <b>’+n+'</b> event buckets, keeping total sort order.’;
var per=(size/n);
for(var i=0;i<n;i++){
var el=document.createElement(‘div’);
el.className=’child’;
el.innerHTML='<span class=”bar”></span>wide_data_20260328_0 · bucket ‘+(32+i)+’ — ‘+per.toFixed(0)+’ MB’;
kids.appendChild(el);
}
var kidEls=kids.querySelectorAll(‘.child’);
for(var j=0;j<kidEls.length;j++){ (function(k){setTimeout(function(){kidEls[k].classList.add(‘show’);},k*120);})(j); }
await wait(n*120+500); setStep(2,’done’);
// 04 Validate
setStep(3,’on’);
status.innerHTML=’Comparing <b>pre-split checksum</b> with <b>post-split checksum</b>…’;
await wait(800);
setStep(3,’done’);
status.innerHTML=’Checksums match → split marked <b>COMPLETED</b>. Partition-keys loaded into Bloom filter.’;
await wait(600);
// 05 Serve reads
setStep(4,’on’);
wide.className=’wide ok’;
status.innerHTML=’Bloom filter hit (µs) → metadata lookup → <b>PartitionReader</b> reads ‘+n+’ partitions in parallel.’;
for(var b=0;b<kidEls.length;b++){ kidEls[b].querySelector(‘.bar’).style.width=’100%’; }
await wait(700); setStep(4,’done’);
var sm=slowMs(size), fm=fastMs(size,n);
animate(mSlow,sm,’ms’,sm>1200);
animate(mFast,fm,’ms’,false);
status.innerHTML=’Done. Tail read latency dropped from <b>’+fmt(sm)+'</b> to <b>’+fm+’ ms</b> using ‘+n+’ parallel child reads.’;
running=false; run.disabled=false;
}
function fmt(ms){ return ms>=1000 ? (ms/1000).toFixed(1)+’ s’ : ms+’ ms’; }
function animate(el,target,unit,big){
var t0=performance.now(), dur=600;
function tick(now){
var p=Math.min(1,(now-t0)/dur), cur=Math.round(target*p);
el.textContent = (big && cur>=1000) ? (cur/1000).toFixed(1)+’ s’ : cur+’ ‘+unit;
if(p<1) requestAnimationFrame(tick);
else el.textContent = big && target>=1000 ? (target/1000).toFixed(1)+’ s’ : target+’ ‘+unit;
}
requestAnimationFrame(tick);
}
run.addEventListener(‘click’,go);
$(‘reset’).addEventListener(‘click’,function(){
if(running)return; clearPipe(); kids.innerHTML=”;
wide.className=’wide’; mSlow.textContent=’—’; mFast.textContent=’—’;
json.innerHTML=’// Kafka detection event appears here on a wide-partition hit’;
status.innerHTML=’Idle. Press <b>Run read</b> to start the pipeline.’;
});
sync();
})();
</script>
<script>
(function(){
function report(){
var el=document.getElementById(‘dwp’);
var h=(el?el.offsetHeight:document.body.offsetHeight)+40;
parent.postMessage({mtpDwpHeight:h},’*’);
}
window.addEventListener(‘load’,report);
window.addEventListener(‘resize’,report);
// report after animations/interactions
new MutationObserver(report).observe(document.getElementById(‘dwp’),{subtree:true,childList:true,attributes:true});
setInterval(report,600);
})();
</script>
“>
Results
Average read latency for wide partitions dropped from seconds to low double-digit milliseconds. Tail latencies fell from several seconds to around 200 ms or better. Read timeouts dropped, and Netflix team reported lower CPU utilization and minimal thread queueing. Overall, the Cassandra clusters became more stable.
For extreme wide rows, the service could paginate and query 500MB+ partitions while remaining available. Such partitions previously caused constant timeouts and unavailability blips. A paginated query returns successfully, trading elevated latency for availability:
{
"next_page_token": "...",
"records": [ { "...": "..." } ],
"response_context": [
{
"namespace": "...",
"time_taken": "41.072410142s"
}
]
}Netflix team lists future work, including splitting mutable wide partitions and reprocessing previously failed splits. Two lessons stand out. Reducing the surface area of a complex change and deploying incrementally pays off operationally. Investing in confidence-building mechanisms is justified by the feature’s complexity, blast radius, and impact.
Check out the Technical details here. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
The post Netflix AI Team Cuts Wide-Partition Read Latency from Seconds to Milliseconds by Splitting Cassandra Partitions Per ID appeared first on MarkTechPost.
MarkTechPost
