![\"Acceleration](\"https://blog.comma.ai/img/bug_stories_ml/output_precision.png\")
“We have solved self driving, we just need to fix the bugs.”
\nRumor has it that the Linux kernel averages around 0.5 bugs per 1,000 lines of code. Some bugs are easy to avoid or catch with tests and metrics. Others are very good at hiding.
\nThis is a sequel to Bugs that broke driving with more Machine Learning related bugs.
\nSome of these bugs affected correctness, others slowed things down or used up extra memory. They all made training or inference worse.
\nLooking back at these bugs helps us understand where our code tends to break and how to catch problems earlier.
\nSeptember 2026.
\nThe world model is a diffusion transformer (DiT) which predicts the next frame and an “expert” driving plan.
\nDuring inference, the model runs in mixed precision bf16_fp8_nvfp4:
In particular, the plan output layers ran in BF16. Around 30 m/s, BF16 can only represent speeds in steps of 0.125 m/s. The acceleration calculation from the plans subtracts nearby speed predictions and divides by a short time interval , magnifying the rounding errors:
\n
The fix is to promote the plan head and its inputs to FP32 before running it, Casting the finished plan to FP32 just gives the staircase more decimal places.
\n+plan_head.float()\n-plan = plan_head(features).float()\n+plan = plan_head(features.float())\nAugust 2026.
\nThe driving model’s vision encoder is a ConvNeXt, with a learned multiplier on each residual branch: x + gamma * f(x). At runtime, it runs in FP16.
The pretrained checkpoint we used we pick had a value of almost 54 at stages[2].blocks[29]. After fine-tuning, some inputs push the residual past FP16’s 65,504 limit leading to inf.
The annoying thing is that this only triggered in some seemingly random cases, and didn’t happen in the various checks we run during model exporting to FP16.
\nThe workaround was to divide that stage’s residual stream by four and carry the scale through its biases, multipliers, and LayerNorm epsilons:
\n stage = encoder.stages[2]\n+stage.downsample[1].weight.div_(4)\n+stage.downsample[1].bias.div_(4)\n+for block in stage.blocks:\n+ block.conv_dw.bias.div_(4)\n+ block.gamma.div_(4)\n+ block.norm.eps /= 16\n+encoder.stages[3].downsample[0].eps /= 16\nJuly 2026.
\nThe driving model predicts “driving desire”, like turning or changing lanes.
\nFor a while, some training data had no desire targets: we use NaNs to distinguish missing targets, and our loss helpers normally handle this by masking those targets out. But the desire loss took an argmax first, and did not have the NaN handling.
>>> \n>>> \n>>> = \n>>> \n\n>>> \n\nAll those NaNs became class 0, “no desire”, with a perfectly normal loss.
The fix replaces NaNs with zeros while keeping a mask, then uses that mask to ignore missing targets:
\n\n = ~\n return , \n+target, valid = num_from_nan(target)\n loss = F.cross_entropy(logits, target.argmax(-1), reduction="none")\n+loss = valid.all(-1) * loss\nApril 2026.
\nThe driving model has many nn.Linear layers. During training, we use torch.compile, BF16 autocast and Distributed Data Parallel (DDP) as “acceleration” recipes.
Sometimes, harmless redundant biases are written in the architecture:
\n = \n = \n = + \nNote that the 8/4 is just illustrative, and keeps the weight gradients different while the bias gradients are identical.
\na.bias and b.bias will be merged by the compiler into one bias, so they get the same gradient. The compiler went one step further: it made both .grad tensors share the same memory.
DDP synchronized them correctly, then clip_grad_norm_ used its foreach implementation to scale that shared buffer from two CUDA blocks at once.
Depending on timing, an entry could be multiplied by the clipping coefficient once or twice, giving each rank a different update.
\nIn one recorded run, rank 0 clipped all 8,192 entries once; rank 1 clipped 384 of them twice. The drift can reach a magnitude of 0.0056 between ranks after only 50 SGD steps, while the weight matrices stayed identical.
\nBoth biases receive the same faulty update within a rank, their sum drifts too, which can change the model’s outputs. DDP synchronizes gradients during training, so it doesn’t repair those parameter differences: the GPUs are now training different copies of the model.
\nWe worked around it by removing the redundant biases.
\nThis was fixed in PyTorch 2.13.0 as a side effect of a compiler optimization change.
\nIt preserved FP32 → BF16 → FP32 cast chains so smaller intermediate tensors could still be materialized: in our backward graph, that happened to keep the two bias gradients in separate FP32 buffers.
Note: The foreach kernel itself still assumes the gradient buffers don’t overlap…
\nAugust 2026.
\nTorchtitan uses the same seed across DDP (replicate), this is correct because weight initialization should match across replicate ranks. But torchtitan then keeps that rank identical across during for training.
\nWith ordinary tensors (not DTensors), different ranks could get identical dropout masks and diffusion noise.
\nThe fix is to reseed each data-parallel rank after initialization (essentially):
\n torch.manual_seed(seed)\n model.init_weights()\n+torch.manual_seed(seed + dp_rank)\n train(model)\nFlux does this after initialization, setting distinct_seed_mesh_dims=["cp", "dp_shard", "dp_replicate"].
![\"Diffusion](\"https://blog.comma.ai/img/bug_stories_ml/ddp_worldmodel_diffusion_loss.png\")
![\"Plan](\"https://blog.comma.ai/img/bug_stories_ml/ddp_worldmodel_plan_loss.png\")
Fixing the ddp seed bug produces better world models overall and less noisy loss curves given that the diffusion noise samples are not the same across all ranks.
\nJune 2021 to April 2026.
\n![\"BatchNorm](\"https://blog.comma.ai/img/bug_stories_ml/allnorm_loss.png\")
Training and validation loss for models trained with BatchNorm and AllNorm.
\nNormalization layers are widely used in Machine Learning. BatchNorm normalizes each channel using the mean and variance across the batch and spatial dimensions. During training, it uses the current batch statistics and updates running averages. During inference, it uses those running averages.
\nThe driving models are multi-task learners, with some tasks being more sparse than others, for example: hard brake events.
\nOne failure mode we encountered was when a channel specializes in a rare feature. Through batches without that feature, the channel stays quiet and its BatchNorm running variance shrinks.
\nWhen validation activates that channel again, it gets divided by a stored standard deviation that is too small, blowing up the outputs. Training can still look fine because its statistics come from the current batch: when the feature returns, the denominator grows with it.
\nLayerNorm fixes this, but needs per input statistics at inference. On device, we want normalization folded into convolutions. Recalibrating BatchNorm running statistics offline also fixes this, but adds overhead and another training pass. Same for lowering momentum which needed extra tuning.
\nThe best fix we came up with was pooling the whole batch, channels, and spatial dimensions into one set of statistics. We called this AllNorm.
\n\n\n \n = \n\n\n return \nAll channels share statistics, making it robust to this failure, and keeping the ability to fold the statistics into the convolutions.
\nNote: With the extra compute that the chestnut gives, we can now afford using LayerNorm and not deal with this!
\nJuly 2026.
\nGigashuffle returns tensors backed by reusable CPU buffers. Each reader allocates these once and fills them again for every batch:
\n# Once per reader:\n = \n\n# For every batch:\n\nNormally, each batch is copied to the GPU before the next read. Our gradient accumulation loop collected several microbatches first, so later reads overwrote batches already in the list.
\nOne way to fix this is to clone both inputs and targets before fetching the next microbatch:
\n for _ in range(grad_accum):\n inputs, targets = next(loader)\n+ inputs = {k: v.clone() for k, v in inputs.items()}\n+ targets = {k: v.clone() for k, v in targets.items()}\n microbatches.append((inputs, targets))\nApril 2025.
\nWith FSDP2, we construct the model on meta, shard it, then use to_empty() to allocate the weights. Initialization has to run after allocation: not defining an init function for a parameter initializes it to the arbitrary memory value it received during allocation.
>>> \n>>> \n... = \n... = \n>>> = \n>>> \n\nNo clear fix here but to be paranoid about not forgetting init methods!
\nJuly 2026.
\nOur RL training runs collect rollouts from the cluster. If the cluster is busy, some ranks can wait for data longer than PyTorch’s default NCCL timeout of 10 minutes.
\nTo work around this, we added a barrier with a timeout of one hour before each training step, thinking we have fixed it. But that timeout was passed around, only to never be used:
\n = \n = \n = \n = \nThe C++ dispatcher passed it along, but NCCL implemented the barrier as an all-reduce and only forwarded asyncOp:
// C++ dispatcher\nBarrierOptions opts;\nopts.timeout = ;\nopts.asyncOp = asyncOp;\nbackend->;\n\n// ProcessGroupNCCL::barrier\nAllreduceOptions arOpts;\narOpts.asyncOp = opts.asyncOp;\nauto work = ;\n// opts.timeout goes nowhere.\nThat all-reduce got its deadline from a different options object: the process group’s configuration. The watchdog checked that deadline; our barrier(timeout=...) argument never entered the calculation:
// Creating the collective's work: options_ belongs to the process group.\n;\n\n// Inside assignTimeoutToWork(work, option), omitting timeout extensions:\nwork->opTimeout_ = option->timeout;\n\n// Watchdog calls work.checkTimeout() with no timeout override:\nif \nThe fix was to give the wait for data its own Gloo group, with the timeout set on that group:
\n+# Once at startup, on every rank:\n+ready = dist.new_group(backend="gloo", timeout=timedelta(hours=1))\n\n # Before each training step:\n-dist.barrier(timeout=timedelta(hours=1))\n+dist.barrier(group=ready)\nMarch 2026.
\nAdamW batches updates into calls like torch._foreach_mul_(params, 1 - lr * weight_decay). FSDP2 exposes its parameters as DTensors, so those calls go through DTensor’s dispatcher to work out how the operation applies to the shards. For foreach’s tensor lists, the dispatcher built a fresh OpSchema on every call, then constructed a cache key from the tensors’ shape and sharding metadata.
In the C++ key builder, PyTuple_Pack increments its arguments’ reference counts, but .release() stopped the C++ wrapper from decrementing its own reference afterward. That left an extra reference to the argument tuple on every call, so the tuples and their metadata kept piling up in CPU RAM.
The fix was to simply not release:
\n // torch/csrc/autograd/python_variable.cpp\n // Inside DTensor_OpSchema_recompute_comparison_key_impl (simplified):\n py::tuple args_to_hash_tup(args_to_hash.size());\n // ... fill the tuple with argument metadata ...\n comparison_key = PyTuple_Pack(\n 2, op,\n- args_to_hash_tup.release().ptr());\n+ args_to_hash_tup.ptr());\nThe bug affected PyTorch 2.10.0 through 2.12.1; the same fix shipped via a later PR in 2.13.0.
\nFebruary 2026.
\nWhen doing inference in our distributed cluster, we use an inference server with dynamic batching: triton inference server. We usually benchmark at the largest batch size in isolation, and assume this will be the max VRAM the model will use. But dynamic batching reserved more VRAM, and we didn’t know why.
\nWe traced it to how PyTorch caches freed GPU allocations, so the order of batch sizes matters. With its default native allocator, this takes just one tensor at a time:
\n\n\n\n # Reset between experiments.\n \n = \n del \n return // 2**20\n\n # 28 MiB\n # 16 MiB\nBatch size 3 leaves a cached 12 MiB block, batch size 4 needs 16 MiB, so the allocator keeps both. Starting with 4 lets batch 3 use 12 MiB of the larger 16 MiB block.
\nWe flipped the warmup order to start with the largest batch:
\n-for bs in range(1, max_batch_size + 1):\n+for bs in range(max_batch_size, 0, -1):\n warmup(bs)\nJune 2022.
\nWhen training with CUDA graphs, we used to capture the whole training step, optimizer step included. CUDA graphs replay only GPU work, so CPU code inside the capture runs just once!
\nBy default, PyTorch’s eager Adam increments its step counter and computes its bias corrections, 1 - beta1**step and 1 - beta2**step, on the CPU:
# torch/optim/adam.py — Adam._init_group() (simplified)\n = \n # Initialize this parameter's optimizer state once.\n = \nBack then, replay kept updating the GPU moment estimates and weights with those frozen corrections, silently reusing the value of step from the capture-time value. We avoided this by running the optimizer outside the graph.
PyTorch 1.12.0 later added Adam(..., capturable=True), putting the step counter and bias-correction math on the GPU so replay advances them too. Today, manual capture with capturable=False raises an error.
The recurring themes were precision, leaked or reused memory, and distributed training. Some bugs were hard to spot but preventable with better code practices. Others became obvious once we visualized model outputs during inference or inspected model state during training.
\nCome fix the bugs that are still hiding (https://comma.ai/jobs).
\nYassine.
\n", "date_modified": "2026-09-16T20:00:00+00:00", "date_published": "2026-09-16T20:00:00+00:00", "external_url": "https://blog.comma.ai/ml-bugs/", "id": "urn:aggr:item:c2fcc7b96110ca6ef552c5a6059f7e544984fdb9", "summary": "“We have solved self driving, we just need to fix the bugs.” Rumor has it that the Linux kernel averages around 0.5 bugs per 1,000 lines of code. Some bugs are easy to avoid or catch with tests and metrics. Others are very good at hiding.…", "tags": [ "programming" ], "title": "Bugs that broke driving: Machine Learning edition", "url": "https://aggr.aymericbeaumet.com/items/blog-comma-ai/2026-09-16-bugs-that-broke-driving-machine-learning-edition/" }, { "content_html": "![\"COMMA_HACK](\"https://blog.comma.ai/img/comma_hack_7/comma_hack_wm.png\")
September 18–20, 2026
\nFriday 6 PM – Sunday 6 PM
\nIRL @ comma HQ in San Diego, CA
-- Days
\n-- Hours
\n-- Minutes
\n-- Seconds
\nCOMMA_HACK is back for a weekend following the chestnut launch, with big models and big GPUs. Come see what openpilot can do with more compute, hack alongside the comma team, and build something worth demoing on Sunday.
\nWhat does your embedded project look like with 172 TFLOPs? Chestnut is the easiest way to throw real GPU compute onto your embedded projects.
\nChestnut connects to any device with a USB port, all the way down to USB2. That means it works with your laptop, Raspberry Pi, comma four, and even your Android phone.
\n![\"chestnut](\"https://blog.comma.ai/img/chestnut/cover.jpg\")
We’re handpicking the strongest applicants on a rolling basis until the event. Great GitHub profiles, leaderboard submissions, and openpilot contributions are a plus.
\nWant us to fly you out? We like winners, so we’ll cover flights for the first two people who get a top 3 score on any active leaderboard challenge.
\nDo I need a chestnut? Nope. We’ll provide the comma hardware and GPUs.
\nWhat kind of projects will I work on? That’s up to you. Build with chestnut, work on openpilot, or make something entirely new.
\nHow are projects judged? Demos on Sunday afternoon, judged by the comma team on quality and creativity.
\n", "date_modified": "2026-08-25T19:00:00+00:00", "date_published": "2026-08-25T19:00:00+00:00", "external_url": "https://blog.comma.ai/comma-hack-7/", "id": "urn:aggr:item:81c44f997285913dd77b101f7f88909b609583bd", "summary": "September 18–20, 2026 Friday 6 PM – Sunday 6 PM IRL @ comma HQ in San Diego, CA --Days --Hours --Minutes --Seconds COMMA_HACK is back for a weekend following the chestnut launch, with big models and big GPUs. Come see what openpilot can do…", "tags": [ "programming" ], "title": "COMMA_HACK 7", "url": "https://aggr.aymericbeaumet.com/items/blog-comma-ai/2026-08-25-comma-hack-7/" }, { "content_html": "Upgrade your comma four’s compute to run openpilot’s biggest models.
\n
Today, we’re launching chestnut, a compute upgrade for comma four. Want to run big models? Nothing beats a desktop GPU. We’re also launching the first chestnut-class model, which is bigger than all openpilot driving models we’ve released combined.
\n![\"chestnut](\"https://blog.comma.ai/img/chestnut/ready_34.png\")
![\"left](\"https://blog.comma.ai/img/chestnut/ready_left.png\")
![\"front](\"https://blog.comma.ai/img/chestnut/ready_front.png\")
![\"right](\"https://blog.comma.ai/img/chestnut/ready_right.png\")
![\"back](\"https://blog.comma.ai/img/chestnut/ready_back.png\")
Since the beginning, openpilot has led on intelligence density. For the last 10 years, we’ve shipped increasingly capable driving models in an on-device compute footprint of 10 W. We shipped fully end-to-end lateral planning, experimental end-to-end longitudinal planning, and even a point-to-point demo to Taco Bell.
\nOver the last two years, we rewrote our stack to scale efficiently with compute and data. With the MLSim architecture, more driving data and training compute translate directly into better openpilot models. The comma fleet now logs nearly one million driving minutes every day, and our scaled-up data pipeline ingests 15% of those drives into the training cluster.
\nBigger models need more compute to run. With chestnut, openpilot is scaling beyond cellphone-sized inference compute. A comma four paired with chestnut has similar compute to Tesla HW4.
\nWe’re launching chestnut with a new 1B parameter driving model in the upcoming openpilot 0.11.2 release. It’s our biggest model yet, and the first in a new line of chestnut-class models.
\n![\"Bignuts\"](\"https://blog.comma.ai/img/chestnut/bignuts.png\")
Compared to the latest on-device model, the first chestnut-class model has 30x more parameters, 100x more FLOPs, and room to go even bigger. Going forward, our autonomy team will focus primarily on maximizing the capabilities of chestnut-class models, and we expect this new class of model to increase development velocity.
\nchestnut comes in a convenient ready to drive kit, which includes chestnut, an AMD Radeon RX 9060 8GB, and everything needed to power, connect, and mount it in your car.
\n![\"Comma](\"https://blog.comma.ai/img/chestnut/comma_four_windshield.png\")
The GPU mounts out of the way in the passenger footwell or under the passenger seat. Run the included cable to comma four, connect to your car’s cigarette lighter, and you’re ready to run openpilot’s biggest models.
\nchestnut is available today in the comma shop.
\nExcited about scaling up? Join us in the #hw-chestnut channel on Discord!
\nWant to help scale up openpilot? We’re hiring for autonomy, product, and operations.
\n", "date_modified": "2026-08-12T21:00:00+00:00", "date_published": "2026-08-12T21:00:00+00:00", "external_url": "https://blog.comma.ai/chestnut/", "id": "urn:aggr:item:2b9dfe8a24749e519842bf3b3be30e97afaf17ff", "summary": "Upgrade your comma four’s compute to run openpilot’s biggest models. Today, we’re launching chestnut, a compute upgrade for comma four. Want to run big models? Nothing beats a desktop GPU. We’re also launching the first chestnut-class…", "tags": [ "programming" ], "title": "Introducing chestnut", "url": "https://aggr.aymericbeaumet.com/items/blog-comma-ai/2026-08-12-introducing-chestnut/" }, { "authors": [ { "name": "Harald" } ], "content_html": "All software is riddled with bugs, and usually bad software is bad because of those bugs, not because of bad algorithmic choices. openpilot is no different; we’ve made great progress over the years by implementing new exciting ML techniques, but most of the driving improvements came from discovering and fixing blatant bugs. Here I walk through some of the most entertaining bugs through the years.
\nIn 2017 comma was not a mature company. We had few users, low quality logging, no metrics in training other than a unitless val loss, and no closed-loop simulation tests. As a result we generally had no clue what was going on with openpilot beyond obvious issues we noticed while driving with it. This is clear from these Slack messages showing we were aware of the driving model’s tendency to bias driving slightly to the right for months, with no root cause or satisfactory explanation.
\nBy the end of the year we did figure out the major cause of this bias. We trained models to predict the future driving trajectory, and lanelines. These targets were ground-truthed with a localizer that relied on gyro measurements, as well as other sensors. Gyros are usually biased, the Android stack we used automatically corrected for the bias when it detected the device was at rest. This meant that when openpilot started and the car was stationary the bias was corrected appropriately. However, when the car then started moving, the openpilot device would heat up causing the bias to change, but the device never came to rest again so the bias was never corrected. The bias correlation with temperature is similar for every device, so all data would skew the same direction. A more comprehensive localizer that accounted for bias accurately fixed this.
\n![\"Road-camera](\"https://blog.comma.ai/img/bug_stories/laneline_middle_of_road.png\")
In 2019 we started getting mysterious complaints of cars departing the lane on roads with clearly marked lanelines. This happened because at the time openpilot still relied on the lanelines for driving decisions, and as the image shows, in these cases it would inexplicably detect a laneline in the middle of the lane. After looking for some more cases like this, we found all examples were in situations with bright daylight.
\n![\"Road-camera](\"https://blog.comma.ai/img/bug_stories/hood_reflection_laneline.png\")
White reflections at front of hood classified as lanelines
\nThe issue became obvious when looking at several examples of the training data with this same flaw. In certain lighting conditions the sun can reflect on the hood showing a bright white area, this gets detected in the ground-truth stack as a laneline marking that is always just in front of the car, and thus gets reconstructed as a laneline in the middle of the road. By adding better ground-truth of specifically these examples to the laneline detector, we retrained the detector and trained a new driving model with the fix.
\n![\"Night](\"https://blog.comma.ai/img/bug_stories/double_yellow_tf_model.png\")
In October 2020 there were several complaints about openpilot crossing the inner laneline in turns. After looking at the examples they all appeared to happen in long sweeping turns at night, in all cases the driving model predictions for the path and lanelines that were biased towards the inside of the turn.
\nAn investigation into the ground-truth showed the localizer incorrectly computed the car’s motion in these cases. Some localizer debugging revealed that the headlights shining onto long continuous lanelines creates problems for the visual odometry. The ORB feature detector selected edges based on the headlight illumination that move vehicle with the vehicle. This means that in scenes where there are little other features for the ORB detector to track, the localizer relies only on these erroneous features causing the localizer to be consistently biased towards those inside of the turn. Smarter ORB feature matching and outlier rejection fixed this, and the retrained model addressed the complaints.
\n![\"Night](\"https://blog.comma.ai/img/bug_stories/garbage_orb.png\")
Poor ORB features on a double-yellow line.
\nPhone cameras use a voice coil actuator to move the lens for focus adjustment. This means the lens is not rigidly mounted to a focus position, but instead mounted on a spring and focus movements are achieved by applying a force with the actuator. This also means that if there are rapid accelerations of the device, focus will shift due to the inertial forces, and to maintain focus a compensatory force needs to be applied with the actuator.
\nWe had an algorithm that would compensate for this focus shift for the OnePlus phones we were using in comma devices in 2019. Later we switched to Leeco phones, but never re-evaluated the focus algorithm. As it turns out our compensation algorithm was causing significant blur during acceleration on these newer devices, and these phones needed no focus compensation at all. This algorithm caused exactly what it was designed to prevent!
\nWith sag compensation (left); without it (right), December 2021.
\nWhen experimental mode was introduced in openpilot, openpilot did not decisively accelerate or brake to a good speeds on the highway. Instead in steady-state, it would just keep driving the speed it was initialized at. Which could for example cause it to drive 55mph forever on an empty long highway.
\nHard braking event at −9.5 m/s², notice the slight pitch movements as a result.
\nopenpilot driving models are trained on-policy in a driving simulator. The inputs to the simulator are 6dof pose changes, we use this instead of actions (acceleration and curvature) so we have more fine-grained control over the physics of the simulation. We noticed that we needed to add a lot of pitch noise to the simulation for the model to learn to brake and accelerate.
\nAfter investigation this was due to the tight coupling of pitch and acceleration. In any normal vehicle, acceleration causes pitch movements. This means acceleration must be simulated with an appropriate pitch change. This lack of realism in the simulator can be exploited by the student model that is trained in it. As the results below show, fixing these artifacts by simulating realistic pitch movements during simulated acceleration made the model perform much better at our speed convergence tests.
\n![\"Baseline](\"https://blog.comma.ai/img/bug_stories/pitch_accel_baseline_report.png\")
Baseline: 0 of 9 runs converged
\n![\"Speed-convergence](\"https://blog.comma.ai/img/bug_stories/pitch_accel_coupled_report.png\")
Realistic coupling: 2 of 6 runs converged
\n![\"Side-by-side](\"https://blog.comma.ai/img/bug_stories/padding_border_comparison.png\")
This bug is so good we wrote it 4 times. Over the years openpilot has supported various cameras and mounting configurations. As a result sometimes the models were trained to take in bigger images than were available in some configurations. To make this work, the images needed to be padded to fit to the model’s desired input shape. It is essential that the padding is the same during train-time and runtime, but the inference and training stacks are completely separate. Over the years they each changed frequently, and were poorly tested to match each other.
\nHere are all the varieties of bugs with padding making runtime different from train-time causing regressed outputs:
\nfc19544 (#22971)8656ef1 (#31495)f24ad7e (#37986)4585e93 (#38091)A lot has changed since 2017, many of these bugs would be unlikely to make it to production with our current tooling and infrastructure. We will still write many silly bugs though, and if you like the idea of fixing those bugs, maybe you should come work here.
\nHarald Schäfer
\nCTO @ comma.ai
The LM GT3 model (#37425) improves phone detection and is the first openpilot DM model trained using labels generated by a locally hosted large Vision-Language Model (VLM).
\nIn this model, the driver’s phone-usage ground truth is generated by asking a VLM a YES/NO question and using the output probability of the YES token. The language model is set to predict only one token. The previous approach relied on a custom classifier network trained on a set of hand-labeled driver images.
\nTo evaluate this new pipeline, we hand-labeled a 1k validation set of driver images with drivers either using or not using phones, called the phone_using set. We also curated a smaller set named phone_mounted, consisting of 80 driver images with a phone mounted on the dash but not in use. The phone_mounted set represents difficult negative samples.
\nThe VLM notably reduces False Positives and improves detection of active handheld phone use as True Positives, while keeping all mounted phones as True Negatives.
\n![\"ROC](\"https://blog.comma.ai/img/0111_release/dm_phone_gt_rocs.png\")
ROC curves for phone_using on the 1k validation set. phone_mounted not depicted since all are negative samples, whose false positives reduce to 0 when phone_using true positive rate is ~50%.
\nWith the model deployed in the field, we were also able to see its positive effects on safe driving habits when using openpilot. We compared phone usage time, as detected by the improved model, across the same group of devices before and after this model change. We saw a significant drop when openpilot was enabled, and even observed a small drop when it was not enabled, suggesting that drivers did reduce their phone usage overall.
\n![\"Phone](\"https://blog.comma.ai/img/0111_release/dm_phone_distraction.png\")
Phone usage behavior progression on the same sample group of 71 devices running master/nightly/staging: ‘before’ segments sampled from 60-day window before new DM model; ‘after’ segments from 20-day window after. All data re-evaluated offline using the same model.
\nThe two road cameras use a different image processing path on the ISP than the driver camera, and have had better image quality due to some missing pieces on the driver camera processing side. We have now enabled equivalent tone mapping blocks (#37873), as well as moved from on-sensor binning to post-demosaic downscaling (#37876). This means the driver camera will get the same high level of dynamic range and pixel details as the road-facing cameras.
\n![\"Driver](\"https://blog.comma.ai/img/0111_release/dm_camera_samples.png\")
Driver camera pictures old vs new, day and night.
\nAlerts from looking around are reduced during active maneuvers (#37751). When steering through a turn, the head pose yaw tolerance is widened in the direction of the steering angle, so glancing where the car is going no longer triggers an alert.
\n![\"Adaptive](\"https://blog.comma.ai/img/0111_release/dm_yaw_cones.png\")
Adaptive head pose thresholds during turns. This drops ~7% of all alerts. Angles shown are for illustrative purposes only, as these thresholds are also based on the scene and speed.
\nThis PR also extends the standstill exemption for DM alerting, so that an audible alert no longer unnecessarily kicks in the moment the car starts moving.
\nThe driverMonitoringState message has been restructured (#37799) to be both more readable and have a cleaner abstraction boundary. Downstream processes, such as selfdrived, can access top-level states and handle alerts without needing to know about DM policy details. Debugging with the new logs should also become easier.
\nIn the comma four onroad UI, we added a color change to the DM icon to subtly indicate distraction (#37826) before any alert is triggered. It also serves as feedback when the driver gets an audible DM alert and tries to look back at the device.
\nDM icon turns orange as soon as alert countdown starts, and returns to green as attention is restored.
\nThe DM icon is now calibrated to the road instead of showing raw pose from the model (#37149), so the green arc will be at the top of the icon when the driver is looking straight ahead.
\n![\"Comparison](\"https://blog.comma.ai/img/0111_release/dm_icon_straight.png\"){kind=icon}
Comparison of DM icon behavior when driver is looking straight ahead.
\nDirect sunlight on a parked comma four can heat the device above the fixed 75°C onroad threshold, temporarily blocking openpilot from going onroad even though the device would cool down once the car started moving.
\nThis release raises the threshold to 85°C and updates the comma four onroad thermal bands (#37891). Fleet analysis projected that raising the threshold from 75°C to 85°C would reduce affected devices by ~90%.
\n![\"Number](\"https://blog.comma.ai/img/0111_release/thermal_threshold.png\")
Number of comma four devices thermally blocked over the last 30 days. Raising the threshold from 75°C to 85°C reduced affected devices from 125 to 11.
\nThermal blocks were not isolated to a specific region, but occurred wherever solar load could push device temperature above the 75°C threshold. Incidents were concentrated in the US, consistent with fleet distribution, with additional clusters across Europe, the Middle East, and East Asia.
\n![\"Worldwide](\"https://blog.comma.ai/img/0111_release/thermal_map.png\")
Worldwide distribution of comma four devices thermally blocked before the threshold change.
\nThe figure below shows temperature traces from three comma four devices with high average offroad temperatures. The raised 85°C threshold allows these devices to go onroad during most peak-heating conditions.
\n![\"Temperature](\"https://blog.comma.ai/img/0111_release/thermal_dongles.png\")
Temperature traces from three comma four devices. Grey offroad points capture the offroad heating and cooling cycle, while green points show onroad driving. The old and new thresholds are shown as dashed lines.
\nThe original 75°C threshold was set conservatively to keep the SoC well below thermal throttling limits. CPU or GPU throttling can cause openpilot processes to lag and eventually disengage, so the onroad thermal policy must keep the device below throttling limits.
\nThermal throttling is managed by the Limits Management Hardware (LMH), which reduces frequency and voltage when a thermal trip point is reached. The CPU clusters, GPU, and RAM each have independent thermal zones and trip points. Since the RAM package is stacked directly on the SoC, its thermal limit can also trigger CPU throttling.
\nQualcomm’s default trip points (95°C) are designed for passively cooled smartphones, while comma four has active cooling and a larger thermal mass. Because of this, the CPU, GPU, and RAM trip points were increased by 10°C to 105°C (agnos-kernel-sdm845#110).
\nThe comma four was designed to operate like this, with a better heatsink design, a second heatsink on the bottom of the chip, and a very quiet fan. The software was still running a conservative thermal policy. Raising the trip points aligns the software with the hardware’s cooling performance.
\nTo validate the new trip points, a comma four was heat-soaked to 84°C before going onroad while CPU, GPU, and RAM temperatures were logged. Under initial load, CPU and GPU temperatures briefly peaked at 96°C before stabilizing around 68°C once the fan ramped up. RAM stabilized near 60°C.
\n![\"CPU,](\"https://blog.comma.ai/img/0111_release/thermal_curve.png\")
CPU, GPU, and RAM temperatures during ignition after heat-soaking the device at 84°C. The new 105°C throttling threshold is shown in red.
\nThe new threshold was merged into master on April 24, 2026. Since then, no comma four running openpilot master has been thermally blocked from going onroad.
\n![\"Daily](\"https://blog.comma.ai/img/0111_release/thermal_blocks.png\")
Daily count of comma four thermal blocks
\nThe longitudinal maneuver report enabled fast iteration on acceleration and jerk tuning by comparing commanded vs. measured response during scripted maneuvers. This release adds the same concept for lateral control. (#37562)
\nOn a straight, level road at constant speed, openpilot commands a set of lateral acceleration profiles and records the car’s response. Three maneuvers are run at 20 and 30 mph:
\n| Maneuver | \nCommand | \n
|---|---|
| Step right | \n+0.5 m/s² for 1 s, then -0.5 m/s² for 1.5 s | \n
| Step left | \n-0.5 m/s² for 1 s, then +0.5 m/s² for 1.5 s | \n
| Sine 0.5 Hz | \n1.0 m/s² amplitude, 2 s period | \n
Each maneuver repeats three times and only starts after the car has held speed for two seconds. Comparing commanded and measured lateral acceleration exposes rate limits, delays, and tuning issues.
\n![\"Sine](\"https://blog.comma.ai/img/0111_release/lateral_report.png\")
Sine 0.5 Hz maneuver at 30 mph on a Tesla Model Y and Kia EV6. The Tesla tracks the command closely, while the Kia is steering-rate limited.
\nLateral maneuver mode can be enabled in developer settings before going onroad. The maneuvers then execute automatically once conditions are met. The route rlogs need to be uploaded to connect, after which the HTML report can be generated with tools/lateral_maneuvers/generate_report.py route_id.
\nopenpilot is built by comma together with the openpilot community. Get involved in the community to push the driving experience forward.
\n#driving-feedback on DiscordWant to build the next release with us? We’re hiring.
\n", "date_modified": "2026-06-05T10:00:00+00:00", "date_published": "2026-06-05T10:00:00+00:00", "external_url": "https://blog.comma.ai/0111release/", "id": "urn:aggr:item:1b0e010ada0e4e9b3b0033682c98e22eecb6fb58", "summary": "Driver MonitoringPermalink New model with good phone detectionPermalink The LM GT3 model (#37425) improves phone detection and is the first openpilot DM model trained using labels generated by a locally hosted large Vision-Language Model…", "tags": [ "programming" ], "title": "openpilot 0.11.1", "url": "https://aggr.aymericbeaumet.com/items/blog-comma-ai/2026-06-05-openpilot-0-11-1/" }, { "authors": [ { "name": "openpilot team" } ], "content_html": "![\"Cover\"](\"https://blog.comma.ai/img/commacare/cover.png\")
Today, we’re launching commacare, an extended warranty option for comma four. With commacare, your comma four has up to two full years of warranty coverage.
\ncommacare is included with a comma prime subscription.
\nExtend your comma four’s warranty up to an additional year in 3 easy steps:
\nAlready have a comma four? It’s eligible for commacare if you sign up before June 18, 2026.
\n![\"You’ll](\"https://blog.comma.ai/img/commacare/connect_commacare.png\")
You’ll know you’re covered when your device shows the commacare badge in comma connect.
\nI already have a comma prime subscription. Congrats, you’re already covered by commacare!
\nI already have a prime lite subscription at $10/mo. Your subscription price will not increase and will stay at $10/mo until it’s cancelled.
\nDoes commacare cover theft, loss, or accidental damage? No, commacare simply extends the duration of the comma four’s included warranty.
\nMy comma four was delivered more than 30 days ago. All comma four are eligible for commacare for 30 days after this announcement (June 18, 2026).
\nI have a comma two/three/3X. We are not currently offering commacare for those devices, but we will continue to offer trade-ins and repair services.
\nIs commacare offered outside of comma prime? No, commacare is only offered through comma prime.
\nThat’s commacare. Read the full terms here.
\n", "date_modified": "2026-05-19T10:00:00+00:00", "date_published": "2026-05-19T10:00:00+00:00", "external_url": "https://blog.comma.ai/commacare/", "id": "urn:aggr:item:f3133468f1da22c77e6be3d1fd6a87d6d27cb02e", "summary": "Today, we’re launching commacare, an extended warranty option for comma four. With commacare, your comma four has up to two full years of warranty coverage. commacare is included with a comma prime subscription. $24/mo comma prime now…", "tags": [ "programming" ], "title": "Introducing commacare", "url": "https://aggr.aymericbeaumet.com/items/blog-comma-ai/2026-05-19-introducing-commacare/" }, { "authors": [ { "name": "openpilot team" } ], "content_html": "This release marks the start of a new learning paradigm, one that relies solely on neural networks, from planning to simulating driving videos.
\n![\"Timeline](\"https://blog.comma.ai/img/011_release/timeline.png\")
Timeline of progress towards end-to-end learning in learned simulator for openpilot’s driving models
\nThis is the result of a decade-long project started at comma in 2016, when we first described this idea in our paper “Learning a Driving Simulator”. In 2021, we started shipping models that were trained end-to-end and did not reference lanelines or any hand-coded features. These models were trained on-policy in classical simulation. In 2023, we released our first transformer-based driving simulator in commaVQ, a.k.a. a World Model. In 2025, with 0.10, we released our first driving model trained using a World Model for planning, which replaced the hand-coded MPCs that were used prior, but still used a reprojective simulator for videos. Today, we are releasing our first driving model trained using both videos and plans generated from a World Model. To the best of our knowledge, this is the first real-world robotics agent shipped to real users that was fully trained in a learned simulation.
\nIn addition to the new driving model and simulator, this release comes with a more polished interface for comma four. And a whopping 77% reduction in idle power draw. We think you are going to love this release.
\nLeft: a simulated video generated by the commaVQ world model. Right: a simulated video generated by the world model at the time of the release. Top: narrow field of view frame. Bottom: wide field of view frame. Note that the commaVQ world model only generated the wide field of view frame, which explains the missing top left frame. The new simulator produces significantly better image quality, temporal consistency, and reduced temporal artifacts such as flickering.
\nAutonomous driving systems generally rely on one or a mix of these approaches:
\nIn this release, the WMI model 🍉 (#36798) is trained in a simple two-step approach:
\nThe classical simulator used to train the driving models of previous releases could only simulate small deviations in longitudinal policy. This meant it was difficult to show examples to the driving model during training of slowing down or speeding up to a desirable speed. As a result, the driving models did not reliably drive at reasonable speeds on the highway.
\nThe new model starting from a stop on the highway.
\n![\"Speed](\"https://blog.comma.ai/img/011_release/speed_convergence.png\")
Speed convergence test: we initialize the simulation in an empty highway scene at different initial speeds and let the model converge to its desired speed. The new model is much more confident at selecting an appropriate speed on the highway than previously released models.
\nThis poor speed convergence is improved in this release by using the learned simulator. We merged the new model into openpilot nightly on January 19th. You can see in our usage data that users on nightly now prefer Experimental mode (end-to-end longitudinal policy) over openpilot’s ACC policy.
\n![\"Percentage](\"https://blog.comma.ai/img/011_release/exp_mode.png\")
Percentage of segments with Experimental mode enabled for platforms supporting Experimental mode by version. 0.11.0 is our best performing version yet for Experimental mode usage.
\nThe new model has noticeably improved reactivity around parked cars.
\nOn top of the longitudinal simulation range issue, training driving models using hand-coded simulators is tricky because the policy will learn to cheat using artifacts of the simulator. The only way to avoid this is to either (i) improve the simulator with more hand-coded rules or (ii) put bottlenecks on the policy’s representations. As a result of this, improving the policy came at the risk of cheating, which limited model improvements we could do. We have described this cheating behavior in more detail in a previous blog post.
\nWith neural network simulators, fidelity can be improved with more computation and more data, and as long as the policy stays in the realm of “The Big World Hypothesis” i.e. is smaller than the simulator, it won’t cheat. For those reasons we expect many improvements to the driving quality to follow very quickly.
\n![\"Quality](\"https://blog.comma.ai/img/011_release/scaling.png\")
Quality of images generated by the World Model measured by the Learned Perceptual Image Patch Similarity (LPIPS) gets better with more data and more parameters.
\nLeft: Reprojective Simulation, Right: World Model Simulation. We command a 1m deviation to the right (top half) or to the left (bottom half), then let the model recover after the deviation is achieved. Notice the accuracy of the World Model Simulation in achieving the deviation compared to the (perfect) Reprojective Simulation, and the consistency between the narrow field of view frame (top) and the wide field of view frame (bottom).
\nThe driving policy gets better at recovering from noise with more training epochs. Top is the narrow field of view frame, bottom is the wide field of view frame.
\nWe describe the World Model and the driving policy training strategy in detail in our Learning to Drive from a World Model CVPR 2025 paper. But the TL;DR is:
\nA World Model is a learned model of an environment’s dynamics: given the current observation and an action, it predicts what will happen next. For driving, the World Model is given a context of frames and a steering/acceleration action and predicts the next frame as well as future actions. In this sense, the World Model is a driving simulator learned from data instead of hand-coded using 3D geometry rules. Our version of the World Model is also conditioned with future frames, making it an “Inverse Dynamics Model” in addition to a World Model.
\nExamples of simulation rollouts using the World Model.
\nFor the curious readers, we provide more technical details about how the models evolved since the first version of the architecture.
\nThe Compressor is a frame encoder/decoder Transformer (ViT), the encoder has 50M parameters, while the decoder has 100M (2✕ deeper). It encodes 2 camera views (narrow and wide) of size 3✕128✕256 each into a single latent space of size 32✕16✕32. It is trained with a mix of image quality losses: Learned Perceptual Image Patch Similarity (LPIPS), adversarial loss, and least squares error.
\nWe adopt the Masked Auto Encoder (MAE) formulation where the input to the encoder has randomly masked patches. The compressor learns to unmask the image, promoting better modelability.
\nThese improvements to the compressor were released in the 0.10.1 version
\n![\"We](\"https://blog.comma.ai/img/011_release/mae.png\")
We train a small Image Diffusion Model using various compressors to evaluate their modelability. The Masked Auto Encoder formulation is 2.7✕ better than the baseline, and comparable to representation-alignment (REPA) while not needing an additional loss.
\nThe Diffusion Transformer is a 2B parameter transformer (n_layer=48, n_head=25, n_embd=1600) adapted to 3 dimensional data. Each sample is comprised of 2 seconds of past context, 1 second of future conditioning, and a 0...7 seconds of simulation window all sampled at 5 frames per second
\nThe actions (6DOF relative positions), the diffusion noise level, and the frame indices within the sample are encoded and injected into the transformer using AdaLN-single (introduced in PixArt-α).
\nThe Transformer is block-causal: all attention layers are equipped with a block-causal mask, allowing tokens to attend to tokens from their frame and past frames, but not future frames.
\n![\"The](\"https://blog.comma.ai/img/011_release/causal_mask.png\")
The block-causal attention masking.
\nThe diffusion model is trained on 2.5M minutes of driving video with the Rectified Flow formulation with logit normal noise sampling. We also augment the noise level input to the model to make it more robust to accumulated errors in the auto-regressive rollout process.
\nSampling is done frame by frame using 15 Euler steps, with Classifier-Free Guidance (CFG) with a strength of 2.0. Since the transformer is block-causal, we are able to use KV-caching, and achieve a 12.2 frames/sec/gpu throughput.
\n![\"For](\"https://blog.comma.ai/img/011_release/cfg.png\")
For the first half of the rollout, we command a deviation (top: left/right lateral deviation, bottom: acceleration/deceleration). We use a pose net to compare the observed deviation (blue) with the expected deviation (red). Using CFG (right) significantly improves the model’s action following compared to not using CFG (left).
\nThe off-policy model is a FastViT trained to predict a variety of outputs, including lane lines, road edges, lead car information, etc. These outputs are only used for visualization and not used as part of the end to end policy (besides the lead car fallback).
\nThe on-policy model is a small transformer with 2 seconds of context at 5 frames per second. It is trained to predict the actions suggested by the world model given a simulation rollout generated by the world model. The rollouts are generated using the world model and the latest policy checkpoint saved by the learning process, which makes this strategy "near on-policy."
\nDuring the rollouts, the policy is subjected to various types of action and observation noise, which is essential to making it robust in the real world.
\nLateral Noise: lag, lag compensation mismatch, lateral acceleration, steering
\nLongitudinal Noise: lag, lag compensation mismatch, pitch, gas/brake
\nWhen comma four is parked, the panda microcontroller stays awake to detect ignition and boot the main SOC back up. Previously, this idle state drew around 225 mW from the car’s 12V battery. In this release, we cut it down by 77% to 52 mW (panda#2340).
\nParasitic draws above 600 mW are generally considered worth investigating. The previous 225 mW were a real chunk of the car’s key-off power budget. Now at 52 mW, that’s the difference between a dead battery in weeks vs months.
\n| \n | Power | \nCurrent | \nvs. car idle | \n
|---|---|---|---|
| Typical car (key-off) | \n250–600 mW | \n20–50 mA | \n\n |
| + comma four (previous) | \n225 mW | \n19 mA | \n38–95% | \n
| + comma four (0.11) | \n52 mW | \n4.2 mA | \n8–21% | \n
| BMW diagnostic threshold | \n600 mW | \n50 mA | \n100–240% | \n
The panda STM32H7 MCU on comma four runs at 240 MHz on VOS1 voltage scaling. With all peripheral clocks enabled it draws 225 mW in power save state. We reduced this in three stages, each measured individually on comma four hardware:
\nDisable unused peripherals. CAN transceivers, audio, ADC, SPI, UART, DMA, timers, GPIO config, USB. Each of these saves a few mW individually. Together they reduce the power from 225 mW to 158 mW.
\nDrop voltage scaling. From VOS1 (high performance) to VOS3 (low power). This setting reduces the core voltage and clock frequency to save power. Listening for ignition, the lowest power mode is sufficient and saves 41 mW.
\nEnter stop mode. The most pronounced saving comes from disabling the PLL, HSE, and SRAM retention, then entering a deep sleep state called “stop mode”. The CPU halts entirely. The EXTI interrupt lines remain active: CAN RX for CAN ignition and both SBU pins for HW ignition. This brings idle power to 52 mW.
\n![\"comma](\"https://blog.comma.ai/img/011_release/power_saving.png\")
With comma four, we introduced a whole new design language to translate the big-screen experience on the comma 3X to a new compact form factor. This release tightens up that interface with a handful of improvements, large and small.
\nAt launch, the setup and onboarding flows used vertical scrolling pages with their own navigation buttons—inconsistent with the rest of the UI. Now, both use horizontal scrollers with informational cards, matching the regular settings menus (#37264).
\nFirst-time user setup flow
\nOnboarding flow before first drive
\nThe Wi-Fi menu has been totally redesigned (#37152). It’s not only simpler to use with one less sub-menu, but it matches the rest of the system visually and functionally.
\nWe now draw a stack of widgets instead of just the active one, so you can see where you are in the UI while swiping back (#37275).
\n![\"Old:](\"https://blog.comma.ai/img/011_release/no_nav_stack.png\")
Old: only top page was drawn
\nNew: background pages are drawn
\nThe steering arc visualizes how close the car is to its lateral actuation limits. Torque control cars are straightforward to visualize since openpilot knows the current and maximum torque it can apply. Angle control cars just command a steering angle, where maximum can change depending on speed, so there’s no direct percentage.
\nThis release adds steering arc support for angle cars by estimating torque from the current lateral acceleration achieved vs. the maximum available to the car (#37078).
\nFord Bronco Sport with steering arc display
\nFactory reset (#37584) and updater (#36235) UI updated with the standardized horizontal layout
\nNew scroll indicator that gently compresses and bounces back (#37162)
\nShimmer effect on sliders, hinting at the slide direction (#37640)
\nAnti-aliased text and overhauled icons for a more refined look
\nOpacity falloff on the edges of horizontal scrollers (#37239)
\nButton labels realigned for a better fit
\n![\"Frame](\"https://blog.comma.ai/img/011_release/frame_comparisons.png\")
We replaced opendbc’s third-party testing infrastructure with stdlib unittest and a few hundred lines of code.
\nmull, the LLVM-based mutation testing framework we used to verify our safety tests, required specific clang versions and took 45 minutes to run. We replaced it with a custom mutation test runner (opendbc#3130) that runs in 30 seconds.
\nThe safety tests also got faster. Since safety checks are threshold comparisons, we only need to test around the boundaries rather than exhaustively. This cut safety test time by 31% (opendbc#3200).
\nWe removed pytest (opendbc#3191), scons (opendbc#3194), and four other dependencies (opendbc#3186, opendbc#3187, opendbc#3188, opendbc#3190). The best dependencies are the ones you delete.
\nopenpilot is built by comma together with the openpilot community. Get involved in the community to push the driving experience forward.
\n#driving-feedback on DiscordWant to build the next release with us? We’re hiring.
\n", "date_modified": "2026-03-17T10:00:00+00:00", "date_published": "2026-03-17T10:00:00+00:00", "external_url": "https://blog.comma.ai/011release/", "id": "urn:aggr:item:e7965abeb04f7fdd45be5d964b83a319b2ea84fe", "summary": "The first robotics agent fully trained in a learned simulationPermalink This release marks the start of a new learning paradigm, one that relies solely on neural networks, from planning to simulating driving videos. Timeline of progress…", "tags": [ "programming" ], "title": "openpilot 0.11", "url": "https://aggr.aymericbeaumet.com/items/blog-comma-ai/2026-03-17-openpilot-0-11/" }, { "content_html": "![\"\"](\"https://blog.comma.ai/img/comma_hack_6/banner.png\")
March 27 – 29, 2026
\nIRL @ comma HQ in San Diego, CA
\n--
\nDAYS
\n--
\nHOURS
\n--
\nMINS
\n--
\nSECS
\nThe hackathon that brought you ChatPID is back! Join us at comma HQ for a weekend of hacking on the present and future of openpilot — cars, bodies, and anything else you can strap a comma four to.
\nWe'll have comma fours, prototype body v2s, eGPUs, and food. Just bring a laptop, your favorite agent, and some good ideas.
\n30
\npeople
5
\nwinners
$10k
\nin prizes
Fri 6 PM
\nDoors open
\nFri 7 PM
\nKickoff + pizza
\nFri - Sun
\nHacking time
\nSun 3 PM
\nDemos + judging
\nSun 6 PM
\nWinners announced, wrap up
\n![\"comma](\"https://blog.comma.ai/img/comma-four/forms.png\")
Do I need a car? Nope.
\nWhat should I bring? Your laptop and any hardware you want to hack on. We'll provide food, drinks, Wi-Fi, and any comma hardware.
\nWhat kind of projects will I work on? That's up to you! Hack on the UI, strap a comma four to your drone, bring your car to port, or anything else openpilot-related.
\nHow are projects judged? Demos on Sunday afternoon, judged by the comma team on quality and creativity.
\nI've never used openpilot. Can I still come? Absolutely, though it's best to clone it and play around with it beforehand.
\n", "date_modified": "2026-03-03T15:00:00+00:00", "date_published": "2026-03-03T15:00:00+00:00", "external_url": "https://blog.comma.ai/comma-hack-6/", "id": "urn:aggr:item:e1b2408515eefcc3ccc27491b2697931f4da183d", "summary": "March 27 – 29, 2026 IRL @ comma HQ in San Diego, CA --DAYS --HOURS --MINS --SECS The hackathon that brought you ChatPID is back! Join us at comma HQ for a weekend of hacking on the present and future of openpilot — cars, bodies, and…", "tags": [ "programming" ], "title": "COMMA_HACK 6", "url": "https://aggr.aymericbeaumet.com/items/blog-comma-ai/2026-03-03-comma-hack-6/" }, { "authors": [ { "name": "Harald Schäfer" } ], "content_html": "These days it seems you need a trillion fake dollars, or lunch with politicians to get your own data center. They may help, but they’re not required. At comma we’ve been running our own data center for years. All of our model training, metrics, and data live in our own data center in our own office. Having your own data center is cool, and in this blog post I will describe how ours works, so you can be inspired to have your own data center too.
\n![\"Our](\"https://blog.comma.ai/img/datacenter/side_view2.png\")
Our data center
\nIf your business relies on compute, and you run that compute in the cloud, you are putting a lot of trust in your cloud provider. Cloud companies generally make onboarding very easy, and offboarding very difficult. If you are not vigilant you will sleepwalk into a situation of high cloud costs and no way out. If you want to control your own destiny, you must run your own compute.
\nSelf-reliance is great, but there are other benefits to running your own compute. It inspires good engineering. Maintaining a data center is much more about solving real-world challenges. The cloud requires expertise in company-specific APIs and billing systems. A data center requires knowledge of Watts, bits, and FLOPs. I know which one I rather think about.
\nAvoiding the cloud for ML also creates better incentives for engineers. Engineers generally want to improve things. In ML many problems go away by just using more compute. In the cloud that means improvements are just a budget increase away. This locks you into inefficient and expensive solutions. Instead, when all you have available is your current compute, the quickest improvements are usually speeding up your code, or fixing fundamental issues.
\nFinally there’s cost, owning a data center can be far cheaper than renting in the cloud. Especially if your compute or storage needs are fairly consistent, which tends to be true if you are in the business of training or running models. In comma’s case I estimate we’ve spent ~5M on our data center, and we would have spent 25M+ had we done the same things in the cloud.
\nOur data center is pretty simple. It’s maintained and built by only a couple engineers and technicians. Your needs may be slightly different, our implementation should provide useful context.
\nTo run servers you need power. We currently use about 450kW at max. Operating a data center exposes you to many fun engineering challenges, but procuring power is not one of them. San Diego power cost is over 40c/kWh, ~3x the global average. It’s a ripoff, and overpriced simply due to political dysfunction. We spent $540,112 on power in 2025, a big part of the data center cost. In a future blog post I hope I can tell you about how we produce our own power and you should too.
\n![\"data](\"https://blog.comma.ai/img/datacenter/power.png\")
data center power usage
\nData centers need cool dry air. Typically this is achieved with a CRAC system, but they are power-hungry. San Diego has a mild climate and we opted for pure outside air cooling. This gives us less control of the temperature and humidity, but uses only a couple dozen kW. We have dual 48" intake fans and dual 48" exhaust fans to keep the air cool. To ensure low humidity (<45%) we use recirculating fans to mix hot exhaust air with the intake air. One server is connected to several sensors and runs a PID loop to control the fans to optimize the temperature and humidity.
\n![\"Filtered](\"https://blog.comma.ai/img/datacenter/air_cooling.jpg\")
Filtered intake fan on right, 2 recirculating fans at the top
\nThe majority of our current compute is 600 GPUs in 75 TinyBox Pro machines. They were built in-house, which saves us money and ensures they suit our needs. Our self-built machines fail at a similar rate to pre-built machines we’ve bought, but we’re capable of fixing them ourselves quickly. They have 2 CPUs and 8 GPUs each, and work as both training machines and general compute workers.
\n![\"Breaker](\"https://blog.comma.ai/img/datacenter/breakers.jpg\")
Breaker panels for all the computers, that’s a lot of breakers!
\nFor data storage we have a few racks of Dell machines (R630 and R730). They are filled with SSDs for a total of ~4PB of storage. We use SSDs for reliability and speed. Our main storage arrays have no redundancy and each node needs to be able to saturate the network bandwidth with random access reads. For the storage machines this means reading up to 20Gbps of each 80TB chunk.
\nOther than storage and compute machines we have several one-off machines to run services. This includes a router, climate controller, data ingestion machine, storage master servers, metric servers, redis servers, and a few more.
\nRunning the network requires switches, but at this scale we don’t need to bother with complicated switch topologies. We have 3 100Gbps interconnected Z9264F switches, which serve as the main ethernet network. We have two more infiniband switches to interconnect the 2 tinybox pro groups for training all-reduce.
\nTo effectively use all these compute and storage machines you need some infra. At this scale, services don’t need redundancy to achieve 99% uptime. We use a single master for all services, which makes things pretty simple.
\nAll servers get ubuntu installed with pxeboot and are managed by salt.
\nAll of our storage arrays use mkv. The main array is 3PB of non-redundant storage hosting our driving data we train on. We can read from this array at ~1TB/s, which means we can train directly on the raw data without caching. Redundancy is not needed since no specific data is critical.
\n![\"Storage](\"https://blog.comma.ai/img/datacenter/mkv_machines.png\")
Storage nodes
\nWe have an additional ~300TB non-redundant array to cache intermediate processed results. And lastly, we have a redundant mkv storage array to store all of our trained models and training metrics. Each of these 3 arrays have a separate single master server.
\nWe use slurm to manage the compute nodes, and compute jobs. We schedule two types of distributed compute. Pytorch training jobs, and miniray workers.
\nTo train models across multiple GPU nodes we use torch.distributed FSDP. We have 2 separate training partitions, each intra-connected with Infiniband for training across machines. We wrote our own training framework which handles the training loop boilerplate, but it’s mostly just pytorch.
![\"reporter;](\"https://blog.comma.ai/img/datacenter/reporter.png\")
reporter; comma’s experiment tracking service
\nWe have a custom model experiment tracking service (similar to wandb or tensorboard). It provides a dashboard for tracking experiments, and shows custom metrics and reports. It is also the interface for the mkv storage array that hosts the model weights. The training runs store the model weights there with a uuid, and they are available to download for whoever needs to run them. The metrics and reports for our latest models are also open.
\nBesides training we have many other compute tasks. This can be anything from running tests, running models, pre-processing data, or even running agent rollouts for on-policy training. We wrote a lightweight open-source task scheduler called miniray that allows you to run arbitrary python code on idle machines. This is a simpler version of dask, with a focus on extreme simplicity. Slurm will schedule any idle machine to be an active miniray worker, and accept pending tasks. All the task information is hosted in a central redis server.
\n![\"Our](\"https://blog.comma.ai/img/datacenter/tbox2.jpg\")
Our main training/compute machines. Notice the 400Gbps switch in the center.
\nMiniray workers with GPUs will spin up a triton inference server to run model inference with dynamic batching. A miniray worker can thus easily and efficiently run any of the models hosted in the model mkv storage array.
\nMiniray makes it extremely easy to scale parallel tasks to hundreds of machines. For example, the controls challenge record was set by just having ~1hr of access to our data center with miniray.
\nAll our code is in a monorepo that we have cloned on our workstations. This monorepo is kept small (<3GB), so it can easily be copied around. When a training job or miniray distributed job is started on any workstation, the local monorepo is cached on a shared NFS drive including all the local changes. Training jobs and miniray tasks are pointed towards this cache, such that all distributed work uses the exact codebase you have locally. Even all the python packages are identical, UV on the worker/trainer syncs the packages specified in the monorepo before starting any work. This entire process of copying your entire local codebase and syncing all the packages takes only ~2s, and is well worth it to prevent the issues mismatches can cause.
\nThe most complex thing we do at comma is train driving models on-policy, these training runs require training data to be generated during training by running simulated driving rollouts with the most recent model weights. Here’s a real-world command we just used to train such a model. This training run uses all of the infrastructure described above. While only this small command is needed to kick everything off, it orchestrates a lot of moving parts.
\n./training/train.sh N=4 partition=tbox2 trainer=mlsimdriving dataset=/home/batman/xx/datasets/lists/train_500k_20250717.txt vision_model=8d4e28c7-7078-4caf-ac7d-d0e41255c3d4/500 data.shuffle_size=125k optim.scheduler=COSINE bs=4\n![\"Diagram](\"https://blog.comma.ai/img/datacenter/onpol.png\")
Diagram of all infrastructure involved in training an on-policy driving model.
\nDoes all this stuff sound exciting? Then build your own datacenter for yourself or your company! You can also come work here.
\nHarald Schäfer
\nCTO @ comma.ai
The Cool People’s Model 😎 (#36249) comes with two minor changes in the model’s architecture and simulator training conditions.
\nWe changed the encoding of the Vision Features, Desire Signal (lane changes, turns, etc.), and Traffic Convention (left-hand drive vs right-hand drive) from a fixed-length non-causal attention layer to a variable-length causal attention layer. While this does not change the behavior of the current training stack, it opens up the possibility to train with variable temporal lengths and makes more sense from an architectural point of view.
\n![\"Temporal](\"https://blog.comma.ai/img/0103_release/temporal_policy.png\")
In order to make the model robust to real-world physics noise, we introduce a wide array of noise types during on-policy training in the simulator [1]. This release includes a small change in those noise models, making them start at a noise-free initial value, which is a required change to train using the World Model Simulation.
\nLeft: a simulation rollout using the previous noise model (starts at a deviated state) and the reprojective simulator (tinysim)
\nRight: a simulation rollout using the new noise model (starts at a noise-free state) and the world model simulator (mlsim)
We are incrementally adding incoming comma four segments to the driver monitoring training dataset, which will improve the experience overall, but especially for comma four users. The comma four driver-facing camera has a different position compared to the comma three and 3X, which is why training on those segments is important. If you just received your comma four, please consider turning on the “Record and Upload Driver Camera” toggle.
\n![\"Device](\"https://blog.comma.ai/img/0103_release/dm_device_types.png\")
Device type distribution in the new training set
\nMSGQ, openpilot’s custom pub-sub IPC, previously allocated a 10MB ring buffer for every service, regardless of the service’s message size. This also meant that tiny low-frequency services, such as clocks, may take weeks of runtime to reach steady state memory usage. While wasteful, this was fine when openpilot had fewer services, however this ballooned to 711MB over the years.
This PR (#36884) reduces MSGQ’s memory usage to 90MB by setting up configurable buffer sizes per-service. In the future, we’ll explore using the capnproto schema and service frequency definitions to automatically determine suitable buffer sizes.
\nWe refreshed our web flasher with a new guided wizard to make it easier to get your device back to a factory state (flash#177). Using our new custom USB Vendor ID, we’re able to cut some steps out of the flashing procedure for comma four.
\n![\"Flash\"](\"https://blog.comma.ai/img/0103_release/flash.png\")
openpilot is built by comma together with the openpilot community. Get involved in the community to push the driving experience forward.
\n#driving-feedback on DiscordWant to build the next release with us? We’re hiring.
\n[1] Peng, Xue Bin, et al. “Sim-to-real transfer of robotic control with dynamics randomization.” 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018.
\n", "date_modified": "2025-12-21T10:00:00+00:00", "date_published": "2025-12-21T10:00:00+00:00", "external_url": "https://blog.comma.ai/0103release/", "id": "urn:aggr:item:58848f7dc269ed2e85dcca1ca01b7486afbd4f73", "summary": "New driving modelPermalink The Cool People’s Model 😎 (#36249) comes with two minor changes in the model’s architecture and simulator training conditions. A more flexible temporal policy architecturePermalink We changed the encoding of the…", "tags": [ "programming" ], "title": "openpilot 0.10.3", "url": "https://aggr.aymericbeaumet.com/items/blog-comma-ai/2025-12-21-openpilot-0-10-3/" } ], "language": "en", "title": "comma.ai blog", "version": "https://jsonfeed.org/version/1.1" }