I have been waking around 4:30 a.m. lately, just before sunrise, and enjoying it. On September 19 I used that morning to start a lab on Kitsune2, the peer-to-peer networking layer underneath Holochain, and went looking for where it was spending time.
By 12:20 p.m., the lab had produced 6 patches. Endpoint lookups: 251× faster at 10,000 retained identities. The peer-update path that calls them: 33× faster. Gossip initiation after a completed fetch: 3,002 ms down to 202 ms. Burst request traffic: 96.9% fewer messages. 3 of the changes repaired work that was waiting unnecessarily, or waiting for something that would never happen.
I've been coding for more than 25 years and building with AI since GPT-3. I use it to isolate something I care about or don't yet understand, work through it, and test what we think we've learned. I love that part: turning a question into an experiment and seeing what holds. This morning, with AI coding agents, I applied that approach to the machinery that lets Holochain nodes keep each other up to date.
Holochain applications share data across participating computers, or nodes. Each participant keeps its own signed history and helps validate and store a portion of the application's shared data. That shared store is the distributed hash table, or DHT. Copies live with several peers, so data can remain available when its author goes offline. Holochain's DHT guide explains how those responsibilities are distributed.
Kitsune2 handles the peer-to-peer communication underneath this. Picture a node catching up: it learns where peers can be reached, compares what it holds with what they hold, and asks for the pieces it lacks. Gossip is the exchange through which peers discover those differences; fetch retrieves the missing operations, the units of data Kitsune2 transfers. The receiving node hands them to its host's operation store. Holochain handles their validation and integration.
Diagram source
graph TD
    A[Learn peer identities and endpoints] --> B[Compare data through gossip]
    B --> C[Identify missing operations]
    C --> D[Fetch operations from peers]
    D --> E[Hand received operations to the host store]
    E --> F[Signal that pending fetches have completed]
    F --> B
The patches sit along that sequence. Some reduce the work a node does; others let it move on when work has already finished. The figures above measure those particular steps on local test nodes.

What each of the 6 changes actually moved

We fixed the starting point at upstream commit 06555ee, development source 0.6.0-dev.1. Each comparison uses that baseline plus the recorded patch sequence. The labels below connect the findings to our experiment records; all 6 patches are now submitted for upstream review.
LabelWhat changedMeasured result
K3aCall the existing drain helper when a fetch completesGossip initiation after a completed fetch: 3,002 → 202 ms
K3bReceiver lock scoped to the dequeueEffective concurrency was 1. Configured workers now genuinely overlap
K3cGeneration-tagged admission ownershipCancelled callers strand 0 pending keys. Baseline stranded 8
K3d-P1Set-based response cleanup1.9–6.1× faster response processing at large pending loads
K3d-P2Opt-in same-peer batching (default 1)Bursts: 87.5–96.9% fewer request messages
K1Reverse URL to agent indexAt 10,000 retained identities: updates 21–33×, lookups 118–251× faster, for 3.10× store memory
The first repair reconnects fetch completion to the gossip task waiting for it. The change calls an existing helper that tells listeners the pending map has emptied. We tested both a completed fetch and a fetch deliberately held open.
Gossip initiation after a fetch completes, with the control that must not move (ms)
After completion, gossip can proceed. While the fetch remains pending, both versions still reach the fallback timer. The isolated test above measured 3,001 → 202 ms; the later combined replay measured 3,002 → 202 ms. Both used a 3,000 ms fallback and a 200 ms initial delay. The roughly 15× difference records which route the task took: waiting out the fallback or waking on completion.

Every change lives in one crate, and 5 of the 6 in a single file

A Rust crate is a package of code. All 6 patches live in the core crate, which contains the peer records and fetch machinery. They change how gossip gets its data and completion signals without editing the gossip algorithm or the underlying network transport. Batching adds an optional core configuration field.
LabelFileLines
K1core/src/factories/core_known_peers.rs+33 / −10
K3acore/src/factories/core_fetch.rs+1
K3bcore/src/factories/core_fetch.rs+11 / −3
K3ccore/src/factories/core_fetch.rs+71 / −53
K3d-P1core/src/factories/core_fetch.rs+6 / −1
K3d-P2core/src/factories/core_fetch.rs, 1 test file+89 / −17
5 patches stack in a recorded order on core_fetch.rs, which requests missing operations and processes responses. The remaining patch sits in core_known_peers.rs, the retained record of which identity is associated with which network endpoint. We then applied all 6 to a single source tree and ran them together.

The data had arrived, but gossip was still waiting

CoreFetch keeps a map of the operations it has asked for and from whom. When a peer's response arrives and the operations are stored successfully, the code removes those identifiers from the map. Other parts of the file, the send-error path and the peer-cleanup path, call a helper named notify_listeners_if_queue_drained right after they remove entries. The success path never called it.
A node can expand the portion of the DHT it stores, called its storage arc. To do that, it needs to catch up on data within the larger range. In the arc-growth path we tested, gossip waits for outstanding fetches to finish before initiating with a peer, with a timer as a fallback. The data arrived and the pending map emptied, but the task waiting for that signal stayed asleep until the timer expired.
Diagram source
sequenceDiagram
    participant G as Gossip arc-growth loop
    participant F as CoreFetch
    participant P as Peer
    G->>F: Register a drain waiter
    P->>F: Final response, operations stored
    F->>F: retain() empties the pending map
    Note over F: helper exists, never called
    F--xG: Notification that never arrives
    G->>G: Initiate after the roughly 3-second fallback
Checking after completion hides the difference: a newly registered waiter sees the empty map and returns immediately. We registered ours before completion. Once the data was stored, that waiter still had no notification on the original code; with the patch, it received one. This repairs the handoff between receiving data and starting the next gossip exchange.
We also checked partial responses, store errors, several listeners, and listeners registering as requests completed or new work arrived. A partial response must leave the waiter pending; only completing the final outstanding fetch should wake it.

5 workers were configured, and 1 was actually sending

The outgoing side of CoreFetch runs a configured number of worker tasks pulling from one shared channel. The loop was written like this:
rust
while let Some(..) = outgoing_request_rx.lock().await.recv().await {
    // send the request
}
The lock lets 1 worker take the next request from the shared queue. But Rust keeps that temporary lock guard alive through the body of this while let, including the wait for the network send. While 1 worker was sending, the other 4 could not even collect their next request. All requests could still arrive eventually, so a delivery test could pass while the configured concurrency was unused.
We moved the receive into a short block so the lock is released before the send begins. Workers still take turns collecting requests, then send them concurrently. A slow send no longer holds the queue lock while other workers have requests ready.
Outgoing sends in flight at the same moment, by configured worker count
The measurement holds each send open with a paused clock and counts how many have entered before any is released, then confirms nothing ever enters beyond the configured bound.

Cancelling a caller left the node waiting for 8 unsent operations

A fetch request passes through 2 records: the pending map says what the node is waiting for, and the outgoing queue holds work for the send workers. request_ops, the method that asks for operations, added a whole batch to the pending map before waiting for room in that queue.
If the caller was cancelled while waiting for capacity, some operations had already been marked pending even though no worker had received them. We filled the queue, cancelled a blocked caller, and let all 16,385 admitted requests finish. 8 operations remained pending. They had never entered the outgoing queue, so there was no response on its way to clear them.
That connects directly to the first repair: a completion notification cannot fire while the map still contains unfinished work. Here, the node's own bookkeeping was keeping it waiting.
The patch reserves queue capacity first, then records and enqueues the operation without another await between those steps. Each entry also carries a generation tag, so cleanup from an older failed send cannot erase a newer request for the same operation. Work already admitted survives the caller's cancellation; work still waiting for admission leaves no pending entry behind.
This repairs ownership of the requests. Total memory limits remain a separate design question, since a caller can still supply a large batch.

Fewer request messages, with a limit on how large each can grow

A catching-up node may need many operations from the same peer. Kitsune2's request format already supports a list of identifiers, but the outgoing code sends 1 per message. Before combining those requests, we checked how larger responses would be handled.
For every pending operation, response cleanup scanned the list of returned identifiers to see whether it could remove that entry. With 8,192 pending entries and 1,024 returned identifiers, that repeats a large amount of searching. Building a HashSet of the returned identifiers makes each membership check cheap; response processing measured 6.1× faster in that case. With only 16 returned identifiers, storing the data dominated and the change bought little.
Then we combined ready requests for the same peer. In a burst of 64 small operations, batch sizes of 8 and 32 reduced request messages by 87.5% and 96.9%, respectively.
Burst behaviour relative to the default batch size of 1 (percent of control)
The burst finished about 27–32% sooner. The same operations still had to be transferred and stored; the saving was in how many requests carried them. The batch-1 control measured 105.4% of baseline elapsed time, showing roughly 5% timing drift even with batching inactive.
A worker combines only requests that are already ready, so a lone request never waits for a batch to fill. Batch splitting uses both a configured identifier limit and a 4,096-byte encoded request threshold. Counting items alone does not constrain their total size; the test with 200 identifiers splits them into several capped messages. A single identifier is still sent if it alone exceeds that threshold, so this is not a universal wire-size bound. Responses carry the actual data and would need a separate size limit on the responding node.
The candidate keeps batch size 1 as its default and accepts existing configurations unchanged. We left larger batches opt-in because the separate test for a lone request produced inconsistent tail-latency results, covered below.

Taking the scan off the update path was worth 33×, and 251× on the lookup itself

CoreKnownPeers remembers identities and the network endpoints associated with them. Several identities can share an endpoint, and a blocked identity still needs to be identifiable after it has been removed from the active peer store. 10,000 retained identities therefore means a history of identities in this store, not 10,000 computers connected at once.
On a peer-store update, access control asks which identities belong to the endpoint. The lookup, get_by_url, scanned the entire retained map under a lock each time. As that history grew, updating current peer information became more expensive.
We added an index from endpoint URL to the identities associated with it: HashMap<Url, HashSet<AgentId>>. The original records remain authoritative. When an identity changes endpoint, the index moves it to the corresponding set; lookup can then go directly to that set. Both versions passed 19,422 result checks covering updates, stale advertisements, removals, and shared endpoints.
Microseconds per 16-advertisement update call (log scale)
Each timed call above inserts 16 advertisements and runs the access-control listeners. At 10,000 retained identities, it dropped from about 155 µs to 4.72 µs, a paired 33× gain. The indexed version stays nearly flat across these sizes because those listeners no longer scan the growing history.
Measured at the API instead of through the caller, the lookup itself goes from a scan to a hash. A miss at 10,000 retained identities dropped from 11,423 ns to 45 ns, a paired 251×, and a single hit from 9,309 ns to 79 ns, 118×. A query that returns many identities still scales with what it returns, so the shared-URL case gains 4.4× rather than 2 orders of magnitude. The 251× is the isolated lookup result. The 33× measures the larger update call that uses it; these gains cannot be added or multiplied.
Requested store memory at 10,000 retained identities, by endpoint shape
3.10× on a mixed store, an extra 2.51 MB at 10,000 identities. Where every identity shares an endpoint the derived map is small and the cost falls to 1.45×. Where no identity has a URL there is nothing to derive and the cost is zero. Those numbers are allocator-requested bytes for the retained store, not total node memory.

With all 6 patches together, the node could finish and continue

The combined replay used Kitsune2's real gossip, fetch, peer-access, and memory-store components, communicating through its in-memory test transport. We ran a 5-step sequence on the original and patched source trees:
  1. Update peer information. An identity changes endpoint. Stale information arriving later leaves the newer record intact, and a blocked identity remains resolvable for access control. Both versions pass; the index preserves existing decisions.
  2. Start gossip after a fetch completes. The patched node initiates after 202 ms. The original takes the fallback path at 3,002 ms.
  3. Receive data out of order. 3 requested operations arrive as 4 responses, including a duplicate. Both versions store them consistently and associate byte counts with the correct operation.
  4. Finish the fetch. The stored operations can be retrieved. The patched node also wakes the listener that registered while those operations were pending.
  5. Continue after a failed peer. A controlled send failure clears that peer's request state. Another peer supplies the operation, the pending map empties, and a subsequent fetch succeeds. On the original, the completion-notification checks fail.
Separate interaction tests combined cancellation with concurrent sends and combined batching with completion notification. The repairs held together: admitted work completed, cancelled admissions left no orphan entries, and listeners woke after the final response.
This gets us as far as data received into the host store and the next networking step able to proceed. Holochain's validation and integration still follow outside this lab. We have yet to measure how much these changes shorten a real node's full catch-up time.

What each gain costs

The index costs memory and makes endpoint changes more expensive. In the original small-store test, updating a fixed set of identities to new URLs ran at 0.946× baseline speed with 100 retained identities. At 10,000, an isolated endpoint-changing write cost 3.68× as much. Maintaining the extra index has a price; the lookup savings must repay it.
Because that trade-off is real, we went looking for the workload where the index loses. 36 configurations were predeclared before measuring, 432 paired runs executed, and the result transcripts hashed identically between variants.
Paired speedup across the 36-config workload envelope, by mix (log scale)
Every workload in this broader matrix had a median above parity, including the mix where every accepted update changed an endpoint: 1.47× at 100 identities and 36.42× at 10,000. Those updates also trigger access-control lookups, and their savings outweighed the additional writes. This broader workload differs from the earlier fixed-set test; its positive result leaves that small-store regression on the record.
The 100-identity, mostly URL-less case remains inconclusive: its 1.09× median included a paired result below parity. We still need traces from real nodes showing retained identity counts, endpoint changes, and lookup frequency before recommending the index for a particular workload.
Concurrent sends exposed a cleanup interaction. Once workers could overlap, cleanup for a failed send could run while another request to the same peer was in flight. That made the existing peer-wide cleanup broader than the work that had failed. The ownership patch narrowed removal to entries still belonging to that request generation. Testing the patches together mattered here.
Batching's lone-request latency remains inconclusive. We measured p95, the time within which 95% of requests finish. For a request sent alone, the worker's final run showed 1.01–1.08× baseline p95; the independent rerun showed 1.44–1.58×. These fixtures take tens of microseconds, and their controls also drifted. The burst improvement reproduced, but the lone-request result did not settle, so larger batches remain opt-in.

Repeated comparisons and stalled sends remain open

We also examined 2 other parts of the same catch-up sequence. Neither produced an accepted patch.
Comparing what peers already hold (K4). Gossip uses compact summaries of stored data to find differences. In the largest test, producing and comparing an unchanged snapshot materialized 262,144 hashes; repeating it 10 times produced identical output each time. Reusing a previous summary looks attractive, but the host store can change without a reliable signal to invalidate that saved result. A cached summary could then conceal a difference between peers. The alternatives we assessed also fell short of the required 20% time saving on the in-memory store. Measurements from a real Holochain store are the next missing piece.
Recovering when a gossip send stalls (K2). Reading the source raised a question about a send holding a lock that the timeout task also needs. We also needed to establish what happens to the underlying Iroh transport stream if that send is cancelled partway through. The agent provider blocked work on the reproduction driver, so the runtime test was never completed. This remains an untested concern, with no patch or reliability result.

What we measured, and what remains to be measured

These experiments ran on development source 0.6.0-dev.1, on 1 shared Apple M4 Max with 16 logical cores. We used local test nodes and in-memory transports, without core pinning or a second physical host. The results cover named components and their interactions. Network throughput, whole-network convergence, total node memory, and Holochain application latency remain unmeasured.
The method follows my work on benchmark-driven development. We recorded each question and acceptance threshold, held the source and fixtures fixed within each comparison, and kept commands, samples, and failures in append-only receipts.
For the 3 repairs, the same tests failed on the original code at the expected assertions and passed with the patch. For the optimizations, both versions had to preserve behavior, then the measurements had to meet the stated threshold. Timing runs alternated order and kept compilation outside the measured window. A supervising agent reran the acceptance commands on the frozen source before accepting each result; the disagreement over lone-request latency stayed in the final report.
The completed package contains 204 receipt directories. The combined tests were accepted at 12:17 p.m., and the terminal results were recorded at 12:20 p.m. A final closure check was retained at 2:38 p.m. All 6 patches are now open for upstream review: completion notification, worker concurrency, admission ownership, response cleanup, request batching, and endpoint indexing.

How this argument was built

What I brought. I commissioned the lab, directed its scope, and kept publication and upstream submissions behind review. I wanted measurable refinements to Holochain's networking layer and an article that makes the work understandable.
What the AI supplied. Coding agents built the experiments, wrote the patches, and ran the measurements in separate worktrees. GPT-6 Astra handled the completion-notification and admission-ownership work; GLM-5.3 handled worker concurrency, workload testing, DHT profiling, batching, and composition. A supervising agent checked the results before accepting them. AI collaborators also drafted and refined this article from those records.
Where I pushed back. I kept yesterday's separate PR out of the story, kept the gains in the opening, and asked for the explanation to follow what nodes actually do. I also cut language that scored points off maintainers or advertised the importance of the work. The agents kept the claims within the measurements, including the memory cost and the inconclusive batching result.
What we arrived at together. 6 candidate patches, 3 of them repairs to completion, concurrency, and request ownership. The Holochain core team's replaceable component factories and in-memory transport let us test those behaviors in real Kitsune2 components. That existing architecture made the lab possible.
Outside review. The supervising reruns are recorded in the experiment reports. External maintainer review of these patches is the next step.
The count. 6 accepted local patches and 204 receipt directories. The article's working ledger records this refinement; it is not a complete count of the conversations that preceded the lab.