Incidents

What went wrong

Every outage and every fault that affected miners here: when it started, how long it lasted, what it cost, why it happened, and what we changed. The figures marked as measured come from the machine's own records. When our monitoring catches a problem, it appears on this page by itself, marked under investigation, until we have written up what happened.

At a glance

Ongoing
0
happening now
Under investigation
2
over, not yet fully explained or fixed
Resolved
3
explained, fixed, written up

Open

Under investigation

A block accepted a moment later than expected locked the pool for three minutes

to · lasted 3 min

Incident 2026-09-28-0556 · noticed by our monitoring

What happened

At 05:56 UTC the pool found a block and sent it to the network. The network accepted it, at its correct place. But the pool expected it one place lower: the work it had handed out was prepared by our node in the few thousandths of a second between two quick rearrangements of the chain, and that work described the block's position inconsistently. The pool software treated the difference as a conflict and stopped submitting any block at all.

Our automatic recovery saw it within a minute, stopped the pool, rebuilt the pool's database and had it running again at 06:00. Nobody had to intervene.

Impact on miners

  • The pool did not answer for 2.1 min.
  • The pool accepted no work from any miner for 2 min.
  • 1 block found by a miner was not submitted to the network — about 162 BDAG of rewards that never reached the pool (at 162.25 BDAG, the median the pool wallet received per block).
  • Payouts from the start until an hour after the end: 50 sent (12,008.270 BDAG), none failed.
  • The block that set this off is on the chain and its reward arrived. Because of the fault the pool recorded no shares for it, so its reward was shared out on the shares of the other blocks found in the same few minutes — practically the same miners.
  • One block found 13 seconds later was thrown away before the pool was stopped.

Measured on 28 September 2026, 07:05 UTC from the status checker's samples, the pool's share log and its own log, and the payout records.

Why it happened

Every block the pool hands out as work carries a height (its position in the chain) and the blocks it builds on. Normally the two agree. Here our node prepared that work 11 thousandths of a second after the chain had rearranged itself twice in 8 thousandths of a second, and the work mixed the two moments: the blocks it builds on came from after the second rearrangement, the height from after the first. The network placed the block where its parents say it belongs, one higher; the pool expected the height written in the work, and treats any such difference as a conflict, and a conflict as a reason to stop submitting every block until its database is started afresh. It is not a fault in the block, the miner or the network. It is rare: of the blocks found since the night before, it happened once.

How it was fixed

The automatic recovery rebuilt the pool's database, as it is built to (the pool's own records only — balances and payout history are kept elsewhere and were not touched). There is no pool setting that avoids this particular case.

What we changed so it does not happen again

  • Reported to the BlockDAG developers with the evidence: an answer that says "accepted" for the expected block should never stop the pool, and a problem with one block should not stop every other one.
  • Until that is fixed, the automatic recovery is the protection: it acts within a minute, and for now it has no limit on how often it may recover.
Under investigation

The same submission lock, twice more; the pool stayed stopped for 100 minutes

to · lasted 2 h 13 min

Incident 2026-09-28-0225

What happened

At 02:25 UTC the fault of the night before came back: the pool received the same block twice from one miner and locked itself against submitting any block.

This time the new automatic recovery caught it. It stopped the pool within a minute, rebuilt the pool's database and had the pool running again at 02:31. No block was lost: the pool was stopped before it found another one.

At 02:58 the same fault came back. The automatic recovery stopped the pool again, and because it had already recovered once in that hour it did not try a second time: it is built to stop and call a person when a fault repeats that fast, instead of looping. It did exactly that, and sent the alert. But nothing moved the miners to our backup pool, so from 02:59 until 04:38 this pool did not mine at all.

Impact on miners

  • The pool did not answer for 103.2 min.
  • The pool accepted no work from any miner for 103 min.
  • No block found by a miner was lost.
  • Payouts from the start until an hour after the end: 106 sent (23,379.339 BDAG), none failed.
  • No block found by a miner was lost: each time the pool was stopped before it found another one.
  • Miners were not moved to the backup pool automatically. Unless a miner had a backup pool set in their own miner, they were not mining for about 100 minutes.
  • Payouts are not affected by a stopped pool: rewards that had already arrived were paid as usual.

Measured on 28 September 2026, 05:37 UTC from the status checker's samples, the pool's share log and its own log, and the payout records.

Why it happened

The same as the night before: one miner's device sends the same solution twice, on two jobs issued a fraction of a second apart with identical work, and the pool software turns the second copy into a permanent lock instead of ignoring it as a duplicate. The second copy went through a setting that lets the pool submit a block found on a job it has just replaced.

How it was fixed

At 04:38 we rebuilt the pool's database by hand and set that grace period to zero (POOL_RECENT_STALE_BLOCK_CANDIDATE_SUBMIT_GRACE_MS=0): the pool no longer submits a block from a job it has already replaced, so the second copy never goes out. In the first hour after the change the pool found and submitted 124 blocks, with no lock and no conflict.

What we changed so it does not happen again

  • The setting above stays for now. It has a cost: a block found on a job that was replaced a moment earlier is no longer submitted. In that first hour this happened twice (on jobs replaced 0.6 and 1 second before), against 124 blocks submitted — about 1.6% of the blocks found, which the old setting would probably have saved.
  • We are watching whether it holds before calling this closed.
  • Planned, not yet in place: when the automatic recovery reaches its limit, move the miners to the backup pool automatically instead of leaving the pool stopped.
  • Reported to the BlockDAG developers: a second copy of a block that was already accepted should be ignored, not lock the pool; and the pool should not issue identical work under a new job.

Resolved

Resolved

The pool found blocks but could not submit them

to · lasted 1 h 4 min

Incident 2026-09-27-2143

What happened

At 21:43 UTC the pool software locked itself: from that moment it refused to submit any block it found to the network. It went on accepting shares as normal, so for a miner nothing looked wrong. Our monitoring noticed within a minute and sent the alert.

At about 22:36 the pool was stopped and the miners moved to our backup pool in Vienna. At 22:47 the pool was running again on a rebuilt database.

Impact on miners

  • The pool answered, but not correctly, for 8.1 min.
  • The pool accepted no work from any miner for 10 min.
  • 87 blocks found by miners were not submitted to the network — about 14,116 BDAG of rewards that never reached the pool (at 162.25 BDAG, the median the pool wallet received per block).
  • Payouts from the start until an hour after the end: 36 sent (6,508.205 BDAG), none failed.
  • The blocks found but not submitted earned nothing: their rewards never reached the pool, so they were not paid to anyone. This is a loss for every miner in the pool, in proportion to their work.
  • Payouts themselves were not affected: every reward that did arrive was paid as usual.

Measured on 28 September 2026, 05:37 UTC from the status checker's samples, the pool's share log and its own log, and the payout records.

Why it happened

One miner's device sent the same winning solution twice, on two jobs issued a fraction of a second apart with identical work. The pool accepted the first copy. It recorded the second as a conflict, and the pool software treats any such record as a reason to stop submitting blocks altogether — permanently. The record cannot be removed; the software has no way to clear it other than starting its database afresh.

How it was fixed

We stopped the pool and rebuilt its database (the pool's own records only — balances and payout history live elsewhere and were not touched). The pool started clean and submitted blocks again at once.

What we changed so it does not happen again

  • An automatic recovery went live about an hour later: it checks every minute, stops the pool within a minute of this fault and rebuilds the database without waiting for a person, with limits so it cannot loop.
  • Reported to the BlockDAG developers, with the evidence.
  • The fault itself came back that night; see the incident of 28 September, 02:25 UTC.
Resolved

One miner's device was paid far less than its work

to · lasted 2 h 17 min

Incident 2026-09-27-1831

What happened

This was not an outage: the pool ran and found blocks all evening. But one miner's device, which joined at 18:31 UTC, did about nine tenths of the pool's work and was paid almost nothing for it, and the page showed its hashrate about 65,000 times too low.

We noticed because the pool suddenly found about twenty times more blocks than before, with no other change. At 20:19 we paused payouts, so that no more rewards would be split the wrong way, fixed the calculation, and paid again at 20:48 on the corrected shares.

Impact on miners

  • Between 18:31 and 20:19 UTC that miner received 7.51 BDAG. By the corrected calculation its share of those payouts would have been about 20,277 BDAG; the other miners received the difference, about 20,269 BDAG.
  • Payouts made before the fix were not recalculated.
  • Payouts were paused for about half an hour (20:19–20:48) while the fix was prepared and checked; nothing was lost, the rewards of that time were paid in the first corrected payout.
  • The miners page showed that device's hashrate about 65,000 times too low until 20:37.

Why it happened

Mining software and the pool must agree on what "difficulty 1" means. This pool uses about 65,537 hashes; that device's software uses Bitcoin's convention, 2³² hashes — 65,536 times more. So it only sent in work worth 65,536 times the difficulty it was asked for, while the pool credited it for the difficulty it asked for: 65,536 times too little. The device did nothing wrong; the two sides simply count differently.

How it was fixed

The payout system now recognises such a device from its own shares: when every share it sends, at least twenty of them in half an hour, is worth over 32,768 times the difficulty it was given, its credit is multiplied by 65,536, and each block's credits are rescaled so the block still pays out exactly what it earned. The same correction applies to the hashrate shown on the site. Anything in between is left as the pool counted it and flagged for a person.

What we changed so it does not happen again

  • The recognition is automatic, for any device, from now on.
  • Every correction is re-checked by the hourly payout audit against the shares it rests on.
  • Reported to the BlockDAG developers, so that the pool can recognise this itself.
Resolved

The node stopped following the network after a short chain reorganisation

to · lasted 1 h 22 min

Incident 2026-09-25-1721

What happened

At 17:21 UTC the node that our pool and our public RPC run on stopped following the network. The network had replaced its last three blocks (an ordinary, short reorganisation), and the node could not go back three blocks to follow it. Without a current node the pool cannot hand out valid work, and the RPC cannot answer.

A restart did not help. We replaced the node's chain data with a copy from our other, healthy node; it was running again at 18:40 and had caught up with the network by 18:42.

Impact on miners

  • The pool did not answer for 75.2 min.
  • The pool answered, but not correctly, for 1.5 min.
  • The node was behind the network for 78.2 min.
  • The pool accepted no work from any miner for 42 min.
  • Payouts from the start until an hour after the end: 15 sent (482.868 BDAG), none failed.
  • Payouts waited for the node and went out normally at 18:42–18:43 UTC.
  • One block reward was counted while the node was stuck and then dropped by the network in the reorganisation. It had already been paid to the miners; it was covered from the pool operator's own funds, so no miner lost anything.

Measured on 28 September 2026, 05:37 UTC from the status checker's samples, the pool's share log and its own log, and the payout records.

Why it happened

The node runs in its default, space-saving mode, in which it no longer keeps the state of older blocks. It had already discarded the state of a block only three below its newest one, and needed exactly that state to follow the reorganisation. It cannot rebuild it from what is on disk, so it waited forever.

How it was fixed

We stopped the node and gave it a copy of the chain data from our other node, which had followed the network normally.

What we changed so it does not happen again

  • The restore is written down step by step, so the next one takes minutes.
  • Blocks the network later drops are no longer counted as found on the site.
  • Reported to the BlockDAG developers: a node should not lose the state it needs for a short reorganisation.

Times are UTC; your own time is added next to each one. For the state of the services right now, see Status. Page updated 28 September 2026, 08:04 UTC.