GHSA-6wxf-7784-62fp

Horcrux Incident Disclosure: Possible Double-Sign

Summary

On March 6, 2025, a Horcrux user (01node) experienced a double-signing incident on the Osmosis network, resulting in a 5% slash penalty (approximately 75,000 OSMO or $20,000 USD). After thorough investigation, we have identified a race condition in Horcrux's signature state handling as the root cause. This vulnerability was introduced in July 2023 as part of PR #169 and affects all Horcrux versions from v3.1.0 through v3.3.1. A fix has been developed and is being deployed immediately.

Probability

The bug has an extremely low probability of occurrence, affecting one validator out of hundreds that have been using the affected software versions to validate over the past few years. In the added tests, the probability on typical hardware is in the range of 1 in 1 billion per signed vote due to the root cause needing two independent events to occur within approximately the same microsecond duration.

Severity

While rare, it is of high severity, as the double-sign (tombstone) slash on most Cosmos chains is typically 5% to the validator’s self stake and the stake of delegators that is delegated to the validator. The bug is not exploitable, so the urgency to apply this patch is purely around avoiding the race condition to remove tombstone risk.

Impact

• One known validator (01node) was affected
• The validator and its delegators were slashed 5% of their stake delegated with the validator (~75,000 OSMO, ~$20,000 USD)
• The incident occurred at Osmosis block height 30968345

Technical Details

Root Cause

The issue was a race condition in the signature state handling code. When two sign requests arrive nearly simultaneously for the same Height-Round-Step (HRS), a split read-write lock pattern allowed both to proceed when they should have been serialized. This vulnerability allowed Horcrux to sign both a "yes" vote (non-nil BlockID) and a "no" vote (nil BlockID) for the same block, which constitutes a double sign violation.

In the affected code:

1. The HRSKey() method used a read lock to check the current signature state
2. The cacheAndMarshal() method used a separate write lock to update the state

Because the usage of these operations unlocked in the middle to perform checks rather than occurring under a single lock, they were not atomic. Very rarely, two concurrent signature requests could both pass the initial safety check before either one updated the state, leading to a double signature.

Evidence from logs shows two different signatures were produced within 1.5 milliseconds of each other:

DuplicateVoteEvidence{
  VoteA: Vote{69:03C016AB7EC3 30968345/00/SIGNED_MSG_TYPE_PREVOTE(Prevote) 000000000000 BEEB4E1F5432 000000000000 @ 2025-03-06T21:35:48.759070033Z}, 
  VoteB: Vote{69:03C016AB7EC3 30968345/00/SIGNED_MSG_TYPE_PREVOTE(Prevote) 817EB28D720F FAE04CB3CF89 000000000000 @ 2025-03-06T21:35:48.760639069Z}
}

This matches the signatures reported in the Horcrux cosigner logs:

• Cosigner-1: sig=FAE04CB3CF89 ts="2025-03-06 21:35:48.760639069 +0000 UTC"
• Cosigner-2: sig=BEEB4E1F5432 ts="2025-03-06 21:35:48.759070033 +0000 UTC"

The race condition allowed both signatures to be validated and broadcast, resulting in the double sign.

Fix

The fix implements a single mutex lock that covers both the reading of the current signature state and the subsequent writing of any updates:

func (signState *SignState) Save(
	ssc SignStateConsensus,
	pendingDiskWG *sync.WaitGroup,
) error {
	signState.mu.Lock()
	if err := signState.getErrorIfLessOrEqual(ssc.Height, ssc.Round, ssc.Step); err != nil {
		signState.mu.Unlock()
		return err
	}

	// HRS is greater than existing state, move forward with caching and saving.
	signState.cache[ssc.HRSKey()] = ssc
	
	// Update state
	// ...
	
	signStateCopy := signState.copy()
…

Horcrux Incident Disclosure: Possible Double-Sign

Summary

On March 6, 2025, a Horcrux user (01node) experienced a double-signing incident on the Osmosis network, resulting in a 5% slash penalty (approximately 75,000 OSMO or $20,000 USD). After thorough investigation, we have identified a race condition in Horcrux's signature state handling as the root cause. This vulnerability was introduced in July 2023 as part of PR #169 and affects all Horcrux versions from v3.1.0 through v3.3.1. A fix has been developed and is being deployed immediately.

Probability

The bug has an extremely low probability of occurrence, affecting one validator out of hundreds that have been using the affected software versions to validate over the past few years. In the added tests, the probability on typical hardware is in the range of 1 in 1 billion per signed vote due to the root cause needing two independent events to occur within approximately the same microsecond duration.

Severity

While rare, it is of high severity, as the double-sign (tombstone) slash on most Cosmos chains is typically 5% to the validator’s self stake and the stake of delegators that is delegated to the validator. The bug is not exploitable, so the urgency to apply this patch is purely around avoiding the race condition to remove tombstone risk.

Impact

• One known validator (01node) was affected
• The validator and its delegators were slashed 5% of their stake delegated with the validator (~75,000 OSMO, ~$20,000 USD)
• The incident occurred at Osmosis block height 30968345

Technical Details

Root Cause

The issue was a race condition in the signature state handling code. When two sign requests arrive nearly simultaneously for the same Height-Round-Step (HRS), a split read-write lock pattern allowed both to proceed when they should have been serialized. This vulnerability allowed Horcrux to sign both a "yes" vote (non-nil BlockID) and a "no" vote (nil BlockID) for the same block, which constitutes a double sign violation.

In the affected code:

1. The HRSKey() method used a read lock to check the current signature state
2. The cacheAndMarshal() method used a separate write lock to update the state

Because the usage of these operations unlocked in the middle to perform checks rather than occurring under a single lock, they were not atomic. Very rarely, two concurrent signature requests could both pass the initial safety check before either one updated the state, leading to a double signature.

Evidence from logs shows two different signatures were produced within 1.5 milliseconds of each other:

DuplicateVoteEvidence{
  VoteA: Vote{69:03C016AB7EC3 30968345/00/SIGNED_MSG_TYPE_PREVOTE(Prevote) 000000000000 BEEB4E1F5432 000000000000 @ 2025-03-06T21:35:48.759070033Z}, 
  VoteB: Vote{69:03C016AB7EC3 30968345/00/SIGNED_MSG_TYPE_PREVOTE(Prevote) 817EB28D720F FAE04CB3CF89 000000000000 @ 2025-03-06T21:35:48.760639069Z}
}

This matches the signatures reported in the Horcrux cosigner logs:

• Cosigner-1: sig=FAE04CB3CF89 ts="2025-03-06 21:35:48.760639069 +0000 UTC"
• Cosigner-2: sig=BEEB4E1F5432 ts="2025-03-06 21:35:48.759070033 +0000 UTC"

The race condition allowed both signatures to be validated and broadcast, resulting in the double sign.

Fix

The fix implements a single mutex lock that covers both the reading of the current signature state and the subsequent writing of any updates:

func (signState *SignState) Save(
	ssc SignStateConsensus,
	pendingDiskWG *sync.WaitGroup,
) error {
	signState.mu.Lock()
	if err := signState.getErrorIfLessOrEqual(ssc.Height, ssc.Round, ssc.Step); err != nil {
		signState.mu.Unlock()
		return err
	}

	// HRS is greater than existing state, move forward with caching and saving.
	signState.cache[ssc.HRSKey()] = ssc
	
	// Update state
	// ...
	
	signStateCopy := signState.copy()
…

Horcrux Double Sign Possibility in github.com/strangelove-ventures/horcrux