Scout on Substrate: Second precision report

February, 2025.

Introduction

As part of the fourth milestone of our initiative to develop the Scout static analyzer for Substrate, we aimed to evaluate the precision of Scout’s detectors. To achieve this, we ran Scout on nine pallets from randomly selected Substrate projects with active support. The results were analyzed within a time-constrained sprint of five days by one of CoinFabrik’s senior security auditors to assess the quality of Scout’s output in terms of precision. This assessment provides a snapshot of the tool’s current performance and helps identify areas for improving the detectors.

Methodology

This was an exploratory study, meaning that the issues that would be reported by Scout were completely unknown to us. This allowed us to work on a non-biased set of results, running the tool on real projects and mimicking as much as possible the conditions that any Scout user will encounter. Given this, the analysis was restricted to the detectors triggered by Scout.

Results

Out of the 17 detectors available by the time of this analysis -Scout Audit version 0.3.4- 7 detectors were triggered and 92 different issues were found. After a thorough evaluation of each of the results, our security auditor concluded that 61 of them were true positives, which represents a precision rate of 66.3%.

Detector	Findings	True positives	False positives
assert-violation	5	5	0
avoid-dispatch-error-other	3	3	0
integer-overflow-or-underflow	12	1	11
invalid-extrinsic-weight	11	0	11
known-vulnerabilities	9	9	0
overflow-check	7	7	0
saturating-arithmetic	41	32	9
unsafe-expect	4	4	0
Total	92	61	31

Detectors refinement

After our first Scout on Substrate precision evaluation, and after analyzing the issues while working on this report, Scout’s development team concluded that some actions could be taken to improve the quality of the detectors with higher false positives rate.

Below is the description of the modifications implemented on 4 detectors.

Integer overflow or underflow

Improved safety-context system for detecting potential overflow in subtraction and division. This system analyzes if statements and their conditions to determine whether a subtraction or division operation is safe before execution. For example, the following code would have previously thrown a warning:

if a >= b {
    let result = a - b;
}

But subtracting b from a is actually safe.

Saturating arithmetic

This detector was triggering false positives in code generated by Substrate macros.

Inconsistent extrinsic weight name

Improved false positive rates in cases where an extrinsic defines its weight inline with an expression, as well as when a single weight function is used in multiple extrinsics with different arguments. For example of the former,

#[pallet::weight(<T as Config>::WeightInfo::bar(2) * 3)]
pub fn foo(/*...*/) -> DispatchResult

And of the latter,

#[pallet::weight(<T as Config>::WeightInfo::foo(FOO_VALUE))]
pub fn foo(/*...*/) -> DispatchResult

//...

#[pallet::weight(<T as Config>::WeightInfo::foo(BAR_VALUE))]
pub fn bar(/*...*/) -> DispatchResult

Unsafe expect

The detector now includes an arithmetic safety checker for expect methods used in arithmetic operations. It analyzes if statements to determine whether the conditions ensure a safe subtraction or division. Previously, the detector would issue a warning in the following scenario:

if a >= b {
    let result = a.checked_sub(b).expect("Subtraction underflow should be prevented by the condition");
}

Now, there is no warning when subtraction happens on known operations.