I compared graph neural networks with a random-forest baseline for detecting illicit Bitcoin transactions on the Elliptic dataset. I used a time-based split and evaluated illicit-class F1, including a check of whether GNNExplainer’s selected edges could reproduce model predictions.
On the test set (time steps 35-49), the illicit-class F1 was 0.79 for the random forest without graph information.
On the test set (time steps 35-49), the illicit-class F1 was 0.53 for GCN.
On the test set (time steps 35-49), the illicit-class F1 was 0.48 for GAT.
Both GNNs also dropped sharply from validation to test: GCN from 0.80 to 0.53, and GAT from 0.77 to 0.48.
The illicit share of labelled transactions changed from 17% in validation to 6.5% in test, which may be part of the reason for the GNN performance drop.
I checked whether GNNExplainer’s top-10 edges were sufficient to reproduce each prediction. They were for 84% of 50 nodes.
A random set of 10 edges worked for 84% of 50 nodes.
I concluded that the explanations did no better than random edges.
I asked: when a simple baseline beats a newer model, what do you check first before trusting the result?
I said I would value opinions from people who have worked on fraud detection or graph models.