An online test for Benford’s Law

I’ve been fascinated with Benford’s Law ever since I read Mark Nigrini‘s “Digital analysis using Benford’s Law“. As part of the fun I’ve used it to check whether email subject length follows it. So I figured, why not make a check available online? And here it is:

You copy-paste your number series and the application displays a column chart of the occurences of the first digit and the theoretical occurencies expected.

The first 50 Fibonacci numbers and Benford's Law
The first 50 Fibonacci numbers and Benford’s Law

Use it and tell me what you think. Many thanks to Vaggelis Tripolitakis for finding the first bug. Be gentle with it as it runs as a free Google App Engine service. Depending on my free time and feedback I get, I may add the rest of the tests that Nigrini describes (second digit, last digit and first two digit distributions).

If you like it very much, you can buy me the kindle edition of the second edition of Nigrini’s book.


Benford’s Law and email subjects

The first book I ever bought from ISACA‘s bookstore, was Nigrini‘s book on Benford’s Law. Briefly stated the law says that in a series of numbers that occur while observing a phenomenon, numbers starting with 1 are more likely to occur than those starting with 2 which in turn are more likely to appear than those that start with 3 and so on up to numbers starting with 9.

P(n) = \log_{10} (1 + \frac{1}{n}), n = 1, ..., 9

The law stands for other bases too.

I’ve had discussions about Benford’s Law applicability on email data over at twitter with Martijn Grooten, but never run any tests. A few hours back I had an interesting discussion with Theodore which reminded me of the law and so I decided to see whether it stands on a number series related to email. The easiest test I could run was on the length of the Subject: lines. Bellow what follows is a graph of Benford’s distribution and actual data from 376916 mails that passed a certain mail server during last week:

Benford's Law vs. length of Subject: lines

It seems that the length of subject lines follow the pattern. For the sake of speed I have omitted from the computation non-latin subject lines, which means that I have to recompute whenever I find a timeslot longer than 15 minutes. But then again if I am to find such a slot, I think I will try to see whether the message body size also follows a Benfordian distribution. It may be more difficult to verify though because of different mail servers imposing different limits on the size of messages sent and received by them. Oh wait, Sotiris just did that! The rest of the tests mentioned in Nigrini’s book are also worth a try.

So what do your logs say about subject lines’ length and Benford’s Law? Do they follow the pattern? I’d be glad to see your answer in the comments section.

PS: I see that there is now a second edition of Nigrini’s book about to be published!