$Id: index-e.html,v 1.5 2004/08/13 16:59:17 nabeken Exp $

bsfilter / bayesian spam filter

japanses english

0. What is bsfilter ?

• a filter which distinguishes spam and non-spam(called "clean" in this page) mail
• support mails written in japanese language
• written in Ruby
• support 3 methods for access
  • traditional unix-style filter. study and judge local files or pipe
  • IMAP. study and judge mails in an IMAP server
  • POP proxy. run between POP server and MUA
• basic concepts come from A Plan for Spam, Better Bayesian Filtering, Spam Detection
• distributed under GPL

1. Contents

• 1. Contents
• 2. Download
• 3. How bsfilter works?
• 4. Lets' get started
• 5. Help
• 6. Usage
• 7. Usage(IMAP)
• 8. Usage(POP proxy)

2. Download

• bsfilter-1.0.4.tgz
• CVS access

3. How bsfilter works?

3.1. using spam proability of each token

3.2. need to prepare

4. Let's get started

preprare

It is necessary to prepare databases before filtering

1. count tokens in clean mails

% bsfilter --add-clean ~/Mail/inbox/*

2. count tokens in spam

% bsfilter --add-spam ~/Mail/spam/*

3. calculate spam probability for each token

% bsfilter --update

filtering

example: specify filenames for filtering as command line argumetns. spam probability numbers(between 0 and 1) are displayed.

% bsfilter ~/Mail/inbox/1
combined probability /home/nabeken/Mail/inbox/1 1 0.012701

example: feed mail for filtering through stdin pipe. exit status is 0 in case of spam

~% bsfilter < ~/Mail/inbox/1 ; echo $status
1
~% bsfilter < ~/Mail/spam/1 ; echo $status
0

procmail sample recipe 1: move spams to spam folder using exit status

:0 HB:
* ? bsfilter -a
spam/.

procmail sample recipe 1: add X-Spam-Flag:, X-Spam-Probability: headers and move spams to black or gray folder based on spam probability at X-Spam-Probability: header

:0 fw
| /home/nabeken/bin/bsfilter --pipe --insert-flag --insert-probability

:0
* ^X-Spam-Probability: *(1|0\.[89])
black/.

:0
* ^X-Spam-Probability: *0\.[67]
gray/.

5. Help

bsfilter forum

6. Usage

formats of command line

there are 2 formats.

1. bsfilter [options] [commands] < MAIL
2. bsfilter [options] [commands] MAIL ...

There are maintenance mode and filtering mode.

• When commands are spcecified, bsfilter is under maintenance mode. It updates databases, but doesn't judge mails.
• When commands aren't specified, bsfilter is under filtering mode. It judges mails, but doen't update databases.
• There is an exception. When "auto-update" is specified in filtering mode, bsfilter updates databases and judge mails also.

Use format 1 in filtering mode in order to feed mail from stdin and judge it. Exit status becomes 0 in case of spam. If bsfilter invoked by MDA(procmailrc and etc), this style are used.

Use format 2 in filtering mode in order to specify multiple mails at command line and judge at once. Results are displayed at stdout.

type "bsfilter --help" to see all commands and options.

7. Usage(IMAP)

bsfilter is able to communicate server by IMAP and study or judge mails stored in it. bsfilter is able to insert headers or move mails to a specified folder

example

sample of bsfilter.conf

imap-server server.example.com
imap-auth login
imap-user hanako
imap-password open_sesame

judge mails without X-Spam-Flag in "inbox", insert X-Spam-Probability header and move spams into "inbox.spam"

% bsfilter --imap --imap-fetch-unflagged --insert-flag --insert-probability --imap-folder-spam inbox.spam inbox

8. Usage(POP proxy)

bsfilter ia able to work as POP proxy and judge mails and insert headers on a path from POP server to MUA.

example

sample of bsfilter.conf

pop-server server.example.com
pop-proxy-port 10110
pop-user alice
insert-flag
insert-probability

invoke bsfilter like the following and change preference of MUA to access port 10110 of a host which bsfilter runs. When MUA receives a mail, X-Spam-Flag, X-Spam-Probability are added into the mail and databases are updated.

% bsfilter --pop --auto-update