FuzzyOCR plugin 2.3b released

The latest release of the FuzzyOCR plugin (2.3b) is out, and
I've
updated the wiki accordingly with new installation and
configuration
instructions: <https://secure.renaissoft.com/maia/wiki/FuzzyOCR23>.

There are a number of key improvements in this version of
the plugin
that make it worth the upgrade, including:

* Handling of interlaced and animated GIFs

As the anti-spam community has come to embrace OCR
technologies,
spammers have been working on ways to confuse OCR engines,
using
interlaced images and animated GIFs.  With interlaced
images, the data
is ordered differently (all the odd-numbered pixel rows
together, all
the even-numbered pixel rows together), so tools that
aren't able to
detect interlaced images and reconstruct them properly would
fail to
load them.  Animated GIFs are also becoming more common,
since tools
that don't know how to handle them will only see the first
frame of the
animation--so spammers simply include a blank first frame
that lasts a
fraction of a second, followed by a long second frame that
contains the
spam message.  This version of the FuzzyOCR plugin uses
tools that can
properly detect and handle interlaced and animated GIFs,
unpacking the
individual frames as necessary.

* Word list in a separate file

In previous versions of the plugin, the list of target words
was stored
in the FuzzyOcr.cf file.  Now they're stored in a separate
file
(FuzzyOcr.words) that won't be overwritten during plugin
upgrades.

* Hashing database cache for previously-scanned images

On the theory that if you see one instance of a given spam
image, you're
likely to see multiple copies of it, this version of the
plugin
maintains a local database to cache scan information about
the images it
has seen.  It's not an MD5 hash, it's a collection of
image property
data that aims to be an invariant "signature"
for a given image, even if
other copies aren't exact pixel-perfect matches.  The
image's score is
cached as well, so that if it is seen again in the future,
the plugin
won't need to run the OCR engine on it again.

* Ability to use multiple, more configurable scan sets

This is perhaps the most powerful addition, as it's a
feature that lets
you configure the plugin to run the OCR scanner on the image
multiple
times with different resolution and tolerance settings, in
order to
catch a wider range of image spam (at the cost of extra
processing time,
naturally; you can still configure the plugin to do just one
pass, if
you prefer).

The trouble with OCR is that a single pass over the image
only tells you
what's visible at a single, fixed resolution, and if the
image is
crafted with a very different resolution the OCR scan may
see a bunch of
dots instead of letters, or misinterpret "noise"
dots as parts of
letters.  If you're only going to do one pass over the
image, you've got
to choose a "compromise" resolution--one that
will read text in most
cases, but will fail for edge cases.

By making a second pass over the image at a different
resolution though,
you can get the best of both worlds by comparing the results
from both
scans and choosing the one with the best result.  You could
even make a
third pass with yet another set of scanner options if you
wanted to test
for even more challenging conditions (e.g. white text on a
dark
background, text in multiple colours, etc.).

These "scan sets" are also highly
configurable--you can construct your
own tool-chains by piping the image through a series of
utilities, as
long as the image begins as a PNM and ends the chain as
input to GOCR.
Thus you can do things like normalize, resize, greyscale,
rotate, etc.
as you see fit, using the Netpbm, Libungif, or ImageMagick
tools to
prepare the image in whatever way you want before it gets
OCR'ed.  I
expect that various "recipes" will be shared
eventually, as people
experiment with tool-chains and scanner settings that catch
particular
image strains.

Upgrade notes:

(1) SpamAssassin 3.1.4 is preferred, due to some
optimizations that make
handling animated GIFs somewhat easier.  The plugin will
still work with
versions as early as 3.1.0 using some (less-efficient)
internal
workarounds, but you should really be using the latest
SpamAssassin in
any case, if only for the newer rules and bug fixes anyway,
so consider
this your excuse to upgrade 

(2) This version of the plugin requires the ImageMagick
suite,
specifically for the "convert" and
"identify" utilities, used to unpack
the animated GIFs.

(3) There's a small patch for the libungif utility
"giftext" which
hardens it against a segfault exploit.  This means you'll
need to get
the libungif sources and patch them and build them, rather
than just
using the binary packages from your favourite repository. 
Sad, but
necessary.

-- 
Robert LeBlanc <rjlrenaissoft.com>
Renaissoft, Inc.
Maia Mailguard <http://www.maiamail
guard.com/>

_______________________________________________
Maia-users mailing list
Maia-usersrenaissoft.com
http://www.renaissoft.com/mailman/listinfo/maia-users

FuzzyOCR plugin 2.3b released

Thread: FuzzyOCR plugin 2.3b released