LOOKUP MAPS (hash, SQL) AND ACCESS LISTS EXPLAINED
==================================================
   April 2002, June 2002, November 2002, December 2002
   Mark Martinec <Mark.Martinec@ijs.si>

New with amavisd-new-20020630: SQL LOOKUPS
New with amavisd-new-20021116: Regular expression lookups


(applies to the semantics of amavisd.conf variables like:
   %virus_lovers,     %bypass_checks,
   @virus_lovers_acl, @bypass_checks_acl,
   $virus_lovers_re,  $bypass_checks_re,
   @local_domains,
   %mailto, ... )

NOTE:
  All lookups are performed with raw (rfc2821-unquoted
  and unbracketed) addresses as a key,
      i.e.:    Bob "Funny" Dude@example.com
      not:     "Bob \"Funny\" Dude"@example.com
      and not: <"Bob \"Funny\" Dude"@example.com>


Several configurable settings in amavisd are controlled through
the use of table (hash) lookups, access control lists (array),
Perl-regexp -based access control lists, or SQL lookups.
The subroutine that does all the lookups is:

  sub lookup ($@) {
      my($addr, lists...) = @_;

It perform a lookup for a key (usually a recipient e-mail address,
unless otherwise noted) against one or more lookup tables / maps.
The set of maps is hard-wired into program, and the order chosen
is: from specific to more general, and from faster to slower,
which is usually flexible enough. Thus the usual sequence of
lookup tables is: SQL, hash, ACL, regexp, constant.

The available types of lookup maps (and their result types) are:
- M   - a Perl hash table
- B   - a Perl array (list), functioning as access control list (ACL)
- B/M - a list of Perl regular expressions
- B   - a lookup into SQL database
- M   - a constant value (hardly worth calling a lookup at all,
        but useful as a final resort default value)

(match return value: B indicates boolean value (true/false, possibly undef),
                     M indicates multivalued result, e.g. some string,
                       usable as boolean as well;

If a match is found, the subroutine lookup() returns whatever the map
returns;  undef is returned if nothing matches (which for Perl is false
as well).


A CONSTANT

Specifying a Perl scalar as an argument to lookup() is a degenerate
case of a lookup table: it matches any key, and the value of the
scalar is returned as the match value, unless it is undef (not defined).
If the value is undef, the search continues with the next argument
(if any, usually not).

Specifying a scalar argument in a call to lookup() is useful as a
last-resort (catchall, default) value.


HASH LOOKUPS

For arguments to subroutine lookup() of type hash-ref, the argument
is passed to subroutine lookup_hash(), which does a lookup into
a Perl hash.

Hash lookups (e.g. for user+foo@sub.example.com) are performed in the
following order:
 - lookup for user+foo@sub.example.com
 - lookup for user@sub.example.com (only if $recipient_delimiter nonempty)
 - lookup for sub.example.com
 - lookup for .sub.example.com
 - lookup for .example.com
 - lookup for .com
 - lookup for .
 - lookup user+foo@
 - lookup user@  (only if $recipient_delimiter is nonempty)

The search sequence stops as soon as a match is found, and the value
of the matched entry determines the result.

The domain part is always matched case-insensitively, the localpart
is matched case-sensitively when $localpart_is_case_sensitive
is true (not case-sensitive by default).

The field value undef implies that the next lookup table (if there are more)
is to be tried. In plain words, undef means "this table does not know
the answer, try the next one". Further searching in this table
(for more general defaults) is terminated.

NOTE: a null reverse path e-mail address used by MTA for delivery status
notifications (DSN) has empty local part and empty domain. As far as the
lookup is concerned (which uses raw, i.e. non-quoted and non-bracketed
address form), this address is @, i.e. a single character "@".
The lookup_hash goes through the following sequence of keys: "@", "", "."
to find a match for null reverse path (double quotes added for clarity,
they are not part of the key).


ACL LOOKUPS

For arguments to subroutine lookup() of type array-ref, the argument
is passed to subroutine lookup_acl(), which does an access list lookup:

  sub lookup_acl($$) {
      my($addr, $acl_ref) = @_;

Domain name of the supplied address is compared with each member of the
access list in turn, the first match wins (terminates the search),
and its value decides whether the result is true (yes, permit, pass)
or false (no, deny, drop). Falling through without a match
produces undef. Search is case-insensitive.

Acl lookup is not aware of the address extensions and they are not
handled specially.

If a list member contains a '@', the full e-mail address is compared,
otherwise if a list member has a leading dot, the domain name part is
matched only, and the domain as well as its subdomains can match. If there
is no leading dot, the domain must match exactly (subdomains do not match).

The presence of character '!' prepended to the list member decides
whether the result will be true (without a '!') or false (with '!')
in case this list member matches.

Because search stops at the first match, it only makes sense
to place more specific patterns before the more general ones.

Although not a special case, it is good to remember that '.' always matches,
so a '.' would stop the search and return true, whereas '!.' would stop
the search and return false (0) (which may not be very useful, if ACL
lookup is the last lookup in a chain of lookup maps), because false (undef)
is also implied at the end of the list).


Examples:

given: @acl = qw( me.ac.uk !.ac.uk .uk )
  'u@me.ac.uk' matches me.ac.uk, returns true and search stops

given: @acl = qw( me.ac.uk !.ac.uk .uk )
  'u@you.ac.uk' matches .ac.uk, returns false (because of '!') and search stops

given: @acl = qw( me.ac.uk !.ac.uk .uk )
  'u@them.co.uk' matches .uk, returns true and search stops

given: @acl = qw( me.ac.uk !.ac.uk .uk )
  'u@some.com' does not match anything, falls through and returns false (undef)

given: @acl = qw( me.ac.uk !.ac.uk .uk !. )
  'u@some.com' similar to the previous, except it returns 0 instead of undef,
  which would only make a difference if this ACL is not the last argument
  in a call to lookup()

given: @acl = qw( me.ac.uk !.ac.uk .uk . )
  'u@some.com' matches catchall ".", and returns true

more complex example: @acl = qw(
  !The.Boss@dept1.xxx.com .dept1.xxx.com
  .dept2.xxx.com .dept3.xxx.com lab.dept4.xxx.com
  sub.xxx.com !.sub.xxx.com
  me.d.aaa.com him.d.aaa.com !.d.aaa.com .aaa.com
);


ACL FOR IP ADDRESSES

A special type of lookup is an IP-matching access list implemented by
lookup_ip_acl(). It performs a lookup for an IP address against
access control list of networks. It is used by amavisd to check
if the SMTP client (your MTA) is allowed to connect - the variable
is @inet_acl .

IP address is compared with each member of the access list in turn,
the first match wins (terminates the search), and its value decides
whether the result is true (yes, permit) or false (no, deny).
Falling through without a match produces false (undef).

The presence of character '!' prepended to the list member decides
whether the result will be true (without a '!') or false (with '!')
in case this list member matches and terminates the search.

Because search stops at the first match, it only makes sense
to place more specific patterns before the more general ones.

Network can be specified in classless notation n.n.n.n/k, or using
a mask n.n.n.n/m.m.m.m . Missing mask implies /32, i.e. a host address.

Although not a special case, it is good to remember that '0/0' always matches.

NOTE: IPv4 syntax is assumed, IPv6 is not supported.

Example
  given: @inet_acl = qw( !192.168.1.12 172.16.3.3 !172.16.3/255.255.255.0
                         10/8 172.16/12 192.168/16 );
matches rfc1918 private address space except host 192.168.1.12
and net 172.16.3/24 (but host 172.16.3.3 within 172.16.3/24 still matches)

More examples at amavisd.conf.


REGULAR EXPRESSION LOOKUPS

For arguments to subroutine lookup() of type Amavis::Lookup::RE
(objects), the object is passed to method lookup_re, which does a
lookup into a list of Perl regular expressions (regexp or RE for short).

The full unmodified key (e.g. an e-mail address) is always used,
so splitting to localpart and domain or lowercasing is NOT performed.
This means the routine is general enough to be useful for other RE tests,
such as looking for banned file names.

Each element of the list can be ref to a pair, or directly a regexp
('Regexp' object created by qr operator, or just a (less efficient)
string containing a regular expression). If it is a pair, the first
element is treated as a regexp, and the second provides a value in case
the regexp matches. If not a pair, the implied result of a match is 1.

The regular expression is taken as-is, no extra anchoring or setting
case insensitivity is done, so do use qr'(?i)^user@example\.com$',
and not a sloppy qr'user@example.com', which is doubly wrong in semantics
but correct in syntax, and can easily backfire. Also, if qr operator
is used with a delimiter other than ' , make sure to quote the @ .

Example (equivalent to the example in lookup_acl):
   $acl_re = new_re->new(
                 qr'@me\.ac\.uk$'i, [qr'[@.]ac\.uk$'i=>0], qr'\.uk$'i );
   ($r,$k) = $acl_re->lookup_re('user@me.ac.uk');
or $r = lookup('user@me.ac.uk', $acl_re);

'user@me.ac.uk'   matches me.ac.uk, returns true and search stops
'user@you.ac.uk'  matches .ac.uk, returns false (because of =>0) and search stops
'user@them.co.uk' matches .uk, returns true and search stops
'user@some.com'   does not match anything, falls through and returns false (undef)

NOTE: new_re is a synonym for the internal subroutine Amavis::Lookup::RE::new

See Perl documentation (or Google the Internet) for the description
of regular expressions. They are just enhanced version of Posix regular
expressions, i.e. what your egrep, awk and sed thrive on. Here are the
most important constructs (simplified):

    .   Match any character                 inter..t
    |   Alternation                         alfa|beta|gamma
    ()  Grouping                            (alfa|beta)|gamma
    []  Character class                     [Aa]lfa[0-9]
    ^   Match the beginning of the string   ^MakeMoney
    $   Match the end of the string         com$
    \   Quote the next metacharacter        \.com$
    most other characters just match themselves

quantifiers are placed after the pattern and modify its meaning
from 'match itself exactly once' into:
    *      Match 0 or more times            ^alfa.*omega$
    +      Match 1 or more times            alfa +beta
    ?      Match 1 or 0 times               (first)?aid


SQL LOOKUPS

For arguments to subroutine lookup() of type Amavis::Lookup::SQLfield
(objects), the object is passed to method lookup_sql_field, which does
a lookup into a SQL table by using Perl module DBI.

The amavisd.conf variable @lookup_sql_dsn controls access to the
SQL server (dsn = data source name). If @lookup_sql_dsn is empty,
no attempts to use SQL will be made, and no code to use DBI will be
loaded or compiled.

@lookup_sql_dsn variable is a list of triples: (dsn,user,passw).
More than one triple may be specified to list multiple (backup)
SQL servers - the first that responds will be used.

@lookup_sql_dsn =
   ( ['DBI:mysql:mail:host1', 'some-username1', 'some-password1'],
     ['DBI:mysql:mail:host2', 'some-username2', 'some-password2'] );

See man page for the Perl module DBI for syntax of the first argument.

SQL 'select' requests all available fields from the specified
tables, and the result is cached (just for this mail message processing).
Individual fields can be extracted one at a time from this cache
very quickly, so there is no penalty in using several calls to lookup
for different fields (for the same key) in different parts of the program.

lookup_sql() performs a lookup for an e-mail address against a SQL map.
If a match is found it returns whatever the map returns (a reference
to a hash containing values of requested fields), otherwise returns undef.
A match aborts further fetching sequence.

lookup_sql_field() also performs a lookup for an e-mail address against
a SQL map. It first calls lookup_sql() if it hasn't been called yet for
this key, but instead of returning all available fields, it returns only
a value of one particular table field. This is the subroutine that gets
called from lookup() for arguments of type Amavis::Lookup::SQLfield
(objects).

The field value NULL is translated to Perl undef, which according
to lookup rules implies that the next lookup table (if there are more)
is to be tried. In plain words, NULL means "this table does not know
the answer, try the next one". Further searching in this table
(e.g. for more general defaults) is terminated.

A special case is a lookup for a nonexistent field name (e.g. the key
matches, but there are no other fields in the table except the key field).
This results in a match with a value 1. Such an interpretation allows
to check just the presence of a key, and avoids a need to provide a field
with a value of true for all the records in the database.

Boolean fields are usually declared as single character (instead of integer)
to minimize storage. The characters N,n,F,f,0 and space represent false (0),
any other character represents true. Leading and trailing whitespace is
ignored. It is customary to use Y for true and N for false.

SQL lookups (e.g. for user+foo@example.com) are performed in order
which is normally requested by 'ORDER BY' in the SELECT statement;
otherwise the order is unspecified, which is only useful if just specific
entries exist in a database (full address always, not only domain part
or mailbox part).

The following order (implemented by sorting on the 'priority' field,
zero means low priority) is recommended, to follow the same
specific-to-general principle as in other lookup tables:

 - lookup for user+foo@example.com
 - lookup for user@example.com (only if $recipient_delimiter nonempty)
 - lookup for user+foo (only if domain part is local)
 - lookup for user     (only local; only if $recipient_delimiter is nonempty)
 - lookup for @example.com
 - lookup for @.       (catchall)

NOTE:
 this is different from hash and ACL lookups in three important aspects:
   - naked key (without '@') implies mailbox name, not domain name;
   - subdomains are not looked at, only full domain names are matched;
   - the naked mailbox name lookups ('user') are only performed
     when the domain part matches the '@local_domains' ACL.

The domain part is always lowercased when constructing a key,
the localpart is not lowercased when $localpart_is_case_sensitive is true.

Table names and field names are hard-wired in the routine
prepare_sql_queries(). Please adjust it to will. Field names should
be unique even without the table prefix.

Here is an example database that I used with MySQL to test the code.


CREATE TABLE users (
  id         int(10) unsigned DEFAULT '0' NOT NULL auto_increment,
  priority   int(10)          DEFAULT '7' NOT NULL,  # 0 is low priority
  policy_id  int(10) unsigned DEFAULT '1' NOT NULL,
  email      varchar(256) NOT NULL,
  fullname   varchar(256),
  PRIMARY KEY (id),
  KEY email (email)
);

CREATE TABLE policy (
  id              int(10) unsigned DEFAULT '0' NOT NULL auto_increment,
  policy_name     varchar(32),

  bypass_virus_checks char(1),  # Y/N
  virus_lover         char(1),  # Y/N
  bypass_spam_checks  char(1),  # Y/N
  spam_tag_level  float,  # higher score declares spam
                          #   (inserts spam info headers)
  spam_kill_level float,  # higher score activates evasive actions, e.g.
                          # reject/drop, quarantine, ...
                          #   (subject to final_spam_destiny setting)
  PRIMARY KEY (id)
);

INSERT INTO users VALUES ( 1, 9, 5, 'user1+foo@y.example.com','Name1 Surname1');
INSERT INTO users VALUES ( 2, 7, 5, 'user1@y.example.com', 'Name1 Surname1');
INSERT INTO users VALUES ( 3, 7, 2, 'user2@y.example.com', 'Name2 Surname2');
INSERT INTO users VALUES ( 4, 7, 7, 'user3@z.example.com', 'Name3 Surname3');
INSERT INTO users VALUES ( 5, 7, 7, 'user4@example.com',   'Name4 Surname4');
INSERT INTO users VALUES ( 6, 7, 1, 'user5@example.com',   'Name5 Surname5');
INSERT INTO users VALUES ( 7, 5, 0, '@sub1.example.com', NULL);
INSERT INTO users VALUES ( 8, 5, 7, '@sub2.example.com', NULL);
INSERT INTO users VALUES ( 9, 5, 5, '@example.com',      NULL);
INSERT INTO users VALUES (10, 3, 8, 'userA', 'NameA SurnameA anywhere');
INSERT INTO users VALUES (11, 3, 9, 'userB', 'NameB SurnameB');
INSERT INTO users VALUES (12, 3,10, 'userC', 'NameC SurnameC');
INSERT INTO users VALUES (13, 3,11, 'userD', 'NameD SurnameD');
# INSERT INTO users VALUES (14, 0, 5, '@.',              NULL);  # catchall

INSERT INTO policy VALUES (1, 'Non-paying',     'Y','N',  'Y',   7, 10);
INSERT INTO policy VALUES (2, 'Uncensored',     'N','Y',  'Y',   5, 999);
INSERT INTO policy VALUES (3, 'Wants all spam', 'N','N',  'Y',   5, 999);
INSERT INTO policy VALUES (4, 'Wants viruses',  'N','Y',  'Y',   5, 6.8);
INSERT INTO policy VALUES (5, 'Normal',         'N','N',  'N', 4.9, 5.4);
INSERT INTO policy VALUES (6, 'Trigger happy',  'N','N',  'N',   3, 5);
INSERT INTO policy VALUES (7, 'Permissive',     'N','N',  'N',   7, 10);
INSERT INTO policy VALUES (8, '6.5/7.8',        'N','N',  'N', 6.5, 7.8);
INSERT INTO policy VALUES (9, 'userB',          'N','N',  'N',   3, 6.3);
INSERT INTO policy VALUES (10,'userC',          'N','N',  'N',   0, 6.5);
INSERT INTO policy VALUES (11,'userD',          'N','Y',  'N',   0, 6);
