=head1 NAME

ovsqlite - SQLite-based overview storage method for INN

=head1 DESCRIPTION

This method uses SQLite to store overview data.  It requires version
3.8.2 or later of the SQLite library (3.20.0+ recommended).

SQLite source, documentation, etc. are available at
L<https://www.sqlite.org/>.  Ones of the stated goals of the SQLite
file format are long-term stability and compatibility, which make that
storage method a perfect choice to store overview data.

Only one protocol version of the ovsqlite storage method currently
exists, implemented since S<INN 2.7.0>.

This method comes with a support utility (B<ovsqlite-util>) and a Perl
module (C<INN::ovsqlite_client>) which permit accessing and manipulating
overview information.  See their manual pages for more details about their
possibilities.

=head1 INSTALLATION

The configure script will automatically enable ovsqlite support if it finds
the SQLite library.  If the library isn't installed in a standard
location, you may have to specify the B<--with-sqlite3> option to help
configure find it.  For complicated cases, you can use separate
B<--with-sqlite3-include> and B<--with-sqlite3-lib> options.  Finally, if
you I<don't> want ovsqlite support even when your system has the SQLite
library, you can use the B<--without-sqlite3> option.

If you have a recent Linux installation, SQLite is most likely already
installed.  You may have to install a separate package with a name similar
to C<libsqlite3-dev> to get the required header files.

Plan on needing at least S<0.93 KB> for every article in your spool
(not counting crossposts).  So, if you have 5 million articles, you'll
need at least S<4.65 GB> of disk space for ovsqlite.  With compression
enabled, this estimate changes to S<0.4 KB> per article, so you'll
need at least S<2 GB> of disk space for 5 million articles.  Plus,
you'll need additional space for transaction logs (a few MB, or up to
a few hundred MB temporarily with I<walmode> enabled).

=head1 CONFIGURATION

To select ovsqlite as your overview method, set the I<ovmethod> parameter
in F<inn.conf> to C<ovsqlite>.  The database file will be created in
the directory specified by the I<pathoverview> parameter in F<inn.conf>.
Restart INN to take the change into account (after having rebuilt your
existing overview with B<makehistory>, if needed).

These additional parameters are read from the F<ovsqlite.conf>
configuration file:

=over 4

=item I<cachesize>

The SQLite in-memory page cache size in kilobytes for the B<ovsqlite-server>
write daemon.  The default value is left up to the SQLite library and seems
to be stable at S<2000 KB>.

=item I<compress>

If INN was built with zlib support and this parameter is true, ovsqlite
will compress overview records whenever this saves space.  This parameter
is consulted only when creating a new database.  Enabling compression
saves about S<55 %> of disk space on standard overview data.  The default
value is false.

=item I<mmapsize>

The maximum number of bytes of the database file that SQLite will
memory-map.  When set, SQLite reads data directly from the operating
system's page cache via mmap instead of copying it into its own page
cache, which can significantly improve read performance for large
databases.  When I<walmode> is enabled, B<nnrpd> reader processes honor
this setting too, which additionally lets them share physical memory
pages for the same database regions through the operating system's page
cache.  This is of little benefit on ZFS or others without a unified
buffer cache, where the mapped pages would simply be cached twice.
A good starting value is the size of the database file
or the amount of available memory, whichever is smaller.  For example,
a value of C<4294967296> allows up to S<4 GB> to be memory-mapped.
The default value of C<0> disables memory-mapped I/O.

=item I<walmode>

If this parameter is true, ovsqlite will use SQLite's WAL (Write-Ahead
Logging) journal mode instead of the default PERSIST mode.  WAL mode
allows concurrent readers and writers without blocking, which significantly
improves read performance when multiple B<nnrpd> processes access the
overview database simultaneously.  It also sets C<synchronous> to
C<NORMAL>, which is safe with WAL and reduces unnecessary fsync overhead.

On shutdown, the WAL is checkpointed back into the main database file
and truncated.

B<Note:> WAL mode requires shared-memory support and does not work on
network filesystems such as NFS.  Do not enable this parameter if
I<pathoverview> is on a network filesystem.

The default value is false.

=item I<readercachesize>

The SQLite in-memory page cache size in kilobytes for each B<nnrpd> reader
process.  When I<walmode> is enabled, B<nnrpd> processes open the overview
database directly for read-only access, bypassing B<ovsqlite-server>.
This eliminates the IPC overhead and allows reads to scale with the number
of reader processes.  Each B<nnrpd> process gets its own cache, so this value
should typically be smaller or equal to I<cachesize>.  The operating system's
file cache (or ZFS ARC) serves as a shared second-level cache across all
readers.

Direct reader mode is only activated when I<walmode> is enabled.  Without
WAL, writers take exclusive locks that would block direct readers during
commits, so the server path is used instead.  The default value
is C<2000> (about S<2 MB>).

=item I<walcheckpointthreshold>

The WAL page count at which B<ovsqlite-server> forces a checkpoint.
When the WAL grows beyond this many pages, the server performs a
TRUNCATE checkpoint during its next idle period, waiting up to
10 seconds for readers to finish their current queries before flushing
WAL content back to the database file.  This prevents unbounded WAL
growth and is no worse than the non-WAL path where every commit takes
an exclusive lock.  If readers do not finish within the timeout, the
checkpoint is skipped and retried on the next idle cycle.

This parameter is only effective when I<walmode> is enabled.  The default
value is C<1000>.

=item I<pagesize>

The SQLite database page size in bytes.  Must be a power of 2, minimum 512,
maximum 65536.  Appropriate values include the virtual memory page size and
the filesystem allocation block size.  This parameter is consulted only
when creating a new database.  The default value is left up to the SQLite
library and varies between versions.

=item I<transrowlimit>

The maximum number of article rows that can be inserted or deleted in a
single SQL transaction.  The default value is C<10000>.

=item I<transtimelimit>

The maximum SQL transaction lifetime in seconds.  The default value is
C<10>.

=back

A transaction occurs every I<transrowlimit> articles or I<transtimelimit>
seconds, whichever is smaller.  You are encouraged to keep the default
value for row limits and, instead, adjust the time limit according to
how many articles your news server usually accepts per second during
normal operation (you can find statistics about incoming articles in
your daily Usenet reports).  Inserting or deleting a database row within
a transaction is very fast whereas committing a transaction is slow,
especially on rotating storage.  Setting transaction limits too low
leads to poor performance.  When rebuilding overview data, it may be
worth temporarily raising these values on systems with slow CPU or rotating
storage (for instance, multiply by 10 the default values).  Performance won't
change much on fast systems.

=head1 RUNNING

Write operations (article ingest, expiration) go through the
B<ovsqlite-server> daemon.  For ordinary operation, B<rc.news> will start
and stop it automatically.  If you want to touch the overview database while
B<innd> isn't running, you'll have to start B<ovsqlite-server> manually
first.  See ovsqlite-server(8).

When I<walmode> is enabled, B<nnrpd> reader processes open the overview
database directly with read-only access, bypassing B<ovsqlite-server>
for read operations.  This eliminates the IPC round-trip and server
serialization bottleneck, allowing read performance to scale with the
number of concurrent readers.

=head1 VACUUMING

ovsqlite does B<not> vacuum the database automatically, and that is
intentional.

When overview rows are deleted (for example by B<expireover>), SQLite does
not usually shrink F<ovsqlite.db>.  It places free pages on a freelist and
reuses them for later inserts.  For a news spool that receives and expires
at a steady rate, the overview database therefore grows to a high-water mark
and then stays there: freelist pages absorb the churn of B<expireover> and
intake without constantly extending or truncating the file.  That behaviour
is efficient.  Keeping the high-water mark avoids repeated filesystem
allocations and means B<expireover> does not pay the cost of a full rewrite on
every run.

A full C<VACUUM> rewrites the entire database into a compact copy and
replaces the original.  It reclaims freelist space (and can apply a new page
size if C<PRAGMA page_size> was set beforehand), but it needs free disk space
on the order of the current file size, takes time proportional to the
database size, and needs exclusive access while it runs.  Automatic
vacuuming after every B<expireover> would therefore be expensive and usually
pointless: the next day of traffic would grow the file back toward the same
high-water mark.

Vacuum manually with C<ovsqlite-util -v> only when the smaller size is
actually wanted for the long term, for example:

=over 2

=item *

After a large, permanent reduction in retention (so the overview will not
grow back to the old size).

=item *

After purging many newsgroups or other bulk deletes that left a large
freelist (check with C<ovsqlite-util -s>; freelist share is reported
there).

=item *

When changing I<pagesize> on an existing database.  The I<pagesize>
parameter in F<ovsqlite.conf> is consulted only when creating a new database.
To change the page size of an existing overview, stop the news system, set
C<PRAGMA page_size> to the desired value, and vacuum (C<ovsqlite-util -v>
reclaims freelist space; applying a new page size can be done with the
B<sqlite3> CLI as C<PRAGMA page_size=N; VACUUM;> on the stopped database),
or rebuild overview with B<makehistory>.  Prefer choosing an appropriate
page size when the database is first created.

=back

For a large vacuum, stop the news system with C<rc.news stop> so
B<ovsqlite-server> and B<nnrpd> are not holding the database open.  In normal
operation, leave freelist pages alone and treat the high-water mark as the
steady-state size of an expiring spool.

=head1 HISTORY

Initial implementation of ovsqlite written by Bo Lindbergh
<2bfjdsla52kztwejndzdstsxl9athp@gmail.com> for InterNetNews.

=head1 SEE ALSO

INN::ovsqlite_client(3pm), inn.conf(5), makehistory(8), ovsqlite-server(8),
ovsqlite-util(8), rc.news(8).

=cut
