OVSQLITE-UTIL(8)

NAME
ovsqlite-util - Overview manipulation utility for ovsqlite

SYNOPSIS
ovsqlite-util [-AFghioOsv] [-a article] [-n newsgroup] [-p path]

DESCRIPTION
ovsqlite-util is an administrative interface to the ovsqlite overview method for INN. It only works on ovsqlite overview databases, not on any other type of INN overview. (See dedicated ovdb_stat(8) and tdx-util(8) programs for ovdb and tradindexed overview methods.) It allows the administrator to dump various information about the internal state of the overview, audit it for errors, fix these errors, and vacuum the database to reclaim free pages.

To audit the entire overview database for problems, use -A. The checks can take an extended period of time, depending on the number of newsgroups and articles present in the overview database. Any problems found will be reported to standard error. Use -F to correct the errors found.

It is safe to run this utility while the server is running. In case fixes should be made to the database (when using the -F flag), ovsqlite-util will properly pause the server before performing its operations, and then resume it. The -v (vacuum) flag also pauses a running innd when possible; for a large vacuum, it is better to stop the news system so ovsqlite-server and readers are not holding the database open.

The number of overview records stored in the database can be obtained with the "inndf -n" command.

As ovsqlite-util directly opens the ovsqlite overview database, it may take a longer time to complete when INN is running owing to the fact that ovsqlite-server may hold locks on certain tables, in which case ovsqlite-util just waits for the locks to be released. (The transtimelimit parameter in ovsqlite.conf controls the duration of a transaction, so you may check you have not set a too high value.)

REQUIREMENTS
The "DBI" Perl module with the "DBD::SQLite" driver must be installed for this program to work. They are present in most distributions as a package named like "libdbd-sqlite3-perl", "perl-DBD-SQLite" or "p5-DBD-SQLite". They can also be directly installed from CPAN ("DBI" and "DBD-SQLite" distributions available at <https://metacpan.org/dist/DBI> and <https://metacpan.org/dist/DBD-SQLite>).

Naturally, an ovsqlite overview database also needs being in use, as detailed in the ovsqlite(5) manual page.

OPTIONS

-a article
    The article number or numbers to act on. article is a valid NNTP range, meaning that it can be either a single article number or a range of article numbers (like "1-5"). Either the start or the end (or both) of the range may be omitted, in which case they will be set to the first or last article number in the group. Passing "-" for article is therefore equivalent to not using the -a option at all.

    Only useful in combination with the -g, -o and -O options to dump overview information. It cannot be combined with -s.

-A  Audit the entire overview database for problems. Currently, only one consistency check is performed: the verification that the low water mark, the high water mark and the article count of each newsgroup match the reality of articles present in overview. No changes will be made to the database, but problems will be reported to standard error. This is the default action.

-F  Audit the entire overview database for problems, fixing them as they're detected where possible. This runs the same checks as the -A flag, and fixes the problems found. Using this option takes care of pausing and resuming INN so that the overview data are not changed by another process during the operations.

    To see what would be changed by -F, run ovsqlite-util with -A first.

-g  Dump overall overview information for a particular group. The fields are, in order, the article number, the length of the overview data, the time (in seconds since epoch) when the article arrived on the server, the time (in seconds since epoch) when the article should expire based on its Expires header field (or 0 if there is no Expires header field), and the storage API token of the article.

    If this option is given, the -n option must also be given to specify the newsgroup on which to act.

-i  Dump newsgroup-related overview information from the database. This contains similar information to the server active file, such as high and low water marks and moderation status, and is the information that nnrpd hands out to clients.

    The fields are, in order, the newsgroup name, the high water mark, the low water mark, the count of articles in the group, the group status, the time (in seconds since epoch) when that newsgroup was last expired (or 0 if expireover hasn't run on it yet), and the deletion status of that newsgroup (0 if not marked for removal during the next expiration, and a positive value otherwise).

    A particular newsgroup can be specified with the -n option. If -n is not given, the entire master index will be dumped.

-n newsgroup
    Specify the newsgroup on which to act, required for the -g, -o, and -O options, and optional for the -i option. It cannot be combined with -s.

-o  Dump the overview information for a newsgroup, in the same format as it would be returned to clients but with one modification. Appended to the end of each entry will be four additional pieces of data: the article number according to overview information labelled with "Article:", the storage API token for that article labelled with "Token:", the arrival date for that article on the server in RFC 5322 date format labelled with "Arrived:", and the expiration date for that article (from the Expires header field) in RFC 5322 date format if there is any, labelled with "Expires:".

    If this option is given, the -n option must also be given to specify the newsgroup on which to act. By default, all of the overview information for that newsgroup is dumped, but the -a option may be given to restrict the dump to the information for a single article or a range of article numbers.

-O  Dump the overview information for a newsgroup in the format used by overchan as input. Each line will start with the storage API token, the arrival timestamp in seconds since epoch, the expires timestamp in the same format (or 0 if there is no Expires header field), and then the overview data.

    If this option is given, the -n option must also be given to specify the newsgroup on which to act. By default, all of the overview information for that newsgroup is dumped, but the -a option may be given to restrict the dump to the information for a single article or a range of article numbers.

-p path
    Act on the overview database rooted in path, overriding the overview path specified in the pathoverview parameter in inn.conf.

-s  Print database storage and overview record statistics. The report contains the SQLite and schema versions, journal and auto-vacuum modes, compression status, page usage, database and journal file sizes, and counts of active and deleted newsgroups, their articles, and any orphaned article records. It remains available when ovsqlite-server is not running.

    Database-derived values are collected from one consistent read snapshot. The logical database size is the page count multiplied by the page size. Non-freelist page capacity is an estimate derived from SQLite's page accounting, not a measure of how full each page is. Freelist pages can be reused by later writes or recovered with a full "VACUUM" (see -v). File sizes are point-in-time values and may change while the server is writing. Files that do not currently exist, such as a WAL or rollback journal, have a reported size of zero.

-v  Vacuum the overview database: rewrite it with SQLite "VACUUM" so freelist pages are removed and the file shrinks toward its logical size. Prints page and freelist statistics before and after the rewrite. It cannot be combined with -n or -a.

    A vacuum needs free disk space roughly equal to the current database file size (SQLite builds a compact copy, then replaces the original). On a large overview, this can take a long time and needs exclusive access to the database. When innd is running, it is paused for the duration. For a reliable vacuum on a busy server, stop the news system with "rc.news stop" first so ovsqlite-server and nnrpd are not holding connections open; otherwise "VACUUM" waits on the busy timeout until other connections release their locks.

    Vacuuming is not needed in normal operation. SQLite keeps deleted pages on a freelist and reuses them for later inserts, so a steady-state spool that expires roughly as much as it receives tends to stabilize near a high-water mark size. That high-water mark is usually desirable: the file does not churn on every expireover run, and subsequent growth can reuse free pages without extending the file. Run -v only when you deliberately want a smaller file, for example after a large permanent reduction in retention, after purging many groups, or when changing the page size of an existing database (which requires a vacuum; see ovsqlite(5)).

    See also "VACUUMING" in ovsqlite(5).

EXAMPLES
Dump newsgroup-related overview information for the overview database in /news/overview, regardless of the overview path specified in inn.conf:

    ovsqlite-util -i -p /news/overview

Dump overall overview information for example.test:

    ovsqlite-util -g -n example.test

Dump the complete overview information for example.test:

    ovsqlite-util -o -n example.test

Dump the overview information for articles 45 and higher in example.test:

    ovsqlite-util -o -n example.test -a 45-

Audit the entire overview database for any problems:

    ovsqlite-util -A

If problems were reported, fix them with:

    ovsqlite-util -F

Print storage and record statistics:

    ovsqlite-util -s

After shrinking retention and letting expireover finish, reclaim freelist space (preferably with the news system stopped):

    ovsqlite-util -v

HISTORY
Written by Julien ELIE for InterNetNews.

SEE ALSO
expireover(8), inndf(8), makehistory(8), nnrpd(8), ovsqlite(5), rc.news(8).