\input texinfo   @c -*-texinfo-*-
@c %**start of header
@setfilename link-controller.info
@settitle Using LinkController
@setchapternewpage odd
@c %**end of header

@dircategory LinkController link checking system
@direntry 
* LinkController: (link-controller).	A system for checking and repairing links
* link-report: (link-controller)Invoking link-report. Reporting the status of broken links
* test-link: (link-controller)Invoking test-link. Testing if links are broken
* extract-links: (link-controller)Invoking extract-links. Finding links in web pages.
* fix-link: (link-controller)Invoking fix-link. Repairing links in your web pages.
* check-page: (link-controller)Invoking fix-lnk. Repairing links in your web pages.
* build-schedule: (build-scheduile)Invoking builld-schedule. Scheduling links for checking.
@end direntry

@c not enough programs and variables to justify separate indexes.
@syncodeindex pg vr
@c fn is used for file names
@syncodeindex fn vr

@titlepage
@title Using LinkController
@author Michael De La Rue

@page
@vskip 0pt plus 1filll
Copyright @copyright{} 1997-2002 Michael De La Rue

Published by ...

Permission is granted to distribute and change this manual under the
terms of the GNU public license.

This manual is not yet complete, but it's better than nothing.  

Manual Revision Code: $Revision: 1.22 $
@end titlepage

@ifnottex
This file documents LinkController

Copyright 1997-2001 Michael De La Rue

Permission is granted to distribute and change this manual under the
terms of the GNU public license.

This is the alpha version of this manual and is very incomplete.

@node Top, Introduction, (dir), (dir)

This document describes LinkController a system for checking and
maintaining links in infostructures

This document applies to version 0.033 of LinkController

Manual Revision Code: $Revision: 1.22 $
@end ifnottex

@contents

@menu
* Introduction::                What LinkController is about.
* Getting Started::             How to get LinkController running.
* Configuration::               Setting up LinkController to check links
* Advanced Configuration::      Optimising configuration; advanced features
* Using LinkController::        Checking and repairing web pages.
* Interfaces::                  Importing and exporting link information.
* Emacs::                       The Emacs interface.
* Administration::              Administrating a LinkController installation.
* Robot Behaviour::             General statements about how to use robots.
* Uncheckable Links::           Links LinkController can't or won't check.
* Absolute and Relative URIs::  Explanation of how relative URIs are handled.
* Bugs::                        What to do if you find a bug.
* History::                     How this came about and who helped.
* Invoking the Programs::       How to run the programs in LinkController.
* Related Packages::            Packages used by or useful with LinkController.
* Terms::                       Glossary of terms used in this documentation.
* Names Index::                 Index of Program and Variable names
* Concept Index::               Index of Concepts.

@detailmenu
 --- The Detailed Node Listing ---

Configuration

* Interactive Configuration::   An easy way to get a basic configuration.
* Setting Configuration Variables::  How to set variables.
* Configuration Variables::     LinkController's main configuration variables.
* Infostructure Configuration::  Defining which pages to check.

Advanced Configuration

* Advanced Infostructure Configuration::  Advanced control of checking
* Authorisation Configuration::  Checking pages which require basic authentication.
* Configuring CGI Programs::    Setting up LinkController's web interface

Using LinkController to Check Links

* Extracting Links::            Getting link information from WWW pages.
* Testing::                     How to run the link testing program.
* Reporting::                   Getting information the state of your links.
* Email Reporting::             Automatic reporting of newly broken links.
* Checking Files::              Checking individual HTML files.
* Repairing Links::             Replacing old URLs with new ones.
* Suggestions::                 Making suggestions for other users.
* CGI Interface::               The LinkController web interface (primitive).

Reporting Problems

* Email Reporting::             Automatic notification of broken links.

The Emacs Interface

* link-report-dired::           An Emacs program to display broken links.
* check-page in Emacs::         Finding broken links in a file.

Administration

* Setting up LinkController::   How to get the system installed
* Default Installation::        A simple multi-user installation
* User Administration::         Adding and removing users
* Cron Scripts::                Running programs automatically
* Link Database Maintenance::   Dealing with database problems
* Link Ageing::                 Clearing outdated data from the databases.

History

* Acknowledgements ::           People and or institutions who helped

Acknowledgements

* Esoterica Internet Portugal::  
* IPPT PAN Poland::             
* The Tardis Project::          
* Other Free Software Authors::  

Invoking the LinkController Programs

* Invoking link-report::        link-report usage summary
* Invoking test-link::          test-link usage summary
* Invoking extract-links::      extract-links usage summary
* Invoking fix-link::           fix-link usage summary
* Invoking check-page::         check-page usage summary
* Invoking build-schedule::     build-schedule usage summary

Packages Which Work With LinkController

* cdb::                         Utilities for the LinkController indexes.
* Tie-Transact-Hash::           Berkeley DB editing tools.

Terms

* Infostructure::               Groups of resources.
* Link::                        A connection between two resources.
* Resource::                    The information on the World Wide Web.
* URIs::                        A type of link including URLs
* URLs::                        The connections in the World Wide Web.
* URNs::                        Names for resources without location.

@end detailmenu
@end menu

@node Introduction, Getting Started, Top, Top
@unnumbered Introduction

LinkController is a system for checking links.

Most HTML pages contain references to other HTML pages (links).  These
allow the readers of those pages to locate other related resources (web
pages etc.).  Unfortunately the location of `resources'
@footnote{resource is a general term for HTML pages and all of the other
things that can be referenced by URLs} can change, resources can
disappear completely or the system providing the resource can break.
When this happens, the link which used to find them will no longer work.
The only reliable way to detect this problem is to periodically check
over the resources and take corrective action for the ones that have
gone missing.

LinkController is designed to make that task much more efficient.  It
automates the task of checking which links have been broken for a period
of time and then of finding which documents they occur in.

LinkController is copyrighted software and is distributed under the GNU
Public License which should have been included as the file
@file{COPYING} in the distribution.

@node Getting Started, Configuration, Introduction, Top
@chapter Getting Started 

This section of the manual assumes that the programs that make up
LinkController are already installed and working on your computer.  If
not, then @xref{Setting up LinkController}.  We assume the standard
setup where your system administrator runs the link checking for you.
Other setups will need slightly different behaviour.  Speak to the
person who set up link controller.

The first thing to do is to run @w{@code{configure-link-control}} to
configure the system.  This will ask a series of questions about your
configuration and create a configuration file which will be used by the
various programs which make up LinkController.

Next you have to work out which links you are interested in.  Do this by
extracting the links from your web pages (@pxref{Extracting Links}).
The output file from this with the list of links found will be stored
in the location you gave during configuration.  

Assuming that you have the default install, your links will be
automatically copied and checked over time following that.  

After a short time (about a day) you will begin to get information about
links which didn't work with @w{@code{link-report --not-perfect}}.  After
some more time (a week or so) you can use @w{@code{link-report}} to find
out which links are really broken.

@node Configuration, Advanced Configuration, Getting Started, Top
@chapter Configuration

@menu
* Interactive Configuration::   An easy way to get a basic configuration.
* Setting Configuration Variables::  How to set variables.
* Configuration Variables::     LinkController's main configuration variables.
* Infostructure Configuration::  Defining which pages to check.
@end menu

@node Interactive Configuration, Setting Configuration Variables, Configuration, Configuration
@section Interactive Configuration
@cindex configuration, interactive
@cindex variables, setting interactively

The @w{@code{configure-link-control}} program can be used by users to
configure LinkController.  This will ask you a series of questions and
then generate a configuration file in your home directory.  This is a
good way to start configuring LinkController.

The configuration that is controlled by this program is related to
reporting and fixing links.  For other configuration see
@xref{Administration}.

@node Setting Configuration Variables, Configuration Variables, Interactive Configuration, Configuration
@section Setting Configuration Variables
@cindex configuration
@cindex variables, setting
@cindex setting variables
@findex ~/.link-control.pl
@findex link-control.pl

If automatic configuration (@pxref{Interactive Configuration}) doesn't
work well enough for you, then you should manually change the
configuration variables.  All of the variable information is stored in
the file @file{.link-control.pl} in your home directory or
@file{/etc/link-control.pl} for system wide configuration.  These
locations are hardwired into the LinkController system and should only
change if your administrator has done something strange.  The
configuration files are written directly in Perl (the programming
language LinkController is written in).  You can set the configuration
variables by putting lines like this.

@example
$::links='/var/lib/link_database.bdbm';
@end example

Please note the semi colon at the end of the line and the use of single
quotes so that Perl doesn't do anything strange to your values.

@node Configuration Variables, Infostructure Configuration, Setting Configuration Variables, Configuration
@section LinkController Configuration Variables

@cindex variables, configuration
@cindex configuration variables

This is a complete list of the configuration variables which a user
should chnage in the @samp{.link-control.pl} file.  
@xref{Setting Configuration Variables}, for how to do this.

@table @code
@item $::user_address
@vindex $user_address
is the email address which the robot declares to the world as it
goes around checking links.  If you want to check links yourself, you
must set this to a valid email address, because if something goes badly
wrong, it is the only way for a user at another site to know how to
contact you.

@item $::base_dir
@vindex $base_dir
This is the base directory for all of the configuration files.  If this
variable is defined then the other variables will default as given
below and do not need to be set individually.

@item $::links
@vindex $links
@findex links.bdbm, location
@findex location of links.bdbm
tells you what file is being used to store information
about links.  This could easily be a shared database used by everyone on
your system.  Defaults to @file{$::base_dir/links.bdbm}.

@item $::schedule
@vindex $schedule
@findex schedule.bdbm, location
@findex location of schedule.bdbm
tells the system where to find the schedule file used to
decide which links should be checked next and when that should be.  You
will need to set this and create the file in order to do link checking. 
Defaults to @file{$::base_dir/schedule.bdbm}.

@item $::page_index
@vindex $page_index
@findex page_index, location
@findex location of page_index
This variable tells LinkController where to find the index which lists
which links are contained on each page.
Defaults to @file{$::base_dir/page_has_link.cdb}.

@item $::link_index
@vindex $link_index
@findex link_index, location
@findex location of link_index
This variable tells LinkController where to find the index which lists
which links are contained on each page.  This is used during reporting
to create the list of pages that should be repaired.  It is also used
during repair to decide which files have to be repaired.  The file can
be regenerated by running @code{extract-links}.
Defaults to @file{$::base_dir/link_on_page.cdb}.

@item $::infostrucs
@vindex $infostrucs
@findex infostrucs, location
@findex location of infostrucs
This variable points to the configuration file where definitions of
infostructures are should be put @ref{Infostructure Configuration}.
Defaults to @file{$::base_dir/infostrucs}.

@item $::link_stat_log
@vindex $link_stat_log
@findex link status log, location
@findex location of link status log
This variable is the name of a file where important link status changes
will be logged.  The current definition is links which have just been
discovered to be broken.  This can be used in email notification
@ref{Email Reporting}.  There is no default value for this file and it
is not generated by default.
@end table

@node Infostructure Configuration,  , Configuration Variables, Configuration
@section Configuring Infostructures

@cindex configuration, infostructure
@findex infostrucs

The infostructure configuration says where our web pages are stored and
how they are accessed.  It is kept in a separate file defined by the
@code{$::infostrucs} configuration variable.  The file is used by
@w{@code{extract-links}} (@pxref{Invoking extract-links}) to find which
files to get links from; it is used by @w{@code{fix-link}}
(@pxref{Invoking fix-link}) to find out where files needing to be
repaired are stored; it is used by @w{@code{check-page}}
(@pxref{Invoking check-page}) to work out the base URI for any file
being checked and finally it is used for certain reports in 
@w{@code{link-report}} (@pxref{Invoking link-report}).

The format of the file is one line for each infostructure with
configuration directives separated by spaces.  For example

@example
directory http://example.com/manual /var/www/html/manual
www http://example.com/strange_database
@end example

The first directive describes how @w{@code{extract-links}} program
should extract the links.  It currently has three possible values.  The
value @code{www} means to actually use the given URL to download the web
pages.  The value @code{directory} means that @w{@code{extract-links}}
should assume that all of the files are stored in a directory and that
the directory structure matches the structure of the infostructure.  The
final value @code{advanced} allows for further configuration at the cost
of extra complexity.  @xref{Advanced Infostructure Configuration}, for
more information about this.

In the case where we use the @code{directory} directive, a third
directive is present on each line with the full path to the base
directory of the infostructure.  In this case @w{@code{fix-link}} will
be able to repair broken links in these files and
@w{@code{extract-links}} will use direct file access to the file system
when extracting links.

@node Advanced Configuration, Using LinkController, Configuration, Top
@chapter Advanced Configuration

There are various advanced ways to configure LinkController.  These are
mostly not needed for simple checking of a small collection of web
pages.  For larger sites and special situations however, they may well
make life much easier.

@menu
* Advanced Infostructure Configuration::  Advanced control of checking
* Authorisation Configuration::  Checking pages which require basic authentication.
* Configuring CGI Programs::    Setting up LinkController's web interface
@end menu

@node Advanced Infostructure Configuration, Authorisation Configuration, Advanced Configuration, Advanced Configuration
@section Advanced Infostructure Configuration
@cindex infostructure, advanced configuration
@cindex regular expression, exclude
@cindex regular expression, include
@cindex filtering links
@findex link-control.pl, infostructures

Using more advanced configuration it is possible to skip over certain
resources when we are doing link extraction and to ignore some of the
links.  You may want to skip over this section initially and come back
to it only when you find that there are links or pages being checked
that you would rather avoid.

For this section, we assume that you already know how to make basic Perl
code.  If not, then please read through the Perl manual pages
@samp{perl}, @samp{perlsyn} and @samp{perldata}.  You may find that the
examples given below are sufficient to get you started.

In order to get @w{@code{extract-links}} to extract links using an
advanced infostructure, you must use the @code{advanced} keyword. In the
infostructure file.  Infostructures not listed there will be ignored,
but won't cause any harm.

Advanced configuration is in the @file{.link-controller.pl}
configuration file by making definitions into the @code{%::infostrucs}
hash.  These look like the following

@example
$::infostrucs@{http://www.mypages.org/@} = @{
   mode => "directory";
   file_base => "/home/myself/www",
   prune_re => "^(/home/myself/www/statistics)" #ignore referrals
              . "|(cgi-bin)", #do CGIs separately
   resource_exclude_re => "\.secret$", #secrets shouldn't stay secret
   link_exclude_re => "^http://([a-z]+\.)+example\.com", 
@};

$::infostrucs@{http://www.mypages.org/cgi-bin/@} = @{
   mode => "www";
   resource_exclude_re => "query", #query space is infinite!!
@};
@end example

There are a number of keywordss that can be used.  

@table @samp
@item mode
@vindex mode
This decides how to download the links.  Either @samp{www} or
@samp{directory}.
@item file_base
@vindex file_base
If defined, this defines the directory which matches the URL where the
infostructure is based.  This must be defined if the mode is set to
directory.
@item resource_include_re
@vindex resource_include_re
If defined, this regular expression must be matched by the @emph{URL}
for every resource before links will be extracted from it.
@item resource_exclude_re
@vindex resource_exclude_re
If defined, this regular expression must @emph{not} be matched by the
@emph{URL} for every resource before links will be extracted from it.
@item link_include_re
@vindex link_include_re
If defined, this regular expression must be matched by every @emph{URL}
found before it will be extracted and saved.
@item link_exclude_re
@vindex link_exclude_re
If defined, this regular expression must @emph{not} be matched by every
@emph{URL} found before it will be extracted and saved.
@item prune_re
@vindex prune_re
Used only in directory mode, this will completely exclude all files and
sub-directories of directories matched by the regular expression.
@end table

N.B. the exclude and include regular expressions can be used together.
For a match, the include regular expression must match and the exclude
must not match.  In other words excludes override includes.

In order for the infostructure to be used by @code{extract-links} an
entry must still be made in the @file{infostrucs} file.  For this use the
@code{advanced} keyword.  The second argument is a URL used to look up
the definition in the $::infostrucs hash.

@example
advanced   http://www.mypages.org/
advanced   http://www.mypages.org/cgi-bin/
@end example

The URL used here must match @emph{exactly} the one used in the hash.
It is important to note that @samp{directory} and @samp{www} definitions
in the @file{infostrucs} file will override any advanced configuration
given.

@node Authorisation Configuration, Configuring CGI Programs, Advanced Infostructure Configuration, Advanced Configuration
@section Authorisation Configuration
@cindex authentication, basic
@cindex basic authentication
@cindex authorisation
@cindex security, risks of authentication
@findex link-control.pl, credentials
@findex link-control.pl, authorisation

One problem when checking links, especially within an intranet situation
is that some pages can be protected with basic authentication.  In order
to extract links from those pages or to simply know that they are there,
we have to get through that authentication.  By using the advanced
Authorisation Configuration we can give LinkController authority to
access these pages and allow link checking to work as normal.

@display
@emph{Using this method to allow LinkController to work in an
environment with authentication is inherently a security issue since
authentication tokens must be stored, effectively in plaintext, in
files.  This risk may, however, not be much higher than the one that you
currently accept, so this can be useful}
@end display

We can store the authentication tokens simply in the %::credentials hash
which we can create in the @file{.link-controller.pl} configuration file.
The keys in the hash are the exact realm string which will be sent by
the web server.  Each value of this hash is a hash with a pair of keys.
The @samp{credentials} key should be associated to the authentication
token.  The @samp{uri_re} key should be a regular expression which
matches the web pages you want to visit.  For security reasons it
shouldn't match any others.

@example
$::credentials = @{
  my_realm => @{ uri_re => "https://myhost.example.com",
                credential => "my_secret" @}
@} );
@end example

As a sanity check, every @samp{uri_re} will be tried on
@samp{http://3133t3hax0rs.rhere.com} and
@samp{http://3133t3hax0rs.rhere.com/secretstuff/www.goodplace.com/}.  If
the expression matches then the credentials will be ignored.  If you
know enough to do this safely then you should definitely know how to get
past this check.  The owners of the domain @samp{3133t3hax0rs.rhere.com}
will just have to hack the code..

For more discussion about the security risks and how to mitigate them
see the file @file{authorisation.pod} included with the LinkController
distribution.  If you didn't understand the security risk from the above
description then probably you should consider avoiding using this
mechanism.

@node Configuring CGI Programs,  , Authorisation Configuration, Advanced Configuration
@section Configuring CGI Programs
@cindex CGI, configuration
@pindex configure-link-cgi@r{, using}

The CGI programs use the same configuration variables as the other
programs, however, to avoid any confusion and related security problems,
a perl script should be written which has the configuration variables
hard wired in then runs the appropriate CGI program.
@code{configure-link-cgi} is a program designed to set up such a
script.  

@strong{FIXME:} this section needs to be rewritten.

@node Using LinkController, Interfaces, Advanced Configuration, Top
@chapter Using LinkController to Check Links
@cindex links, extracting
@cindex extracting links
@pindex extract-links@r{, using}

This chapter covers in reasonable detail how to use each of the programs
in LinkController.  

@menu
* Extracting Links::            Getting link information from WWW pages.
* Testing::                     How to run the link testing program.
* Reporting::                   Getting information the state of your links.
* Email Reporting::             Automatic reporting of newly broken links.
* Checking Files::              Checking individual HTML files.
* Repairing Links::             Replacing old URLs with new ones.
* Suggestions::                 Making suggestions for other users.
* CGI Interface::               The LinkController web interface (primitive).
@end menu

@node Extracting Links, Testing, Using LinkController, Using LinkController
@section Extracting Links
@cindex links, extracting
@cindex extracting links
@pindex extract-links@r{, using}

This section is written assuming that you are using a standard HTML
infostructure in a directory or on the World Wide Web

The first part of using link controller is to extract the links.  When
doing this, a pair of index files is built which list which URLs happen
on which pages along with a file listing all of the URLs in the
infostructure.  

@strong{FIXME:} compare and contrast multi-user configuration with
single user

@pindex extract-links
The first stage of the process is done by @w{@code{extract-links}}
@footnote{the command names in link controller are quite long.. you
might want to make your life easier by using command completion which
will finish what you have started.. once you've typed a little of the
command use @kbd{escape escape} or @kbd{tab} depending on your
shell.. if this doesn't work then you may like to upgrade to a newer
shell such as bash or zsh.}.

There are two modes for extract links @code{directory} and @code{www}.
The key difference between them is that the latter actually downloads
from a server so it is less efficient but will work in more
circumstances and is more likely to represent your site as seen by
users.  This is assuming that all of your WWW pages are interconnected
so it can find them.

@strong{FIXME} : need to describe modes of operation of extract link

@findex link index, creating
@findex page index, creating

@w{@code{extract-links}} creates three files.  The first two files
(@file{*.cdb}) are the index files for your infostructure and are
located wherever you have configured them to by default they are called
@file{link_on_page.cdb}, @file{page_has_link.cdb}.  The third file is
the database file @file{links.db}.  @w{@code{extract-links}} can also
optionally create a text file which lists all of the URLs in the
infostructure, one per line.

There are a number of other ways of using @w{@code{extract-links}} and
it has many options. @xref{Invoking extract-links}, for more information
about using extract links.

@node Testing, Reporting, Extracting Links, Using LinkController
@section Testing Links
@pindex fix-link@r{, using}

If you are using someone else's link information then you may be able to
skip this part and go straight on to the next one on generating reports.
If not then the next stage is to test your links using @w{@code{test-link}}.

Testing links takes a long time.  Reporting of broken links will not
begin until after several days.  This is a deliberate feature of
LinkController.  Most problems that will be found in a well maintained
web page will be temporary configuration or system problems.  By
wainting to report problems we give people responsible for the other end
of the problem link a chance to repair their resources.  Once we have
made this decision, we may as well check slowly and in a way which will
reduce the amount of network bandwidth LinkController uses at a given
time and so its impact on other people's Internet usage.

@pindex test-link@r{, using}
The key program which you want to use is @w{@code{test-link}}.  I run
this from a shell script which directs its output to a log file

@strong{FIXME} actually I now just use a cron job.

@example
#!/bin/sh
#this is just a little sample script of how I run the program.

LOGDIR=$HOME/log
test-link >> \
        $LOGDIR/runlog-`/bin/date +%Y-%m-%d`.log 2>&1 
#assumes the use of a gnu style date command which can print 
#out full dates.
@end example

And I run this shell script from my @file{crontab} with a command like
this

@cindex crontab, example
@example
42 02 * * *     /..directories./run-daily-test.sh
@end example

The string @w{@code{/..directories./}} should be replaced with the directory
where you have the script.  Remember to make the script executable.  

This will now run until completion each night.   However, you should
make sure that it does actually finish.  If you have too many links to
check in the given time, then you can end up with a backlog and the
system will take a long time to stop.  To avoid this, either make
testing less frequent or make checking run faster.  This will have to be
done by editing the program itself at present.

The @w{@code{test-link}} program has a number of options.  These control
the limits on checking and the speed of checking.  @xref{Invoking
extract-links}, for more information on these.

@node Reporting, Email Reporting, Testing, Using LinkController
@section Reporting Problems
@cindex reports
@cindex broken links, finding
@cindex links, examining
@pindex link-report@r{, using}

The easiest way to find out which links are broken is to use the command
line interface.  The simplest report you can generate is just a list of
all the known broken links.  Do this like so:

@example
link-report
@end example

On the system I'm testing on right now, this gives:

@example 
broken:-       file://ftp.ncsa.uiuc.edu/Web/httpd/Unix/ncsa_httpd/cgi
        http://www.ippt.gov.pl/docs-1.4/cgi/examples.html
broken:-       file://ftp.ncsa.uiuc.edu/Web/httpd/Unix/ncsa_httpd/curr
ent/httpd_1.4_irix5.2.Z
        http://www.ippt.gov.pl/docs-1.4/setup/PreExec.html
broken:-       file://ftp.ncsa.uiuc.edu/Web/httpd/Unix/ncsa_httpd/curr
ent/httpd_1.4_linux.Z
        http://www.ippt.gov.pl/docs-1.4/setup/PreExec.html
broken:-       file://ftp.ncsa.uiuc.edu/Web/httpd/Unix/ncsa_httpd/curr
ent/httpd_1.4_osf3.0.Z
        http://www.ippt.gov.pl/docs-1.4/setup/PreExec.html
broken:-       file://ftp.ncsa.uiuc.edu/Web/httpd/Unix/ncsa_httpd/curr
ent/httpd_1.4_solaris2.4.Z
        http://www.ippt.gov.pl/docs-1.4/setup/PreExec.html
broken:-       file://ftp.ncsa.uiuc.edu/Web/httpd/Unix/ncsa_httpd/curr
ent/httpd_1.4_solaris2.4.tar.Z
        http://www.ippt.gov.pl/docs-1.4/setup/PreCompiled.html
Sorry, couldn't find info for url file://ftp.ncsa.uiuc.edu/Web/httpd/U
nix/ncsa_httpd/current/httpd_1.4_source.tar.Z
please remember to check you have put it in full format
broken:-       file://ftp.ncsa.uiuc.edu/Web/httpd/Unix/ncsa_httpd/docu
ments/usage.ps
        http://www.ippt.gov.pl/docs-1.4/postscript-docs/Overview.html
..etc...
@end example

Which just tells you which links are broken.  We also know which page
they are broken on and can go and look at that on the World Wide Web or
directly as a file on the server.

There are many different options which control the output of
@w{@code{link-report}}.  These include options which select which kinds
of problems to report, options which select which pages to report from
and options which allow other output formats such as HTML.
@xref{Invoking link-report}, for more information about these.

For more advanced reporting and editing of documents with broken links
you may want to use the Emacs interface (@pxref{Emacs}).

@menu
* Email Reporting::             Automatic notification of broken links.
@end menu

@node Email Reporting, Checking Files, Reporting, Using LinkController
@section Email Reporting of Newly Broken Links
@cindex automatic notification
@cindex broken links, automatic reporting
@pindex link-report@r{, reporting new problems}
@pindex test-link@r{, recording new problems}
@vindex $::link_stat_log@r{, automatic notification using}

It's possible to arrange automatic reporting by email of links which
have become newly broken.  This is done by getting @w{@code{test-link}}
to make a list of links that become broken using the @samp{$::link_stat_log}
variable (@pxref{Configuration Variables}) and calling @w{@code{link-report}} to
report on those links.

Typically, you may don't want to have a report every time that
@w{@code{test-link}} runs, but probably once a day instead.  In this
case, run a script like the following from your crontab.

@example
#!/bin/sh
STAT_LOG=$HOME/link-data/stat-log
WORK=$STAT_LOG.work
EMAIL=me@@example.com
mv $STAT_LOG  $WORK
if [ -s $WORK ]
then
   link-report --broken --url-file=$STAT_LOG | 
      mail -s "link-report for `date`" $EMAIL
fi
@end example

Every time that this script is run, it will rename the status change
log file and then mail a report with all of the new broken links
to the specified email address.  

@node Checking Files, Repairing Links, Email Reporting, Using LinkController
@section Examining Individual Files
@cindex page, checking
@cindex file, individual, checking
@cindex checking individual pages
@pindex check-page@r{, usage}

When you have just written an HTML page, you often want to check it
before you put it up for use.  You can do this immediately using the
@w{@code{check-page}} program.  Simply run something like

@example
check-page filename.html
@end example

And it will list all of the links that it is unsure about along with the
line number the problem occurred on.  This program works particularly
well when you editing with Emacs (@pxref{check-page in Emacs, The Emacs Interface}).

@node Repairing Links, Suggestions, Checking Files, Using LinkController
@section Repairing Links
@cindex links, repairing
@cindex repairing links

@pindex fix-link
The program responsible for repairing links is @w{@code{fix-link}}.  It
simply accepts two URLs and changes all of the occurrences of the first
link in your documents into the second link.  It assumes that you have
permission to edit all of the problem files and that there is a
replacement link.  For example

@example
fix-link http://www.ed.ac.uk/~mikedlr/climbing/ \
        http://www.tardis.ed.ac.uk/~mikedlr/climbing/
@end example

Typed at the shell prompt would have updated the location of my Climbing
pages when they moved some while ago and 

@example
fix-link http://www.tardis.ed.ac.uk/~mikedlr/climbing/ \
        http://scotclmb.org.uk/
fix-link http://www.tardis.ed.ac.uk/climb/ \
        http://scotclmb.org.uk/
@end example

Will change them to the very latest location.  More information about
@w{@code{fix-link}} can be found in @xref{Invoking fix-link}.

At present, there's no facility for automatically updating the databases
when you do this.  Instead, you have to run @w{@code{extract-links}}
after some time so that new links are noticed.  In practice this doesn't
matter because you shouldn't be creating new pages with broken links and
can check that you don't with @w{@code{check-page}}.  A later version of
LinkController will may change this.

The other way to fix links is to edit the files by hand.  This is the
only solution where a link has disappeared forever and so text changes
have to be made to the web site.  This can be made more convenient by
using the @samp{link-report-dired} emacs module included in the
distribution.  This is covered elsewhere in this manual (@pxref{Emacs}). 

@node Suggestions, CGI Interface, Repairing Links, Using LinkController
@section Making Suggestions
@cindex suggestions

A link in the database can have suggestions associated with it.  These
are normally alternative URLs which somebody or something has decided
would make a good replacement for the URL of the Link.  Humans can add
to the database with the @code{suggest} program.  For example use:

@pindex suggest
@example
suggest file://ftp.ncsa.uiuc.edu/Web/httpd/Unix/ncsa_httpd/current/htt
pd_1.4_linux.Z \
        http://delete.me.org/
Link suggestion accepted.  Thank you
@end example

If you try the same thing again you get

@example
suggest file://ftp.ncsa.uiuc.edu/Web/httpd/Unix/ncsa_httpd/current/htt
pd_1.4_linux.Z \
        http://delete.me.org/ 
Already knew about that suggestion.  Thanks though.
@end example

These suggestions will make it easier for others to repair links,
especially if they are using the CGI interface.

@node CGI Interface,  , Suggestions, Using LinkController
@section CGI Interface
@cindex interface, CGI
@cindex CGI interface
@pindex link-report.cgi
@pindex fix-link.cgi

The CGI interface is not fully developed and has a number of issues
related to security to be considered.  I have however used it and shown
that it can work, so if you want to you could try the same.  The two
programs @w{@code{fix-link.cgi}} and @w{@code{link-report.cgi}} replace the
normal ones @w{@code{fix-link}} and @w{@code{link-report}}.  They should be
interfaced through an HTML page which feeds the needed information to
@w{@code{link-report.cgi}}.

@cindex authentication
The main security question is how to do authentication of the user.
This will have to be set up using the features of the web server.   You
should not leave these programs available for non-authenticated users
since that would give them the ability to edit your web pages directly
and probably do worse.  

@node Interfaces, Emacs, Using LinkController, Top
@chapter Interfacing to other programs
@cindex Interfaces to other programs
@pindex extract-links@r{, importing links}
@pindex link-report@r{, exporting results}
@cindex importing links from other programs
@cindex exporting test results to other programs

Probably not all of your links are directly in web pages.  If this is
the case, it's still possible to use LinkController to check those
links, but it won't be possible to use the repair facilities.

In this case, you have to generate the list of URIs you want checked
yourself.  This should be a file with one URI per line.  Then
@w{@code{extract-links}} can be used to import those links into
LinkControllers database.  For example, if you had put those links into
the file @samp{links} the following command would import them.

@example
extract-links --in-url-list=links
@end example

Now, when you want to report on your links you can give the links file
as an argument to link-report and it will only report those links which
are in your file.  This can be done with the following command

@example
link-report --url-file=links
@end example

The usual options can be given to control which links are reported
for example @samp{--all-links} to list all links 
(@pxref{Invoking link-report}).

Another possibility for interfacing to programs is to use output from
LinkController to automatically remove links from your web pages.  That
would be a very suitable solution, for example, if you keep a list of
links to other related pages, but don't mind if some of them disappear
temporarily.  

In this case, it's probably best to use the @w{@code{link-report}}
option for machine oriented output @samp{--uri-report} and to choose
either the @samp{--broken} report for deleting links or the
@samp{--good}@footnote{Note: this option doesn't output links which
can't be checked.} option to choose which links should be shown on your
web pages.  For example, run something like the following each night
from your @samp{.crontab} file.

@example
link-report --url-file=links --uri-report --broken \
   | automatic-link-deleter
@end example

You should probably mail someone with the information that the link has
been deleted so that if there's an easy way to fix it they can do that.

@node Emacs, Administration, Interfaces, Top
@chapter The Emacs Interface
@cindex Emacs interface
@cindex interface, Emacs

LinkController's reporting system is designed to be independent of the
interface to it, and often the shell interface will be all that is
needed.  However another convenient interface is through
@w{@code{emacs}}.  There are two parts to this integration.

@menu
* link-report-dired::           An Emacs program to display broken links.
* check-page in Emacs::         Finding broken links in a file.
@end menu

@node link-report-dired, check-page in Emacs, Emacs, Emacs
@section Finding Files with Broken Links
@cindex links, examining, in Emacs
@cindex broken links, finding in Emacs
@pindex link-report-dired

There is a special Emacs mode called @w{@code{link-report-dired}}
written for locating files with broken links.  The mode is based on
@w{@code{find-dired}} and works very similarly.  It runs the program
@w{@code{link-report}} with an option which makes it list file names in
the same way as the @code{ls} program does.  The user can then move
around the buffer as normal in Emacs and enter files using a single key
press (normally @kbd{f}).

@node check-page in Emacs,  , link-report-dired, Emacs
@section Finding Broken Links in Files Within Emacs
@pindex check-page@r{, in emacs}

The program @w{@code{check-page}} was specially designed so that it outputs
in a format which can be read by Emacs' @w{@code{compile}} mode.  You can
use it within Emacs and then step from error to error correcting them.

To do this, after you have set up your system and run
@w{@samp{test-link}} a few times.  checking use the command @kbd{M-x}
@kbd{compile} @kbd{RET} @kbd{check-page filename} @kbd{RET} .  You will
now see another buffer open up with all of the errors shown there.  You
can use the key @kbd{M-`} (that's a real back quote, not an apostrophe)
to step between errors.

The one problem with @w{@samp{check-page}} is that if you have just
created a file containing new links it should really verify them by
testing each one.  This makes it more suitable for use during link
correction of existing pages than during writing new pages.  

@node Administration, Robot Behaviour, Emacs, Top
@chapter Administration
@cindex administration

There are various aspects of administration.  This is mostly related to
testing links.  

@menu
* Setting up LinkController::   How to get the system installed
* Default Installation::        A simple multi-user installation
* User Administration::         Adding and removing users
* Cron Scripts::                Running programs automatically
* Link Database Maintenance::   Dealing with database problems
* Link Ageing::                 Clearing outdated data from the databases.
@end menu

@node Setting up LinkController, Default Installation, Administration, Administration
@section Setting up LinkController
@cindex installation
@cindex configuration, installation
@pindex default-install@r{, using}

This chapter is aimed at administrators setting up LinkController or who
want to have a better understanding of the way that their installation
is set up.  

The first stage is to actually build and install the programs.  This is
covered in the document @file{INSTALL} which is included with the
distribution.

Once you have installed the software, the next step is to configure
LinkController so that it knows where you have all of your data.  The
program @code{default-install} provides one model of this.  

@node Default Installation, User Administration, Setting up LinkController, Administration
@section Default Installation
@cindex installation, basic multi user
@pindex default-install@r{, usage}

Running @code{default-install -all} should set everything up
correctly.  There are various variations on this command which do
different things, but the summary is 

@itemize @bullet
@item
Create a linkcont user-id and group which will be used for running programs
@item
Create a working directory where LinkController will keep it's data
@item
Create configuration files, especially /etc/link-control.pl
@item
Create cron scripts which will run LinkController automatically.
@end itemize

Using this command it is also possible to activate users and groups
e.g. @code{default-install -user @var{username}} or  
@code{default-install -group @var{groupname}} in which case the specified
users will become a member of the @code{linkcont} group.  


@node User Administration, Cron Scripts, Default Installation, Administration
@section User Administration
@pindex copy-links-from-users@r{, usage}

User administration is really only needed if you are running link
testing centrally for your users.  This makes sense since it means that
if several users have a link to the same place (likely in any given
site) then you will only have to check that link once.  

In this case, the important question is which links are copied into the
checking database.  This is controlled by the program
@w{@code{copy-links-from-users}} and decides copies data from users which
are in the @code{lnctusr} group.  

The command @w{@code{default-install}} can be used to manipulate which
users are in the group e.g. @code{default-install --user @var{username}} or  
@code{default-install -group @var{groupname}} in which case the specified
users will become a member of the lcntusr group.

Another form of user administration is limitation on which users have
access to the database.  This can be done with normal file permissions.
There isn't any specific control to stop users from seeing which links
other users have put into the database.  

@node Cron Scripts, Link Database Maintenance, User Administration, Administration
@section Cron Scripts
@cindex cron scripts for multi user use
@cindex automatic testing

In order to be effective, link testing should be done every day.
Furthermore, it is a good idea to do the testing at low usage times,
which normally means at night.  For this reason normally a cron script
will be used.  

@itemize @bullet
@item 
copy the links from each user with @code{copy-links-from-users} 
@item 
add them to the database with @code{extract-links --in-url-list} 
@item 
build a schedule for testing the links with @code{build-schedule} 
@item 
test the links with @code{test-link} 
@end itemize

The program @w{@code{default-install}} can create these scripts
@ref{Default Installation}.

@node Link Database Maintenance, Link Ageing, Cron Scripts, Administration
@section Link Database Maintenance

For the most part the link database shouldn't need much maintenance.
There are a number of cases where it might, however.  If it becomes
corrupt, you may try the db_recover command.  Probably, however, it's
just better to recover the database and link checking schedule from a
recent backup.  Time is not really critical since the work is normally
easy to regenerate.  You should make sure that link checking doesn't run
in at the time you do backups, however.

The other thing is that occasionally you may want to recover space in
the link database by dumping and un-dumping it.  See the Berkeley database
documentation for more details.

@node Link Ageing,  , Link Database Maintenance, Administration
@section Link Ageing
@cindex link ageing
@cindex database cleaning, automatic
@pindex test-link@r{, link ageing in}

Sometimes we have a link which is no longer in use within our
infostructure.  However, it's not a good idea to throw away
information about it immediately.  It could be that certain files have
been temporarily deleted and will come back.  Alternatively, a link
could have been found broken and been corrected, but someone has a
copy of the old page.  When they re-install the copy, we will have to
deal with that link again.  

If we had not kept that link around and they immediately do a link check
on their document they may see nothing wrong, and, because of the nature
of LinkController, we won't start reporting the link as a problem until
some days later, when we have confirmed that it really is broken.

If, on the other hand, we keep checking all of the links which have ever
been in our web pages, we will cause ourselves considerable extra work.  

To handle this, links are aged.  Once a link has reached greater than a
certain age, @w{@code{test-link}} will not be checked it any more.  Once
the link has reached a much larger age, it will be completely deleted
from the link database.  The age is reset to nothing each time the link
is extracted from the infostructure, so links which are still in use
will continue to be checked.

In the meantime, the link will still be repeatedly scheduled based to
it's normal checking time.  This causes us to examine it quite
regularly, but that is okay, since we won't actually check it.

By default links which have not been refreshed will be ignored after
one week and will be deleted from the database after two months.  

@node Robot Behaviour, Uncheckable Links, Administration, Top
@appendix Robots and Sensible Behaviour
@cindex robots
@cindex dangers
@cindex bandwidth usgae

The most important thing about a program like this is to realise that if
you set it up incorrectly and used it in the wrong way, you could upset
a large number of people who have set up their web servers in the
assumption that they would be used normally by human beings browsing
through on Netscape.

It is true that LinkController is very careful to limit resource usage
on remote sites, but the other site may not know that or may have a real
reason not to want their pages visited too often.  

Probably it's true that the only safe way forward is for every WWW site
to begin to set up robot defences and detect when someone starts to
download from them at an unreasonable rate and then cut off the person
doing the downloading.  I suggest that you don't make people have to do
this to protect themselves against you for at least two reasons.

@itemize @bullet
@item 
respect for the person's time
@item 
a wish not to be the person who is cut off
@end itemize

There are probably many other reasons, but that's one for the good side
in you and one for the selfish.  What more do you need.

For suggestions about what constitutes `correct' behaviour, it's worth
seeing the Robots World Wide Web page.
@url{http://www.robotstxt.org/wc/robots.html}

There are a number of points which make LinkController relatively safe
as a link.  These are all related to the design and limitations on
@w{@code{test-link}}.  

@itemize @bullet
@item 
@w{@code{test-link}} does @emph{not} recurs.  It only tests links that
are specifically listed in the schedule database.
@item 
There is a limit to the number of links that will be tested in one run.
This defaults to 1000, but can be configured.  
@item
The schedule for link testing is designed to spread the testing of links
across time
@item
The testing system will not test links at a given site faster than a
certain rate.
@end itemize

The last limitation is inherited from the @code{LWP::RobotUA} module and
the documentation for that covers the details of how it works.
@w{@code{test-link}} tries to re-order testing of links as needed so that
a limit on the rate of visits to one site does not cause a limit on
overall testing speed.

@node Uncheckable Links, Absolute and Relative URIs, Robot Behaviour, Top
@appendix Uncheckable Links
@cindex uncheckable links
@cindex link, uncheckable
@cindex mailto, can't be checked
@cindex news, can't be checked

Some links can't be checked because the target url doesn't have an easy
method of verification or because the link checker doesn't have the
facilities needed for verification.  Examples of this include
@samp{mailto} and @samp{news} URLs.

Although it's possible to verify, to a certain degree, many mail
addresses the only absolute way to check that a mail address reaches
the person it's meant to reach is to send a mail and ask for them to
reply.  Obviously, if everybody checking every link in the world
started to do this some unlucky recipients would get very upset at
being bombarded with so much mail.  

A low level of verification could be done in some circumstances.  This
requires that the persons actual mail server (the place where their mail
is kept) can be contacted directly and is willing to be helpful.  This
will be implemented in a later version of LinkController.

Checking news urls (not possible at the present moment anyway) requires
access to a @emph{correctly set up} news server which has the feed for
that newsgroup.  Even then, it doesn't talk about the access for the end
user.

Other links cannot be checked because @code{libwww-perl} doesn't yet
support them.  In this case the solution is to add support to
@code{libwww-perl}.

@node Absolute and Relative URIs, Bugs, Uncheckable Links, Top
@appendix Absolute and Relative URIs
@pindex fix-link@r{, relative link support}

LinkController is designed to handle absolute and relative URIs in a
consistent but sensible fasion, but unfortunately there isn't any
totally clear correct way.  For link extraction we simply conver
relative URLs to absolute form.  For link testing, this means that we
don't ever think of relative URLs.  For link fixing on the other hand
the situation is more complex.  For this reason there is a
@code{--relative} option to @code{fix-link}.  

If we run @code{fix-link} without the @code{--relative} option then we
only substitute absolute links in the existing document to the link
given on the command line.  This is safer because the subsitution is
unlikely to mistake other strings which accidentally match the link.

If we run @code{fix-link} with the @code{--relative} option on the other
hand then we will handle relative links.  What this means depends on
whether the links to be fixed can be expressed as links relative to the
pages being fixed.  

If the original (broken) link can be expressed as a relative link then
we will do substitutions where we find relative links.  If the target
(corrected) link can be expressed as a relative link then we will always
substitute broken links with a relative link.  

Taken together, this means that if a resource has moved to the same
server as your pages, substitution with the @code{--relative} option
will correction convert all of your absolute links into relative links
and if a resource has moved from your server to another then we will
correctly substitute relative links with absolute links. 

The only undesirable effect would be if a resource is moved within your
pages and you have a mixture of relative and absolute links to that
resource (e.g. for absolute links on page which is mirrored on other
sites).  In this case, first do substitution without the
@code{--relative} option and then afterwards with the @code{--relative}
option.

@node Bugs, History, Absolute and Relative URIs, Top
@appendix Bugs and bug reporting
@cindex bugs, reporting
@cindex reporting bugs

This version of LinkController is still in early development.  There are
many changes to come.  Undoubtedly there are many bugs in the software
already and will soon be more.

A bug is when 

@itemize @bullet
@item 
the software doesn't do something the documentation says it should
@item 
the software does something the documentation says it shouldn't
@item 
the software does something surprising and that isn't documented
@item 
the software does something strange but the documentation doesn't
explain why
@item 
it is difficult or impossible to understand what the documentation is
trying to say
@end itemize

some of these mean fixing the documentation and some the software.  All
of them are bugs and should be reported and fixed.  

If you find a bug, I will be grateful to hear about it.  Even if you
don't know how to fix it or anything, it is useful to know what is wrong
so that other people don't get caught out but @emph{read the BUGS file
first} please.  If the bug is listed there then the only useful thing
that you can do is fix it.  If you do this and contribute it to me then
that is very useful.

When you report a bug, please tell me what release of link controller
you were using.  This is the number which was in the name of the file
that LinkController came in.  If your problem was with a specific
program, please also run @samp{program --version} and send the output.
This tells me exactly which version of that program you were running.

Since this is a developers release, I'd hope most users would be able to
make some level of fixes.  If you do this, send me context differences
(use @samp{diff -u} if it works or try @samp{diff -c} otherwise).  I use
CVS, so as long as I know which version you have I will be able to find
the original file and see your changes.  However it's also important to
explain them because I won't be able to use them unless I (relatively
stupid computer type) understand them.

Send bug reports to the address you get by changing words into
punctuation in the following.

@example
link minus controller at scotclimb dot org dot uk
@end example

This mailing address is sent only to me right now, but may become a list
in future.  Use my (Michael De La Rue) personal address to contact me
please.  N.B. I am @strong{extremely} inefficient about answering email.
Don't worry if you don't get a reply.

	The ideas, and history

@node History, Invoking the Programs, Bugs, Top
@appendix History
@cindex history MOMspider

LinkController was originally inspired by MOMspider and having the
MOMspider code available was very useful when starting the creation of
this kit, but, it shares almost no code with MOMspider, other than what
has comes to it from the LibWWW-Perl library.

Philosophically, the MOMspider heritage is obvious in the wish to handle
big jobs efficiently.  In the working practice there are far more
differences than similarities, partly caused by Perl language changes.

I decided to completely separate the exploration of the local
infostructure, looking for links to be checked, from the actual checking
process.  This means that checking can be spread over a large number of
days and still run efficiently. 

The basic aim of this link checking kit is to be able to efficiently
handle any size of link checking job.  At the bottom end we have
checking new pages as they are written.  Here we want to use
information from previous checks to avoid having to check all of each
page every time.  At the other end we have massive info structures
(sites) which deal in many thousands of links and could not possibly
all be checked in one day.  For this latter case the aim is to be able
to efficiently spread the link checking load into all available low
usage periods.

My primary aim in writing this was not to write very efficient code for
the small scale case (takes minimum time to do everything), but rather
code which would scale well.  If your system can check 1000 links in two
days, it will hopefully be able to check almost 7000 links in two weeks.
I'm trying to make sure all data structures which grow with the number
of links are kept on disk.

@menu
* Acknowledgements ::           People and or institutions who helped
@end menu

@node Acknowledgements ,  , History, History
@section Acknowledgements
@cindex acknowledgements

Although I wrote this system by myself, this would not have been nearly
as easy and almost certainly wouldn't have ever been finished without
the help of the following people and organisations.

@menu
* Esoterica Internet Portugal::  
* IPPT PAN Poland::             
* The Tardis Project::          
* Other Free Software Authors::  
@end menu

@node Esoterica Internet Portugal, IPPT PAN Poland, Acknowledgements , Acknowledgements
@unnumberedsubsec Esoterica Internet Portugal

@c I'd like to have these people's names correct but I'm not sure how to
@c do that in texinfo.. oh well.

Esoterica provided me with full access to the Internet in Portugal and
use of their computers for free which allowed me to keep up on both this
software and the Linux Access HOWTO.  In particular I'd like to thank
all of the members of staff who helped me very much.  These people include
@c M\'ario Francisco Valente
Mario Francisco Valente (the instigator of Mini Linux) who first
agreed to me using their kit, set me up to use their machines, and along
with 
@c Lu\'\i{}s Sequeira
Luis Sequeira provided a sounding board for some ideas.
@c Lu\'\i{}s 
Luis also provided the odd lift home in the evening.  Also 
@c Martim de Magalhães
Martim de Magalhaes Pereira and Mr Mendes.  See them all on 

@url{http://www.esoterica.pt/esoterica/quemsomos.html}

For more about esoterica (Internet Services in Portugal) see:

@url{http://www.esoterica.pt/esoterica/}

These pages are in Portugese@footnote{Whilst the above names are mangled
here.  See the correct versions in the original texinfo or on the Web
pages.} of course.

@node IPPT PAN Poland, The Tardis Project, Esoterica Internet Portugal, Acknowledgements
@unnumberedsubsec IPPT PAN Poland

Thanks go to IPPT PAN (part of PAN - Polska Akademia Naukowa) in Poland
and in particular Piotr Pogorzelski who allowed me use of facilities for
testing this software, provided a willing victim for having his web
pages tested and made a number of suggestions which have been
incorporated into the software.

@node The Tardis Project, Other Free Software Authors, IPPT PAN Poland, Acknowledgements
@unnumberedsubsec The Tardis Project

Supported by the Computing Science department of the University of
Edinburgh, the Tardis project provides an experimental framework in
which students, former students and other related people to do their own
work on fully Internet connected Unix and Linux hosts.

The use of the facilities of the Tardis Project has made it much easier
for me to develop software like this.  In particular, the large amount
of disk space the administrators have allow me to use is very useful.

@node Other Free Software Authors,  , The Tardis Project, Acknowledgements
@unnumberedsubsec Other Free Software Authors

It is through the software provided by the Free Software Foundation
(such as the @w{@code{gcc}} C compiler, Emacs, the file utilities), the
authors of the various packages which make up a working Linux System
(Linux by Linus Torvalds, Alan Cox, etc.... filesystems and support by
Theodore Tytso, Stefan Tweedie etc.. Linux-Libc by HJ Lu, based on GNU
@w{@code{glibc}} from the FSF.. the list is indefinite) and the authors
of Perl and its modules, especially Gisle Aas and Martijn Kostler for
LibWWW-Perl that I was able to set this up.

I'd particularly like to thank Tim Goodwin the author of the Perl CDB
module who made and accepted a number of alterations to that, at my
request.  These alterations made this package simpler to write and
easier to maintain.

The Free Software Foundation web pages are at

@url{http://www.gnu.ai.mit.edu/}

@node Invoking the Programs, Related Packages, History, Top
@appendix Invoking the LinkController Programs
@cindex command line options
@cindex usage of programs

Because they use the Perl @code{Getopt::Mixed} module, all of the
LinkController command line programs respond to the standard POSIX style
command line options.  At least the following two options will be implemented. 

@table @samp
@item --help
This option will give a list of all of the options understood by the
program along with brief explanations of what they do.  
@item --version
This option will give some version information for the program.
@end table

You can use the @samp{--help} option to get help on each program, for
example:

@example
extract-links --help
@end example

You can then use that information to get the program to do what you
want.

@menu
* Invoking link-report::        link-report usage summary
* Invoking test-link::          test-link usage summary
* Invoking extract-links::      extract-links usage summary
* Invoking fix-link::           fix-link usage summary
* Invoking check-page::         check-page usage summary
* Invoking build-schedule::     build-schedule usage summary
@end menu

@node Invoking link-report, Invoking test-link, Invoking the Programs, Invoking the Programs
@appendixsec Invoking link-report
@cindex invoking link-report
@pindex link-report@r{, invocation}

The @samp{link-report} program prints out status information about links
allowing the user to see what needs to be fixed.  The default is to
print out all of the broken and redirected links that currently occur on
the users web pages and which are either redirected or broken.  

Before running @samp{link-report} you should probably use
@w{@code{test-link}} (@pxref{Invoking test-link}) to check which links
are broken.  That may not be needed if your system administrator does it
for you.  After you have identified broken links you may want to use
@w{@code{fix-link}} (@pxref{Invoking fix-link}) to repair the broken
links.

The primary configuration file used by @w{@code{link-report}} is the
@samp{.link-control.pl} file.  This tells it where the schedule file and
LinkController database are. @xref{Setting Configuration Variables}, for
how to control the contents of this file.

In the case of the @samp{--long-list} report, a second configuration
file, the @samp{infostrucs} file, is used.  This contains the
information needed to know where to extract links from by default.
@xref{Infostructure Configuration}, for more details on configuring
this.

@strong{FIXME} this section should give a better description of each option.

@example
link-report [options]

 -V --version            Give version information for this program
 -h --help --usage       Describe usage of this program.
    --help-opt=OPTION    Give help information for a given option
 -v --verbose[=VERBOSITY] Give information about what the program is 
                         doing.  Set value to control what information
                         is given.

 -U --uri=URIs           Give URIs which are to be reported on.
 -f --uri-file=FILENAME  Read all URIs in a file (one URI per line).
 -E --uri-exclude=EXCLUDE RE Add a regular expressions for URIs to 
                         ignore.
 -I --uri-include=INCLUDE RE Give regular expression for URIs to check
                         (if this option is given others aren't 
                         checked).
 -e --page-exclude=EXCLUDE RE Add a regular expressions for pages to 
                         ignore.
 -i --page-include=INCLUDE RE Give regular expression for URIs to check
                         (if this option is given others aren't 
                         checked).

 -a --all-links          Report information about every URI.
 -b --broken             Report links which are considered broken.
 -n --not-perfect        Report any URI which wasn't okay at last test.
 -r --redirected         Report links which are redirected.
 -o --okay               Report links which have been tested okay.
 -d --disallowed         Report links for which testing isn't allowed.
 -u --unsupported        Report links which we don't know how to test.
 -m --ignore-missing     Don't complain about links which aren't in the
                         database.
 -g --good               Report links which are probably worth listing.

 -N --no-pages           Report without page list.
    --config-file=FILENAME Load in an additional configuration file
    --link-index=FILENAME Use the given file as the index of which file
                         has what link.
    --link-database=FILENAME Use the given file as the dbm containing 
                         links.

 -l --long-list          Where possible, identify the file and long 
                         list it (implies infostructure).  This is used
                         for emacs link-report-dired.
 -R --uri-report         Print URIs on separate lines for each link.
 -H --html               Report status of links in html format.
@end example

@node Invoking test-link, Invoking extract-links, Invoking link-report, Invoking the Programs
@appendixsec Invoking test-link
@cindex invoking test-link
@pindex test-link@r{, invocation}

The @w{@code{test-link}} program tests all of the links in the
LinkController database storing information about any problems found.
It works as a robot contacting the servers where the target of each link
is stored and verifying that the resource the link points to is really there.

Before running @w{@code{test-link}} you should probably use
@w{@code{extract-links}} (@pxref{Invoking extract-links}) to collect all of
the links you want to test and then @w{@code{build-schedule}}
(@pxref{Invoking build-schedule}).

The configuration file used by @w{@code{test-link}} is the
@samp{.link-control.pl} file.  This tells it where the schedule file and
LinkController database are. @xref{Setting Configuration Variables}, for
how to control the contents of this file.


@strong{FIXME} this section should give a better description of each option.

@example
test-link [arguments]

 -V --version            Give version information for this program
 -h --help --usage       Describe usage of this program.
    --help-opt=OPTION    Give help information for a given option
 -v --verbose[=VERBOSITY] Give information about what the program is
                         doing.  Set value to control what information
                         is given.
 --quite -q --silent     Program should generate no output except in
                         case of error.
    --no-warn            Avoid issuing warnings about non-fatal 
                         problems.

 -c --config-file=FILENAME Load in an additional configuration file
 -u --user-address=STRING Email address for user running link testing.
 -H --halt-time=MINUTES  stop after given number of minutes

    --never-stop         keep running without stopping
    --no-robot           Don't follow robot rules.  Dangerous!!!
 -w --no-waitre=NETLOC-REGEX Home HOST regex: no robot rules.. 
                         (danger?)!!!
    --test-now           Test links now not when scheduled (testing 
                         only)
    --untested           Test all links which have not been tested.
    --sequential         Put links into schedule in order tested (for 
                         testing)
 -H --halt-time=MINUTES  stop after given number of minutes
 -m --max-links=INTEGER  Maximum number of links to test (-1=no limit)
@end example

Several of the options could potentially lead to overloading networks
and even other people's computer systems:

Don't use --no-robot, except for when you are doing local
testing (that is, you aren't connected to the internet proper).

Don't use --never-stop or --test-now except when you are watching what
is happening.

Generally you should be somewhat careful about running this program
since it does automatically connect to other servers on the internet.
Reasonable care has been taken to ensure it does this in a responsible
way, but you must make sure that anybody who is inconvenienced has a
good route for communicating this problem back to you.

@node Invoking extract-links, Invoking fix-link, Invoking test-link, Invoking the Programs
@appendixsec Invoking extract-links
@cindex invoking extract-links
@pindex extract-links@r{, invocation}

The @w{@code{extract-links}} program walks through the users web pages
collecting all of the links from those pages and storing them into a
database for later checking by the @w{@code{test-link}} program
(@pxref{Invoking test-link}).  It can also list the links found into a
given file.

After running @w{@code{extract-links}} you should use @w{@code{build-schedule}}
(@pxref{Invoking build-schedule}) which will make sure that any new
links discovered are scheduled for checking..

There are two configuration files used by @w{@code{extract-links}}.  The
@samp{.link-control.pl} file is the first.  This tells it where the
various files it uses are.  @xref{Setting Configuration Variables}, for
how to control the contents of this file.  The second file is the
@samp{infostrucs} file.  This contains the information needed to know
where to extract links from by default.  
@xref{Infostructure Configuration}, for more details on configuring this.

@strong{FIXME} this section should give a better description of each option.

@example
extract-links [arguments] [url-base [file-base]]

 -V --version            Give version information for this program
 -h --help --usage       Describe usage of this program.
    --help-opt=OPTION    Give help information for a given option
 -v --verbose[=VERBOSITY] Give information about what the program is
                         doing.  Set value to control what information
                         is given.
 --quiet -q --silent     Program should generate no output except in
                         case of error.

 -e --exclude-regex=REGEX Exclude expression for excluding files.
 -p --prune-regex=REGEX  Regular expression for excluding entire 
                         directories.
 -d --default-infostrucs handle all default infostrucs (as well as ones 
                         listed on command line)

 -l --link-database=FILENAME Database to create link records into.
 -c --config-file=FILENAME Load in an additional configuration file

 -o --out-url-list=FILENAME File to output the URL of each link found to
 -i --in-url-list=FILENAME File to input URLs from to create links
@end example

@node Invoking fix-link, Invoking check-page, Invoking extract-links, Invoking the Programs
@appendixsec Invoking fix-link
@cindex invoking fix-link
@pindex fix-link@r{, invocation}

The @w{@code{fix-link}} program is designed to repair a broken links across
all of the files which LinkController is managing.  It does this by
looking up index files and seeing files contain the broken link then
doing a textual substitution in each of these files.  This makes it much
faster than searching through all of the files in a set of web pages to
see which pages have the broken link.

In order to work properly, @w{@code{extract-links}} 
(@pxref{Invoking extract-links}) must have been run first to build up
the index databases used by @w{@code{fix-link}}.

There are two configuration files used by @w{@code{fix-link}}.  The
file @samp{.link-control.pl} is the first.  This tells it where the
other configuration file and index files are.  @xref{Setting Configuration
Variables}, for how to control the contents of this file.  The second
file is the @samp{infostrucs} file.  This contains the information
needed to relate broken links to the files which need to be repaired.
@xref{Infostructure Configuration}, for more details on confiuguring
this.

@example
fix-link [options] old-link new-link

 -V --version            Give version information for this program
 -h --help --usage       Describe usage of this program.
    --help-opt=OPTION    Give help information for a given option
 -v --verbose[=VERBOSITY] Give information about what the program is 
                         doing. Set value to control what information is 
                         given.
 -q --quiet --silent     Program should generate no output except in
                         case of error.
    --no-warn            Avoid issuing warnings about non-fatal 
                         problems.

    --directory=DIRNAME  correct all files in the given directory.

 -r --relative           Fix relative links (expensive??).
 -t --tree               Fix the link and any others based on it.
 -b --base=FILENAME      Base URI of the document or directory to be 
                         fixed.

    --config-file=FILENAME Load in an additional configuration file
@end example

@node Invoking check-page, Invoking build-schedule, Invoking fix-link, Invoking the Programs
@appendixsec Invoking check-page
@cindex invoking check-page
@pindex check-page@r{, invocation}

Check page is useful where broken links in files need to be manually
corrected.  It outputs a list of line numbers where interesting links
occur allowing the user to find those lines and correct the mistakes.
The output format is compatible with the @code{emacs} @code{compile}
mode which allows fast access to the problem locations.

There are two configuration files used by @w{@code{extract-links}}.  The
file @samp{.link-control.pl} is the first.  This tells it where the link
database is.  @xref{Setting Configuration Variables}, for how to control
the contents of this file.  The second file is the @samp{infostrucs}
file.  This allows @w{@code{check-page}} to know what the base URI of
the file being checked is and so check relative links within the page
corectly.  @xref{Infostructure Configuration}, for more details on
configuring this.

@example
check-page [options] filename...

 -V --version            Give version information for this program
 -h --help --usage       Describe usage of this program.
    --help-opt=OPTION    Give help information for a given option
 -v --verbose[=VERBOSITY] Give information about what the program is 
                         doing.  Set value to control what information 
                         is given.

 -r --redirect           Report links which are redirected.
 -m --ignore-missing     Don't complain about links which aren't in 
                         database.

    --link-index=FILENAME Use the given file as the index of which 
                         file has what link.
    --link-database=FILENAME Use the given file as the dbm containing
                         links.
@end example

@node Invoking build-schedule,  , Invoking check-page, Invoking the Programs
@appendixsec Invoking build-schedule
@cindex invoking build-schedule
@pindex build-schedule@r{, invocation}

The @w{@code{build-schedule}} program makes a schedule for testing links.
If run with no options it will make sure that all the links in the
LinkController database will be checked at some point in the future.

Before running @w{@code{build-schedule}} you should probably use
@w{@code{extract-links}} (@pxref{Invoking extract-links}) to collect all of
the links you want to test.  Afterwards you should use @w{@code{test-link}}
to check which ones are broken (@pxref{Invoking test-link}).

The configuration file used by @w{@code{build-schedule}} is the
@samp{.link-control.pl} file.  This tells it where the schedule file and
LinkController database are. @xref{Setting Configuration Variables}, for
how to control the contents of this file.

@example
build-schedule [options]

 -V --version            Give version information for this program
 -h --help --usage       Describe usage of this program.
    --help-opt=OPTION    Give help information for a given option
 -v --verbose[=VERBOSITY] Give information about what the program is
                         doing.  Set value to control what information 
                         is given.
 --quite -q --silent     Program should generate no output except in
                         case of error.
    --no-warn            Avoid issuing warnings about non-fatal 
                         problems.

 -l --url-list=FILENAME  File with complete list of URLs to schedule
 -s --schedule=FILENAME  Override location of the schedule
 -t --spread-time=SECONDS Time over which to spread checking; default 10 
                        days
 -S --start-offset=SECONDS Time offset from now for starting work (can
                         be negative)
 -d --ignore-db          Set the time with no regard to curent setting
 -i --ignore-link        Set the time with no regard to link status
    --no-warn            Avoid issuing warnings about non-fatal 
                         problems.
    --config-file=FILENAME Load in an additional configuration file
@end example

@node Related Packages, Terms, Invoking the Programs, Top
@appendix Packages Which Work With LinkController

LinkController uses several programs and can work with several others.
This section covers the most important ones.

@menu
* cdb::                         Utilities for the LinkController indexes.
* Tie-Transact-Hash::           Berkeley DB editing tools.
@end menu

@node cdb, Tie-Transact-Hash, Related Packages, Related Packages
@appendixsec The CDB utilities
@cindex cdb files
@cindex database format, cdb

In order to have LinkController working you must have installed these.
It is worth looking at the utilities that are provided, especially
@code{cdbdump} which will let you look at the contents of the file.  You
should be aware that @code{cdbget} program which is provided
@emph{won't} be able to get at the full contents of the index files
since they contain repeated keys.

More information on cdb and new releases can be got from the www page.

@url{http://cr.yp.to/cdb.html}

When using link controller you are advised to use FreeCDB which, because
of its better license terms, has the extra guarantee that it will be
possible for anybody to distribute fixed versions and provide support
for them.

@url{http://packages.debian.org/freecdb}

@node Tie-Transact-Hash,  , cdb, Related Packages
@section The Tie-Transact-Hash Perl Module and Programmes
@cindex database, editing
@cindex editing the links database

This is a Perl module written by myself which includes a program which
allows direct examination and editing of Berkeley databases.  It can be
useful for debugging and correcting problems in the LinkController Link
database or schedule file.

Tie::TransactHash can be downloaded from CPAN, the Comprehensive Perl
Archive Network get there via:

@url{http://www.perl.com/language/info/software.html}

@node Terms, Names Index, Related Packages, Top
@appendix Terms

@menu
* Infostructure::               Groups of resources.
* Link::                        A connection between two resources.
* Resource::                    The information on the World Wide Web.
* URIs::                        A type of link including URLs
* URLs::                        The connections in the World Wide Web.
* URNs::                        Names for resources without location.
@end menu

@node Infostructure, Link, Terms, Terms
@appendixsec Infostructure
@cindex infostructure
@cindex web pages, groups of

An infostructure is a concept which was introduced in Link Checking in
the MOMspider package.  It is a collection of related resources.  For us
it's mostly just a way of saying `web pages' but includes things like
databases which may not have any real identifiable `pages' that we can
read through directly.

@node Link, Resource, Infostructure, Terms
@appendixsec Link
@cindex link, definition

The term link in LinkController is used for a connection between two
resources.  It's existence really comes from the `class' or piece of
type of computer data which is used to store information about `links'.
Properties of a link include:

@node Resource, URIs, Link, Terms
@appendixsec Resource
@cindex resource

A resource is almost anything.  `It' can range from a person to an HTML
file to a computer to a database or presumably eventually to phone
numbers, possibly physical hardware.  This generality is a very
important concept for the World Wide Web.  Really the key thing about a
resource is that it can be `identified'.  @xref{URLs}, for more
details.

@node URIs, URLs, Resource, Terms
@appendixsec URIs
@cindex URI, definition
@cindex URL, relation to URI
@cindex URI, relation to URL

A URI or `Uniform Resource Identifier' is a more generic form of the URL
@ref{URLs} which also includes URNs @ref{URNs}.  It also allows links to
abstract objects which can't be reached through a network server.  Since
all URLs are URIs we mostly try to talk about URIs when we can since
that includes both.  Often people say URL when they mean URI.  We try to
use correct usage always so that in future we can support all forms of
URI without confusing existing users.  URIs and URLs are defined in RFC
2396.

@node URLs, URNs, URIs, Terms
@appendixsec URLs
@cindex URL, definition
@cindex WWW

A URL or `Uniform Resource Locator' are the essence of the World Wide
Web.  Approximately, they are addresses through which `resources' can be
located.  The idea is that almost anything can be given some kind of
address in a form that a machine can work with.  By defining a set of
rules, this can then be converted into a URL.  A URL has two parts.  The
first tells us what rules to use and the second tells us what the
address is.  URLs and URIs are defined in RFC 2396.  URLs are not the
only kind of link, but they are the most common and currently the only
ones LinkController really handles well.

@node URNs,  , URLs, Terms
@appendixsec URIs
@cindex URN, definition
@cindex URN, relation to URI
@cindex URN, relation to URL

A URN or `Uniform Resource Name' is a URI @ref{URIs} which is not a URL
@ref{URLs}.  This means a way of specifying a resource without saying
how to get it.  For example, a scheme which has been considered is for
ISBN (International Standardised Book Numbers) numbers.  This would
allow us to specify a book as a resource but wouldn't tell us how to get
it.

It's not totally clear where these will be useful in link checking (they
are used internally in several computer systems), but LinkController
intends to support them whenever needed, wherever possible.

@itemize @bullet
@item 
Knowing what the URL of the target resource of the connection is.
@item
Knowing whether the connection to the target resource has been working
recently.
@item
Knowing when the connection to the target resource was last checked.
@end itemize

Within the programs, a link is different from a URL in that it is
specifically aimed at checking connections, where a URL just specifies
what the connection should be if it is working.

@node Names Index, Concept Index, Terms, Top
@unnumbered Program, Variable and File Name Index

This index includes all programs, variables and files.

@printindex vr

@node Concept Index,  , Names Index, Top
@unnumbered Concept Index

@printindex cp

@bye
