\input texinfo   @c -*-texinfo-*-
@c %**start of header
@setfilename link-controller.info
@settitle Using LinkController
@setchapternewpage odd
@c %**end of header

@c not enough programs and variables to justify separate indexes.
@syncodeindex pg vr

@titlepage
@title Using LinkController
@author Michael De La Rue

@page
@vskip 0pt plus 1filll
Copyright @copyright{} 1997-2001 Michael De La Rue

Published by ...

Permission is granted to distribute and change this manual under the
terms of the GNU public license.

This is the alpha version of this manual and is very incomplete.
@end titlepage

@ifnottex
This file documents LinkController

Copyright 1997-2001 Michael De La Rue

Permission is granted to distribute and change this manual under the
terms of the GNU public license.

This is the alpha version of this manual and is very incomplete.

@node Top, Introduction, (dir), (dir)

This document describes LinkController a system for checking and
maintaining links in infostructures

This document applies to version 0.033 of LinkController
@end ifnottex

@menu
* Introduction::                What LinkController is about.
* Getting Started::             Configuration.
* Configuration::               
* Advanced Configuration::      
* Configuring CGI Programs::    
* Extracting Links::            Getting link information from WWW pages.
* Testing::                     Testing Links.
* Reporting::                   Getting information about broken links.
* Files::                       Checking individual HTML files.
* Repair::                      Replacing old URLs with new ones.
* Suggestions::                 Making suggestions for other users.
* CGI::                         The CGI interface (to be).
* Emacs::                       The Emacs interface.
* Setting up LinkController::   
* Administration::              
* Robots::                      General statements about robots.
* Link Database Maintenance::   
* Bugs::                        What to do if you find a bug.
* History ::                    
* Acknowledgements ::           People who helped with this package.
* Invoking the Programs::       How to run the programs in LinkController.
* Related Packages::            Packages used by or useful with LinkController.
* Terms::                       Glossary of terms used in this documentation.
* Names Index::                 Index of Program and Variable names
* Concept Index::               Index of Concepts.

@detailmenu
 --- The Detailed Node Listing ---

Configuration

* Automatic Configuration::     
* Setting Configuration Variables::  
* Link Variables::              General configuration variables.
* Infostructure Configuration::  

Advanced Configuration

* Advanced Infostructure Configuration::  
* Authorisation Configuration::  

Reporting Problems

* Email Reporting::             

The Emacs Interface

* link-report-dired::           An Emacs program to display broken links.
* check-page in Emacs::         Finding broken links in a file.

Setting up LinkController

* Default Installation::        

Administration

* Default Installation::        
* User Administration::         
* Cron Scripts::                

Link Database Maintenance

* Link Ageing::                 

Acknowledgements

* Esoterica Internet Portugal::  
* IPPT PAN Poland::             
* The Tardis Project::          
* Other Free Software Authors::  

Packages Which Can Be Useful with LinkController

* cdb::                         Utilities for the LinkController indexes.
* Tie-Transact-Hash::           Berkeley DB editing tools.

Terms

* Resource::                    The information on the World Wide Web.
* URLs::                        The connections in the World Wide Web.
* Infostructure::               Groups of resources.
* Link::                        A connection between two resources.

@end detailmenu
@end menu

@node Introduction, Getting Started, Top, Top
@unnumbered Introduction

LinkController is a system for checking links.

Most HTML pages contain references to other HTML pages (links).  These
allow the readers of those pages to locate other related resources (web
pages etc.).  Unfortunately the location of `resources'
@footnote{resource is a general term for HTML pages and all of the other
things that can be referenced by URLs} can change, resources can
disappear completely or the system providing the resource can break.
When this happens, the link which used to find them will no longer work.
The only reliable way to detect this problem is to periodically check
over the resources and take corrective action for the ones that have
gone missing.

LinkController is designed to make that task much more efficient.  It
automates the task of checking which links have been broken for a period
of time and then of finding which documents they occur in.

LinkController is copyrighted software and is distributed under the GNU
Public License which should have been included as the file
@file{COPYING} in the distribution.

@node Getting Started, Configuration, Introduction, Top
@chapter Getting Started 

This section of the manual assumes that the programs that make up LinkController are already
installed and working on your computer.  If not, then @xref{URLs}.  We
assume the standard setup where your system administrator runs the link
checking for you.  Other setups will need slightly different behaviour.
Speak to the person who set up link controller.

The first thing to do is to run @w{@code{configure-link-control}} to
configure the system.  This will ask a series of questions about your
configuration and create a configuration file which will be used by the
various programs which make up LinkController.

Next you have to work out which links you are interested in.  Do this by
extracting the links from your web pages (@pxref{Extracting Links}).
The output file from this with the list of links found will be stored
in the location you gave during configuration.  

Assuming that you have the default install, your links will be
automatically copied and checked over time following that.  

After a short time (about a day) you will begin to get information about
links which didn't work with @w{@code{link-report --not-perfect}}.  After
some more time (a week or so) you can use @w{@code{link-report}} to find
out which links are really broken.

@node Configuration, Advanced Configuration, Getting Started, Top
@chapter Configuration

@menu
* Automatic Configuration::     
* Setting Configuration Variables::  
* Link Variables::              General configuration variables.
* Infostructure Configuration::  
@end menu

@node Automatic Configuration, Setting Configuration Variables, Configuration, Configuration
@section Setting Configuration Variables

@cindex configuration
@cindex variables, setting
@cindex setting variables

The @w{@code{configure-link-control}} program can be used by users to
configure LinkController.  This will ask you a series of questions and
then generate a configuration in your home directory.

The configuration that is controlled by this program is related to
reporting and fixing links.  For other configuration see
@xref{Administration}.

@node Setting Configuration Variables, Link Variables, Automatic Configuration, Configuration
@section Setting Configuration Variables

@cindex configuration
@cindex variables, setting
@cindex setting variables

All of the variable information is stored in the file
@file{.link-control.pl} in your home directory or
@file{/etc/link-control.pl} for system wide configuration.  The
configuration files are written directly in Perl (the programming
language LinkController is written in).  You can set the configuration
variables by putting lines like this.

@example
$::links='/var/lib/link_database.bdbm';
@end example

Please note the semi colon at the end of the line and the use of single
quotes so that Perl doesn't do anything strange to your values.

@node Link Variables, Infostructure Configuration, Setting Configuration Variables, Configuration
@section Link Control Configuration Variables

@cindex variables, configuration
@cindex configuration variables

@table @code
@item $::user_address
@vindex $user_address
is the email address which the robot declares to the world as it
goes around checking links.  If you want to check links yourself, you
must set this to a valid email address, because if something goes badly
wrong, it is the only way for a user at another site to know how to
contact you.

@item $::base_dir
@vindex $base_dir
This is the base directory for all of the configuration files.  If this
variable is defined then the other variables will default as given
below and do not need to be set individually.

@item $::links
@vindex $links
tells you what file is being used to store information
about links.  This could easily be a shared database used by everyone on
your system.  Defaults to @file{$::base_dir/links.bdbm}.

@item $::schedule
@vindex $schedule
tells the system where to find the schedule file used to
decide which links should be checked next and when that should be.  You
will need to set this and create the file in order to do link checking. 
Defaults to @file{$::base_dir/schedule.bdbm}.

@item $::page_index
@vindex $page_index
tells the system where to find the schedule file used to
decide which links should be checked next and when that should be.  You
will need to set this and create the file in order to do link checking.
Defaults to @file{$::base_dir/page_has_link.cdb}.

@item $::link_index
@vindex $link_index
tells the system where to find the schedule file used to
decide which links should be checked next and when that should be.  You
will need to set this and create the file in order to do link checking.
Defaults to @file{$::base_dir/link_on_page.cdb}.

@item $::infostrucs
@vindex $infostrucs
This variable points to the configuration file where definitions of
infostructures are should be put @xref{Infostructure Configuration}.
Defaults to @file{$::base_dir/infostrucs}.

@item $::link_stat_log
@vindex $infostrucs
This variable is the name of a file where important link status changes
will be logged.  The current definition is links which have just been
discovered to be broken.  This can be used in email notification.  
@xref{Email Reporting}.
@end table

@node Infostructure Configuration,  , Link Variables, Configuration
@section Configuring Infostructures

@cindex configuration, infostructure

The infostructure configuration is used to find links within the pages
when we are building our databases.  It is kept in a separate file
defined by the @code{$::infostrucs} configuration variable.

The format of the file is one line for each infostructure with
configuration directives separated by spaces.  For example

@example
directory http://example.com/manual /var/www/html/manual
www http://example.com/strange_database
@end example

The first directive describes how @w{@code{extract-links}} program
should extract the links.  It currently has three possible values.  The
value @code{www} means to actually use the given URL to download the web
pages.  The value @code{directory} means that @w{@code{extract-links}}
should assume that all of the files are stored in a directory and that
the directory structure matches the structure of the infostructure.  The final value "advanced"

In the case where we use the @code{directory} directive, a third
directive is present on each line with the full path to the base
directory of the infostructure.

More advanced configuration is possible by defining the information
directly in Perl.  

@node Advanced Configuration, Configuring CGI Programs, Configuration, Top
@chapter Advanced Configuration

There are various advanced ways to configure LinkController.  These are
mostly not needed for simple checking of a small collection of web
pages.  For larger sites and special situations however, they may well
make life much easier.

@menu
* Advanced Infostructure Configuration::  
* Authorisation Configuration::  
@end menu

@node Advanced Infostructure Configuration, Authorisation Configuration, Advanced Configuration, Advanced Configuration
@section Advanced Infostructure Configuration
@cindex infostructure, configuration
@cindex regular expression, exclude
@cindex regular expression, include
@cindex filtering links

Using more advanced configuration it is possible to skip over certain
resources when we are doing link extraction and to ignore some of the
links.  You may want to skip over this section initially and come back
to it only when you find that there are links or pages being checked
that you would rather avoid.

For this section, we assume that you already know how to make basic Perl
code.  If not, then please read through the Perl manual pages
@samp{perl}, @samp{perlsyn} and @samp{perldata}.  You may find that the
examples given below are sufficient to get you started.

In order to get @w{@code{extract-links}} to extract links using an
advanced infostructure, you must use the @code{advanced} keyword. In the
infostructure file.  Infostructures not listed there will be ignored,
but won't cause any harm.

Advanced configuration is in the @file{.link-controller.pl}
configuration file by making definitions into the @code{%::infostrucs}
hash.  These look like the following

@example
$::infostrucs@{http://www.mypages.org/@} = @{
   mode => "directory";
   file_base => "/home/myself/www",
   prune_re => "^(/home/myself/www/statistics)" #ignore referrals
              . "|(cgi-bin)", #do CGIs separately
   exclude_re => "\.secret$", #secrets shouldn't get into link database
@};

$::infostrucs@{http://www.mypages.org/cgi-bin/@} = @{
   mode => "www";
   exclude_re => "query", #query space is infinite!!
@};
@end example

There are a number of keywords that can be used.  

@table @samp
@item mode
This decides how to download the links.  Either @samp{www} or
@samp{directory}.
@item file_base
If defined, this defines the directory which matches the URL where the
infostructure is based.  This must be defined if the mode is set to
directory.
@item resource_include_re
If defined, this regular expression must be matched by the @emph{URL}
for every resource before links will be extracted from it.
@item resource_exclude_re
If defined, this regular expression must @emph{not} be matched by the
@emph{URL} for every resource before links will be extracted from it.
@item link_include_re
If defined, this regular expression must be matched by every @emph{URL}
found before it will be extracted and saved.
@item link_exclude_re
If defined, this regular expression must @emph{not} be matched by every
@emph{URL} found before it will be extracted and saved.
@item prune_re
Used only in directory mode, this will completely exclude all files and
sub-directories of directories matched by the regular expression.
@end table

N.B. the exclude and include regular expression can be used together.
For a match, the include regular expression must match and the exclude
must not match.  In other words excludes override includes.

In order for the infostructure to be used by @code{extract-links} an
entry must still be made in the @file{infostrucs} file.  For this use the
@code{advanced} keyword.  The second argument is a URL used to look up
the definition in the $::infostrucs hash.

@example
advanced   http://www.mypages.org/
advanced   http://www.mypages.org/cgi-bin/
@end example

The URL used here must match @emph{exactly} the one used in the hash.
It is important to note that @samp{directory} and @samp{www} definitions
in the @file{infostrucs} file will override any advanced configuration
given.

@node Authorisation Configuration,  , Advanced Infostructure Configuration, Advanced Configuration
@section Authorisation Configuration
@cindex authentication, basic
@cindex basic authentication
@cindex authorisation
@cindex security

One problem when checking links, especially within an intranet situation
is that some pages can be protected with basic authentication.  In order
to extract links from those pages or to simply know that they are there,
we have to get through that authentication.  By using the advanced
Authorisation Configuration we can give LinkController authority to
access these pages and allow link checking to work as normal.

@display
@emph{Using this method to allow LinkController to work in an
environment with authentication is inherently a security issue since
authentication tokens must be stored, effectively in plaintext, in
files.  This risk may, however, not be much higher than the one that you
currently accept, so this can be useful}
@end display

We can store the authentication tokens simply in the %::credentials hash
which we can create in the @file{.link-controller.pl} configuration file.
The keys in the hash are the exact realm string which will be sent by
the web server.  Each value of this hash is a hash with a pair of keys.
The @samp{credentials} key should be associated to the authentication
token.  The @samp{uri_re} key should be a regular expression which
matches the web pages you want to visit.  For security reasons it
shouldn't match any others.

@example
$::credentials = @{
  my_realm => @{ uri_re => "https://myhost.example.com",
                credential => "my_secret" @}
@} );
@end example

As a minor sanity check, every @samp{uri_re} will be tested against the
strings @samp{http://3133t3hax0rs.rhere.com} and
@samp{http://3133t3hax0rs.rhere.com/secretstuff/www.goodplace.com/}.  If
they match then those credentials will be disallowed.  The owners of
@samp{3133t3hax0rs.rhere.com} will just have to hack the code.. 

For more discussion about the security risks and how to mitigate them
see the file @file{authorisation.pod} included with the LinkController
distribution.  If you didn't understand the security risk from the above
description then probably you should consider avoiding using this
mechanism.


@node Configuring CGI Programs, Extracting Links, Advanced Configuration, Top
@chapter Configuring CGI Programs
@cindex CGI, configuration

The CGI programs have separate configuration variables and configuration.  

@table @samp
@item $WWW::Link::Repair::infostrucbase
@vindex $Link/Repair/infostrucbase
The URL that gets to the base directory of the infostructure.
@item $WWW::Link::Repair::filebase
@vindex $Link/Repair/filebase
The file-name of the directory which is equivalent to the URL
@end table

@strong{FIXME:} this section needs to be rewritten.

@node Extracting Links, Testing, Configuring CGI Programs, Top
@chapter Extracting Links
@cindex links, extracting
@cindex extracting links

This section is written assuming that you are using a standard HTML
infostructure in a directory or on the World Wide Web

The first part of using link controller is to extract the links.  When
doing this, a pair of index files is built which list which URLs happen
on which pages along with a file listing all of the URLs in the
infostructure.  

@strong{FIXME:} compare and contrast multi-user configuration with
single user

@pindex extract-links
The first stage of the process is done by @w{@code{extract-links}}
@footnote{the command names in link controller are quite long.. you
might want to make your life easier by using command completion which
will finish what you have started.. once you've typed a little of the
command use @kbd{escape escape} or @kbd{tab} depending on your
shell.. if this doesn't work then you may like to upgrade to a newer
shell such as bash or zsh.}.

There are two modes for extract links @code{directory} and @code{www}.
The key difference between them is that the latter actually downloads
from a server so it is less efficient but will work in more
circumstances and is more likely to represent your site as seen by
users.  This is assuming that all of your WWW pages are interconnected
so it can find them.

@strong{FIXME} : need to describe modes of operation of extract link

@cindex link index, creating
@cindex page index, creating

@w{@code{extract-links}} creates three files.  The first two files
(@file{*.cdb}) are the index files for your infostructure and are
located wherever you have configured them to by default they are called
@file{link_on_page.cdb}, @file{page_has_link.cdb}.  The third file is
the database file @file{links.db}.  @w{@code{extract-links}} can also
optionally create a text file which lists all of the URLs in the
infostructure, one per line.

@node Testing, Reporting, Extracting Links, Top
@chapter Testing Links

If you are using someone else's link information then you may be able to
skip this part and go straight on to the next one on generating reports.

Testing links takes a long time.  To begin to detect broken ones will
take about ten days.  This is a deliberate feature of LinkController.
It is designed this way firstly to give people at the other end of links
a chance to repair their resources and secondly to reduce the amount of
network bandwidth LinkController uses at a given time and so its impact
on other people's Internet usage.

@pindex test-link, using
The key program which you want to use is @w{@code{test-link}}.  I run
this from a shell script which directs its output to a log file

@strong{FIXME} actually I now just use a cron job.

@example
#!/bin/sh
#this is just a little sample script of how I run the program.

LOGDIR=$HOME/log
test-link >> \
        $LOGDIR/runlog-`/bin/date +%Y-%m-%d`.log 2>&1 
#assumes the use of a gnu style date command which can print 
#out full dates.
@end example

And I run this shell script from my @file{crontab} with a command like
this

@cindex crontab, example
@example
42 02 * * *     /..directories./run-daily-test.sh
@end example

The string @w{@code{/..directories./}} should be replaced with the directory
where you have the script.  Remember to make the script executable.  

This will now run until completion each night.   However, you should
make sure that it does actually finish.  If you have too many links to
check in the given time, then you can end up with a backlog and the
system will take a long time to stop.  To avoid this, either make
testing less frequent or make checking run faster.  This will have to be
done by editing the program itself at present.

@node Reporting, Files, Testing, Top
@chapter Reporting Problems
@cindex reports
@cindex broken links, finding
@cindex links, examining
@pindex link-report

The easiest way to find out which links are broken is to use the command
line interface.  The simplest report you can generate is just a list of
all the known broken links.  Do this like so:

@example
link-report
@end example

On the system I'm testing on right now, this gives:

@example 
broken:-       file://ftp.ncsa.uiuc.edu/Web/httpd/Unix/ncsa_httpd/cgi
        http://www.ippt.gov.pl/docs-1.4/cgi/examples.html
broken:-       file://ftp.ncsa.uiuc.edu/Web/httpd/Unix/ncsa_httpd/curr
ent/httpd_1.4_irix5.2.Z
        http://www.ippt.gov.pl/docs-1.4/setup/PreExec.html
broken:-       file://ftp.ncsa.uiuc.edu/Web/httpd/Unix/ncsa_httpd/curr
ent/httpd_1.4_linux.Z
        http://www.ippt.gov.pl/docs-1.4/setup/PreExec.html
broken:-       file://ftp.ncsa.uiuc.edu/Web/httpd/Unix/ncsa_httpd/curr
ent/httpd_1.4_osf3.0.Z
        http://www.ippt.gov.pl/docs-1.4/setup/PreExec.html
broken:-       file://ftp.ncsa.uiuc.edu/Web/httpd/Unix/ncsa_httpd/curr
ent/httpd_1.4_solaris2.4.Z
        http://www.ippt.gov.pl/docs-1.4/setup/PreExec.html
broken:-       file://ftp.ncsa.uiuc.edu/Web/httpd/Unix/ncsa_httpd/curr
ent/httpd_1.4_solaris2.4.tar.Z
        http://www.ippt.gov.pl/docs-1.4/setup/PreCompiled.html
Sorry, couldn't find info for url file://ftp.ncsa.uiuc.edu/Web/httpd/U
nix/ncsa_httpd/current/httpd_1.4_source.tar.Z
please remember to check you have put it in full format
broken:-       file://ftp.ncsa.uiuc.edu/Web/httpd/Unix/ncsa_httpd/docu
ments/usage.ps
        http://www.ippt.gov.pl/docs-1.4/postscript-docs/Overview.html
..etc...
@end example

Which just tells you which links are broken.  (In this chapter examples
are folded at the 70th row, so that they fit well on narrow screens and
in @TeX{}.  I wanted to use real text rather than making something up.)

We also know which page they are broken on and can go and look at that
on the World Wide Web or directly as a file on the server.

You can get a complete list of options for @w{@code{link-report}} (or
any other program which is part of LinkController) using

@example
link-report -h
@end example

For more advanced reporting and editing of documents with broken links
you may want to use the Emacs interface (@pxref{Emacs, The Emacs
Interface}).

@menu
* Email Reporting::             
@end menu

@node Email Reporting,  , Reporting, Reporting
@section Email Reporting of Newly Broken Links
@cindex links, examining, in Emacs
@cindex broken links, finding in Emacs
@pindex link-report-dired

It's possible to arrange automatic reporting by email of links which
have become newly broken.  This is done by getting @w{@code{test-link}}
to make a list of links that become broken using the $::link_stat_log
variable (@pxref{Link Variables}) and calling @w{@code{link-report}} to
report on those links.

Typically, you may don't want to have a report every time that
@w{@code{test-link}} runs, but probably once a day instead.  In this
case, run a script like the following from your crontab.

@example
#!/bin/sh
STAT_LOG=$HOME/link-data/stat-log
WORK=$STAT_LOG.work
EMAIL=me@@example.com
mv $STAT_LOG  $WORK
if [ -s $WORK ]
then
   link-report --broken --url-file=$STAT_LOG --infostructure | 
      mail -s "link-report for `date`" $EMAIL
fi
@end example

Every time that this script is run, it will rename the status change
log file and then mail a report with all of the new broken links
to the specified email address.  

In future this feature may be folded into @w{@code{link-report}}
directly.

@node Files, Repair, Reporting, Top
@chapter Examining Individual Files
@cindex page, checking
@cindex file, individual, checking
@cindex checking individual pages

When you have just written an HTML page, you often want to check it
before you put it up for use.  You can do this immediately using the
@w{@code{check-page}} program.  Simply run something like

@example
check-page filename.html
@end example

And it will list all of the links that it is unsure about along with the
line number the problem occurred on.  This program works particularly
well when you editing with Emacs (@pxref{check-page in Emacs, The Emacs Interface}).

@node Repair, Suggestions, Files, Top
@chapter Repairing Links
@cindex links, repairing
@cindex repairing links

@pindex fix-link
The program responsible for repairing links is @w{@code{fix-link}}.  It
simply accepts two URLs and changes all of the occurrences of the first
link in your documents into the second link.  It assumes that you have
permission to edit all of the problem files and that there is a
replacement link.  For example

@example
fix-link http://www.ed.ac.uk/~mikedlr/climbing/ \
        http://www.tardis.ed.ac.uk/~mikedlr/climbing/
@end example

Typed at the shell prompt would have updated the location of my Climbing
pages when they moved some while ago and 

@example
fix-link http://www.tardis.ed.ac.uk/~mikedlr/climbing/ \
        http://www.tardis.ed.ac.uk/climb/
@end example

Will change them to the very latest location.  

At present, there's no facility for automatically updating the databases
when you do this.  Instead, you have to run @w{@code{extract-links}}
regularly so that new links are noticed.  Maybe a later version of
LinkController will change this.

@node Suggestions, CGI, Repair, Top
@chapter Making Suggestions
@cindex suggestions

A link in the database can have suggestions associated with it.  These
are normally alternative URLs which somebody or something has decided
would make a good replacement for the URL of the Link.  Humans can add
to the database with the @code{suggest} program.  For example use:

@pindex suggest
@example
suggest file://ftp.ncsa.uiuc.edu/Web/httpd/Unix/ncsa_httpd/current/htt
pd_1.4_linux.Z \
        http://delete.me.org/
Link suggestion accepted.  Thank you
@end example

If you try the same thing again you get

@example
suggest file://ftp.ncsa.uiuc.edu/Web/httpd/Unix/ncsa_httpd/current/htt
pd_1.4_linux.Z \
        http://delete.me.org/ 
Already knew about that suggestion.  Thanks though.
@end example

These suggestions will make it easier for others to repair links,
especially if they are using the CGI interface.

@node CGI, Emacs, Suggestions, Top
@chapter CGI Interface
@cindex interface, CGI
@cindex CGI interface
@pindex link-report.cgi
@pindex fix-link.cgi

The CGI interface is not fully developed and has a number of issues
related to security to be considered.  I have however used it and shown
that it can work, so if you want to you could try the same.  The two
programs @w{@code{fix-link.cgi}} and @w{@code{link-report.cgi}} replace the
normal ones @w{@code{fix-link}} and @w{@code{link-report}}.  They should be
interfaced through an HTML page which feeds the needed information to
@w{@code{link-report.cgi}}.

@cindex authentication
The main security question is how to do authentication of the user.
This will have to be set up using the features of the web server.   

@node Emacs, Setting up LinkController, CGI, Top
@chapter The Emacs Interface
@cindex Emacs interface
@cindex interface, Emacs

LinkController's reporting system is designed to be independent of the
interface to it, and often the shell interface will be all that is
needed.  However another convenient interface is through
@w{@code{emacs}}.  There are two parts to this integration.

@menu
* link-report-dired::           An Emacs program to display broken links.
* check-page in Emacs::         Finding broken links in a file.
@end menu

@node link-report-dired, check-page in Emacs, Emacs, Emacs
@section Finding Files with Broken Links
@cindex links, examining, in Emacs
@cindex broken links, finding in Emacs
@pindex link-report-dired

There is a special Emacs mode called @w{@code{link-report-dired}}
written for locating files with broken links.  The mode is based on
@w{@code{find-dired}} and works very similarly.  It runs the program
@w{@code{link-report}} with an option which makes it list file names in
the same way as the @code{ls} program does.  The user can then move
around the buffer as normal in Emacs and enter files using a single key
press (normally @kbd{f}).

@node check-page in Emacs,  , link-report-dired, Emacs
@section Finding Broken Links in Files Within Emacs
@pindex check-page@r{, in emacs}

The program @w{@code{check-page}} was specially designed so that it outputs
in a format which can be read by Emacs' @w{@code{compile}} mode.  You can
use it within Emacs and then step from error to error correcting them.

To do this, after you have set up your system and completed link
checking use the command @kbd{M-x} @kbd{compile} @kbd{check-page
filename}.  You will now see another buffer open up with all of the
errors shown there.  You can use the key @kbd{M-`} (that's a real back
quote, not an apostrophe) to step between errors.  

@node Setting up LinkController, Administration, Emacs, Top
@chapter Setting up LinkController

This chapter is aimed at administrators setting up LinkController or who
want to have a better understanding of the way that their installation
is set up.  

The first stage is to actually build and install the programs.  This is
covered in the document @file{INSTALL} which is included with the
distribution.

Once you have installed the software, the next step is to configure
LinkController so that it knows where you have all of your data.  The
program @code{default-install} provides one model of this.  

@menu
* Default Installation::        
@end menu

@node Administration, Robots, Setting up LinkController, Top
@chapter Administration

There are various aspects of administration.  This is mostly related to
testing links.  

@menu
* Default Installation::        
* User Administration::         
* Cron Scripts::                
@end menu

@node Default Installation, User Administration, Administration, Administration
@section Default Installation

Running @code{default-install -all} should set everything up
correctly.  There are various variations on this command which do
different things, but the summary is 

@itemize @bullet
@item
Create a linkcont user-id and group which will be used for running programs
@item
Create a working directory where LinkController will keep it's data
@item
Create configuration files, especially /etc/link-control.pl
@item
Create cron scripts which will run LinkController automatically.
@end itemize

Using this command it is also possible to activate users and groups
e.g. @code{default-install -user @var{username}} or  
@code{default-install -group @var{groupname}} in which case the specified
users will become a member of the @code{linkcont} group.  


@node User Administration, Cron Scripts, Default Installation, Administration
@section User Administration

User administration is really only needed if you are running link
testing centrally for your users.  This makes sense since it means that
if several users have a link to the same place (likely in any given
site) then you will only have to check that link once.  

In this case, the important question is which links are copied into the
checking database.  This is controlled by the program
@w{@code{copy-links-from-users}} and decides copies data from users which
are in the @code{lnctusr} group.  

The command @w{@code{default-install}} can be used to manipulate which
users are in the group e.g. @code{default-install --user @var{username}} or  
@code{default-install -group @var{groupname}} in which case the specified
users will become a member of the lcntusr group.

Another form of user administration is limitation on which users have
access to the database.  This can be done with normal file permissions.
There isn't any specific control to stop users from seeing which links
other users have put into the database.  

@node Cron Scripts,  , User Administration, Administration
@section Cron Scripts

In order to be effective, link testing should be done every day.
Furthermore, it is a good idea to do the testing at low usage times,
which normally means at night.  For this reason normally a cron script
will be used.  

@itemize @bullet
@item 
copy the links from each user with @code{copy-links-from-users} 
@item 
add them to the database with @code{extract-links --in-url-list} 
@item 
build a schedule for testing the links with @code{build-schedule} 
@item 
test the links with @code{test-link} 
@end itemize

The other thing which is done is to remove old links from the database.
This only needs to be done weekly.  

The program @w{@code{default-install}} can create these scripts.

@node Robots, Link Database Maintenance, Administration, Top
@chapter Robots and Sensible Behaviour
@cindex robots
@cindex dangers
@cindex bandwidth

The most important thing about a program like this is to realise that if
you set it up incorrectly and used it in the wrong way, you could upset
a large number of people who have set up their web servers in the
assumption that they would be used normally by human beings browsing
through on Netscape.

Probably it's true that the only way forward is for every WWW site to
begin to set up robot defences and detect when someone starts to
download from at an unreasonable rate and then cut off the person doing
the downloading.  I suggest that you don't do this for at least two
reasons.

@itemize @bullet
@item 
respect for the person's time
@item 
a wish not to be the person who is cut off
@end itemize

There are probably many other reasons, but that's one for the good side
in you and one for the selfish.  What more do you need.

For suggestions about what constitutes `correct' behaviour, it's worth
seeing the Robots World Wide Web page.
@url{http://info.webcrawler.com/mak/projects/robots/robots.html}

There are a number of points which make LinkController relatively safe
as a link.  These are all related to the design and limitations on
@w{@code{test-link}}.  

@itemize @bullet
@item 
@w{@code{test-link}} does @emph{not} recurs.  It only tests links that
are specifically listed in the schedule database.
@item 
There is a limit to the number of links that will be tested in one run.
This defaults to 1000, but can be configured.  
@item
The schedule for link testing is designed to spread the testing of links
across time
@item
The testing system will not test links at a given site faster than a
certain rate.
@end itemize

The last limitation is inherited from the @code{LWP::RobotUA} module and
the documentation for that covers the details of how it works.
@w{@code{test-link}} tries to re-order testing of links as needed so that
a limit on the rate of visits to one site does not cause a limit on
overall testing speed.

@node Link Database Maintenance, Bugs, Robots, Top
@chapter Link Database Maintenance

For the most part the link database shouldn't need much maintenance.
There are a number of cases where it might, however.  If it becomes
corrupt, you may try the db_recover command.  Probably, however, it's
just better to recover the database and link checking schedule from a
recent backup.  Time is not really critical since the work is normally
easy to regenerate.  You should make sure that link checking doesn't run
in at the time you do backups, however.

The other thing is that occasionally you may want to recover space in
the link database by dumping and un-dumping it.  See the Berkeley database
documentation for more details.

@menu
* Link Ageing::                 
@end menu

@node Link Ageing,  , Link Database Maintenance, Link Database Maintenance
@section Link Ageing

Sometimes we have a link which is no longer in use within our
infostructure.  However, it's not a good idea to throw away
information about it immediately.  It could be that certain files have
been temporarily deleted and will come back.  Alternatively, a link
could have been found broken and been corrected, but someone has a
copy of the old page.  When they re-install the copy, we will have to
deal with that link again.  

If we had not kept that link around and they immediately do a link
check on their document they may see nothing wrong, and, because of
the nature of link-controller, we won't start reporting the link as
broken until we have confirmed that it really is, some days later.

If, on the other hand, we keep checking all of the links which are
aged, we will cause ourselves considerable extra work.  

To handle this, links are aged.  Once a link has reached greater than
a certain age, the link will not be checked any more.   Once the link
has reached a much larger age, it will be completely deleted from the
link database.  The age is reset each time the link is extracted from
the infostructure, so links which are still in use will continue to be
checked.  

In the meantime, the link will still be repeatedly scheduled based to
it's normal checking time.  This causes us to examine it quite
regularly, but that is okay, since we will do almost no work when we
look at it.

By default links which have not been refreshed will be ignored after
one week and will be deleted from the database after two months.  

@node Bugs, History , Link Database Maintenance, Top
@chapter Bugs and bug reporting

This version of LinkController is still in early development.  There are
many changes to come.  Undoubtedly there are many bugs in the software
already and will soon be more.

A bug is when 

@itemize @bullet
@item 
the software doesn't do something the documentation says it should
@item 
the software does something the documentation says it shouldn't
@item 
the software does something surprising and that isn't documented
@item 
the software does something strange but the documentation doesn't
explain why
@item 
it is difficult or impossible to understand what the documentation is
trying to say
@end itemize

some of these mean fixing the documentation and some the software.  All
of them are bugs and should be reported and fixed.  

If you find a bug, I will be grateful to hear about it.  Even if you
don't know how to fix it or anything, it is useful to know what is wrong
so that other people don't get caught out but @emph{read the BUGS file
first} please.  If the bug is listed there then the only useful thing
that you can do is fix it.  If you do this and contribute it to me then
that is very useful.

When you report a bug, please tell me what release of link controller
you were using.  This is the number which was in the name of the file
that LinkController came in.  If your problem was with a specific
program, please also run @samp{program --version} and send the output.
This tells me exactly which version of that program you were running.

Since this is a developers release, I'd hope most users would be able to
make some level of fixes.  If you do this, send me context differences
(use @samp{diff -u} if it works or try @samp{diff -c} otherwise).  I use
CVS, so as long as I know which version you have I will be able to find
the original file and see your changes.  However it's also important to
explain them because I won't be able to use them unless I (relatively
stupid computer type) understand them.

Send bug reports to the address you get by changing words into
punctuation in the following.

@example
link minus controller at scotclimb dot org dot uk
@end example

This mailing address is sent only to me right now, but may become a list
in future.  Use my (Michael De La Rue) personal address to contact me
please.  N.B. I am @strong{extremely} inefficient about answering email.
Don't worry if you don't get a reply.

	The ideas, and history

@node History , Acknowledgements , Bugs, Top
@chapter History
@cindex history MOMspider

LinkController was originally inspired by MOMspider and having the
MOMspider code available was very useful when starting the creation of
this kit, but, it shares almost no code with MOMspider, other than what
has comes to it from the LibWWW-Perl library.

Philosophically, the MOMspider heritage is obvious in the wish to handle
big jobs efficiently.  In the working practice there are far more
differences than similarities, partly caused by Perl language changes.

I decided to completely separate the exploration of the local
infostructure, looking for links to be checked, from the actual checking
process.  This means that checking can be spread over a large number of
days and still run efficiently. 

The basic aim of this link checking kit is to be able to efficiently
handle any size of link checking job.  At the bottom end we have
checking new pages as they are written.  Here we want to use
information from previous checks to avoid having to check all of each
page every time.  At the other end we have massive info structures
(sites) which deal in many thousands of links and could not possibly
all be checked in one day.  For this latter case the aim is to be able
to efficiently spread the link checking load into all available low
usage periods.

My primary aim in writing this was not to write very efficient code for
the small scale case (takes minimum time to do everything), but rather
code which would scale well.  If your system can check 1000 links in two
days, it will hopefully be able to check almost 7000 links in two weeks.
I'm trying to make sure all data structures which grow with the number
of links are kept on disk.

@node Acknowledgements , Invoking the Programs, History , Top
@chapter Acknowledgements
@cindex acknowledgements

Although I wrote this system by myself, this would not have been
nearly as easy and almost certainly wouldn't have been finished without
the help of the following people and organisations.

@menu
* Esoterica Internet Portugal::  
* IPPT PAN Poland::             
* The Tardis Project::          
* Other Free Software Authors::  
@end menu

@node Esoterica Internet Portugal, IPPT PAN Poland, Acknowledgements , Acknowledgements
@unnumberedsec Esoterica Internet Portugal

@c I'd like to have these people's names correct but I'm not sure how to
@c do that in texinfo.. oh well.

Esoterica provided me with full access to the Internet in Portugal and
use of their computers for free which allowed me to keep up on both this
software and the Linux Access HOWTO.  In particular I'd like to thank
all of the members of staff who helped me very much.  These people include
@c M\'ario Francisco Valente
Mario Francisco Valente (the instigator of Mini Linux) who first
agreed to me using their kit, set me up to use their machines, and along
with 
@c Lu\'\i{}s Sequeira
Luis Sequeira provided a sounding board for some ideas.
@c Lu\'\i{}s 
Luis also provided the odd lift home in the evening.  Also 
@c Martim de Magalhães
Martim de Magalhaes Pereira and Mr Mendes.  See them all on 

@url{http://www.esoterica.pt/esoterica/quemsomos.html}

For more about esoterica (Internet Services in Portugal) see:

@url{http://www.esoterica.pt/esoterica/}

These pages are in Portugese@footnote{Whilst the above names are mangled
here.  See the correct versions in the original texinfo or on the Web
pages.} of course.

@node IPPT PAN Poland, The Tardis Project, Esoterica Internet Portugal, Acknowledgements
@unnumberedsec IPPT PAN Poland

Thanks go to IPPT PAN (part of PAN - Polska Akademia Naukowa) in Poland
and in particular Piotr Pogorzelski who allowed me use of facilities for
testing this software, provided a willing victim for having his web
pages tested and made a number of suggestions which have been
incorporated into the software.

@node The Tardis Project, Other Free Software Authors, IPPT PAN Poland, Acknowledgements
@unnumberedsec The Tardis Project

Supported by the Computing Science department of the University of
Edinburgh, the Tardis project provides an experimental framework in
which students, former students and other related people to do their own
work on fully Internet connected Unix and Linux hosts.

The use of the facilities of the Tardis Project has made it much easier
for me to develop software like this.  In particular, the large amount
of disk space the administrators have allow me to use is very useful.

@node Other Free Software Authors,  , The Tardis Project, Acknowledgements
@unnumberedsec Other Free Software Authors

It is through the software provided by the Free Software Foundation
(such as the @w{@code{gcc}} C compiler, Emacs, the file utilities), the
authors of the various packages which make up a working Linux System
(Linux by Linus Torvalds, Alan Cox, etc.... filesystems and support by
Theodore Tytso, Stefan Tweedie etc.. Linux-Libc by HJ Lu, based on GNU
@w{@code{glibc}} from the FSF.. the list is indefinite) and the authors
of Perl and its modules, especially Gisle Aas and Martijn Kostler for
LibWWW-Perl that I was able to set this up.

I'd particularly like to thank Tim Goodwin the author of the Perl CDB
module who made and accepted a number of alterations to that, at my
request.  These alterations made this package simpler to write and
easier to maintain.

The Free Software Foundation web pages are at

@url{http://www.gnu.ai.mit.edu/}

@node Invoking the Programs, Related Packages, Acknowledgements , Top
@appendix Invoking the LinkController Programs

Because they use the Perl @code{Getopt::Mixed} module, all of the
LinkController command line programs respond to the standard POSIX style
command line options.  At least the following two options will be implemented. 

@table @samp
@item --help
This option will give a list of all of the options understood by the
program along with brief explanations of what they do.  At present it
may lie a little, but that is being corrected.
@item --version
This option will give some version information for the program.
@end table

You can use the @samp{--help} option to get help on each program, for
example:

@example
extract-links --help
@end example

will give something like

@example
extract-links [arguments] [url-base [file-base]]

 -V --version            Give version information for this program
 -h --help --usage       Describe usage of this program.
    --help-opt=OPTION    Give help information for a given option
 -v --verbose[=VERBOSITY] Give information about what the program is doing.  Set
                         value to control what information is given.

 -e --exclude-regex=REGEX Exclude expression for excluding files.
 -p --prune-regex=REGEX  Regular expression for excluding entire directories.
 -d --default-infostrucs handle all default infostrucs (as well as ones listed
                         on command line)

 -l --link-database=FILENAME Database to create link records into.
 -c --config-file=FILENAME Load in an additional configuration file

 -o --out-url-list=FILENAME File to output the url of each link found to
 -i --in-url-list=FILENAME File to input urls from to create links

Extract the link and index information from a directory containing
HTML files or from a set of WWW pages with URLs which begin with the
given URL and which can be found by starting from that URL and
searching other such pages.
@end example

You can then use that information to get the program to do what you
want.

@node Related Packages, Terms, Invoking the Programs, Top
@chapter Packages Which Can Be Useful with LinkController

@menu
* cdb::                         Utilities for the LinkController indexes.
* Tie-Transact-Hash::           Berkeley DB editing tools.
@end menu

@node cdb, Tie-Transact-Hash, Related Packages, Related Packages
@section The CDB utilities

In order to have LinkController working you must have installed these.
It is worth looking at the utilities that are provided, especially
@code{cdbdump} which will let you look at the contents of the file.  You
should be aware that @code{cdbget} program which is provided
@emph{won't} be able to get at the full contents of the index files
since they contain repeated keys.

More information on cdb and new releases can be got from the www page.

@url{http://pobox.com/~djb/cdb.html}

@node Tie-Transact-Hash,  , cdb, Related Packages
@section The Tie-Transact-Hash Perl Module and Programmes

This is a Perl module written by myself which includes a program which
allows direct examination and editing of Berkeley databases.  It can be
useful for debugging and correcting problems in the LinkController Link
database or schedule file.

Tie::TransactHash can be downloaded from CPAN, the Comprehensive Perl
Archive Network get there via:

@url{http://www.perl.com/perl/info/software.html}

@node Terms, Names Index, Related Packages, Top
@appendix Terms

@menu
* Resource::                    The information on the World Wide Web.
* URLs::                        The connections in the World Wide Web.
* Infostructure::               Groups of resources.
* Link::                        A connection between two resources.
@end menu

@node Resource, URLs, Terms, Terms
@section Resource
@cindex resource

A resource is almost anything.  `It' can range from a person to an HTML
file to a computer to a database or presumably eventually to phone
numbers, possibly physical hardware.  This generality is a very
important concept for the World Wide Web.  Really the key thing about a
resource is that it can be `identified'.  @xref{URLs}, for more
details.

@node URLs, Infostructure, Resource, Terms
@section URLs
@cindex URL
@cindex WWW

A URL or `Uniform Resource Locator' are the essence of the World Wide
Web.  Approximately, they are addresses through which `resources' can be
located.  The idea is that almost anything can be given some kind of
address in a form that a machine can work with.  By defining a set of
rules, this can then be converted into a URL.  A URL has two parts.  The
first tells us what rules to use and the second tells us what the
address is.

@node Infostructure, Link, URLs, Terms
@section Infostructure
@cindex infostructure
@cindex HTML pages, groups of

An infostructure is a concept which was introduced in Link Checking in
the MOMspider package.  It is a collection of related resources.

@node Link,  , Infostructure, Terms
@section Link

The term link in LinkController is used for a connection between two
resources.  It's existence really comes from the `class' or piece of
type of computer data which is used to store information about `links'.
Properties of a link include:

@itemize @bullet
@item 
Knowing what the URL of the target resource of the connection is.
@item
Knowing whether the connection to the target resource has been working
recently.
@item
Knowing when the connection to the target resource was last checked.
@end itemize

Within the programs, a link is different from a URL in that it is
specifically aimed at checking connections, where a URL just specifies
what the connection should be if it is working.

@node Names Index, Concept Index, Terms, Top
@unnumbered Program and Variable Name Index

Perl variables, which use :: to separate the different parts, are
separated here using / instead to make it easier to work in info with
this file.  

@printindex vr

@node Concept Index,  , Names Index, Top
@unnumbered Concept Index

@printindex cp

@bye
