=head1 NAME

CHANGES Changelog for the Ngram Statistics Package (Text-NSP)

=head1 SYNOPSIS

Revision history for Perl module Text::NSP

=head1 DESCRIPTION

=over

=item 1.23

Release March 31, 2011 all changes by YL

=over

=item

Changed printf to print in huge-split.pl, huge-sort.pl, 
huge-merge.pl, and count2huge.pl. 

Replaced the tail hash of huge-merge.pl by without 
use hash.  

=back

=item 1.21

Released November 12, 2010 all changes by BTM

=over

=item

Added the Log Likelihood Measure for 4-grams

=back

>>>>>>> 1.24
=item 1.19

Released November 1, 2010 all changes by YL

=over

=item

Created find-compounds.pl and its testing files. find-compounds.pl helps 
to pick out the compound words in the text file. 

=back

=item 1.17

Released April 26, 2010 all changes by YL

=over

=item

Created count2huge.pl and its testing files. count2huge.pl helps 
to convert the output of count.pl to huge-count.pl.  

=back

=item 1.15

Released April 7, 2010 all changes by YL

=over

=item

Created huge-split.pl and huge-delete.pl in order to remove this 
functionality from huge-count.pl and huge-merge.pl (and make it 
easier to use these different components in more flexible ways).

=back

=item 1.13

Released March 5, 2010 all changes by TDP and YL

=over

=item 

Replaced huge-count.pl with a more efficient version that counts
large number of bigrams by creating multiple files, sorting, and
merging them. The sorting and merging are carried out by huge-sort and 
huge-merge.pl. Note that the previous versions of huge-count.pl
and associated utilities can be found in /Text-NSP/bin/utils/deprecated 
and will remain there for at least one more release. They will 
not however be installed automatically. (YL) 

=item 

Added --uremove and --ufrequency options to count.pl. This allow for 
frequency cutoffs based on ngrams occuring more than a given number of 
times (rather than just less than, which is what --remove and 
--frequency enable). This is a long standing item on the NSP Todo 
list that has finally been checked off! (YL)

=item

Introduced /bin/utils/contributed to allow for the distribution of user 
contributed programs that might be useful to other users. These programs
do not get installed automatically with NSP, and are not included in our
standard testing streams, but could still prove very useful to users.
Please let us know if you have code you might like to include here. (TDP)

=item 

Added nsp-stoplist.regex to distribution (in /Text-NSP/bin/utils), to 
serve as a default stoplist. (TDP)

 Reported here : <L http://tech.groups.yahoo.com/group/ngram/message/280>

This was not added in 1.11 due to failure to rebuilt MANIFEST.

=item 

Added support for 4-d log-likelihood (Text::NSP::Measures::4D::MI:ll). (TDP)

=back

=item 1.11

Released Nov 5, 2009 all changes by TDP

=over

=item 

Fixed bug in statistic.pl which caused long form of pmi (Text::NSP::Measures::3D::MI::pmi)
not to be handled correctly on the command line, and that caused pmi_exp 
not to be properly initialize when using the long form of pmi.

  Reported here : <L http://tech.groups.yahoo.com/group/ngram/message/240>

=item  

Added nsp-stoplist.regex to distribution (in /Text-NSP/bin/utils), to 
serve as a default stoplist.

  Reported here : <L http://tech.groups.yahoo.com/group/ngram/message/280>

=item 

Fixed link to class diagram in FAQ.pod. 

  Reported here : <L http://tech.groups.yahoo.com/group/ngram/message/230>

=item 

Fixed documentation Text::NSP::Measures::3D::MI::pmi to correctly show 
how we are computing expected values. 

  Reported here : <L http://tech.groups.yahoo.com/group/ngram/message/290>

=item 

Fixed a few broken links in README.pod that were discovered while 
preparng this release

=back

=item 1.09

Released March 26, 2008 all changes by TDP

=over

=item 

Spell checked the modules

=item 

Relaxed test cases 27, 29 for ll, 20 for x2, and 13 for phi, due to 
arithmetic differences on 64 bit architectures

=item

Modified Makefile.PL to go back to more standard methods of testing and 
installation. 

=item

Modified structure of /t directory for 'make test'. It appears that the 
use of subdirectories in /t with test cases might have been causing 
problems for Windows testing, so we have moved all test files to the top 
level of /t, and also removed the TEST program so that things are called 
in a more standard or generic fashion.

=back

=item 1.07

Released March 24, 2008 all changes by TDP

=over

=item

Updated Makefile.PL to no longer require 5.8.5 - have dropped back to 5.6

=item

Updated FAQ with some explanation of ALL-TESTS.sh

=item

Renamed /docs as /doc to be consistent with other packages

=item

Added descriptive labels in POD in NAME field of .pl programs to provide 
that info on CPAN display

=item 

Fixed duplicate Copyright message bug in documentation of Measures.pm

=item

Removed "help" messages from Makefile.PL execution so as to (hopefully) 
avoid problems with installations on Windows.

=item

Corrected error in INSTALL instructions - csh ./ALL-TESTS.sh must be 
performed after 'make install'

=back

=item 1.05 

Released March 20, 2008 all changes by TDP

=over

=item

Fixed problem with file Testing/statistic/t2 would appear 
(mysteriously) but not be in the MANIFEST. This file was left behind 
during /Testing/statistic/normal-op.sh and is now being removed.


=item

Fixed problem in /Testing where .sh files are sometimes not executable. 
Those files are now invoked via 'csh test.sh' rather than './test.sh', 
meaning that they no longer need to be executable.

=item

Fixed ticket number 24061 from rt.cpan.org regarding incorrect 
version information coming from Measures.pm

=item 

Archived all old ChangeLogs to doc/ChangeLogs directory. Began to use 
pod in CHANGES directory instead

=item 

Added doc/update-pod.sh to automatically refresh top level read only 
documentation including README, CHANGES, TODO and INSTALL

=item 

Fixed Makefile.PL to avoid problems during Windows install. This 
problem and fix was reported by Richard Churchill to the ngram mailing 
list. This may also address ticket #20371 from rt.cpan.org. 

=item 

Modified Makefile.PL to allow for use of 'make dist' and also creation 
of META.yml

=back

=back

=head1 BUGS

There is a limitation in huge-count.pl. When the size of the 
corpus is very large (>16G)  and the some of the terms of the 
bigrams is very long (>30 chars), the program could run out of 
memory at huge-merge.pl step. This is because huge-merge use 
two hashes to count the frequencies of the first and second 
term of the bigrams. These two hashes could use up the memory
with the increase of the length of the terms and the increase
of the number of the terms. If just for normal text, terms 
are within limited length and numbers, the software won't 
use up the memory.   

=head1 AUTHORS

 Ying Liu, University of Minnesota, Twin Cities 
 liux0395 at umn.edu

 Ted Pedersen, University of Minnesota, Duluth
 tpederse at d.umn.edu

This document last modified by :
$Id: CHANGES.pod,v 1.29 2011/03/31 22:41:12 tpederse Exp $

=head1 SEE ALSO

L<http://ngram.sourceforge.net>

=head1 COPYRIGHT AND LICENSE

Copyright (c) 2004-2011 Ted Pedersen

Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.2
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
Texts.

Note: a copy of the GNU Free Documentation License is available on
the web at L<http://www.gnu.org/copyleft/fdl.html> and is included in
this distribution as FDL.txt.
 Ted Pedersen, University of Minnesota, Duluth
 tpederse at d.umn.edu

This document last modified by :
$Id: CHANGES.pod,v 1.29 2011/03/31 22:41:12 tpederse Exp $

=head1 SEE ALSO

L<http://ngram.sourceforge.net>

=head1 COPYRIGHT AND LICENSE

Copyright (c) 2004-2011 Ted Pedersen

Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.2
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
Texts.

Note: a copy of the GNU Free Documentation License is available on
the web at L<http://www.gnu.org/copyleft/fdl.html> and is included in
this distribution as FDL.txt.

=cut

