NAME
    CHANGES Changelog for the Ngram Statistics Package (Text-NSP)

SYNOPSIS
    Revision history for Perl module Text::NSP

DESCRIPTION
    1.23
        Release March 31, 2011 all changes by YL

        *   Changed printf to print in huge-split.pl, huge-sort.pl,
            huge-merge.pl, and count2huge.pl.

            Replaced the tail hash of huge-merge.pl by without use hash.

    1.21
        Released November 12, 2010 all changes by BTM

        *   Added the Log Likelihood Measure for 4-grams

        >>>>>>> 1.24 =item 1.19

        Released November 1, 2010 all changes by YL

        *   Created find-compounds.pl and its testing files.
            find-compounds.pl helps to pick out the compound words in the
            text file.

    1.17
        Released April 26, 2010 all changes by YL

        *   Created count2huge.pl and its testing files. count2huge.pl helps
            to convert the output of count.pl to huge-count.pl.

    1.15
        Released April 7, 2010 all changes by YL

        *   Created huge-split.pl and huge-delete.pl in order to remove this
            functionality from huge-count.pl and huge-merge.pl (and make it
            easier to use these different components in more flexible ways).

    1.13
        Released March 5, 2010 all changes by TDP and YL

        *   Replaced huge-count.pl with a more efficient version that counts
            large number of bigrams by creating multiple files, sorting, and
            merging them. The sorting and merging are carried out by
            huge-sort and huge-merge.pl. Note that the previous versions of
            huge-count.pl and associated utilities can be found in
            /Text-NSP/bin/utils/deprecated and will remain there for at
            least one more release. They will not however be installed
            automatically. (YL)

        *   Added --uremove and --ufrequency options to count.pl. This allow
            for frequency cutoffs based on ngrams occuring more than a given
            number of times (rather than just less than, which is what
            --remove and --frequency enable). This is a long standing item
            on the NSP Todo list that has finally been checked off! (YL)

        *   Introduced /bin/utils/contributed to allow for the distribution
            of user contributed programs that might be useful to other
            users. These programs do not get installed automatically with
            NSP, and are not included in our standard testing streams, but
            could still prove very useful to users. Please let us know if
            you have code you might like to include here. (TDP)

        *   Added nsp-stoplist.regex to distribution (in
            /Text-NSP/bin/utils), to serve as a default stoplist. (TDP)

             Reported here : <L http://tech.groups.yahoo.com/group/ngram/message/280>

            This was not added in 1.11 due to failure to rebuilt MANIFEST.

        *   Added support for 4-d log-likelihood
            (Text::NSP::Measures::4D::MI:ll). (TDP)

    1.11
        Released Nov 5, 2009 all changes by TDP

        *   Fixed bug in statistic.pl which caused long form of pmi
            (Text::NSP::Measures::3D::MI::pmi) not to be handled correctly
            on the command line, and that caused pmi_exp not to be properly
            initialize when using the long form of pmi.

              Reported here : <L http://tech.groups.yahoo.com/group/ngram/message/240>

        *   Added nsp-stoplist.regex to distribution (in
            /Text-NSP/bin/utils), to serve as a default stoplist.

              Reported here : <L http://tech.groups.yahoo.com/group/ngram/message/280>

        *   Fixed link to class diagram in FAQ.pod.

              Reported here : <L http://tech.groups.yahoo.com/group/ngram/message/230>

        *   Fixed documentation Text::NSP::Measures::3D::MI::pmi to
            correctly show how we are computing expected values.

              Reported here : <L http://tech.groups.yahoo.com/group/ngram/message/290>

        *   Fixed a few broken links in README.pod that were discovered
            while preparng this release

    1.09
        Released March 26, 2008 all changes by TDP

        *   Spell checked the modules

        *   Relaxed test cases 27, 29 for ll, 20 for x2, and 13 for phi, due
            to arithmetic differences on 64 bit architectures

        *   Modified Makefile.PL to go back to more standard methods of
            testing and installation.

        *   Modified structure of /t directory for 'make test'. It appears
            that the use of subdirectories in /t with test cases might have
            been causing problems for Windows testing, so we have moved all
            test files to the top level of /t, and also removed the TEST
            program so that things are called in a more standard or generic
            fashion.

    1.07
        Released March 24, 2008 all changes by TDP

        *   Updated Makefile.PL to no longer require 5.8.5 - have dropped
            back to 5.6

        *   Updated FAQ with some explanation of ALL-TESTS.sh

        *   Renamed /docs as /doc to be consistent with other packages

        *   Added descriptive labels in POD in NAME field of .pl programs to
            provide that info on CPAN display

        *   Fixed duplicate Copyright message bug in documentation of
            Measures.pm

        *   Removed "help" messages from Makefile.PL execution so as to
            (hopefully) avoid problems with installations on Windows.

        *   Corrected error in INSTALL instructions - csh ./ALL-TESTS.sh
            must be performed after 'make install'

    1.05
        Released March 20, 2008 all changes by TDP

        *   Fixed problem with file Testing/statistic/t2 would appear
            (mysteriously) but not be in the MANIFEST. This file was left
            behind during /Testing/statistic/normal-op.sh and is now being
            removed.

        *   Fixed problem in /Testing where .sh files are sometimes not
            executable. Those files are now invoked via 'csh test.sh' rather
            than './test.sh', meaning that they no longer need to be
            executable.

        *   Fixed ticket number 24061 from rt.cpan.org regarding incorrect
            version information coming from Measures.pm

        *   Archived all old ChangeLogs to doc/ChangeLogs directory. Began
            to use pod in CHANGES directory instead

        *   Added doc/update-pod.sh to automatically refresh top level read
            only documentation including README, CHANGES, TODO and INSTALL

        *   Fixed Makefile.PL to avoid problems during Windows install. This
            problem and fix was reported by Richard Churchill to the ngram
            mailing list. This may also address ticket #20371 from
            rt.cpan.org.

        *   Modified Makefile.PL to allow for use of 'make dist' and also
            creation of META.yml

BUGS
    There is a limitation in huge-count.pl. When the size of the corpus is
    very large (>16G) and the some of the terms of the bigrams is very long
    (>30 chars), the program could run out of memory at huge-merge.pl step.
    This is because huge-merge use two hashes to count the frequencies of
    the first and second term of the bigrams. These two hashes could use up
    the memory with the increase of the length of the terms and the increase
    of the number of the terms. If just for normal text, terms are within
    limited length and numbers, the software won't use up the memory.

AUTHORS
     Ying Liu, University of Minnesota, Twin Cities 
     liux0395 at umn.edu

     Ted Pedersen, University of Minnesota, Duluth
     tpederse at d.umn.edu

    This document last modified by : $Id: CHANGES.pod,v 1.29 2011/03/31
    22:41:12 tpederse Exp $

SEE ALSO
    <http://ngram.sourceforge.net>

COPYRIGHT AND LICENSE
    Copyright (c) 2004-2011 Ted Pedersen

    Permission is granted to copy, distribute and/or modify this document
    under the terms of the GNU Free Documentation License, Version 1.2 or
    any later version published by the Free Software Foundation; with no
    Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.

    Note: a copy of the GNU Free Documentation License is available on the
    web at <http://www.gnu.org/copyleft/fdl.html> and is included in this
    distribution as FDL.txt. Ted Pedersen, University of Minnesota, Duluth
    tpederse at d.umn.edu

    This document last modified by : $Id: CHANGES.pod,v 1.29 2011/03/31
    22:41:12 tpederse Exp $

SEE ALSO
    <http://ngram.sourceforge.net>

COPYRIGHT AND LICENSE
    Copyright (c) 2004-2011 Ted Pedersen

    Permission is granted to copy, distribute and/or modify this document
    under the terms of the GNU Free Documentation License, Version 1.2 or
    any later version published by the Free Software Foundation; with no
    Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.

    Note: a copy of the GNU Free Documentation License is available on the
    web at <http://www.gnu.org/copyleft/fdl.html> and is included in this
    distribution as FDL.txt.

