Link Grammar Parser ------------------- Version 4.5.7 The Link Grammar Parser implements the Sleator/Temperley/Lafferty theory of natural language parsing. This version of the parser is an extended, expanded version of the last official CMU release, and includes many enhancements and fixes created by many different developers. This latest version incorporates a substantial part (but not all) of the BioLG project. This code is released under the BSD license (which is GPL compatible), making it freely available for private and commercial use, with few restrictions. The terms of the license are given in the LICENSE file included with this software, and also available at http://www.link.cs.cmu.edu/link/license.html. Please see the web page http://www.abisource.com/projects/link-grammar/ for more information. This version is a continuation of the original parser posted at http://www.link.cs.cmu.edu/link CONTENTS of this directory: --------------------------- LICENSE The license describing terms of use link-grammar/*.c The program. (Written in ANSI-C) link-grammar/corpus/*.c Optional corpus statistics database. link-grammar/java/* Optional Java language bindings. link-grammar/minisat/* Optional SAT Solver. (Written in C++) link-grammar/sat-solver Optional SAT Solver. (Written in C++) data/en/* English language dictionaries. data/en/4.0.dict The file containing the dictionary definitions. data/en/4.0.knowledge The post-processing knowledge file. data/en/4.0.constituents The constituent knowledge file. data/en/4.0.affix The affix file. data/en/4.0.regex Regex-based morphology guesser. data/en/tiny.dict A small sample dictionary. data/en/words/* A directory full of word lists. data/en/4.0*.batch These files contain sentences (both grammatical and ungrammatical ones) that are handled correctly by this release of the system. These can be run through the parser with the command "./link-parser < 4.0.*.batch" COPYING The license for this code and data ChangeLog A compendium of recent changes. configure The GNU configuration script autogen.sh Developer's configure maintenance tool msvc6, msvc9 Microsoft Visual-C project files UNPACKING and signature verification: ------------------------------------- The system is distributed using the normal tar.gz format; it can be extracted using the "tar -zxf link-grammar.tar.gz" command at the command line. The files have been digitally signed to make sure that there was no corruption of the dataset during download, and to help ensure that no malicious changes were made to the code internals by third parties. The signatures can be checked with the gpg command: gpg --verify link-grammar-4.5.7.tar.gz.asc which should generate output identical to (except for the date): gpg: Signature made Tue Feb 26 19:41:07 2008 CST using RSA key ID C0389241 gpg: Good signature from "Linas Vepstas (current active email; others are inactive) " Alternately, the md5 checksums can be verified. These do not provide cryptographic security, but they can detect simple corruption. To verify the checksums, issue "md5sum -c MD5SUM" at the command line. CREATING the system: -------------------- To compile the link-grammar shared library and demonstration program, at the command line, type: ./configure make To install, change user to "root" and say make install ldconfig This will install the lib-link-grammar.so library into /usr/local/lib, the header files in /usr/local/include/link-grammar, and the dictionaries into /usr/local/share/link-grammar. Running 'ldconfig' will rebuild the shared library cache. By default, the Makefiles attempt to build the Java bindings. The use of the Java bindings is *OPTIONAL*; you do not need these if you do not plan to use link-grammar with Java. You can skip building the Java bindings by disabling as follows: ./configure --disable-java-bindings The /usr/local install target can be over-ridden using the standard GNU configure --prefix option, so for example: ./configure --prefix=/opt/link-grammar By using pkg-config (see below), non-standard install locations can be automatically detected. Additional config options are printed by ./configure --help The system has been tested and works well on 32 and 64-bit Linux systems, FreeBSD, MacOSX, as well as on many Microsoft Windows systems, under various different Windows development environments. Specific OS-dependent notes follow. BUILDING on MacOS: ------------------ Apple MacOSX users will probably need to obtain missing packages from MacPorts in order to successfully build on MacOS. See http://www.macports.org/ for details. See also: http://trac.macports.org/browser/trunk/dports/textproc/link-grammar/Portfile Users who intend to use the java bindings with java6 will need to make sure to compile link-grammar to produce a 64-bit binary, as this is not automatic on OSX. Do this during configure: ./configure CFLAGS="-arch x86_64" BUILDING on Windows: -------------------- The easiest way to have link-grammar working on MS Windows, especially if you want the Java bindings, is to use MinGW/MSYS, a minimalist GNU environment for Windows that includes gcc, bash, automake, and many other utilities. Download and install MinGW, MSYS and MSYS-DTK from http://mingw.org . Link-grammar now requires a working version of POSIX-standard regex libraries. Since these are not provided by Microsoft, a copy must be obtained elsewhere. One possibility is from the gnuwin32 system: http://gnuwin32.sourceforge.net/packages/regex.htm http://ftp.gnome.org/pub/gnome/binaries/win32/dependencies/regex.README Then build and install link-grammar with ./configure make make install If you used the standard installation paths, the directory /usr/ is mapped to C:\msys\1.0, so after 'make install', the libraries and executable will be found at C:\msys\1.0\local\bin and the dictionary files at C:\msys\1.0\local\share\link-grammar. In order to use the Java bindings you'll need to build two extra DLLs, running from the link-grammar base directory the following commands: cd link-grammar gcc -g -shared -Wall -D_JNI_IMPLEMENTATION_ -Wl,--kill-at \ .libs/analyze-linkage.o .libs/and.o .libs/api.o .libs/build-disjuncts.o .libs/command-line.o \ .libs/constituents.o .libs/count.o .libs/error.o .libs/extract-links.o .libs/fast-match.o \ .libs/idiom.o .libs/massage.o .libs/post-process.o .libs/pp_knowledge.o .libs/pp_lexer.o \ .libs/pp_linkset.o .libs/prefix.o .libs/preparation.o .libs/print-util.o .libs/print.o \ .libs/prune.o .libs/read-dict.o .libs/resources.o .libs/string-set.o .libs/tokenize.o \ .libs/utilities.o .libs/word-file.o .libs/word-utils.o \ -o /usr/local/bin/link-grammar.dll gcc -g -shared -Wall -D_JNI_IMPLEMENTATION_ -Wl,--kill-at \ .libs/jni-client.o /usr/local/bin/link-grammar.dll \ -o /usr/local/bin/link-grammar-java.dll This will create link-grammar.dll and link-grammar-java.dll in the directory c:\msys\1.0\local\bin . These files, together with link-grammar-*.jar, will be used by Java programs. Make sure that this directory is in the %PATH setting, as otherwise, the DLL's will not be found. Alternately, Windows users who have access to the "Microsoft Visual C" product may wish to use the project files in the msvc6 directory, or alternately, those in the msvc9 directory. RUNNING the program: -------------------- To run the program issue the Unix command: ./link-parser This starts the program. The program has many user-settable variables and options. These can be displayed by entering !var at the link-parser prompt. Entering !help will display some additional commands. The dictionaries contain some utf-8 punctuation. These may generate errors for users in a non-utf-8 locale, such as the "C" locale. The locale can be set, for example, by saying export LANG=en_US.UTF-8 at the shell prompt. By default, the parser will use dictionaries at the installed location (typically in /usr/local/share). Other locations can be specified on the command line; for example: link-parser ../path/to-my/modified/data/en When accessing dictionaries in non-standard locations, the standard filenames are still assumed (i.e. 4.0.dict, 4.0.affix, etc.) TESTING the program: -------------------- The program can run in batch mode for testing the system on a large number of sentences. The following command runs the parser on a file called 4.0.batch ./link-parser < 4.0.batch The line "!batch" near the top of 4.0.batch turns on batch mode. In this mode sentences labeled with an initial "*" should be rejected and those not starting with a "*" should be accepted. The current batch file does report some errors, as do the files "4.0.biolg.batch" and "4.0.fixes.batch". Work is ongoing to fix these. The "4.0.fixes.batch" file contains sentences that have been fixed since the original 4.1 release of link-grammar. The "4.0.biolg.batch" contains biology/medical-text sentences from the BioLG project. USING the parser in your own applications: ------------------------------------------ There is a API (application program interface) to the parser. This makes it easy to incorporate it into your own applications. The API documented on the web site. SAT solver: ----------- The current parser uses an algorithm that runs in O(N^3) time, for a sentence containing N words. This algorithm is more or less a chart parser, a variant of the Earley parser. The SAT solver aims to replace the chart parser with an algorithm based on Boolean Satisfiability Theory; specifically using the MiniSAT solver. The SAT solver has a bit more overhead for shorter sentences, but is significantly faster for long sentences. To work properly, it needs to be attached to a parse ranking system. This work is incomplete, although the prototype works. The SAT solver is enabled by specifying ./configure --enable-sat-solver prior to compiling. Corpus Statistics: ------------------ The parser now contains some experimental code for using corpus statistics to provide a parse ranking, and to assign WordNet word senses to word, based on their grammatical usage. An overview of the idea is given on the OpenCog blog, here: http://brainwave.opencog.org/2009/01/12/determining-word-senses-from-grammatical-usage/ It is planned that the Corpus statistics database will be used to guide the SAT solver. To enable the corpus statistics, specify ./configure --enable-sat-solver prior to compiling. The database itself can be downloaded from http://gnucash.org/linas/nlp/data/linkgrammar-wsd/ The data is contained in an sqlite3 database file, disjuncts.20090430.db.bz2 Unzip this file (using bunzip2) and place it in the same directory with the other data files. Spell Checking: --------------- The parser will now run a spell-checker at an early stage, if it encounters a word that it does not know, and cannot guess, based on morphology. The spell-checker used is hunspell; support for it is automatically compiled in if the hunspell development environment is found. Spell checking may be disabled in the link-parser client with the !spell flag. Enter !help for more details. USING pkg-config: ----------------- To make compiling and linking easier, the current release uses the pkg-config system. To determine the location of the link-grammar header files, say `pkg-config --cflags link-grammar` To obtain the location of the libraries, say `pkg-config --libs link-grammar` Thus, for example, a typical makefile might include the targets: .c.o: cc -O2 -g -Wall -c $< `pkg-config --cflags link-grammar` $(EXE): $(OBJS) cc -g -o $@ $^ `pkg-config --libs link-grammar` JAVA bindings: -------------- This release includes Java bindings. Their use is optional. The bindings will be built automatically if jni.h can be found. Some common java JVM distributions (most notably, the ones from Sun) place this file in unusual locations, where it cannot be automatically found. To remedy this, make sure that JAVA_HOME is set. If jni.h still cannot be found, specify the location with the CPPFLAGS variable: so, for example, export CPPFLAGS="-I/opt/jdk1.5/include/:/opt/jdk1.5/include/linux" or export CPPFLAGS="-I/c/java/jdk1.6.0/include/ -I/c/java/jdk1.6.0/include/win32/" Please note that the use of /opt is non-standard, and most system tools will fail to find packages installed there. The building of the Java bindings can be disabled by configuring as below: ./configure --disable-java-bindings MULTI-THREADED USE: ------------------- The link-grammar library should be thread-safe as of version 4.3.6, although this has not yet been tested very well, and may possibly be buggy. To enable the thread-safe version of the library, run ./configure like so: ./configure --enable-pthreads The resulting library is thread-safe for all parsing and general use, with the exception of the startup/initialization code. That is, the initial open/creation of the parser dictionary should be done only once, from a single thread, and parsing should not be started until after the dictionary has been read and initialized. Parse options can be set on a per-thread basis, with the exception of verbosity, which is treated as a global. The library uses pthreads for the few spots where it needs to manage per-thread storage. It is possibly quite reasonable to abolish these few spots, thus making the library not only thread-safe, but thread agnostic (i.e. not require pthreads). The following exceptions and special notes apply: error.c -- uses pthreads to access error message area. utilities.c -- has global "verbosity". Uses pthreads for tracking memory usage. The memory usage code is just about obsolete, and could probably be discarded. jni-client.c - uses per-thread struct, but needs to be attached to JNIEnv somehow. malloc-dbg.c - not thread safe, not normally used; only for debugging. prefix.c - not thread-safe, but doesn't need to be; used only during initialization, and only if binreloc turned on. pp_lexer.c -- autogened code, original lex sources lost. This is only used when reading dictionaries, during initialization, and so doesn't need to be thread safe. utilities.h -- Windows doesn't have a thread-safe random-number generator, which means link-grammar for Windows is not currently thread-safe. COMMERCIAL use: --------------- The original authors would be pleased to discuss alternate licensing terms and/or additional research work, with those wanting to use this system under a paid contract. Academic products or papers should reference the various link grammar publications. ADDRESSES --------- If you have any questions, or find any bugs, please feel free to send a note to the mailing list: link-grammar@googlegroups.com Although all messages should go to the mailing list, the current maintainers can be contacted at: Dom Lachowicz - Linas Vepstas - The original authors of the Link Grammar parser are: Daniel Sleator sleator@cs.cmu.edu Computer Science Department 412-268-7563 Carnegie Mellon University www.cs.cmu.edu/~sleator Pittsburgh, PA 15213 Davy Temperley dtemp@theory.esm.rochester.edu Eastman School of Music 716-274-1557 26 Gibbs St. www.link.cs.cmu.edu/temperley Rochester, NY 14604 John Lafferty lafferty@cs.cmu.edu Computer Science Department 412-268-6791 Carnegie Mellon University www.cs.cmu.edu/~lafferty Pittsburgh, PA 15213 TODO -- Working Notes: ---------------------- Some working notes. incremental parsing: to avoid a combinatorial explosion of parses, it would be nice to have an incremental parsing, phrase by phrase, using a Viterbi-like algorithm to obtain the parse. Thus, for example, the parse of the last half of a long, run-on sentence should not be sensitive to the parse of the beginning of the sentence. Doing so would help with combinatorial explosion. So, for example, if the first half of a sentence has 4 plausible parses, and the last half has 4 more, then link-grammar reports 16 parses total. It would be much, much more useful to instead be given the factored results: i.e. the for plausible parses for the first half, and the four plausible parses for the last half. The lower combinatoric stress would ease the burden on downstream users of link-grammar. (This somewhat resembles the application of construction grammar ideas to the link-grammar dictionary). Assorted minor cleanup: -- Should provide a query that returns compile-time consts, e.g. the max number of characters in a word, or max words in a sentence -- Should remove compile-time constants, e.g. max words, max length etc. -- Split out parse-options, so that the command-line client options are not in the parse-options struct. Hand-refining verb patterns: A good reference for refining verb usage patterns is: COBUILD GRAMMAR PATTERNS 1: VERBS from THE COBUILD SERIES /from/ THE BANK OF ENGLISH HARPER COLLINS online at https://arts-ccr-002.bham.ac.uk/ccr/patgram/ http://www.corpus.bham.ac.uk/publications/index.shtml Quotations: Currently, tokenize.c ignores all ASCII double-quotes (grep for "quote_found" in the source). However, it does not do this for the various "curly" UTF8 quotes, such as ‘these’ and “these”. This results is some ugly parsing for sentences containing such quotes. (Note that these are in 4.0.affix). no-links-cross: Link Grammar uses a constraint that all linkages must be planar graphs. While this is mostly correct, it would probably be more correct to use "landmark transitivity" as articulated by Hudson in the Word Grammar theory. This might allow the elimination of most or all post-processing rules. This is done by making each link directional: one end of the link is the parent. (e.g. noun, noun-modifier: the noun is the parent). Parents are landmarks for children. Transitivity is applied to parent-child relationships. Specifically, the no-links-cross rule is replaced by two landmark transitivity rules: -- If B is a landmark for C, then A is also a type-L landmark for C -- If A is a landmark for C, then B is also a landmark for C where type-L means either a right-going or left-going link. See http://goertzel.org/ProwlGrammar.pdf for details. See also: http://www.phon.ucl.ac.uk/home/dick/enc/syntax.htm "to be fishing": Link grammar offers four parses of "I was fishing for evidence", two of which are given low scores, and two are given high scores. Of the two with high scores, one parse is clearly bad. Its links "to be fishing.noun" as opposed to the correct "to be fishing.gerund". That is, I can be happy, healthy and wise, but I certainly cannot be fishing.noun. This is perhaps not just a bug in the structure of the dictionary, but is perhaps deeper: link-grammar has little or no concept of lexical units (i.e. collocations, idioms, institutional phrases), which thus allows parses with bad word-senses to sneak in. The goal is to introduce more knowledge of lexical units into LG. Different word senses can have different grammar rules (and thus, the links employed reveal the sense of the word): for example: "I tend to agree" vs. "I tend to the sheep" -- these employ two different meanings for the verb "tend", and the grammatical constructions allowed for one meaning are not the same as those allowed for the other. Yet, the link rules for "tend.v" have to accommodate both senses, thus making the rules rather complex. Worse, it potentially allows for non-sense constructions. If, instead, we allowed the dictionary to contain different rules for "tend.meaning1" and "tend.meaning2", the rules would simplify (at the cost of inflating the size of the dictionary). "holes" in collocations: The link-grammar provides a mechanism to support circumpositions (or even more complicated multi-word structures) by means of the post-processing rules. However, post-processing rules for many common forms have not yet been written. The general problem is of supporting structures that have "holes" in the middle. For example, the adposition: ... from [xxx] on. "He never said another word from then on." "I promise to be quiet from now on." "Keep going straight from that point on." "We went straight from here on." ... from there on. "We went straight, from the house on to the woods." "We drove straight, from the hill onwards." Note that multiple words can fit in the slot [xxx]. Note the tangling of another prepositional phrase: "... from [xxx] on to [yyy]" More complicated collocations with holes include "First.. next..." "If ... then ..." 'Then' is optional, for example: "If it is raining, stay inside!" "if ... only ..." "If there were only more like you!" "... not only, ... but also ..." "Either ... or ..." "Both ... and ..." "Both June and Tom are coming" "ought ... if ..." "That ought to be the case, if John is not lying" "Someone ... who ..." "Someone is outside who wants to see you" The above are not currently supported. An example that is supported is the "non-referential it", e.g. "It ... that ..." "It seemed likely that John would go" The above is supported by means of special disjuncts for 'it' and 'that', which must occur in the same post-processing domain. See also: http://www.phon.ucl.ac.uk/home/dick/enc/syntax.htm#relative-clause http://www.phon.ucl.ac.uk/home/dick/enc/syntax.htm#extraposition