Several Perl-modules are required for the HTTP-requests. These include:
  • LWP (Library for WWW access in Perl), which is part of the libwww-bundle, freely available at CPAN (check the modules/by-module/LWP-directory for a file called libwww-perl-5.36.tar.gz).
  • HTML::Parser, necessary for parsing HTML-commands, also available from CPAN (check the modules/by-module/HTML-directory for a file called HTML-Parser-3.10.tar.gz).
  • The former two modules depend on the URI-module to be installed (check the modules/by-module/URI-directory at CPAN for a file called URI-1.08.tar.gz).
  • The former modules depend on the MIME::Base64-module to be installed (check the modules/by-module/MIME-directory at CPAN for a file called MIME-Base64-2.11.tar.gz).

    •  

     

    Copies of all of these four packages are also available from the Perl-modules-directory at the PubCrawler-FTP-site.

    These modules can be easily installed.
    Check out the Perl-module installation documentation or follow these steps:

  • Since some modules are dependent on others you should install the packages in the following order:
    1. MIME-Base64-2.11.tar.gz
    2. URI-1.08.tar.gz
    3. HTML-Parser-3.10.tar.gz
    4. libwww-perl-5.36.tar.gz
    This helps avoiding error-messages when testing the compiled modules before installing them (make test).
  • If you plan to install modules into one of your own directories, you should assign this path to the PERLLIB environmental variable before starting the installation (the following examples presume that you installed your modules into the directory 'lib' below your home-directory):
    • To check if you succeeded to include the directory in Perl's search path enter the following command:
    perl -e 'foreach (@INC) {print "$_\n";}'
    This will produce all the paths in which Perl will look for modules.
    If the directory you specified doesn't show up something went wrong and you will have to try a different method.
  • download, uncompress and untar the module:
    gzip -cd <name-of-package> | tar xvf -
    This will create a new directory with the name of the module.
  • change into the newly created directory (cd <name-of-package>) and create a Makefile by running

    • perl Makefile.PL
    or, if you would like to install it into a specific file:
    perl Makefile.PL LIB=your_local_directory
    eg. perl Makefile.PL LIB=/home/fiona/lib
    (the latter version is very useful, if you don't have the permissions for installing modules into the Perl-tree).
  • compile and test the module with the following commands:

    • make
    make test
  • if the tests were successful, install the module by running

    • make install
  • Remember: If you explicitly specify an install-directory for a module, make sure that it is included in the INC-array (see LOCATION OF MODULES at the beginning of pubcrawler.pl)!
  • As an alternative to these modules you can use a browser that works from the command line, like lynx. It is probably already installed, but if it isn't you can get it from http://lynx.browser.org.
    • If you don't have the modules installed and want to use an alternative browser instead, make sure that the three lines following the section titled #### ADDITIONAL MODULES #### at the beginning of pubcrawler.pl are commented out (prepend a # before each line). You also have to comment out the line
    @proxy_config = split /\n/, (get "$proxy_conf");
    somewhere in the middle of the script.
    To use the alternative browser pass the command for evoking it via the command line option -lynx or specify it in the configuration file.


    Last modified at $Date: 2018/05/25 12:12:10 $