-
Notifications
You must be signed in to change notification settings - Fork 43
Running the test suite
To run the Links test suite execute:
./run-tests
This will run all the tests defined in *.tests files inside tests directory. If you want to run specific test suites you can do this by executing
./test-harness tests/foo.tests
where foo.tests is the name of the file with tests you want to run.
There are database tests in the subdirectories tests/{database,shredding,relational-lenses}. run-tests does not run the database tests by default. If you want to run tests requiring database access then execute one of the following:
./run-tests unit
./run-tests database
./run-tests shredding
./run-tests relational-lenses
The unit tests run against the local database configuration (tests/unit/config) and if that file is not present, copies it from the config.sample file in the same directory. This file gives the default PostgreSQL configuration used in the CI. Currently, running unit tests on other databases is not supported and the unit tests are not managed by the run-database-tests script or testsuite.json configuration files.
To run the tests from run-tests you need to setup a database, preferably PostgreSQL since other databases are not yet fully supported. Refer to Database setup for instructions how to compile Links with PostgreSQL support. You also need to create a links database and links user account and configure the database permissions appropriately, as described on the wiki page just mentioned.
To configure tests to run using an alternate database configuration (user+password/host), copy the appropriate config files in tests/<dbtest>/config.<dbname> to a local config file in the same directory. For the time being, this is recommended only if you know what you are doing.
The run-tests script is set up to run the database tests using a local configuration (tests/<dbtestdir>/config) if present, or the default config.pgsql configuration otherwise. Other database configurations (either PostgreSQL with a different username/password, or MySQL or Sqlite3 databases) can be tested using the run-database-tests script, which has the following usage:
usage: run-database-tests [-h] [-c CONFIG_FILE]
[-d {all,mysql,pgsql,sqlite3}] [-l] [-v]
working_dir
positional arguments:
working_dir Working directory of tests
optional arguments:
-h, --help show this help message and exit
-c CONFIG_FILE, --config_file CONFIG_FILE
Test configuration file (default: testsuite.json)
-d {all,mysql,pgsql,sqlite3}, --database {all,mysql,pgsql,sqlite3}
Run tests for specific database (all, mysql, pgsql,
sqlite3). "all" selects all databases listed in the
testsuite configuration file. Default: "pgsql"
-l, --local Test local database configuration
(<working_dir>/config) if present; otherwise, the
database(s) specified by the --database parameter are
tested.
-v, --verbose Print verbose debugging messages
The configuration file option is for the testsuite configuration testsuite.config, a JSON file that lists the possible databases, test setup filenames, and test filenames. Normally, using the default configuration files already present in the directories is sufficient, but another one can be specified if needed for debugging purposes.
Test setup filenames have either suffix .sql (for generic SQL that should work on any database by default) or .<dbname> (for SQL that needs to be customized to a specific database), where <dbname> is one of pgsql, mysql or sqlite3. The Links configuration file used is also chosen using <dbname>, that is, is of the form config.<dbname>. For example, to run the database tests in relational-lenses on MySQL using the default testsuite configuration and the Links configuration file config.mysql do:
./run-database-tests tests/relational-lenses -d mysql
The --local option implements the behavior used in §rnu-tests: if an unsuffixed local configuration file configis present in the database test directory, then that is used; the database configuration (including the database name / driver name) is parsed from that config file and the-doption is ignored. If noconfigfile is present then the standard behavior of using the database specified by the-d` operator is used.
The -d all argument indicates that for each testsuite config file, all databases listed there should be tested. It is for use in the CI environment, or for anyone doing database development that has all three databases installed and configured with configurations matching the CI environment. Since most people won\t do this, the default behavior in run-tests is to run all of the tests locally on eitehr Postgres or whatever database is specified in the local config files.
Each database test should consist of a pair of files: SQL database setup script (.sql extension) + Links test code (.links extension). Both files should have identical base name and should be placed in a directory containing database tests (at the moment tests/database/ and tests/shredding are such directories). Each directory contains:
-
config.<dbname>- the default database configuration for tests of database<dbname>in a given directory -
config- local database configuration for tests in a given directory. This file is ignored by git and can be freely edited to match the local database steup. Differently to previous incarnations of the testing infrastructure,configis not automatically created; if a local test configuration is needed, the appropriateconfig.<dbname>file should be copied and modified manually. -
testsuite.json- list of tests to run from a given directory. This has the following format:
"databases" :
[ <dbname>, ... ],
"setups" :
[ <setupname>, ... ],
"tests" :
[ <testname>, ... ]
}
The §run-database-testsscript first decides whcih databases/config files to use. If--local is used, then the configuration specified by the §tests/<dbtest>/config file is used if present; otherwise, the -d option determines the databases that will be tested. If -d all is specified then all of the databases listed in the databases part of the JSON file will be used; otherwise if -d <dbname> is specified then database <dbname> will be used. In this case, <dbname> is not checked to be one of the databases listed in testsuite.json; this is to allow trying other databases for debugging purposes without having to make temporary modifications to a version-controlled file.
The default behavior is -d pgsql and local is unset, but the run-tests script calls run-database-tests with --local set, so this is the recommended way of running database tests in a local repository.
Next, for each selected database, each setup SQL file in the list is run in order. If there is a database-specific SQL file, named <setupname>.<dbname>, this is run instead of the generic file <setupname>.sql; otherwise the default one is used. This is useful, for example, where different databases have different ways of declaring automatically-populated fields such as AUTO_INCREMENT in MySQL or serial in PostgreSQL.
Finally, for each selected database, each Links test file is run with the associated configuration/parameters. Again, the order of the list elements matters, so care should be taken if reordering or adding tests. The test files all have suffix .links and database-specific behavior is not supposed to be reflected in the Links code. Not all Links features are supported by all databases, though, and in particular the shredding and relational-lenses tests are not yet supported by all three.
To run the Links test suite execute:
./run-tests
This will run all the tests defined in *.tests files inside tests directory, as well as all database tests defined inside tests/database directory. If you only want to run some of the tests you can do this by executing
./test-harness tests/foo.tests
where foo.tests is the name of the file with tests you want to run. If you want to skip tests requiring database access run
./run-tests no-db
If you want to run only those tests that require database access run
./run-tests db-only
To run the tests requiring database access you need to setup PostgreSQL database. Refer to Database setup for instructions how to compile Links with PostgreSQL support. You also need to create links database and have permissions to access it.
To run shredding tests, run
./run-tests db-only shredding
To configure tests to run using an alternate database configuration (user+password/host), edit the config files in tests/{database,shredding}/config. These are created from the respective sample files if they do not exists, but will not be overwritten if they are already there.
Each database test should consist of a pair of files: SQL database setup script (.sql extensions) + Links test code (.links extension). Both files should have identical base name and should be placed in a directory containing database tests (at the moment tests/database/ and tests/shredding are such directories). Each directory contains:
-
config.sample- a default database configuration for tests in a given directory -
config- actual database configuration for tests in a given directory. This file is ignored by git and can be freely edited to match the local database steup. Ifconfigfile does not exist it will be created fromconfig.samplebyrun-testsscript. -
testsuite.config- list of tests to run from a given directory. Each line should contain a name of a test to run. If a test is known to fail precede its name with wordBROKEN, e.g.BROKEN xpath.