The filosofy to test is:
- Test multiple distributions. (see .travis.yml)
- Test multiple version of Ansible, previous, current and next previous and future. (see tox.ini)
In Travis CI these combinations are called a matrix. You can consider this overview per role:
| Distribution | Ansible 2.9 | Ansible 2.10 | Ansible devel |
|---|---|---|---|
| Alpine latest | yes | yes | yes |
| Alpine edge | yes | yes | yes |
| Archlinux (base) | yes | yes | yes |
| CentOS 7 | yes | yes | yes |
| CentOS latest | yes | yes | yes |
| Debian stable | yes | yes | yes |
| Debian latest | yes | yes | yes |
| Debian unstable | yes | yes | yes |
| Fedora latest | yes | yes | yes |
| Fedora rawhide | yes | yes | yes |
| OpenSuse Leap | yes | yes | yes |
| OpenSuse Tumbleweed | yes | yes | yes |
| Ubuntu Artful (17) | yes | yes | yes |
| Ubuntu latest | yes | yes | yes |
| Ubuntu devel | yes | yes | yes |
Read [this page to understand the tools (Travis, Molecule and Tox)](tox-molecule-travis.html] better.
There are multiple tests configured, here is how they relate.
To test an Ansible role, Travis CI runs molecule on a commit. This verifies that the role does it's job, but does not ensure that it works in combination with other roles.
An example for the unit test for the Ansible role java.
Because distributions, molecule, and ansible change over time, a monthly test is done to all roles using this schedule:
|------------|------------| |Day of month|Ansible Role| |1|aide| |1|alternatives| |1|anaconda| |1|ansible| |1|ansible_lint| |1|apt_autostart| |1|apt_repository| |1|ara| |1|artifactory| |1|at| |1|atom| |1|auditd| |1|autofs| |1|auto_update| |1|azure_cli| |2|backup| |2|bios_update| |2|bootstrap| |2|buildtools| |3|ca| |3|ca_certificates| |3|cargo| |3|certbot| |3|cis| |3|cntlm| |3|collabora_online| |3|collectd| |3|common| |3|container_docs| |3|consul| |3|core_dependencies| |3|cron| |3|cups| |3|clamav| |4|debug| |4|dhcpd| |4|digitalocean_agent| |4|diskspace| |4|dns| |4|dnsmasq| |4|docker| |4|docker_ce| |4|docker_compose| |4|dovecot| |4|dsvpn| |5|earlyoom| |5|eclipse| |5|environment| |5|epel| |5|etherpad| |6|facts| |6|fail2ban| |6|filesystem| |6|firewall| |6|forensics| |6|functions| |7|git| |7|gitlab_runner| |7|glusterfs| |7|gnome| |7|grub| |7|go| |7|gotop| |8|haproxy| |8|hashicorp| |8|haveged| |8|httpd| |8|hostname| |9|investigate| |9|irslackd| |10|java| |10|jenkins| |10|jitsi| |11|keepalived| |11|kernel| |11|kubectl| |12|libvirt| |12|logrotate| |12|logwatch| |12|luks| |12|lvm| |12|locale| |12|lynis| |13|memcached| |13|maintenance| |13|mate| |13|mediawiki| |13|microsoft_repository_keys| |13|mitogen| |13|modprobe| |13|minikube| |13|molecule| |13|moodle| |13|mount| |13|mssql| |13|mysql| |14|natrouter| |14|nextcloud| |14|nfsserver| |14|nginx| |14|node_red| |14|nomad| |14|npm| |14|ntp| |15|obsproject| |15|omsagent| |15|openssh| |15|openssl| |15|owncloud| |16|packer| |16|php| |16|php_fpm| |16|phpmyadmin| |16|postfix| |16|postgres| |16|powertop| |16|powertools| |16|python_pip| |18|qemu| |18|rabbitvcs| |18|reboot| |18|redis| |18|remi| |18|restore| |18|revealmd| |18|roundcubemail| |18|rpmfusion| |18|rsyslog| |18|ruby| |18|rundeck| |19|scl| |19|selinux| |19|service| |19|snmpd| |19|software| |19|sosreport| |19|spamassassin| |19|squid| |19|storage| |19|stratis| |19|subversion| |19|sudo-pair| |19|swap| |19|sysctl| |19|sysstat| |19|systemd| |20|terraform| |20|test_connection| |20|tftpd| |20|tigervnc| |20|tlp| |20|tomcat| |20|travis| |20|tune2fs| |20|turn| |21|types| |21|ulimit| |21|umask| |21|update| |21|update_package_cache| |21|update_pip_packages| |21|upgrade| |21|unbound| |21|unowned_files| |21|users| |22|vagrant| |22|vault| |22|vdo| |22|virtualbox| |24|xinetd| |24|xrdp| |26|zabbix_agent| |26|zabbix_proxy| |26|zabbix_repository| |26|zabbix_server| |26|zabbix_web|
To test a combination of Ansible roles, Travis CI runs terraform and a complex playbook.
There is currently one integration test. A report is saved every run.