v811_spc009/external/autotest/client/site_tests/platform_PrinterPpds

Tests naming convention
-----------------------
Names of tests (suffixes of control.* files) are build from 2 or 3 words
separated by an underscore character. The first one is 'core' or 'ext', what
defines the set of PPD files to test. These two are described in the section
below. The second word of the name is either 'test' or 'dump'. The third and
optional word is 'debug'. All these three words influence the testing
procedure used by the autotest, their meanings are described in the next
paragraphs.

Input sets of PPD files ('core' and 'ext')
------------------------------------------
There are two input sets of PPD files to test: 'core' and 'ext'. 'core' is the
smaller one and represents clusters defined in the file large_clusters.txt; it
is built by taking the first element (PPD filename) from every line of this
file. The set 'ext' is built in similar way by taking the first element (PPD
filename) from every line of small_clusters.txt file; however the elements
already present in 'core' are omitted. Currently, 'core' contains around ~130
PPD files; the exact size equals the number of lines in large_clusters.txt.
The set 'ext' contains around ~1400 PPD files; the exact size equals the number
of lines in small_clusters.txt minus the number of lines in large_clusters.txt.

Overall testing procedure
-------------------------
The purpose of this autotest is to verify that given subset of PPD files work
in ChromeOS. Each PPD file is tested with the following procedure:
1. A printer driver is added to CUPS server.
2. Test documents are printed on the configured printer.
3. Raw output from the CUPS server is intercepted by, so called, FakePrinter.
4. CUPS logs are parsed to make sure that no errors occured.
5. Obtained outputs are verified (see below) - test only.
6. A printing pipeline used by CUPS is rerun in shell (see below) - debug only.
7. All obtained outputs & logs are saved on the device (see below) - dump only.
8. The printer driver is removed from CUPS server.
This procedure is repeated for every PPD file. The number of PPD files may be
large (~2K files). To decrease amount of time needed by the autotest, several
PPD files are tested simultaneously in parallel threads. Autotest always run
the procedure for all given PPD files and print a summary report at the end.
If at least one of PPD files fails, whole autotest is finished with failure
(but always all PPD files are processed).

Output verification ('test')
----------------------------
Intercepted output is verified by comparision with the previous results
obtained for the same PPD. We cannot store outputs directly, because their
total size may have hundreds of megabytes. Instead of that short digest is
calculated for each obtained document and it is used for comparison.
A function for digests calculation is in the 'helpers.py' file. Not all
outputs can be tested this way because for some PPD files produced contents
differ between runs. List of PPD files for which we cannot calculate
constant digest is saved in the file digests_denylist.txt. Files with
expected digests for every test document are stored in the directory "digests".
If a digests for given pair (test document, PPD file) is missing, the test
checks only check if the output is not empty (or not too short).

Rerun printing pipeline ('debug')
---------------------------------
Every PPD file defines printing pipeline used by CUPS, consisting of external
binaries (so called filters). To make sure that every part of the printing
pipeline works correctly, it can be run outside CUPS in Linux shell. In tests
with 'debug' suffix in their names, whole pipeline is rerun outside CUPS. In
this mode every document is processed twice: the first time by CUPS and the
second time by extracted pipeline run as bash script. Additionally, each
pipeline's step is verified by checking its return code (it is supposed to be
equal 0) and the final output is compared with the output returned by CUPS
server. If the test is also defined as 'dump', each intermediate content is
saved to the directory given in path_outputs parameter.
This mode requires more disk space on tested device (~2GB in /tmp) and takes
more execution time (~2 times more). All information needed to reconstruct
a pipeline used by CUPS can are extracted from CUPS logs after the job is
completed. To do that, CUPS configuration must be modified: the LogLevel
option in /etc/cups/cupsd.conf must be set to "debug". This modification is
done when the autotest is initialized; the original setting is restored in
the cleaup stage. To make these modifications, 'debug' tests remount the root
partition in read-write mode. The partition is NOT remounted back to read-only
mode.

Save outputs and logs ('dump')
------------------------------
All obtained outputs and logs are saved on the device in the directory
/tmp/PrinterPpds_outputs/. Results obtained from PPD files with the same prefix
are grouped together and stored in single archive to limit usage of disk space.

Test parameters
---------------
path_docs - path to directory with test documents (PDF files)
path_ppds - path to directory with PPD files, it is supposed to be compressed
            as .tar.xz files (with a command "tar cJf ...")
path_digests - path to directory with files containing digests for
            verification, if not set then outputs are not verified
path_outputs - if set, then all outputs are dumped there (given directory is
            deleted if already exists); also all digests files are recalculated
            and saved in the same directory
debug_mode - True or False, if True then for every test the printing pipeline
            is extracted from CUPS logs and executed again outside CUPS

Generating new digests
----------------------
The following procedure can be used to update digests:
1. Run the test defined in control.all_dump:
        test_that <device IP>  platform_PrinterPpds.all_dump
2. Download generated files with digests to your workstation
        rsync root@<device IP>:/tmp/PrinterPpds_outputs/*.digests <local dir>
3. Replace the files from the "digests" directory and commit changes

Updating the archives with PPD files
------------------------------------
Currently, all tests are based on PPD files stored in local directories. The
autotest can download all PPD files by itself, but we do not use this option
to limit the number of possible points of failures during testing. Archives
with PPD files are prepared with ppdTool.go:
1. Delete old files:
    rm ppds_core.tar.xz ppds_ext.tar.xz large_clusters.txt small_clusters.txt
2. Download all PPD files to ppds_all directory:
    go run ppdTool.go download
3. Calculate new clusters:
    go run ppdTool.go compare
4. Compress new directories with PPD files:
    tar cJf ppds_core.tar.xz ppds_core
    tar cJf ppds_ext.tar.xz ppds_ext

Others
------
* How to get comma-separated list of all used ghostscript devices?
  1. Go to the directory generated by all_dump_debug
  2. Unpack all tar.xz archives (not on the device)
        for d in *.pdf;
        do
            cd $d;
            for f in *.tar.xz; do tar xf $f; done ;
            cd ..;
        done
  3. Run the following
        grep -o sDEVICE=[^\ ]*\  ./*.pdf/*.err[12345] | cut -d \= -f 2 | sort | uniq | tr -d ' ' | tr '\n' ','
  4. Add to the list uniprint (it is not caught by the command above)
* How to get a list of all used cups filters?
  1. Go to the directory generated by all_dump_debug
  2. Unpack all tar.xz archives (not on the device)
        for d in *.pdf;
        do
            cd $d;
            for f in *.tar.xz; do tar xf $f; done ;
            cd ..;
        done
  3. Run the following
        grep -o '(exec -a "Fake[^"]*" [^ ]* ' ./*.pdf/*.sh | cut -d \  -f 4 | sort | uniq