107 lines · plain
1Overview2==================================================3This directory contains a random test generator for the gmp4compatibility layer of imath. The tests cases are randomly generated5and run with both gmp and imath. The results are compared and any6mismatching results are flagged as failures.7 8You should not see any failures when running these tests.9 10Requirements11==================================================12These tests use the python ffi to run the imath and gmp functions. To13run these tests you will need the following items14 15 * libimath.so16 * python 317 * gmp library and header files18 19 20Running21==================================================22All the tests cases will be generated and run automatically by the23makefile. First, make sure you have built libimath.so in the top level24imath directory.25 26Use the following command to generate and run the tests27 28 $ make TESTS=random.tests29 30This should generate all the needed wrapper files along with the31randomized unit tests. By default the unit tests are output to the32random.tests file. The tests can be run by hand using the following33command line34 35 $ ./runtest random.tests36 37You can also write your own unit tests and run them by passing the38file name to the runtest script.39 40 41Testing Methodology42==================================================43The goal of our testing is to ensure that we are compatible with the44gmp api. To test our compatibility layer we generate inputs for each45gmp api that we wrap and then call the gmp version of the api and46compare the results to the imath version. The results should be47identical. Any output mismatch is considered an error.48 49 50Testcase Generation51--------------------52The test data generation is inspired by the QuickCheck testing53methodolgy. We want to test both random data and important values such54as 0,1,-1, etc. We generate input data based on the type of the input55parameter. Most apis take either an mpz_t or a signed/unsigned long.56 57For each parameter of the given type we generate the following values:58 59 mpz_t: 0,1,-1, all min/max values, all min/max values +/- 160 small numbers: 1-4 digits61 medium numbers: 5-20 digits62 large numbers: 20-100 digits63 long : 0,1,-1 short,int,long min/max values64 random long values65 unsigned long: 0,1,-1, unsigned short, unsigned int, unsigned long min/max values66 random unsigned long values67 68The generated data for each paramater is combined to produce a series69of inputs to the function. The input data format looks like:70 71mpz_add|0,1,272 73Which represents the call mpz_add(0, 1, 2). Additional test cases can be written by hand.74 75Test Structure76--------------------77The tests are run using the python ffi (the ctypes module). We have a78single description of each api that is used to generate the following:79 80 * intput data81 * c function wrapper to call the libgmp or libimath function82 * python wrapper to call both libgmp and libimath wrappers and compare results83 84The test_gmp.so and test_imath.so libraries are loaded at runtime by85the python runner scrip and used to run each test input and compare86the results.87 88The code generation pattern looks something like this:89 90genctest.py ~~generates~~> gmp_test.c <--includes-- gmp_custom_test.c91gmp_test.c ~~generates~~> gmp_tests.so <--links-- libgmp.so92genpytest.py ~~generates~~> wrappers.py <--calls-- gmp_test.so93gmp_test.so --loads--> runtest.py <--includes-- wrappers.py94 ^95 |96 reads97 |98gendata.py ~~generates~~> random.tests99 100Adding tests for new APIs101==================================================102New apis can be tested by modifying the gmpapy.py file. This file103contains a list of all apis that will be tested. Adding an api to the104apis list will cause code and input data to be generated for that api.105Rerunning make will generate the test data and code to run the test.106 107