264 lines · plain
1Let's pick test/settings/TestSettings.py as our example. First, notice the file2name "TestSettings.py", the Test*.py pattern is the default mechanism that the3test driver uses for discovery of tests. As to TestSettings.py, it defines a4class:5 6class SettingsCommandTestCase(TestBase):7 8derived from TestBase, which is defined in test/lldbtest.py and is itself9derived from Python's unittest framework's TestCase class. See also10http://docs.python.org/library/unittest.html for more details.11 12To just run the TestSettings.py test, chdir to the lldb test directory, and then13type the following command:14 15/Volumes/data/lldb/svn/trunk/test $ ./dotest.py settings16----------------------------------------------------------------------17Collected 6 tests18 19----------------------------------------------------------------------20Ran 6 tests in 8.699s21 22OK (expected failures=1)23/Volumes/data/lldb/svn/trunk/test $ 24 25Pass '-v' option to the test driver to also output verbose descriptions of the26individual test cases and their test status:27 28/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -v settings29----------------------------------------------------------------------30Collected 6 tests31 32test_set_auto_confirm (TestSettings.SettingsCommandTestCase)33Test that after 'set auto-confirm true', manual confirmation should not kick in. ... ok34test_set_output_path (TestSettings.SettingsCommandTestCase)35Test that setting target.process.output-path for the launched process works. ... expected failure36test_set_prompt (TestSettings.SettingsCommandTestCase)37Test that 'set prompt' actually changes the prompt. ... ok38test_set_term_width (TestSettings.SettingsCommandTestCase)39Test that 'set term-width' actually changes the term-width. ... ok40test_with_dsym (TestSettings.SettingsCommandTestCase)41Test that run-args and env-vars are passed to the launched process. ... ok42test_with_dwarf (TestSettings.SettingsCommandTestCase)43Test that run-args and env-vars are passed to the launched process. ... ok44 45----------------------------------------------------------------------46Ran 6 tests in 5.735s47 48OK (expected failures=1)49/Volumes/data/lldb/svn/trunk/test $ 50 51Underneath, the '-v' option passes keyword argument verbosity=2 to the52Python's unittest.TextTestRunner (see also53http://docs.python.org/library/unittest.html#unittest.TextTestRunner). For very54detailed descriptions about what's going on during the test, pass '-t' to the55test driver, which asks the test driver to trace the commands executed and to56display their output. For brevity, the '-t' output is not included here.57 58Notice the 'expected failures=1' message at the end of the run. This is because59of a bug currently in lldb such that setting target.process.output-path to60'stdout.txt' does not have any effect on the redirection of the standard output61of the subsequent launched process. We are using unittest to decorate (mark)62the particular test method as such:63 64 @unittest.expectedFailure65 # rdar://problem/843579466 # settings set target.process.output-path does not seem to work67 def test_set_output_path(self):68 69See http://docs.python.org/library/unittest.html for more details.70 71Now let's look inside the test method:72 73 def test_set_output_path(self):74 """Test that setting target.process.output-path for the launched process works."""75 self.build()76 77 exe = os.path.join(os.getcwd(), "a.out")78 self.runCmd("file " + exe, CURRENT_EXECUTABLE_SET)79 80 # Set the output-path and verify it is set.81 self.runCmd("settings set target.process.output-path 'stdout.txt'")82 self.expect("settings show target.process.output-path",83 startstr = "target.process.output-path (string) = 'stdout.txt'")84 85 self.runCmd("run", RUN_SUCCEEDED)86 87 # The 'stdout.txt' file should now exist.88 self.assertTrue(os.path.isfile("stdout.txt"),89 "'stdout.txt' exists due to target.process.output-path.")90 91 # Read the output file produced by running the program.92 with open('stdout.txt', 'r') as f:93 output = f.read()94 95 self.expect(output, exe=False,96 startstr = "This message should go to standard out.")97 98The self.build() statement is used to build a binary for this99test instance. This will build the binary for the current debug info format. If100we wanted to avoid running the test for every supported debug info format we101could annotate it with @no_debug_info_test. The test would then only be run for102the default format. The logic for building a test binary resides in the builder103modules (packages/Python/lldbsuite/test/builders/builder.py)104 105After the binary is built, it is time to specify the file to be used as the main106executable by lldb:107 108 # Construct the path to a file "a.out" inside the test's build folder.109 exe = self.getBuildArtifact("a.out")110 self.runCmd("file " + exe, CURRENT_EXECUTABLE_SET)111 112The runCmd() method is defined in the TestBase base class and its purpose is to113pass the specified command to the lldb command interpreter. It's like you're114typing the command within an interactive lldb session.115 116The CURRENT_EXECUTABLE_SET is an assert message defined in the lldbtest module117so that it can be reused from other test modules.118 119By default, the runCmd() is going to check the return status of the command120execution and fails the test if it is not a success. The assert message, in our121case CURRENT_EXECUTABLE_SET, is used in the exception printout if this happens.122 123There are cases when we don't care about the return status from the command124execution. This can be accomplished by passing the keyword argument pair125'check=False' to the method.126 127After the current executable is set, we'll then execute two more commands:128 129 # Set the output-path and verify it is set.130 stdout = self.getBuildArtifact('stdout.txt')131 self.runCmd("settings set target.process.output-path '%s'" %stdout)132 self.expect("settings show target.process.output-path",133 SETTING_MSG("target.process.output-path"),134 startstr = "target.process.output-path (string) = '.*stdout.txt'")135 136The first uses the 'settings set' command to set the static setting137target.process.output-path to be 'stdout.txt', instead of the default138'/dev/stdout'. We then immediately issue a 'settings show' command to check139that, indeed, the setting did take place. Notice that we use a new method140expect() to accomplish the task, which in effect issues a runCmd() behind the141door and grabs the output from the command execution and expects to match the142start string of the output against what we pass in as the value of the keyword143argument pair:144 145 startstr = "target.process.output-path (string) = '%s'" %stdout146 147Take a look at TestBase.expect() within lldbtest.py for more details. Among148other things, it can also match against a list of regexp patterns as well as a149list of sub strings. And it can also perform negative matching, i.e., instead150of expecting something from the output of command execution, it can perform the151action of 'not expecting' something.152 153This will launch/run the program:154 155 self.runCmd("run", RUN_SUCCEEDED)156 157And this asserts that the file 'stdout.txt' should be present after running the158program.159 160 # The 'stdout.txt' file should now exist.161 self.assertTrue(os.path.isfile(stdout),162 "stdout.txt' exists due to target.process.output-path.")163 164Also take a look at main.cpp which emits some message to the stdout. Now, if we165pass this assertion, it's time to examine the contents of the file to make sure166it contains the same message as programmed in main.cpp:167 168 # Read the output file produced by running the program.169 with open(stdout, 'r') as f:170 output = f.read()171 172 self.expect(output, exe=False,173 startstr = "This message should go to standard out.")174 175We open the file and read its contents into output, then issue an expect()176method. The 'exe=False' keyword argument pair tells expect() that don't try to177execute the first arg as a command at all. Instead, treat it as a string to178match against whatever is thrown in as keyword argument pairs!179 180There are also other test methods present in the TestSettings.py mode:181test_set_prompt(), test_set_term_width(), test_set_auto_confirm(),182test_with_dsym(), and test_with_dwarf(). We are using the default test loader183from unittest framework, which uses the 'test' method name prefix to identify184test methods automatically.185 186This finishes the walkthrough of the test method test_set_output_path(self).187Before we say goodbye, notice the little method definition at the top of the188file:189 190 @classmethod191 def classCleanup(cls):192 system(["/bin/sh", "-c", "rm -f "+self.getBuildArtifact("output.txt")])193 system(["/bin/sh", "-c", "rm -f "+self.getBuildArtifact("stdout.txt")])194 195This is a classmethod (as shown by the @classmethod decorator) which allows the196individual test class to perform cleanup actions after the test harness finishes197with the particular test class. This is part of the so-called test fixture in198the unittest framework. From http://docs.python.org/library/unittest.html:199 200A test fixture represents the preparation needed to perform one or more tests,201and any associate cleanup actions. This may involve, for example, creating202temporary or proxy databases, directories, or starting a server process.203 204The TestBase class uses such fixture with setUp(self), tearDown(self),205setUpClass(cls), and tearDownClass(cls). And within teraDownClass(cls), it206checks whether the current class has an attribute named 'classCleanup', and207executes as a method if present. In this particular case, the classCleanup()208calls a utility function system() defined in lldbtest.py in order to remove the209files created by running the program as the tests are executed.210 211This system() function uses the Python subprocess module to spawn the process212and to retrieve its results. If the test instance passes the keyword argument213pair 'sender=self', the detailed command execution through the operating system214also gets recorded in a session object. If the test instance fails or errors,215the session info automatically gets dumped to a file grouped under a directory216named after the timestamp of the particular test suite run.217 218For simple cases, look for the timestamp directory in the same directory of the219test driver program dotest.py. For example, if we comment out the220@expectedFailure decorator for TestSettings.py, and then run the test module:221 222/Volumes/data/lldb/svn/trunk/test $ ./dotest.py -v settings223----------------------------------------------------------------------224Collected 6 tests225 226test_set_auto_confirm (TestSettings.SettingsCommandTestCase)227Test that after 'set auto-confirm true', manual confirmation should not kick in. ... ok228test_set_output_path (TestSettings.SettingsCommandTestCase)229Test that setting target.process.output-path for the launched process works. ... FAIL230test_set_prompt (TestSettings.SettingsCommandTestCase)231Test that 'set prompt' actually changes the prompt. ... ok232test_set_term_width (TestSettings.SettingsCommandTestCase)233Test that 'set term-width' actually changes the term-width. ... ok234test_with_dsym (TestSettings.SettingsCommandTestCase)235Test that run-args and env-vars are passed to the launched process. ... ok236test_with_dwarf (TestSettings.SettingsCommandTestCase)237Test that run-args and env-vars are passed to the launched process. ... ok238 239======================================================================240FAIL: test_set_output_path (TestSettings.SettingsCommandTestCase)241Test that setting target.process.output-path for the launched process works.242----------------------------------------------------------------------243Traceback (most recent call last):244 File "/Volumes/data/lldb/svn/trunk/test/settings/TestSettings.py", line 125, in test_set_output_path245 "'stdout.txt' exists due to target.process.output-path.")246AssertionError: False is not True : 'stdout.txt' exists due to target.process.output-path.247 248----------------------------------------------------------------------249Ran 6 tests in 8.219s250 251FAILED (failures=1)252/Volumes/data/lldb/svn/trunk/test $ ls 2010-10-19-14:10:49.059609253 254NOTE: This directory name has been changed to not contain the ':' character255 which is not allowed in windows platforms. We'll change the ':' to '_'256 and get rid of the microsecond resolution by modifying the test driver.257 258TestSettings.SettingsCommandTestCase.test_set_output_path.log259/Volumes/data/lldb/svn/trunk/test $ 260 261We get one failure and a timestamp directory 2010-10-19-14:10:49.059609.262For education purposes, the directory and its contents are reproduced here in263the same directory as the current file.264