417 lines · plain
1Open Projects2=============3 4The following is a mostly unordered set of the ideas for improvements to the5LLDB debugger. Some are fairly deep, some would require less effort.6 7Speed up type realization in lldb8---------------------------------9 10The type of problem I'm addressing here is the situation where you are11debugging a large program (lldb built with debug clang/swift will do) and you12go to print a simple expression, and lldb goes away for 30 seconds. When you13sample it, it is always busily churning through all the CU's in the world14looking for something. The problem isn't that looking for something in15particular is slow, but rather that we somehow turned an bounded search (maybe16a subtype of "std::string" into an unbounded search (all things with the name17of that subtype.) Or didn't stop when we got a reasonable answer proximate to18the context of the search, but let the search leak out globally. And quite19likely there are other issues that I haven't guessed yet. But if you end up20churning though 3 or 4 Gig of debug info, that's going to be slow no matter how21well written your debug reader is...22 23My guess is the work will be more in the general symbol lookup than in the24DWARF parser in particular, but it may be a combination of both.25 26As a user debugging a largish program, this is the most obvious lameness of27lldb.28 29Symbol name completion in the expression parser30-----------------------------------------------31 32This is the other obvious lameness of lldb. You can do:33 34::35 36 (lldb) frame var foo.b37 38and we will tell you it is "foo.bar". But you can't do that in the expression39parser. This will require collaboration with the clang/swift folks to get the40right extension points in the compiler. And whatever they are, lldb will need41use them to tell the compiler about what names are available. It will be42important to avoid the pitfalls of #1 where we wander into the entire DWARF43world.44 45Make a high speed asynchronous communication channel46----------------------------------------------------47 48All lldb debugging nowadays is done by talking to a debug agent. We used the49gdb-remote protocol because that is universal, and good enough, and you have to50support it anyway since so many little devices & JTAG's and VM's etc support51it. But it is really old, not terribly high performance, and can't really52handle sending or receiving messages while the process is supposedly running.53It should have compression built in, remove the hand-built checksums and rely54on the robust communication protocols we always have nowadays, allow for55out-of-order requests/replies, allow for reconnecting to a temporarily56disconnected debug session, regularize all of the packet formatting into JSON57or BSON or whatever while including a way to do large binary transfers. It must58be possible to come up with something faster, and better tunable for the many59communications pathways we end up supporting.60 61Fix local variable lookup in the lldb expression parser62-------------------------------------------------------63 64The injection of local variables into the clang expression parser is65currently done incorrectly - it happens too late in the lookup. This results66in namespace variables & functions, same named types and ivars shadowing67locals when it should be the other way around. An attempt was made to fix68this by manually inserting all the visible local variables into wrapper69function in the expression text. This mostly gets the job done but that70method means you have to realize all the types and locations of all local71variables for even the simplest of expressions, and when run on large72programs (e.g. lldb) it would cause unacceptable delays. And it was very73fragile since an error in realizing any of the locals would cause all74expressions run in that context to fail. We need to fix this by adjusting75the points where name lookup calls out to lldb in clang.76 77Support calling SB & commands everywhere and support non-stop debugging78-----------------------------------------------------------------------79 80There is a fairly ad-hoc system to handle when it is safe to run SB API's and81command line commands. This is actually a bit of a tricky problem, since we82allow access to the command line and SB API from some funky places in lldb. The83Operating System plugins are the most obvious instance, since they get run84right after lldb is told by debugserver that the process has stopped, but85before it has finished collating the information from the stop for presentation86to the higher levels. But breakpoint callbacks have some of the same problems,87and other things like the scripted stepping operations and any fancier88extension points we want to add to the debugger are going to be hard to89implement robustly till we work on a finer-grained and more explicit control90over who gets to control the process state.91 92We also won't have any chance of supporting non-stop debugging - which is a93useful mode for programs that have a lot of high-priority or real-time worker94threads - until we get this sorted out.95 96Finish the language abstraction and remove all the unnecessary API's97--------------------------------------------------------------------98 99An important part of making lldb a more useful "debugger toolkit" as opposed to100a C/C++/ObjC/Swift debugger is to have a clean abstraction for language101support. We did most, but not all, of the physical separation. We need to102finish that. And then by force of necessity the API's really look like the103interface to a C++ type system with a few swift bits added on. How you would104go about adding a new language is unclear and much more trouble than it is105worth at present. But if we made this nice, we could add a lot of value to106other language projects.107 108Add some syntax to generate data formatters from type definitions109-----------------------------------------------------------------110 111Uses of the data formatters fall into two types. There are data formatters for112types where the structure elements pretty much tell you how to present the113data, you just need a little expression language to express how to turn them114into what the user expects to see. Then there are the ones (like pretty much115all our Foundation/AppKit/UIKit formatters) that use deep magic to figure out116how the type is actually laid out. The latter are pretty much always going to117have to be done by hand.118 119But for the ones where the information is expressed in the fields, it would be120great to have a way to express the instructions to produce summaries and121children in some form you could embed next to the types and have the compiler122produce a byte code form of the instructions and then make that available to123lldb along with the library. This isn't as simple as having clang run over the124headers and produce something from the types directly. After all, clang has no125way of knowing that the interesting thing about a std::vector is the elements126that you get by calling size (for the summary) and [] for the elements. But it127shouldn't be hard to come up with a generic markup to express this.128 129Allow the expression parser to access dynamic type/data formatter information130-----------------------------------------------------------------------------131 132This seems like a smaller one. The symptom is your object is Foo child of133Bar, and in the Locals view you see all the fields of Foo, but because the134static type of the object is Bar, you can't see any of the fields of Foo.135But if you could get this working, you could hijack the mechanism to make136the results of the value object summaries/synthetic children available to137expressions. And if you can do that, you could add other properties to an138object externally (through Python or some other extension point) and then139have these also available in the expression parser. You could use this to140express invariants for data structures, or other more advanced uses of types141in the debugger.142 143Another version of this is to allow access to synthetic children in the144expression parser. Otherwise you end up in situations like:145 146::147 148 (lldb) print return_a_foo()149 (SomeVectorLikeType) $1 = {150 [0] = 0151 [1] = 1152 [2] = 2153 [3] = 3154 [4] = 4155 }156 157That's good but:158 159::160 161 (lldb) print return_a_foo()[2]162 163fails because the expression parser doesn't know anything about the164array-like nature of SomeVectorLikeType that it gets from the synthetic165children.166 167Recover thread information lazily168---------------------------------169 170LLDB stores all the user intentions for a thread in the ThreadPlans stored in171the Thread class. That allows us to reliably implement a very natural model for172users moving through a debug session. For example, if step-over stops at a173breakpoint in an function in a younger region of the stack, continue will174complete the step-over rather than having to manually step out. But that means175that it is important that the Thread objects live as long as the Threads they176represent. For programs with many threads, but only one that you are debugging,177that makes stepping less efficient, since now you have to fetch the thread list178on every step or stepping doesn't work correctly. This is especially an issue179when the threads are provided by an Operating System plugin, where it may take180non-trivial work to reconstruct the thread list. It would be better to fetch181threads lazily but keep "unseen" threads in a holding area, and only retire182them when we know we've fetched the whole thread list and ensured they are no183longer alive.184 185Make Python-backed commands first class citizens186------------------------------------------------187 188As it stands, Python commands have no way to advertise their options. They are189required to parse their arguments by hand. That leads to inconsistency, and190more importantly means they can't take advantage of auto-generated help and191command completion. This leaves python-backed commands feeling worse than192built-in ones.193 194As part of this job, it would also be great to hook automatically hook the195"type" of an option value or argument (e.g. eArgTypeShlibName) to sensible196default completers. You need to be able to over-ride this in more complicated197scenarios (like in "break set" where the presence of a "-s" option limits the198search for completion of a "-n" option.) But in common cases it is unnecessary199busy-work to have to supply the completer AND the type. If this worked, then it200would be easier for Python commands to also get correct completers.201 202Reimplement the command interpreter commands using the SB API203-------------------------------------------------------------204 205Currently, all the CommandObject::DoExecute methods are implemented using the206lldb_private API's. That generally means that there's code that gets duplicated207between the CommandObject and the SB API that does roughly the same thing. We208would reduce this code duplication, present a single coherent face to the users209of lldb, and keep ourselves more honest about what we need in the SB API's if210we implemented the CommandObjects::DoExecute methods using the SB API's.211 212BTW, it is only the way it was much easier to develop lldb if it had a213functioning command-line early on. So we did that first, and developed the SB214API's when lldb was more mature. There's no good technical reason to have the215commands use the lldb_private API's.216 217Documentation and better examples218---------------------------------219 220We need to put the lldb syntax docs in the tutorial somewhere that is more221easily accessible. On suggestion is to add non-command based help to the help222system, and then have a "help lldb" or "help syntax" type command with this223info. Be nice if the non-command based help could be hierarchical so you could224make topics.225 226There's a fair bit of docs about the SB API's, but it is spotty. Some classes227are well documented in the Python "help (lldb.SBWhatever)" and some are not.228 229We need more conceptual docs. And we need more examples. And we could provide a230clean pluggable example for using LLDB standalone from Python. The231process_events.py is a start of this, but it just handles process events, and232it is really a quick sketch not a polished expandable proto-tool.233 234Make a more accessible plugin architecture for lldb235---------------------------------------------------236 237Right now, you can only use the Python or SB API's to extend an extant lldb.238You can't implement any of the actual lldb Plugins as plugins. That means239anybody that wants to add new Object file/Process/Language etc support has to240build and distribute their own lldb. This is tricky because the API's the241plugins use are currently not stable (and recently have been changing quite a242lot.) We would have to define a subset of lldb_private that you could use, and243some way of telling whether the plugins were compatible with the lldb. But244long-term, making this sort of extension possible will make lldb more appealing245for research and 3rd party uses.246 247Use instruction emulation to reduce the overhead for breakpoints248----------------------------------------------------------------249 250At present, breakpoints are implemented by inserting a trap instruction, then251when the trap is hit, replace the trap with the actual instruction and single252step. Then swap back and continue. This causes problems for read only text, and253also means that no-stop debugging must either stop all threads briefly to handle254this two-step or risk missing some breakpoint hits. If you emulated the255instruction and wrote back the results, you wouldn't have these problems, and256it would also save a stop per breakpoint hit. Since we use breakpoints to257implement stepping, this savings could be significant on slow connections.258 259Use the JIT to speed up conditional breakpoint evaluation260---------------------------------------------------------261 262We already JIT and cache the conditional expressions for breakpoints for the C263family of languages, so we aren't re-compiling every time you hit the264breakpoint. And if we couldn't IR interpret the expression, we leave the JIT'ed265code in place for reuse. But it would be even better if we could also insert266the "stop or not" decision into the code at the breakpoint, so you would only267actually stop the process when the condition was true. Greg's idea was that if268you had a conditional breakpoint set when you started the debug session, Xcode269could rebuild and insert enough no-ops that we could instrument the breakpoint270site and call the conditional expression, and only trap if the conditional was271true.272 273Broaden the idea in "target stop-hook" to cover more events in the debugger274---------------------------------------------------------------------------275 276Shared library loads, command execution, User directed memory/register reads277and writes are all places where you would reasonably want to hook into the278debugger.279 280Mock classes for testing281------------------------282 283We need "ProcessMock" and "ObjectFileMock" and the like. These would be real284plugin implementations for their underlying lldb classes, with the addition285that you can prime them from some sort of text based input files. For classes286that manage changes over time (like process) you would need to program the287state at StopPoint 0, StopPoint 1, etc. These could then be used for testing288reactions to complex threading problems & the like, and also for simulating289hard-to-test environments (like bare board debugging).290 291Expression parser needs syntax for "{symbol,type} A in CU B.cpp"292----------------------------------------------------------------293 294Sometimes you need to specify non-visible or ambiguous types to the expression295parser. We were planning to do $b_dot_cpp$A or something like. You might want296to specify a static in a function, in a source file, or in a shared library. So297the syntax should support all these.298 299Add a "testButDontAbort" style test to the UnitTest framework300-------------------------------------------------------------301 302The way we use unittest now (maybe this is the only way it can work, I don't303know) you can't report a real failure and continue with the test. That is304appropriate in some cases: if I'm supposed to hit breakpoint A before I305evaluate an expression, and don't hit breakpoint A, the test should fail. But306it means that if I want to test five different expressions, I can either do it307in one test, which is good because it means I only have to fire up one process,308attach to it, and get it to a certain point. But it also means if the first309test fails, the other four don't even get run. So though at first we wrote a310bunch of test like this, as time went on we switched more to writing "one at a311time" tests because they were more robust against a single failure. That makes312the test suite run much more slowly. It would be great to add a313"test_but_dont_abort" variant of the tests, then we could gang tests that all314drive to the same place and do similar things. As an added benefit, it would315allow us to be more thorough in writing tests, since each test would have lower316costs.317 318Convert the dotest style tests to use lldbutil.run_to_source_breakpoint319-----------------------------------------------------------------------320 321run_to_source_breakpoint & run_to_name_breakpoint provide a compact API that322does in one line what the first 10 or 20 lines of most of the old tests now do323by hand. Using these functions makes tests much more readable, and by324centralizing common functionality will make maintaining the testsuites easier325in the future. This is more of a finger exercise, and perhaps best implemented326by a rule like: "If you touch a test case, and it isn't using327run_to_source_breakpoint, please make it do so".328 329Unify Watchpoint's & Breakpoints330--------------------------------331 332Option handling isn't shared, and more importantly the PerformAction's have a333lot of duplicated common code, most of which works less well on the Watchpoint334side.335 336Reverse debugging337-----------------338 339This is kind of a holy grail, it's hard to support for complex apps (many340threads, shared memory, etc.) But it would be SO nice to have...341 342Non-stop debugging343------------------344 345By this I mean allowing some threads in the target program to run while346stopping other threads. This is supported in name in lldb at present, but lldb347makes the assumption "If I get a stop, I won't get another stop unless I348actually run the program." in a bunch of places so getting it to work reliably349will be some a good bit of work. And figuring out how to present this in the UI350will also be tricky.351 352Fix and continue353----------------354 355We did this in gdb without a real JIT. The implementation shouldn't be that356hard, especially if you can build the executable for fix and continue. The357tricky part is how to verify that the user can only do the kinds of fixes that358are safe to do. No changing object sizes is easy to detect, but there were many359more subtle changes (function you are fixing is on the stack...) that take more360work to prevent. And then you have to explain these conditions the user in some361helpful way.362 363Unified IR interpreter364----------------------365 366Currently IRInterpreter implements a portion of the LLVM IR, but it doesn't367handle vector data types and there are plenty of instructions it also doesn't368support. Conversely, lli supports most of LLVM's IR but it doesn't handle369remote memory and its function calling support is very rudimentary. It would be370useful to unify these and make the IR interpreter -- both for LLVM and LLDB --371better. An alternate strategy would be simply to JIT into the current process372but have callbacks for non-stack memory access.373 374Teach lldb to predict exception propagation at the throw site375-------------------------------------------------------------376 377There are a bunch of places in lldb where we need to know at the point where an378exception is thrown, what frame will catch the exception.379 380For instance, if an expression throws an exception, we need to know whether the381exception will be caught in the course of the expression evaluation. If so it382would be safe to let the expression continue. But since we would destroy the383state of the thread if we let the exception escape the expression, we currently384stop the expression evaluation if we see a throw. If we knew where it would be385caught we could distinguish these two cases.386 387Similarly, when you step over a call that throws, you want to stop at the throw388point if you know the exception will unwind past the frame you were stepping in,389but it would annoying to have the step abort every time an exception was thrown.390If we could predict the catching frame, we could do this right.391 392And of course, this would be a useful piece of information to display when stopped393at a throw point.394 395Add predicates to the nodes of settings396---------------------------------------397 398It would be very useful to be able to give values to settings that are dependent399on the triple, or executable name, for targets, or on whether a process is local400or remote, or on the name of a thread, etc. The original intent (and there is401a sketch of this in the settings parsing code) was to be able to say:402 403::404 405 (lldb) settings set target{arch=x86_64}.process.thread{name=foo}...406 407The exact details are still to be worked out, however.408 409Resurrect Type Validators410-------------------------411 412This half-implemented feature was removed in413https://reviews.llvm.org/D71310 but the general idea might still be414useful: Type Validators look at a ValueObject, and make sure that415there is nothing semantically wrong with the object's contents to416easily catch corrupted data.417