63

gdb implemented support for reverse debugging in 2009 (with gdb 7.0). I never heard about it until 2012. Now I find it extremely useful for certain types of debugging problems. I wished that I heard of it before.

Correct me if I'm wrong but my impression is that the technique is still rarely used and most people don't know that it exists. Why?

Do you know of any programming communities where the use of reverse debugging is common?

Background information:

Community
  • 1
Philipp Claßen
  • 1,387
  • 1
  • 12
  • 26
  • 47
    For the benefit of those who don't know it exists, what *is* reverse debugging? – Mason Wheeler Jan 04 '13 at 14:58
  • 5
    MS called their system intellitrace which appears to be similar to what you call revese debugging, it could be a matter of multiple names depending on enviorment making it seem less used. – Ryathal Jan 04 '13 at 15:12
  • @Ryathal Thanks, you are right. There seem to be multiple terms for identical or similar techniques: reverse debugging / stepping back / IntelliTrace / omniscient debugging – Philipp Claßen Jan 04 '13 at 15:25
  • 1
    Just for what it's worth: it's really *much* older than you realize -- Microsoft supported it in QuickC (around 1989 or '90, if memory serves). – Jerry Coffin Jan 04 '13 at 19:41
  • @JerryCoffin Interesting that they already had it. Now, 20 years later, the don't support it in Visual Studio. At least not for unmanaged C++ code. It's still a feature request. – Philipp Claßen Jan 04 '13 at 20:26
  • @PhilippClaßen: based on my experience back then, I don't find it very surprising (i.e., I didn't find it especially useful). – Jerry Coffin Jan 04 '13 at 20:33
  • 1
    FYI OCaml has this and calls it time travel. See: http://caml.inria.fr/pub/docs/manual-ocaml-4.00/manual030.html – Guy Coder Jan 04 '13 at 21:20
  • 42
    @MasonWheeler, clearly reverse debugging is the act of adding bugs to code. I disagree with the OP's premise that this is an uncommon practice. – Ben Lee Jan 07 '13 at 19:51
  • 3
    @BenLee, we call that rebugging. – OldFart Jan 31 '13 at 15:14
  • 1
    @MasonWheeler here is an article (written by the reverse debugging company I work for) on reverse debugging and its history: (https://undo.io/resources/blog-articles/reverse-debugging-time-travel-debugging-brief-hist/) – Edward Moffett Aug 03 '18 at 10:53

5 Answers5

28

For one, running in debug mode with recording on is very expensive compared to even normal debug mode; it also consumes a lot more memory.

It is easier to decrease the granularity from line level to function call level. For example, the standard debugger in eclipse allows you to "drop to frame," which is essentially a jump back to the start of the function with a reset of all the parameters (nothing done on the heap is reverted, and finally blocks are not executed, so it is not a true reverse debugger; be careful about that).

Note that this has been available for several years now and works hand in hand with hot-code replacement.

Laurent Bourgault-Roy
  • 5,606
  • 4
  • 20
  • 25
ratchet freak
  • 25,706
  • 2
  • 62
  • 97
  • 1
    Very good answer. I can confirm that recording is expensive. You have to enable it just before you enter the critical part of your application, which is not always trivial. I also agree that "drop to frame" is often good enough. It does not work well with loops or recursive algorithms, though. – Philipp Claßen Jan 04 '13 at 15:49
  • 3
    That's until rr (http://rr-project.org/). speed while recording an execution using rr is barely slower. You can then replay, step in, rewind, set watchers in your favourite IDE (https://github.com/mozilla/rr/wiki/Using-rr-in-an-IDE)... The way you debug your code will never be the same. – jyavenard Aug 24 '17 at 12:31
  • rr doesn't seem to support anything outside x86/amd64... – Dr. Koutheir Attouchi Feb 29 '20 at 04:51
11

As mentioned already, performance is key e.g. with gdb's reversible debugging, running something like gzip sees a slowdown of 50,000x compared to running natively. There are commercial alternatives however: I work for Undo undo.io, and our UndoDB product does the same but with a slowdown of less than 2x. There are other commercial reversible debuggers available too.

Edward Moffett
  • 103
  • 4
user1839284
  • 119
  • 2
  • 1
    Interesting, I will definitely give it a try. In several articles, it is stated that it is free for non-commercial usage but I found no information confirming that on your homepage. Is it still true? Thanks for disclosing your affiliation. – Philipp Claßen Jan 06 '13 at 15:18
  • 3
    What are the prices for the Starter and Professional versions of UndoDB? I don't see them on the UndoDB editions page. – tcrosley Jan 06 '13 at 20:18
  • 3
    How are you guys comparing to Mozilla's `rr`? – Ciro Santilli OurBigBook.com Sep 09 '17 at 03:27
10

For an overview of the technology choices and products, see a series of blog posts I wrote about a year ago (and some follow-ups since):

My feeling for why it is used so little is that it requires special hardware, or using a special debugger, or setting up your system right. Most people do not invest the time to get maximum value from their debug tools, unfortunately.

And the fact that the "cheap default" of gdb is almost unusably slow and has quite a few stability problems for everything but the most common target systems.

jakobengblom2
  • 211
  • 1
  • 3
4

From my experience as a sales engineer for TotalView debugger, people do know that it exists but they don't think it works, regardless of the (acceptable or not) slowdown.

University of Cambridge recently did a survey entitled "Failure to Adopt Reverse Debugging Costs Global Economy $41 Billion Annually".

And coming back to GDB, I've heard (a lot) that slowdown makes it quite unusable on a "real-life" application.

I'd personally love to hear back from more people using reverse debugging on applications others than "Hello world!"

Ladlestein
  • 103
  • 4
SebGR
  • 381
  • 2
  • 7
  • 6
    I think it would be far better to link this study rather than 'Undo Software' spam pretending to be about study. – Bulwersator Mar 14 '14 at 04:36
  • 1
    At my former job, I worked on building a reverse debugger (http://www.ghs.com/products/timemachine.html). We used the debugger internally heavily, and I can tell you, it was incredible. Of course it was great on the really hairy race conditions, but it was more than that. When reverse debugging is on all the time, your mentality about debugging changes. You iterate less, and you can be less careful with how you test your code. You can also save an execution run and send it to someone, so it's great for finding bugs in other people's code. – speedplane Oct 24 '16 at 15:55
2

I think it is important to expand a little further on this "reverse" or "historic" debugging. I think to understand complex systems and behavior in those, to replay "events" which make state explicit is absolutely crucial.

What I want to express is that you are not alone in wondering why this technique is not so much applied today or why the related problems are rarely discussed clearly.

So let’s emphasize two very important concepts here:

1.To understand a programming system it is helpful to make state explicit

2.To even further understand a programming system replaying sequences of state (events) can help a lot.

Here are some sources which tackled the problem and proposed or designed solutions for the problem (dealing with state in complex systems):

-Out of the tar bit, paper: http://shaffner.us/cs/papers/tarpit.pdf Main ideas: avoid, isolate or make state explicit

-CQRS http://www.cqrs.nu/ This is a combination of two concepts: Command Query Segregation and Event Sourcing. There exists different implementations ( Java,C# , Scala). The replaying of Tate sequences and the evolving of a domain model are the crucial parts here.

If you really zoom out and see the very broad picture you can already see that with the "rise" of functional programming people are already ((un)consciously ) attracted to fp because it makes state explicit! But that only deal with point one, to address the second one you need another concept which could be "loosely" described as functional reactive programming.

So you might say all well and good but who actually uses CQRS and FRP? I would say (IMO because I don’t have concrete numbers) actually a lot of companies its just that they don’t know the work they do has this terminology. Maybe you google a bit around and you hear from enterprises which use CQRS, there are some success stories already out there. FRP too is rising slowly as an example I could give Netflix: http://techblog.netflix.com/2013/02/rxjava-netflix-api.html Which just released an implementation of RX which is actually .NET based (but has a Javascript implementation too). So People are using these techniques today already, IN THE LARGE to understand complex systems and to make them even better. That is why they use reverse debugging techniques.

AndreasScheinert
  • 454
  • 3
  • 7