Help debugging 100.000 lines C++ project

cslr

New Member
Credits
21
I have a project which has 100.000 lines of C++ source code which now crashes for unknown reasons. I use GCC compiler.

Can you give advices how to debug it?

I currently use
- GCC -g switch
- GCC -D_GLIBCXX_DEBUG switch for debugging
- run in GDB
- run using Valgrind
- internal logging mechanism telling what is happening in source code.

Especially, code now crashes on MSYS2 (Windows) but don’t otherwise.
 


diorthotis

New Member
Credits
139
There are several options, all of which are tedious. You can try to comment out code until it does not crash, which will pinpoint the fault. Your debugger should provide some error information, what did it say?
 

cslr

New Member
Credits
21
I use GDB and -g -fsanitize=address -fsanitze=undefined and other GCC’s command line switches and valgrind but still I get large floating point numbers which causes code to crash/not functioning correctly.

There seem to be also problem in my threaded code where one of the threads stopped running for unknown reasons although it should.
 

JasKinasis

Well-Known Member
Credits
7,746
What kind of crash is it? What error messages do you get?

Also does the crash happen randomly/inconsistently? Or can the crash be reproduced reliably using a particular set of actions?

The reason I ask is - if the crash can reliably be reproduced using a fixed set of actions - that should give you some idea which module, or modules in your project might be failing.
Run a debug build through gdb, once you've managed to crash your program in gdb, you should be able to perform a backtrace and look at the call-stack to see where the crash occurred.

Then set some breakpoints in that area and you should be able to step through the code and see where the crash is occurring.
Once you know where the crash is occurring, you need to then determine why it occurred. It could be some pointer that isn't initialised, or an uninitialised variable, a race condition etc. etc.

If your project is under any kind of source control and this crash is a newly discovered bug - another thing you could try is, to bisect through your projects commit-history and try to determine which commit caused the bug.
So for example:
You know that the crash occurs at the current HEAD of your branch.
The next step is to sync your branch to some point in the past.
So you might sync your branch back to the last stable release.
Build and run that - if the crash occurs, go back another release etc..until you see no crash.
And make a note of your findings:
e.g.
Code:
changelist #55 - Current HEAD - Crash << Bug occurs here
changelist #50 - Version 0.5 - Crash << Bug occurs here
changelist #40 - Version 0.4 - Crash  << Bug occurs here
changelist #30 - Version 0.3 - No Crash << Good
Now we know the bug was introduced by a changelist that was submitted between changelist #30 (version 0.3) and changelist #40 (version 0.4). So the next thing to do is to look at the changelists that were submitted between those dates.
In the above example, there were only 10 changelists between each version. In a real world scenario, you could be looking at lots of changelists. It's basically like doing a manual, binary search.

You know the error was introduced between changelist 30 and changelist 40, so you sync to changelist 35, build and run. No crash.
So again, tab out what you find:
Code:
Changelist #30 - good
Changelist #35 - good
Changelist #40 - bad
Now you know it's between changelist #35 and #40. So next you'd sync to changelist #37 - #37 is bad.
Now you know it occurs between #35 and #37.
You sync to changelist #36 - #36 is good.
Therefore, in this case, we know that changelist #37 introduced the bug.

So you keep bisecting until you identify the exact changelist that introduced the bug.
Once you have done that - you can look at the changes that were made in that changelist and try to determine which of those changes is likely to be causing your bug.
Then it's just a case of syncing to the HEAD of the source tree, sticking some breakpoints in the areas that were changed in changelist #37 and verifying the cause of the bug before applying, testing and submitting your fix and moving on!

Bisecting through the code-base can be a time-consuming task. But in a project that is under source control - it allows you to quickly narrow in on the cause of a reproducible bug.

Another idea might be to introduce some logging capabilities into your project. You could use conditional compilation to only include the logging functionality in your debug builds. That can aid you with monitoring the internal state of your project and identify problematic areas.

Coding defensively should also allow you to prevent a lot of common bugs, or catch them early, either at compile time, or at runtime in debug builds (via debug assertions). I'm not going to teach you to suck eggs here. If you're any kind of C++ programmer, you probably already know all the things I'm referring to.

Basic memory management, using smart pointers, sanitizing and range checking user input, initializing all variables/objects/pointers. Checking pointers are valid before dereferencing them, using const correctness, avoid using mutable global variables, treat warnings as errors, etc etc etc.

Also, you could try running some kind of lint, or static analysis tools over the codebase. That might pick up problematic areas, security vulnerabilities, memory leaks, threading issues, smells in the code etc.
 

diorthotis

New Member
Credits
139
Unfortunately there are not many more options besides commenting out. The command prompt should give some sort of error output, please post the error messages. The complex nature of code requires that prior to each change, a backup should be made, in case you find some error. Otherwise, locating a random crash in a large program becomes very difficult. I use Visual Studio for debugging, because it allows many tools such as step-by-step execution and variable information.
 
$100 Digital Ocean Credit
Get a free VM to test out Linux!


Top