Before completing almost any project, you'll usually have a bug or ten in your system. That's where debugging comes in handy, however it's not a miracle worker; you have to have an idea where in general the issue may lie, or you'll be scrawling through a 130,000 line long debug log with no idea what you're looking for (yes, I speak with experience on this point). The best way to debug is to create theories in your head, and prove/disprove them as you go (you may dynamically create more theories or merge them as you go through the debugs and collate more information surrounding the problem).

Using Diagnostics Tools

Introduction

Before starting to play with the actual debugger and other fun toys, you may want to read this section about actually getting the data you need to diagnose issues. This section goes into detail about using Firefox Developer edition, amongst other seemingly obvious and menial tasks.

Firefox Developer Edition

Without a doubt, this is one of the most useful browsers created for development. This section will go into detail on how to use the inbuilt tools and functions for diagnosing and resolving issues (it may even mean you can avoid debugging at all!). For the purposes of this, we're going to assume you know most of the basic functions of Firefox Developer.

Using Network
In the devlopment tools, there is a curious tool called 'Network' that we can use to see what data is transferred. First and foremost, it can be most useful for getting the filename and path of the most recent debug. This can be achieved by going into the response of the XML that has been transferred and going right to the end of the document, where it written like:

<session_errors>Debugs being written to : /tmp/dafz28231aafq53tc0odmmab9uzsb2.1535011074.64373.hfnexrcqiu.log</session_errors>

This file can be used to either interactively debug or go through the debugs manually, which is explained later. However, most importantly this feature can be used to debug solo filters; by refreshing the filter and not the page you can get a much more concise, accurate debug. This is the only way to get that type of debug - the other method explained later only works for a whole page.

Using Console
Console is pretty self-explanatory - it throws up errors and warnings. If a program dies before it outputs what it is meant to, the last error usually is the root cause of the issue, hence it helps to narrow down your search criteria.

Using Inspector
Another extremely useful (and hopefully familiar) tool is Inspector, in which you can get the ZName of most objects on the page to assist in theorising and debugging. It is coded in HTML, and provides the basic necessities for a lot of diagnostical issues: for example, you may be getting the definition of an object but no results appear - you can use the Inspector to get the ZName and get_def using zymonic_toolkit.pl to see if the data you are trying to pull actually exists in the first place.

Using the Web-Based Debug Viewer
On a system with the ZymonicDebugs module installed you can type into the URL bar:

http://[server]/zymonicdebugs?branch=[trunk|qa|live|mgt-live]&zymonic_rev=[zymonic rev]&zcps_rev=[zcps rev]&file=[the filename of the debugs without the path or .log extension]

Terminal

The Linux terminal is Linux's answer to Window's command prompt. Mixed with Zymonic, it becomes a whole lot better - specific tools are included within Zymonic to help with debugging and diagnosis. Click here for more information on Zymonic terminal-based commands; this section will be purely focused on useful Linux commands.

Vim
As your friendly neighbourhood command line text editor, vim can be launched as read-only or editable - to edit in vim you would usually use sudo. This section briefly explains certain functions which are used regularly in vim, the following commands launch vim read-only and editable (respectively):

vi FILENAME
sudo vi FILENAME

The following is a list of common commands and what they do (for usage, use google - the information is out there, easily available):

G - skips to the end of the document
/ - searches the document for a text pattern
i - Allows editing of the document
esc - Stops editing of the document to allow other commands to be used
:w! - forcibly writes the documents to memory
:q! - forcibly quits the editor, wiping all unsaved changes
gg - goes to the start of the document
dd - delete line
:x! - forcibly save and quit

pscp
Whilst not strictly Linux terminal, pscp comes with PuTTY and is used to transfer files between a remote computer and your own. It is used in command prompt, and usage is as follows:

pscp username@remote_ip_or_computer_name:/file/location/filename ~\Downloads\FILENAME.extension

It's that simple, obviously you substitute my fake details for real information.

Module Loading
Using perls in built functions, we can test to see if a module loads by running the following command (replace the non-existent modules though):

perl -e 'use Zymonic::Module::MaybeAnother;'

GUI Based Debug Analysis (Performance)

TODO

Debugging with the Interactive Debugger

Introduction

Whilst creating code, you may find that you cannot diagnose an error in the potential 6000 lines it could be in - that's where debugging comes in! Zymonic can make use of Perl's inbuilt debugger to go through code, line by line, to diagnose errors (for more information on using the Perl debugger visit the Perl docs website).

The sections below explain how to access the debugger in perl.

Debug the Config Build

If you have changed part of the config build and it results in an error/outputs unexpected results, you may want to try the following (replace 'test_system' with your system):

sudo perl -d `which zymonic_toolkit.pl` System config_build --system test_system

From here, you will be presented with a command line - use perl debugger commands to get to the correct area of code (example below) and test your variables to see where the error might be coming from.

DB<1> use Zymonic::SystemDefinition
DB<2> c Zymonic::SystemDefinition::some_random_function

You can then either use 'c' or 'n' to navigate through the code, searching for the problematic area. Don't forget, in the Perl debugger you can assign temporary variables, view a variable's definition, change existing variables in the code and actually look what the output of a command would be (examples all below, respectively). Line DB<7> would have an extremely long output, hence we won't view that on here.

5652:       my $self = shift;

DB<3> c 5704
Zymonic::SystemDefinition::some_random_function(/usr/local/lib64/perl5/5.24.3/Zymonic/SystemDefinition.pm:60000):
5704:       foreach my $test ( keys %{$hash_ref} )
5705:       {

DB<4> n
5706:           my $hash_ref_2 = $self->get_def( 'Object_Type', $test, 'no_error' );

DB<4> $tempvar = 'Zymonic is the greatest'

DB<5> x $tempvar
0  'Zymonic is the greatest'

DB<6> $test = 'This has now changed'

DB<7> x $self->get_def( 'Object_Type', $test, 'no_error' )

Using this method, we can achieve a step-by-step analysis of the code for the config build.

Debug in Runtime

There are, of course, times where you cannot debug in the config build, and the error is in the runtime code. This is more complicated to debug, but still possible. A diagram is below to help with finding GUI objects.

You must enable debugs in the UI
Once you have done that, you will have a message button appear. Hover over it, and carefully highlight the filename that comes up and copy it
Now, you will want to run the following command, inserting the filename you copied where FILENAME is.
```
grep DEBUG [FILENAME] | perl -pe 'use HTML::Entities; decode_entities($_);' >~/grepoutput.log
```
Cat the file grepped to.
```
cat grepoutput.log
```
There will be a very long string of text after "PERL DEBUGGER COMMAND: "
Copy all of it, and paste it into the command line. It should begin "REMOTE_ADDR="
It will then begin loading the debugger; use the same commands as you did with the config build debugger to diagnose issues. There is another way to easily get the debug filename, see Using Firefox Developer Tools for more details

Finding the 'Enable Debugs' option and message button in the GUI

Debugging without the Interactive Debugger

Introduction

Sometimes it's not appropriate to use Perl's inbuilt debugger to diagnose issues - for example, when you are working on a live copy of the code and have no access to the systems, only the debug logs. Fortunately, Zymonic debug logs are able to support you.

Adding a Working Copy Debug

This is generally used to debug code that you have already - you can add a debug line in the code for maximum error reporting, so you know exactly what area in the code the error is, and which object is being currently used (it depends on how you set it up).

Locate the working copy - check with your sysadmin.
Change directory to the correct 'module' containing folder
Using the following command, open the module you want in vim (or your preferred editor):
```
sudo vi FILENAME
```
Add the debug - for more information, see the below section titled 'Adding a Debug to your Code'
Run the following command:
```
sudo make install
```
To view the difference or to create a patch to send (respectively):
```
sudo svn diff FILENAME
sudo svn diff FILENAME ~/mypatch.patch
```
You may wish to make a more meaningful patch name to send to someone.

To remove the debug:

sudo svn revert FILENAME ; sudo make install

Depending on the change you may need to do an apache reload:
```
sudo /etc/init.d/apache2 reload
```

Adding a Debug to your Code To add a debug to give more detailed debug log files, just enter the following second line in your code (where you want to get the debug data, it's not going to magically know where you want it):

my $yourvariablehere = "Success";
debug( "The Current State is: " . $yourvariablehere);

This debug would output a line saying: "The Current State is: Success"
The potential advantages are high, for example if one item in a foreach loop were failing, you could identify it and resolve the issue.

Debugging using the Debug Logs

Sometimes, you have access to the debug logs alone - you can still find errors in there using debug commands within the code and searching through the code to find the correct areas.

Debugging SQL errors
This section will allow you to find the exact command QueryBuilder creates for you, this will help you see if your changes have took effect, or precisely what your changes have done to the SQL.

Refresh results with debugs enabled and Network tab of developer console open - in the response find the session_errors line:

<session_errors>Debugs being written to : /tmp/dafz28231aafq53tc0odmmab9uzsb2.1535011074.64373.hfnexrcqiu.log</session_errors>

Next we need to find the 'main' SQL query; to do this we need the SQL name of the base table:
- Find the id (ident) of the filter using inspect element - will be the first part of all the ids within the block:
- Now you need to get the definition of the filter to find the BaseTable:
```
zymonic_toolkit.pl Definition get_def --definition test_system --zname test_filter >~/output.xml
vi ~/output.xml
```
  The BaseTable element will look something like (you want the ZName contents):
```
<BaseTable>
  <ZName>test_table</ZName>
</BaseTable>
```
- Now you will want to get the definition of the BaseTable ZName to find the SQLName of the table:
```
zymonic_toolkit.pl Definition get_def --definition test_system --zname test_table >~output.xml              
vi ~/output.xml
```
  The SQLName element will look something like:
```
<SQLName>test_sqlname</SQLName>
```
- Now we can grep for the SQL section of the debugs, however the entities will be encoded, so you will want to run:
```
grep "FROM test_sqlname" /tmp/dafz28231aafq53tc0odmmab9uzsb2.1535011074.64373.hfnexrcqiu.log | perl -pe 'use HTML::Entities; decode_entities($_);'
```
- With these results, you want the full SQL command (it will usually be very long). You can then see if changes you made took effect (you can also copy the command into an SQL query to test the data that comes up).

Debugging Perl errors
You may also want to debug Perl issues from a debug - not an issue.

Refresh results with debugs enabled and Network tab of developer console open - in the response find the session_errors line:

<session_errors>Debugs being written to : /tmp/dafz28231aafq53tc0odmmab9uzsb2.1535011074.64373.hfnexrcqiu.log</session_errors>

From the command line, type in the following command to open the debug logs - use vim's in built commands to search through to find what you need.
```
vi /tmp/dafz28231aafq53tc0odmmab9uzsb2.1535011074.64373.hfnexrcqiu.log
```
You can also add extra data into your debug by setting up working copy debugs in your code (see above); this will allow you to collect extra information to resolve issues faster and with more accuracy.

Error Logs

When you have an error come up in a Zymonic system, the error can be viewed in more detail from the error logs. An easy way to find the correct one is to view the most recent doing the below command in the terminal:

view `ls -1tr /var/errors/*.xml | tail -n1`

The top error is the most recent and will give you a description of the error along with a line number. Below this will be a list of all the functions in the caller stack. You may be able to save time debugging at this point by simply looking at the code with the error message in mind, to spot simple mistakes.

A point to note, this trick can be used in /tmp/...log by doing:

view `ls -1tr /tmp/*.log | tail -n1`