Thursday, May 25, 2017

VSCode JTAG Debugging of ESP32 - Part 2

This in a continuation from part 1. If you need help connecting the ESP32 to JTAG, see this post.

After setting up the ESP32 toolchain and confirming regular GDB debugging is working as described in my previous post, now it is time to configure VSCode for remote debugging of the ESP32.

I've also added Part 3 - using ESP WIP OpenOCD.

Everything is pretty straightforward here, well except for the details of actually debugging in VSCode. I had quite an ordeal with that, as described in github issue  #763: Remote Debugging of ESP32 causes EXCEPTION: System.ArgumentOutOfRangeException.

In the end, it seems that somehow VSCode was confused on the initial *state* of the ESP32 when debugging is first initiated. The first MI commands would get the ESP32 into some sort of weird state "internal to GDB":

C next: {"threadId":-2}
1: (812785) <-1022-exec-next
1: (812793) ->1022^running
1: (812793) ->*running,thread-id="all"
1: (812793) ->(gdb)
1: (812793) 1022: elapsed time 8
1: (812794) ->(gdb)
 R: {"success":true,"message":null,"request_seq":11,"command":"next","body":null,"running":false,"refs":null,"seq":0,"type":"response"}
1: (812903) ->@"esp32.cpu0: Target halted, pc=0x40080E77\n"
E output: {"category":"stdout","output":"@\"esp32.cpu0: Target halted, pc=0x40080E77\\n\"\n","data":null,"type":"output"}
@"esp32.cpu0: Target halted, pc=0x40080E77\n"
1: (812932) ->~"/home/ivan/e/crosstool-NG/.build/src/gdb-7.10/gdb/inline-frame.c:171: internal-error: inline_frame_this_id: Assertion `!frame_id_eq (*this_id, outer_frame_id)' failed.\nA problem internal to GDB has been detected,\nfurther debugging may prove unreliable.\nQuit this debugging session? "
1: (812932) ->~"(y or n) [answered Y; input not from terminal]\n"
E output: {"category":"stdout","output":"/home/ivan/e/crosstool-NG/.build/src/gdb-7.10/gdb/inline-frame.c:171: internal-error: inline_frame_this_id: Assertion `!frame_id_eq (*this_id, outer_frame_id)' failed.\nA problem internal to GDB has been detected,\nfurther debugging may prove unreliable.\nQuit this debugging session? ","data":null,"type":"output"}
1: (812932) ->&"\nThis is a bug, please report it."
/home/ivan/e/crosstool-NG/.build/src/gdb-7.10/gdb/inline-frame.c:171: internal-error: inline_frame_this_id: Assertion `!frame_id_eq (*this_id, outer_frame_id)' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session?

I opened an espressif issue on giuthub for this one: [TW#12876] VSCode Debugging via MI causes internal GDB error  #645. Fortunately - with the help of @pieandcakes - I was able to determine a (yes, somewhat wonky) sequence of steps to persuade VSCode to debug the ESP32!

I have my openocd running on Ubuntu with samba, and have mapped my windows Z: drive to the root of the ubuntu file system. For that, my smb.conf looks like this:

## edit the file /etc/samba/smb.conf and put in these lines at the end (without single # comment markers!)
##*******************************************************************************************************
[c$]
   comment= root
   path=/
   browseable=Yes
   writeable=Yes
   only guest=no
   create mask=0777
   directory mask=0777
   public=no

## also, make sure the authentication line is not commented out
####### Authentication #######
#   security = user
##*******************************************************************************************************

# try to keep away WannaCry
[global]
min protocol = SMB2

First, browse to the Hello World "Build" directory in Windows File Explorer. Right click and click "Open with Code".  The select Debug - Open Configurations. This should create a file called launch.json - in my case, in:

Z:\home\gojimmypi\esp\hello_world\build\.vscode\

Here's my launch.json for debugging the ESP32:


{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "OpenOCD Debug",
            "type": "cppdbg",
            "request": "launch",
            "miDebuggerPath": "C:/msys32/opt/xtensa-esp32-elf/bin/xtensa-esp32-elf-gdb.exe",
            "program": "Z:/home/gojimmypi/esp/hello_world/build/hello-world.elf",
            "setupCommands": [
                {
                    "description": "Enable pretty-printing for gdb",
                    "text": "-enable-pretty-printing",
                    "ignoreFailures": true
                },
                {
                    "text": "file 'Z:/home/gojimmypi/esp/hello_world/build/hello-world.elf'"
                },
                {
                    "text": "target remote 192.168.174.129:3333"
                },
                {
                    "text": "monitor reset init"
                }
            ],
            "externalConsole": false,
            "cwd": "Z:/home/gojimmypi/esp/hello_world/build/",
            "logging": {
                "trace": true,
                "traceResponse": true,
                "engineLogging": true
            }
        }
    ]
}
Note that you need to choose between 
"miDebuggerServerAddress": "192.168.174.129:3333"
and 
"text": "target remote 192.168.174.129:3333"
- but not both! (see that first question in github issue 763). You will, of course - need to edit the IP address to point to your own OpenOCD server, otherwise you will see this error in VSCode:

Unable to start debugging. Unexpected GDB output from command "-interpreter-exec console "target remote 192.168.174.129:3333""


If you don't have a VM or a Linux box, perhaps consider a Raspberry Pi.  :)


The line

            "logging": {
                "trace": true,
                "traceResponse": true,
                "engineLogging": true

is optional, but tremendously helpful in actually seeing what's going on between VSCode and the ESP32.

I created a few gists that can either be put into script files, or copy/pasted for some basic operations. I run each of these in a separate putty shell connection. First, the compile & flash: 

#!/bin/bash
export PATH=$PATH:$HOME/esp/xtensa-esp32-elf/bin
export IDF_PATH=~/esp/esp-idf

sudo chmod 777 /dev/ttyUSB0

cd ~/esp/hello_world
make flash

In this same "compile window", it can be helpful to run the serial monitor. I like keeping it in the same window as make/building, so that I am sure to stop the monitor when reflashing. 

make monitor
Next, run openocd server in yet another putty connection (when successful, the openocd will not exit, rather it continues running as a server for your VSCode GDB connection as a foreground task): 

#!/bin/bash
export PATH=$PATH:$HOME/esp/xtensa-esp32-elf/bin
export IDF_PATH=~/esp/esp-idf

cd  ~/workspace/openocd-esp32/tcl
sudo openocd -f interface/ftdi/olimex-arm-usb-ocd-h-1MHz.cfg -c "transport select jtag"  -f target/ESP32-RTOS-none.cfg

A successful launch of openocd will look something like this: 

0 $ sudo openocd -f interface/ftdi/olimex-arm-usb-ocd-h-1MHz.cfg -c "transport select jtag"  -f target/ESP32-RTOS-none.cfg
[sudo] password for gojimmypi:
Open On-Chip Debugger 0.10.0-dev-g372bb59 (2017-05-19-10:35)
Licensed under GNU GPL v2
For bug reports, read
        http://openocd.org/doc/doxygen/bugs.html
adapter speed: 1000 kHz
jtag
force hard breakpoints
Info : clock speed 1000 kHz
Info : JTAG tap: esp32.cpu0 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
Info : JTAG tap: esp32.cpu1 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
Info : esp32.cpu0: Target halted, pc=0x400D1158
then the xtensa-esp32-elf-gdb debug client (the find is of course optional) this one with gdb commands in hello_world,gdb: 

#!/bin/bash
export PATH=$PATH:$HOME/esp/xtensa-esp32-elf/bin
export IDF_PATH=~/esp/esp-idf
#find /. -name xtensa-esp32-elf-gdb 2>/dev/null

cd ~/esp/hello_world/build
xtensa-esp32-elf-gdb -tui -x hello_world.gdb hello-world.elf


Alternatively launch the xtensa gdb client simply like this (assume paths properly set as shown above):


xtensa-esp32-elf-gdb  hello-world.elf

A successful launch of gdb will look something like this: 

gojimmypi@ubuntu : ~/esp/hello_world/build
0 $ xtensa-esp32-elf-gdb  hello-world.elf
GNU gdb (crosstool-NG crosstool-ng-1.22.0-61-gab8375a) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later &lthttp://gnu.org/licenses/gpl.html&gt
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-build_pc-linux-gnu --target=xtensa-esp32-elf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
          &lthttp://www.gnu.org/software/gdb/bugs/&gt.
Find the GDB manual and other documentation resources online at:
          &lthttp://www.gnu.org/software/gdb/documentation/&gt.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from hello-world.elf...done.
(gdb) 

target remote localhost:3333
Remote debugging using localhost:3333
0x400d1158 in esp_vApplicationIdleHook () at /home/gojimmypi/esp/esp-idf/components/esp32/./freertos_hooks.c:52
52              asm("waiti 0");
(gdb)


We just wanted to confirm OpenOCD was working and that we could connect with linux GDB, so exit/quit.

The Windows 10 executable gdb is needed to debug in VSCode on Windows, so as shown on: http://esp-idf.readthedocs.io/en/latest/get-started/windows-setup.html it can be downloaded from here

(mine took HOURS to download)

When ready to start debugging in Windows, locate your xtensa-esp32-elf-gdb.exe, Mine is located in:

C:\msys32\opt\xtensa-esp32-elf\bin

Note the line in the VSCode launch.json:

"miDebuggerPath": "C:/msys32/opt/xtensa-esp32-elf/bin/xtensa-esp32-elf-gdb.exe",


That's the one that calls the Windows GDB client to talk to the Ubuntu OpenOCD server. This gives us the best of both worlds: assimilation! Here we can have all the reliable compile of ESP32 code in linux, yet the ability to edit and debug from Windows!

Pay attention to the forward and back-slashes in the launch.json file. See vscode-cpptools issue #706.

The details of these steps are shown in my github issue (in particular the 8th comment after the question) if you want to see more details, but in summary:

Be sure the OpenOCD server is running as described above. Press F5 to start debugging. You should see "Exception has occurred.Trace/breakpoint trap." It appears we are debugging, but we are not. Target (appears to be) running. VSCode thinks it is paused.

If you try to single step here, all bets are off; xtensa GDB crashes, VSCode debugging stops. Game over.

If instead VSCode immediately stops debugging and you see something in the debug log like: 

E output: {"category":"stderr","output":"ERROR: Unable to start debugging. Failed to find thread 1 for break event\r\n","data":null,"type":"output"}
ERROR: Unable to start debugging. Failed to find thread 1 for break event
...then go back to part 1 and read about how I slightly modified my esp32.cfg

Otherwise if all is still going according to plan: type -exec continue at the VSCode > prompt.



Observe debug output. Note that despite -exec continue we are now actually in halt state. One of the lines should say:

(16086) ->@"esp32.cpu0: Target halted, pc=0x400F4AB3\n"

Press the VSCode Play button (F5). Observe error:


ERROR: Cannot execute this command while the selected thread is running.

Only after the above steps are completed, now press the pause button. Observe output. There should be no errors.

Press a code-stepping button, such as Shift-F11.... tada! Single step debugging!



For an even better experience - see FHFS comment on espressif/esp-idf issue 303,
"if esp-idf is in the project folder symbols get found automatically. for windows you can also add the msys2, mingw32 and xtensa-esp32-elf bin folders to the path in tasks.json. make monitor has some trouble starting a console, you can edit the idf_monitor.py to output to stderr(task output window), or make a .cmd file with the python command in it and run that command from tasks.
You can see how I work with it esp-idf-VSCode-template"


I'm still working on polishing these instructions. Hopefully you've found this useful. Feedback is welcome and appreciated. Send me a message on twitter, github pull request  or gmail.


Various tips:


GDB tip: Ctrl-x Ctrl-a in gdb to enter TUI mode.

You can press the EN button on the ESP32; it acts like reset and resolves many problems without having to power cycle (which will likely also need permissions re-applied,etc)

Note that as of May 2017, there's an issue in VSCode regarding the direction and quantity of path slashes as note in gitbub vscode-cpptools issue #706.

When unplugging usb devices such as your debugger and the ESP32 itself, be sure to note the order when re-inserting. Sometimes there's an unexpected recognition delay on the VM; you'll want to be consistent (e.g. having ESP32 on /dev/ttyUSB0). Trust me, you'll wait all day long trying to flash ESP32 firmware onto your JTAG debugger if it somehow ends up as /dev/ttyUSB0

If you see something like this there's a good change the JTAG adapter ended up on the default ttyUSB0 port. Edit the config or change the plug order.
>
gojimmypi@ubuntu : ~/esp/hello_world
0 $ sudo chmod 777 /dev/ttyUSB0
[sudo] password for gojimmypi:
gojimmypi@ubuntu : ~/esp/hello_world
0 $ make flash
Flashing binaries to serial port /dev/ttyUSB0 (app at offset 0x10000)...
esptool.py v2.0-beta3
Connecting........_____....._____....._____....._____....._____....._____....._____....._____....._____....._____

A fatal error occurred: Failed to connect to ESP32: Timed out waiting for packet header
/home/gojimmypi/esp/esp-idf/components/esptool_py/Makefile.projbuild:52: recipe for target 'flash' failed
make: *** [flash] Error 2


Debugging the ESP32 can be rather finicky; I spent a ton of time on the thread problem https://github.com/espressif/openocd-esp32/issues/10

If you don't see the debug console in VSCode: see View - Debug Console (Ctryl-Shift-Y)

Somehow over the course of time, my old friend "ifconfig" fell out of vogue, It is now simply "ip a". I don't really like it; harder to read. So you may need to install net-tools if ifconfig does nto work for you to determine ip addy.

Don't leave OpenOCD / GDB running when recompiling.

Don't leave OpenOCD running when disconnecting JTAG or ESP32. When the process gets stuck, closing the putty window does NOT terminate the process. Regular kill does not work. You'll need -SIGKILL


gojimmypi@ubuntu : ~/esp
1 $ ps aux | grep openocd
root      25099  0.0  0.3  54796  3964 pts/8    S+   10:30   0:00 sudo ../src/openocd -f interface/ftdi/olimex-arm-usb-ocd-h.cfg -f target/esp32-single-thread.cfg
root      25100  0.5  0.4  28660  4252 pts/8    Sl+  10:30   0:16 ../src/openocd -f interface/ftdi/olimex-arm-usb-ocd-h.cfg -f target/esp32-single-thread.cfg
gojimmy+  25176  0.0  0.0  14224   936 pts/9    S+   11:23   0:00 grep --color=auto openocd
gojimmypi@ubuntu : ~/esp
0 $ sudo kill -SIGKILL 25100
[sudo] password for gojimmypi:
gojimmypi@ubuntu : ~/esp
0 $ ps aux | grep openocd
gojimmy+  25180  0.0  0.0  14224   984 pts/9    S+   11:24   0:00 grep --color=auto openocd
gojimmypi@ubuntu : ~/esp


Resources, Inspiration, and Other Links: 


Completely unrelated, but cool:






By at

No comments:

Post a Comment

comments are welcome, but I prefer not to allow links to promotions or other unrelated services.

Subscribe to: Post Comments (Atom)

Find gojimmypi at gojimmypi.github.io

I'm currently working on my new blog home at  gojimmypi.github.io After implementing a variety of features such as dark mode , syntax hi...