How to debug vsock communication issues ?

[schaefi]

Hi,

I'm using firecracker as VM engine to register/run applications in the following project:

• https://github.com/Elektrobit/flake-pilot

For the purpose to support resume type apps I read about the vsock implementation in firecracker and how
it is supposed to be used. As source of information I read:

• https://github.com/firecracker-microvm/firecracker/blob/main/docs/vsock.md

For my tests I used Firecracker v1.3.2

To run an app in a firecracker VM I simply registered a bash shell as follows:

flake-ctl firecracker pull --name leap --kis-image https://download.opensuse.org/repositories/home:/marcus.schaefer:/delta_containers/images/firecracker-basesystem.x86_64.tar.xz
flake-ctl firecracker register --vm leap --app $HOME/mybash --target /bin/bash --overlay-size 20GiB

after that I can call:

 $HOME/mybash 

and get a shell inside of a firecracker VM. In the process table this provides:

14213 pts/1    Sl+    0:01 firecracker --no-api --id 14206 --config-file /tmp/.tmpGZf5B3

The config file used for the instance looks like the following

{
   "boot-source":{
      "kernel_image_path":"/var/lib/firecracker/images/leap/kernel",
      "initrd_path":"/var/lib/firecracker/images/leap/initrd",
      "boot_args":"run=\"/bin/bash\"  PILOT_DEBUG=1 overlay_root=/dev/vdb init=/usr/sbin/sci console=ttyS0 root=/dev/vda acpi=off rd.neednet=1 ip=dhcp quiet sci_resume=1"
   },
   "drives":[
      {
         "drive_id":"rootfs",
         "path_on_host":"/var/lib/firecracker/images/leap/rootfs",
         "is_root_device":true,
         "is_read_only":true,
         "cache_type":"Writeback"
      },
      {
         "drive_id":"overlay",
         "path_on_host":"/var/lib/firecracker/storage/mybash.ext2",
         "is_root_device":false,
         "is_read_only":false,
         "cache_type":"Writeback"
      }
   ],
   "network-interfaces":[
      {
         "iface_id":"eth0",
         "guest_mac":"AA:FC:00:00:00:01",
         "host_dev_name":"tap-mybash"
      }
   ],
   "machine-config":{
      "vcpu_count":2,
      "mem_size_mib":4096
   },
   "vsock":{
      "guest_cid":3,
      "uds_path":"/run/sci.sock"
   }
}

As you can see there is also a vsock setup for this instance and firecracker correctly created

ls -l /run/sci.sock
srwxr-xr-x 1 root root 0 Jun 13 21:51 /run/sci.sock

From this point I tried to test the vsock communication between host and guest as it's documented.
So I called inside of the guest

socat VSOCK-LISTEN:52,fork -

and next on the host

socat - UNIX-CONNECT:/run/sci.sock
CONNECT 52

Problem is, nothing happened. So no connection was established and I'm looking for a way to debug it better

What I checked is:

• vhost_vsock enabled on the host

  lsmod | grep sock
  
  vhost_vsock            24576  0
  vmw_vsock_virtio_transport_common    49152  1 vhost_vsock
  vhost                  53248  1 vhost_vsock
  vsock                  53248  2 vmw_vsock_virtio_transport_common,vhost_vsock
  

  looks good imho

• virtio transport enabled in the guest

  lsmod  | grep sock
  
  vhost_vsock            24576  0
  vmw_vsock_virtio_transport_common    49152  1 vhost_vsock
  

  looks good too imho

• VSOCK registered in the guest kernel at boot time

  dmesg | grep VSOCK
  
  [    1.078154] NET: Registered PF_VSOCK protocol family
  

At this point I was running out of ideas and a research on the web did not bring more answers.

Thus I'm kindly asking for your help to identify the issue. I don't think it's related to firecracker itself

Thanks much

[schaefi]

For the record. I solved all my issues related to this topic

[wearyzen]

Hi @schaefi ,
Sorry we were not able to reply in time for this. But I am glad you were able to fix the issues.
For future reference I can suggest enabling logging with log-level set to debug e.g.:

./firecracker --api-sock /tmp/firecracker.socket --log-path logs.fifo --level Debug --show-level --show-log-origin

vsock has lot of debug logs which can help point to the issue.
The vsock examples we have documented are with socat version 1.7.4.0 or later.

If you are debugging something in Firecracker code you can also run Firecracker with debuggers like gdb/lldb/rr.

Also, if you can, it would be helpful for other contributors to know what the issue was and how you were able to resolve it.

[schaefi]

Hi @sudanl0,

thanks for your reply. Indeed the log params you gave me are valuable information. Thanks

Also, if you can, it would be helpful for other contributors to know what the issue was and how you were able to resolve it.

yep sorry I missed that. Ok the issue for me with regards to this topic was not a problem in firecracker itself. As you know for testing firecracker you need an image and I'm building my own ones. Thus one needs to understand which requirements are needed in the guest VM to make vsocks work. After reading how vsocks are implemented it was clear that the guest requires the
vsock transport kernel code either compiled in or loaded. In my case the kernel offers it as a module. Thus making sure that vmw_vsock_virtio_transport is loaded made everything to work as you documented it.

If you ask me for a suggestion how to improve, I think it would be great if the documentation from here: https://github.com/firecracker-microvm/firecracker/blob/main/docs/vsock.md could mention very briefly about the host and guest requirements. Just very briefly such that a user can check if custom VM images meet the needs.

Thanks much, for me it's working really nicely now :)

[wearyzen]

Hi @schaefi ,
Thank you for the valuable feedback. I have created an issue #3872 to update the document.