netlab status is much slower to respond in 1.9.2 compared to the old netlab status in version (1.7.2) for example
#1526
Comments
|
Interesting, could you do a tcpdump during |
|
I've done a few tcpdump to see if I could see something special happening during the Although, when doing a Ctrl+C during the ~/co/netsim-main-lab on revival_q3_2024 ⇡5 !4 ❯ netlab status
Lab 1 in /home/sa/code/netsim-main-lab
status: started
provider(s): libvirt,clab
^C[FATAL] libvirt: Cannot execute "vagrant status --machine-readable": {ex}
┏━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
┃ ┃ ┃ ┃ mgmt ┃ ┃ ┃ ┃ ┃
┃ node ┃ device ┃ image ┃ IPv4 ┃ connec… ┃ provid… ┃ VM/con… ┃ status ┃
┡━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
│ c0xr01 │ vsrx │ junipe… │ 10.194… │ netconf │ libvirt │ ml_1_c… │ Unknown │
├─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤Now if I test this command, it takes ~38s to execute ~/code/netsim-main-lab on revival_q3_2024 ⇡5 !4 ❯ vagrant status --machine-readable ✘ INT netsim-main-lab with sa@sa-netlab01a
1731420511,lb3x01,metadata,provider,libvirt
1731420511,c0xr01,metadata,provider,libvirt
1731420511,c0xr02,metadata,provider,libvirt
1731420511,c0xr03,metadata,provider,libvirt
1731420511,d1xr01,metadata,provider,libvirt
1731420511,d1xr02,metadata,provider,libvirt
1731420511,d1xfw01,metadata,provider,libvirt
1731420511,c0xr04,metadata,provider,libvirt
1731420511,c0xr05,metadata,provider,libvirt
1731420511,c0xr06,metadata,provider,libvirt
1731420512,d2xfw01,metadata,provider,libvirt
1731420512,s2xgw1,metadata,provider,libvirt
1731420512,s2xsw01,metadata,provider,libvirt
1731420512,d3xr01,metadata,provider,libvirt
1731420512,d3xr02,metadata,provider,libvirt
1731420512,d4xr01,metadata,provider,libvirt
1731420512,d4xr02,metadata,provider,libvirt
1731420512,d5xr01,metadata,provider,libvirt
1731420512,s5xr02,metadata,provider,libvirt
1731420512,s5xr03,metadata,provider,libvirt
1731420513,s5xr04,metadata,provider,libvirt
1731420513,s5xr05,metadata,provider,libvirt
1731420513,s5xr06,metadata,provider,libvirt
1731420537,lb3x01,provider-name,libvirt
1731420537,lb3x01,state,inaccessible
1731420537,lb3x01,state-human-short,inaccessible
1731420537,lb3x01,state-human-long,translation missing: en.vagrant_libvirt.states.inaccessible
1731420540,c0xr01,provider-name,libvirt
1731420540,c0xr01,state,running
1731420540,c0xr01,state-human-short,running
1731420540,c0xr01,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420540,c0xr02,provider-name,libvirt
1731420540,c0xr02,state,running
1731420540,c0xr02,state-human-short,running
1731420540,c0xr02,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420540,c0xr03,provider-name,libvirt
1731420540,c0xr03,state,running
1731420540,c0xr03,state-human-short,running
1731420540,c0xr03,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420540,d1xr01,provider-name,libvirt
1731420540,d1xr01,state,running
1731420540,d1xr01,state-human-short,running
1731420540,d1xr01,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420541,d1xr02,provider-name,libvirt
1731420541,d1xr02,state,running
1731420541,d1xr02,state-human-short,running
1731420541,d1xr02,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420541,d1xfw01,provider-name,libvirt
1731420541,d1xfw01,state,running
1731420541,d1xfw01,state-human-short,running
1731420541,d1xfw01,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420541,c0xr04,provider-name,libvirt
1731420541,c0xr04,state,running
1731420541,c0xr04,state-human-short,running
1731420541,c0xr04,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420541,c0xr05,provider-name,libvirt
1731420541,c0xr05,state,running
1731420541,c0xr05,state-human-short,running
1731420541,c0xr05,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420541,c0xr06,provider-name,libvirt
1731420541,c0xr06,state,running
1731420541,c0xr06,state-human-short,running
1731420541,c0xr06,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420541,d2xfw01,provider-name,libvirt
1731420541,d2xfw01,state,running
1731420541,d2xfw01,state-human-short,running
1731420541,d2xfw01,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420541,s2xgw1,provider-name,libvirt
1731420541,s2xgw1,state,running
1731420541,s2xgw1,state-human-short,running
1731420541,s2xgw1,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420542,s2xsw01,provider-name,libvirt
1731420542,s2xsw01,state,running
1731420542,s2xsw01,state-human-short,running
1731420542,s2xsw01,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420542,d3xr01,provider-name,libvirt
1731420542,d3xr01,state,running
1731420542,d3xr01,state-human-short,running
1731420542,d3xr01,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420542,d3xr02,provider-name,libvirt
1731420542,d3xr02,state,running
1731420542,d3xr02,state-human-short,running
1731420542,d3xr02,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420542,d4xr01,provider-name,libvirt
1731420542,d4xr01,state,running
1731420542,d4xr01,state-human-short,running
1731420542,d4xr01,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420542,d4xr02,provider-name,libvirt
1731420542,d4xr02,state,running
1731420542,d4xr02,state-human-short,running
1731420542,d4xr02,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420542,d5xr01,provider-name,libvirt
1731420542,d5xr01,state,running
1731420542,d5xr01,state-human-short,running
1731420542,d5xr01,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420542,s5xr02,provider-name,libvirt
1731420542,s5xr02,state,running
1731420542,s5xr02,state-human-short,running
1731420542,s5xr02,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420543,s5xr03,provider-name,libvirt
1731420543,s5xr03,state,running
1731420543,s5xr03,state-human-short,running
1731420543,s5xr03,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420543,s5xr04,provider-name,libvirt
1731420543,s5xr04,state,running
1731420543,s5xr04,state-human-short,running
1731420543,s5xr04,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420543,s5xr05,provider-name,libvirt
1731420543,s5xr05,state,running
1731420543,s5xr05,state-human-short,running
1731420543,s5xr05,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420543,s5xr06,provider-name,libvirt
1731420543,s5xr06,state,running
1731420543,s5xr06,state-human-short,running
1731420543,s5xr06,state-human-long,The Libvirt domain is running. To stop this machine%!(VAGRANT_COMMA) you can run\n`vagrant halt`. To destroy the machine%!(VAGRANT_COMMA) you can run `vagrant destroy`.
1731420543,,ui,info,Current machine states:\n\nlb3x01 inaccessible (libvirt)\nc0xr01 running (libvirt)\nc0xr02 running (libvirt)\nc0xr03 running (libvirt)\nd1xr01 running (libvirt)\nd1xr02 running (libvirt)\nd1xfw01 running (libvirt)\nc0xr04 running (libvirt)\nc0xr05 running (libvirt)\nc0xr06 running (libvirt)\nd2xfw01 running (libvirt)\ns2xgw1 running (libvirt)\ns2xsw01 running (libvirt)\nd3xr01 running (libvirt)\nd3xr02 running (libvirt)\nd4xr01 running (libvirt)\nd4xr02 running (libvirt)\nd5xr01 running (libvirt)\ns5xr02 running (libvirt)\ns5xr03 running (libvirt)\ns5xr04 running (libvirt)\ns5xr05 running (libvirt)\ns5xr06 running (libvirt)\n\nThis environment represents multiple VMs. The VMs are all listed\nabove with their current state. For more information about a specific\nVM%!(VAGRANT_COMMA) run `vagrant status NAME`.In this, it takes ~25s between those 2 lines: 1731420513,s5xr06,metadata,provider,libvirt
1731420537,lb3x01,provider-name,libvirtlb3x01 is an unmanaged device from a netlab perspective, it's a F5 vagrant box. I now think this is what is causing the issue, when I simply do ~/code/netsim-main-lab on revival_q3_2024 ⇡5 !4 ❯ vagrant status took 38s netsim-main-lab with sa@sa-netlab01a
Current machine states:
lb3x01 inaccessible (libvirt)
c0xr01 running (libvirt)
c0xr02 running (libvirt)
c0xr03 running (libvirt)
d1xr01 running (libvirt)
... |
|
hopefully it won't be affecting others, it seems to be something happening locally, with this specific box. |
I'm wondering what changed since 1.7.2 (February 2024) - it might be worth investigating. I'd suggest to wait for @ipspace to respond before closing |
The early versions of "netlab status" executed "docker ps" and "virsh list". You would get all the VMs and containers, not just those relevant to the current lab. Someone (rightfully) complained about that and so I changed the code to execute "vagrant status" and "containerlab status" (or some such), and as you discovered, "vagrant status" takes a long time. |
'inaccessible' usually means "Vagrant can't reach it via SSH" (and the 30+ seconds is probably the TCP timeout). What happens if you try to do "vagrant ssh lb3x01"? |
|
Good suggestions, thanks for helping me with this! ~/co/netsim-main-lab on revival_q3_2024 ⇡5 !4 ❯ vagrant ssh lb3x01 took 40s netsim-main-lab with sa@sa-netlab01a
VM must be running to open SSH connection. Run `vagrant up`
to start the virtual machine.
~/co/netsim-main-lab on revival_q3_2024 ⇡5 !4 ❯ took 18s netsim-main-lab with sa@sa-netlab01aSo it took 18s for it to fail. But the ~/co/netsim-main-lab on revival_q3_2024 ⇡5 !4 ❯ netlab connect lb3x01 netsim-main-lab with sa@sa-netlab01a
Connecting to 10.194.56.101 using SSH port 22
Last login: Mon Oct 7 11:49:35 2024 from 10.194.50.6
[vagrant@lb3x01:Active:Standalone] ~ #If I go back to the topology, this is the definition of lb3x01: groups:
[...]
unprovisioned:
members: [ lb3x01 ]
nodes:
# Load Balancer Site 3
lb3x01:
id: 36 # to get the ip address .101 (36+65)
libvirt.uuid: 8e7604de-b67f-41ec-b7ef-3d14a167a04e
box: f5/bigip
device: linux
provider: libvirt
memory: 4096
cpu: 2 |
|
I can't remember if on this current lab I have 'altered' the state of vagrant. It could be the reason for this. I will restart the lab in the next few days, and will try again BEFORE doing my mess! |
Yeah, Vagrant doesn't like anyone messing around with what it expects to be running. I managed to get simple labs into a very similar state followed a container-induced crash of the TCP/IP stack (or whatever was going on). Looking through the vagrant-libvirt code (and understanding ~10% of it), it seems like the DHCP lease might have expired (as in, the VM grabs an IP address via DHCP, but then does not renew it?). Would that make sense? |
|
FWIW, added a note explaining why netlab status might take a while. |
I forgot to activate my virtual environment, and realised that netlab
1.7.2-post1was responding tonetlab statusmuch faster than with the version1.9.2:0.5sversus40sNot sure if there are some quick wins doable to improve this, otherwise, not a big issue.
1.7.2-post11.9.2The text was updated successfully, but these errors were encountered: