Podman Machine configured to use hacked NTP server?

[adjavaherian]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I noticed yesterday that after leaving my Podman machine running, my terminal had reset and I got a warning about a datagram proxy request from what appears to be Podman machine to host machine. The request seems to have failed but was for NTP to what appears to be a rogue or compromised NTP server in Switzerland. Not sure how / why Podman would choose to use this NTP server, possibly configured when the VM was created? A little concerning, and wanted to make a note of it here.

Here's the screenshot of the error

Here's some IP info regarding 85.195.227.163. Looks to be a compromised Wordpress site that just so happens to be listening for NTP too. 🤔

https://www.threatcrowd.org/ip.php?ip=85.195.227.163
https://otx.alienvault.com/indicator/ip/85.195.227.163/?utm_medium=InProduct&&utm_source=ThreatCrowd

Here's an lsof showing a different (current) gvproxy request, for reference. Looks like its using a NTP in Sweden at the moment, so thinking that Podman is perhaps using some European NTP pool by default and maybe the compromised Swiss IP is just randomly chosen from the pool?

lsof -nP -iUDP:123
COMMAND   PID             USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
gvproxy 15517 xxxxxx    8u  IPv4 0x167xxxx      0t0  UDP 10.x.x.x:51719->194.58.202.148:123

Steps to reproduce the issue:

1. podman machine init

2. podman machine start

3. sleep / wake the host computer?

Describe the results you received:

See description

Describe the results you expected:

Something a little more local in the NTP request. eg. ntp.apple.com here in the US.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version
Client:
Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.17.3
Built:        Wed Dec  8 13:41:11 2021
OS/Arch:      darwin/amd64

Server:
Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.16.8
Built:        Wed Dec  8 16:45:07 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

podman info --debug
host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    variant: coreos
    version: "35"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.16.16-200.fc35.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1641881600
  memTotal: 2061004800
  ociRuntime:
    name: crun
    package: crun-1.4.3-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.3
      commit: 61c9600d1335127eba65632731e2d72bc3f0b9e8
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 1m 26.6s
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 25
    paused: 0
    running: 0
    stopped: 25
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 163
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 1638999907
  BuiltTime: Wed Dec  8 21:45:07 2021
  GitCommit: ""
  GoVersion: go1.16.8
  OsArch: linux/amd64
  Version: 3.4.4

Package info (e.g. output of rpm -q podman or apt list podman):

brew info podman 
podman: stable 3.4.4 (bottled), HEAD
Tool for managing OCI containers and pods

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

No

Additional environment details (AWS, VirtualBox, physical, etc.):

[mheon]

I see no mention of that IP, or NTP in general, in our ignition config. Possible this is a more general FCOS issue?

@baude PTAL

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@Luap99 @dustymabe @baude @mccv1r0 @giuseppe @mtrmac Any ideas?

[dustymabe]

chrony is configured to use 2.fedora.pool.ntp.org:

$ head -n 5 /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (https://www.pool.ntp.org/join.html).
pool 2.fedora.pool.ntp.org iburst

but it also will pick up NTP servers from DHCP if they are advertised there.

what does chronyc sources and chronyc -n sources show?

If you run journalctl and look at the logs for NetworkManager do you see anything about a dispatcher getting run to set an NTP server?

[mtrmac]

The pool is inherently going to point at various unpredictable IP addresses over time.

Insisting on having control over the individual destinations is fine, of course, but requires not using the configured-by-default pool.

[rhatdan]

Is this an issue with coreos, the user or Podman?

[dustymabe]

Assuming the user didn't change anything and their DHCP server didn't provide an NTP server then chrony should be using 2.fedora.pool.ntp.org. I guess it's possible the "bad" server in question came from that pool.

Either way it's not really something the FCOS team can do anyting with. If there is a server in 2.fedora.pool.ntp.org that shouldn't be there the Fedora infra team would need to fix it. So.. confirm that is the case and then open an infra ticket at https://pagure.io/fedora-infrastructure/issues/

[rhatdan]

Moving this issue to a discussion.