Speed too slowly if set cpu-limit for pod in Kubernetes. Or high cpu usage with cpu-nolimit.

[penglongli]

I want to develop a file distributor based on torrent and deploy it in a Kubernetes cluster using DaemonSet. I deployed the chihaya/chihaya as the tracker.

Every node i have a torrent client:

	clientConfig := torrent.NewDefaultClientConfig()
	clientConfig.DataDir = th.op.TransferPath
	clientConfig.Seed = true
	clientConfig.ListenPort = int(th.op.TorrentPort)
	clientConfig.DefaultStorage = storage.NewFileByInfoHash(th.op.TransferPath)
	clientConfig.DisableUTP = true
	clientConfig.MaxUnverifiedBytes = 64 << 30
	clientConfig.NoDHT = false
	clientConfig.DisablePEX = false
	clientConfig.EstablishedConnsPerTorrent = 200
	clientConfig.HalfOpenConnsPerTorrent = 100
	clientConfig.TorrentPeersHighWater = 2000
	clientConfig.MaxUnverifiedBytes = 671088640

When i need distribute a file from one node, i'll generate the torrent first

pieceLength = metainfo.ChoosePieceLength(layerFileInfo.Size())
info := metainfo.Info{
	PieceLength: pieceLength,
}
if err := info.BuildFromFilePath(layerFile); err != nil {
	return nil, errors.Wrapf(err, "build torrent metainfo from file '%s' failed", layerFileInfo)
}
mi := metainfo.MetaInfo{
	InfoBytes: bencode.MustMarshal(info),
}
ih := mi.HashInfoBytes()
to, _ := th.client.AddTorrentOpt(torrent.AddTorrentOpts{
	InfoHash: ih,
	Storage: storage.NewFileOpts(storage.NewFileClientOpts{
		ClientBaseDir: layerFile,
		FilePathMaker: func(opts storage.FilePathMakerOpts) string {
			return filepath.Join(opts.File.BestPath()...)
		},
		TorrentDirMaker: nil,
		PieceCompletion: th.pc,
	}),
	ChunkSize: 128 * 1024,
})
err := to.MergeSpec(&torrent.TorrentSpec{
	InfoBytes: mi.InfoBytes,
	Trackers:  [][]string{{th.op.TorrentAnnounce}},
})

Then every torrent client(running in K8S pod) will start download this torrent. But the speed is too slowly, about 8-9MiB/s.
(The K8S pod is set cpu limit to 1c)

If i set cpu no-limit, the torrent client will use about 6c CPU, speed 60MiB/s. I don't know where the problem is.

pprof file
cpu.pb.gz

[anacrolix]

To clarify, if you don't limit the CPU, it uses 600% but you get about 7x the speed than when you limit it to 100%?

[anacrolix]

You probably don't want to set some of those constants so high, particularly max unverified bytes. The request algorithm will try to read ahead much to far and expend a lot of computation excessively parallelizing downloads.

More is not better here. On resource constrained systems you actually want to lower those values.

Could you let me know how you go with mostly defaults?

[anacrolix]

The default storage implementation, file, is not very good and has very high system overhead.

If you are caching, consider possum, otherwise mmap, if you are able to control file consistency/availability.

[anacrolix]

The CPU profile you submitted makes me think that the request upload routine could do with some optimization, it probably needs to batch a bit. Otherwise it's a pretty healthy looking profile. I think you could mostly improve on that by reducing EstablishedConnsPerTorrent to maybe 25 or so.