GetVfsStats (lib/fs/vfs/stats.go) runs statfs in a goroutine and returns ctx.Err() after two seconds. The goroutine assigns the named results (err at line 46, total to inodesFree at lines 49 to 53) and the timeout branch at line 59 writes the same variables on its way out. When the timeout path returns without receiving the worker's result, those two sets of writes are not synchronized.
The buffered resultChan keeps the goroutine from blocking after the caller has returned, but it does nothing to order those writes against the timeout return.
Expected: no race when the timeout path is taken.
Seen with lib v0.60.6 in the Kubernetes job pull-kubernetes-kind-dra-all, which runs the kubelet with the race detector. Six reports, one per result, on a node whose kubelet log lines were arriving up to 22 seconds late at the time: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/142411/pull-kubernetes-kind-dra-all/2103785290568044544
WARNING: DATA RACE
Write at 0x00c003a40010 by goroutine 569:
github.com/google/cadvisor/lib/fs/vfs.GetVfsStats()
github.com/google/cadvisor/lib@v0.60.6/fs/vfs/stats.go:59 +0x6ad
github.com/google/cadvisor/lib/fs/tmpfs.(*tmpfsPlugin).GetStats()
github.com/google/cadvisor/lib@v0.60.6/fs/tmpfs/plugin.go:51 +0x4f
github.com/google/cadvisor/lib/fs.(*RealFsInfo).GetFsInfoForPath()
github.com/google/cadvisor/lib@v0.60.6/fs/fs.go:401 +0x80b
github.com/google/cadvisor/lib/fs.(*RealFsInfo).GetGlobalFsInfo()
github.com/google/cadvisor/lib@v0.60.6/fs/fs.go:536 +0x28
github.com/google/cadvisor/lib/container/raw.(*rawContainerHandler).getFsStats()
github.com/google/cadvisor/lib@v0.60.6/container/raw/handler.go:200 +0xfa
github.com/google/cadvisor/lib/container/raw.(*rawContainerHandler).GetStats()
github.com/google/cadvisor/lib@v0.60.6/container/raw/handler.go:241 +0xbb
github.com/google/cadvisor/lib/manager.(*containerData).updateStats()
github.com/google/cadvisor/lib@v0.60.6/manager/container.go:441 +0x59
github.com/google/cadvisor/lib/manager.(*containerData).housekeepingTick()
github.com/google/cadvisor/lib@v0.60.6/manager/container.go:355 +0x22a
github.com/google/cadvisor/lib/manager.(*containerData).housekeeping()
github.com/google/cadvisor/lib@v0.60.6/manager/container.go:330 +0x689
Previous write at 0x00c003a40010 by goroutine 26416:
github.com/google/cadvisor/lib/fs/vfs.GetVfsStats.func1()
github.com/google/cadvisor/lib@v0.60.6/fs/vfs/stats.go:49 +0x204
The same report also comes through the overlay plugin. The job hit it in eight runs between September 21 and 26. Kubernetes pins the module in go.mod, the file is stats.go at v0.60.6, and master still has the same code. The goroutine and the timeout came in with #3541.
A test that lets statfs finish after the timeout reproduces it with -race every time. Filling a local result in the goroutine and sending only that over the channel fixes it. The change and the test are in #3938.
Related: kubernetes/kubernetes#142439 tracks picking up the fixed module on the Kubernetes side.
GetVfsStats(lib/fs/vfs/stats.go) runsstatfsin a goroutine and returnsctx.Err()after two seconds. The goroutine assigns the named results (errat line 46,totaltoinodesFreeat lines 49 to 53) and the timeout branch at line 59 writes the same variables on its way out. When the timeout path returns without receiving the worker's result, those two sets of writes are not synchronized.The buffered
resultChankeeps the goroutine from blocking after the caller has returned, but it does nothing to order those writes against the timeout return.Expected: no race when the timeout path is taken.
Seen with
lib v0.60.6in the Kubernetes jobpull-kubernetes-kind-dra-all, which runs the kubelet with the race detector. Six reports, one per result, on a node whose kubelet log lines were arriving up to 22 seconds late at the time: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/142411/pull-kubernetes-kind-dra-all/2103785290568044544The same report also comes through the
overlayplugin. The job hit it in eight runs between September 21 and 26. Kubernetes pins the module in go.mod, the file is stats.go at v0.60.6, and master still has the same code. The goroutine and the timeout came in with #3541.A test that lets
statfsfinish after the timeout reproduces it with-raceevery time. Filling a localresultin the goroutine and sending only that over the channel fixes it. The change and the test are in #3938.Related: kubernetes/kubernetes#142439 tracks picking up the fixed module on the Kubernetes side.