Repository navigation
Spotlight indexing causes additional events on macOS #15
Description
@paulhammond reported this issue at howeyc/fsnotify#62
A temporary workaround is to add your folder(s) to the Spotlight Privacy settings.
It looks like the extra update update is being caused by mds (the daemon responsible for updating the file metadata used by spotlight). See lines 6 onwards from the output of fs_usage, which shows the name of any process performing file operations:
$ sudo fs_usage | grep TestFsnotifyAttrib.testfile 11:46:35 open _test/TestFsnotifyAttrib.testfile 0.000065 fsnotify.tes 11:46:35 lstat64 _test/TestFsnotifyAttrib.testfile 0.000010 fsnotify.tes 11:46:35 open _test/TestFsnotifyAttrib.testfile 0.000009 fsnotify.tes 11:46:35 chmod _test/TestFsnotifyAttrib.testfile 0.000024 fsnotify.tes 11:46:35 access /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile 0.000008 dbfseventsd 11:46:35 getattrlist /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile 0.000192 mds 11:46:35 lstat64 /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile 0.000018 mdworker 11:46:35 stat64 /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile 0.000013 mdworker 11:46:35 lstat64 /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile 0.000009 mdworker 11:46:35 getattrlist /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile 0.000116 mdworker 11:46:35 open /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile 0.000024 mdworker 11:46:35 getattrlist /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile 0.000041 mdworker 11:46:35 getattrlist /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile 0.000116 mdworker 11:46:35 getattrlist go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile/.DS_Store 0.000009 mdworker 11:46:35 getattrlist go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile/.DS_Store 0.000006 mdworker 11:46:35 getattrlist go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile/.DS_Store 0.000005 mdworker 11:46:35 getattrlist go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile/.DS_Store 0.000005 mdworker 11:46:35 getattrlist /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile 0.000024 mds 11:46:35 getattrlist /Users/ph/go/src/github.com/howeyc/fsnotify/_test/TestFsnotifyAttrib.testfile 0.000010 mds 11:46:35 unlink _test/TestFsnotifyAttrib.testfile 0.000131 fsnotify.tes(It's also worth noting that this test case consistently passes as of 2c4a662, because mds ignores temporary directories. But the problem is still there.)
I tried implementing throttling but found it ineffective. Spotlight will sometimes touch several files, whereas the throttling was for multiple events on the same file. At the time I tried this, NOTE_ATTRIB would trigger Write events, so it may be worth another shot (as of a100b1a).
Hopefully Spotlight will behave better with FSEvents #11.
Activity
Do not rebuild on extra Spotlight filesystem events
Do not rebuild on extra Spotlight filesystem events
[-]Spotlight indexing causes additional events on OS X[/-][+]Spotlight indexing causes additional events on macOS[/+]danield137 commentedon Feb 27, 2017
any updates or plans on fixing this?
nathany commentedon Feb 27, 2017
no updates yet. the plan to fix this is to use FSEvents (#11) which currently exists as a stand-alone project but isn't integrated into fsnotify:
https://github.com/fsnotify/fsevents
kevinburke commentedon May 4, 2017
Ugh, I just spent two hours trying to debug this :( I wasn't searching for the right terms... I was searching for kqueue sends multiple events, kqueue atime, etc.
For people who find this later the key to debugging this is to run
sudo fs_usage | grep myfileand check for open/utimes events.nathany commentedon May 4, 2017
@kevinburke sorry to hear that. 😢
Is there something that could be added to the Spotlight mention in the README to improve finding the issue?
kevinburke commentedon May 4, 2017
I didn't know Spotlight was responsible until the very end, I just saw multiple CHMOD events firing for the same file. I figured "go install" (another step in the build pipeline) was modifying the atime or something and that was triggering another rerun, or there was an error in fsnotify or justrun (the tool I use for watching for changes)
kevinburke commentedon May 4, 2017
(I started reading the code and adding print statements right away, then I searched Github issues for "kqueue", I didn't bother to check the README)
Do not rebuild on extra Spotlight filesystem events
arp242 commentedon Oct 17, 2022
I'm not sure if there's anything more we can do here; if file attributes get changed and kqueue sends a NOTE_ATTRIB event then, well, we got to report it.
The problem isn't isolated to just Spotlight or even kqueue; it can happen on any backend if some software generates a lot of attribute changes; I've seen issues for it on Windows and Linux too. This is why attributes are now ignored in Windows (Windows reports them as "writes", and they weren't useful anyway). I don't think FSEvents will help much here either: if Spotlight updates the metadata, then FSEvents will still have to report that. Even if we could exclude Spotlight, do we really want to do that? What if I want to listen for Spotlight events? At best we could add some option to filter some things, but this probably shouldn't be enabled by default.
With the benefit of hindsight, maybe adding a Chmod event wasn't even a good idea to start with. It's rarely useful, and has a lot of caveats and problems associated with it. I'll document that a bit clearer in godoc, which should prevent confusion in general.