{{ message }}
Repository navigation
Migration fails with "ghc table does not exist" #1622
Description
This was running off of master against an 8.4.6 database.
+ ./gh-ost -allow-on-master -attempt-instant-ddl -dml-batch-size 50 -exact-rowcount -force-named-cut-over -force-named-panic -postpone-cut-over-flag-file /tmp/queries/bb2d953d-431d-4bf0-
b6b2-d7ed1a2ae9d0/postpone-cutover -throttle-additional-flag-file /tmp/queries/bb2d953d-431d-4bf0-b6b2-d7ed1a2ae9d0/throttle -timestamp-old-table --skip-metadata-lock-check -database test -alter 'add index idx_user_start_time_end_time (user_id, start_time, end_time)' -table queries -host $hostname -user root -ask-pass
[2026/01/30 17:57:10] [info] binlogsyncer.go:191 create BinlogSyncer with config {ServerID:99999 Flavor:mysql Host:[redacted] Port:3306 User:root Password: Localhost: Charset: SemiSyncEnabled:false RawModeEnabled:false TLSConfig:<nil> ParseTime:false TimestampStringLocation:UTC UseDecimal:true RecvB
ufferSize:0 HeartbeatPeriod:0s ReadTimeout:0s MaxReconnectAttempts:0 DisableRetrySync:false VerifyChecksum:false DumpCommandFlag:0 Option:<nil> Logger:0x40004c23c0 Dialer:0x329d40 RowsEv
entDecodeFunc:<nil> TableMapOptionalMetaDecodeFunc:<nil> DiscardGTIDSet:false EventCacheCount:10240 SynchronousEventHandler:<nil>}
[2026/01/30 17:57:10] [info] binlogsyncer.go:443 begin to sync binlog from position (mysql-bin-changelog.027237, 3002)
[2026/01/30 17:57:10] [info] binlogsyncer.go:409 Connected to mysql 8.4.6 server
[2026/01/30 17:57:10] [info] binlogsyncer.go:868 rotate to (mysql-bin-changelog.027237, 3002)
# Migrating `test`.`queries`; Ghost table is `test`.`_queries_gho`
# Migrating `test`.`queries`; Ghost table is `test`.`_queries_gho`
# Migrating ip-172-17-5-122:3306; inspecting ip-172-17-5-122:3306; executing on bastion-0
# Migration started at Fri Jan 30 17:57:10 +0000 2026
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 50; max-load: ; critical-load: ; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/queries/bb2d953d-431d-4bf0-b6b2-d7ed1a2ae9d0/throttle
# postpone-cut-over-flag-file: /tmp/queries/bb2d953d-431d-4bf0-b6b2-d7ed1a2ae9d0/postpone-cutover [set]
# Serving on unix socket: /tmp/gh-ost.test.queries.sock
# Migrating ip-172-17-5-122:3306; inspecting ip-172-17-5-122:3306; executing on bastion-0
# Migration started at Fri Jan 30 17:57:10 +0000 2026
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 50; max-load: ; critical-load: ; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/queries/bb2d953d-431d-4bf0-b6b2-d7ed1a2ae9d0/throttle
# postpone-cut-over-flag-file: /tmp/queries/bb2d953d-431d-4bf0-b6b2-d7ed1a2ae9d0/postpone-cutover [set]
# Serving on unix socket: /tmp/gh-ost.honeycomb_prod.queries.sock
Copy: 0/0 100.0%; Applied: 0; Backlog: 0/1000; Time: 0s(total), 0s(copy); streamer: mysql-bin-changelog.027237:7368; Lag: 0.07s, HeartbeatLag: 0.07s, State: migrating; ETA: due
Copy: 0/426432577 0.0%; Applied: 0; Backlog: 0/1000; Time: 0s(total), 0s(copy); streamer: mysql-bin-changelog.027237:7368; Lag: 0.07s, HeartbeatLag: 0.07s, State: migrating; ETA: N/A
CREATE TABLE `_queries_gho` (
`id` int NOT NULL AUTO_INCREMENT,
...
PRIMARY KEY (`id`),
) ENGINE=InnoDB AUTO_INCREMENT=404517417 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin
[2026/01/30 17:57:10] [info] binlogsyncer.go:225 syncer is closing...
[2026/01/30 17:57:10] [info] binlogsyncer.go:988 kill last connection id 109
[2026/01/30 17:57:10] [info] binlogsyncer.go:255 syncer is closed
2026-01-30 17:57:10 ERROR Error 1146 (42S02): Table 'test._queries_ghc' doesn't exist
Interestingly, I was not running in either "revert" or "resume" mode, so I have no idea why gh-ost didn't actually try to create the changelog table. After the gh-ost process exited, there were no gh-ost tables on the database at all:
(MySQL):test>show tables like '%gh%';
+---------------------------------+
| Tables_in_test (%gh%) |
+---------------------------------+
+---------------------------------+
Activity
saimitta-looker commentedon Feb 21, 2026
Ran into the same exact issue today, was there any investigation into the root cause? Thanks!
+1 to " not running in either "revert" or "resume" mode"
meiji163 commentedon Mar 9, 2026
@ajm188 @saimitta-looker what gh-ost version are you using? I couldn't reproduce this. Does it also error with
--execute?ajm188 commentedon Apr 5, 2026
I was using the 1.18 release candidate. If I get some time this week I can try with 1.18 GA to see if it is happening there as well.
Unfortunately I don't remember if this also errored with
--executebut I will try to remember to grab that info next time aroundNoskyOrg commentedon Aug 3, 2026
Has this issue been fixed?
#1664
ajm188 commentedon Sep 18, 2026
this happened again yesterday, version 1.1.11. i'm going to poke around a bit at logs to see if i can find anything useful and will report back!
ajm188 commentedon Sep 18, 2026
i'm back, and i think this is an intermittent but ultimately harmless error log noise. it's a race between the cleanup and the throttler's lag collection goroutine.
The race between
Throttler.collectReplicationLagandMigrator.finalCleanup:collectReplicationLag's ticker (default 100ms) spawnsgo collectFunc()on every tick,gated only on
thlr.finishedMigrating— which is set inThrottler.Teardown(), called fromMigrator.teardown()afterfinalCleanup()(and the_ghcdrop) has already run.collectFunc()does checkCleanupImminentFlagbefore querying, but it's a check-then-actrace: if the goroutine reads the flag as
0a moment beforefinalCleanupsets it, itproceeds anyway.
readChangelogState("heartbeat")) runs on the inspector connection, i.e. theread replica — not the primary where
DROP TABLE _ghcactually executes. So the race windowis the full replication lag needed for the drop to propagate to the replica.
HeartbeatLagspikes from0.06sto1.07sat the exact moment of theerror, and that spike gave the stale in-flight SELECT enough time to land on the replica
after the drop had already replicated there.
it's pretty rare since it needs a
collectFunc()goroutine spawned in the small window right beforeCleanupImminentFlagis set, and enough replica lag for the drop to beat it there.Planning to look at a fix (tighter check in the collection loop, and/or having
finalCleanupwait out in-flight throttler queries before dropping_ghc).ajm188 commentedon Oct 13, 2026
hey @meiji163 i've got a repro and a bugfix, if you have a few moments to look! :)