Skip to content

Fix flaky distributed subscriptions test by renaming the peer nodes - #670

Open
vheathen wants to merge 1 commit into
commanded:mainfrom
Qarma-inspect:fix-distributed-test-node-prefix
Open

vheathen wants to merge 1 commit into
commanded:mainfrom
Qarma-inspect:fix-distributed-test-node-prefix

Conversation

@vheathen

Copy link
Copy Markdown
Contributor

distributed_subscriptions_test.exs sometimes kills the test node's Mox.Server, and every later test that uses MockEventStoreCase then fails on GenServer.call({:global, Mox.Server}, ...) with "no process". On OTP 27 it happened in 1 of 20 runs of four test files. On OTP 29, three full runs of mix test --include distributed on main had 75, 73 and 35 failures out of 501 tests.

Each peer starts :commanded with all its applications, Mox included, and Mox registers {:global, Mox.Server}. When a peer registers before :global has synced with the test node, :global settles the clash by killing the copy on the node whose name sorts higher. The peers are named commanded1 to commanded3, which sort before manager, so the test node's copy is the one that dies.

This renames the peers to peer1 to peer3, which sort after manager, so the test node's copy always survives. A comment in the test explains why the prefix matters. After the change the same 20 runs on OTP 27 and three full runs on OTP 29 pass. Only test code changes.

The test starts three LocalCluster peers named "commanded1" to
"commanded3", while the test node is named "manager". Every peer starts
Mox, which registers {:global, Mox.Server}. If a peer registers the name
before it has synced with the test node, :global resolves the conflict
with random_exit_name/3. That function keeps the process on the node
whose name sorts first and kills the other one. "commanded..." sorts
before "manager", so the test node's Mox.Server was sometimes killed.
Every later test that uses Mox then failed with no process or no
connection errors for {:global, Mox.Server}.

Use the prefix "peer" so the test node always sorts first and keeps its
Mox.Server, whatever the timing of the sync.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant