2 posts tagged with "global"

When working with distributed Erlang applications at Logflare/Supabase, we encountered an interesting issue where the :global_name_server would become overwhelmed with messages, leading to a boot loop situation. This issue is particularly relevant when dealing with the prevent_overlapping_partitions feature.

Understanding the Boot Loop​

The boot loop behaviour comes about when the global name server becomes overwhelmed with messages in scenarios involving network partitions, where many nodes are connecting or disconnecting simultaneously. This can create a cascade effect where:

1. The global name server receives too many messages
2. Message processing delays lead to timeouts
3. Node reconnection attempts trigger more messages
4. GOTO 1

This behaviour is closely related to OTP issue #9117, and within the issue, I highlighted several main potential factors that could be causing the issue depsite the throw fix that Rickard Green had implemented.

We also observed that this behaviour occurs even when not using :global at all. For Logflare, we had migrated our distributed name registration workloads to use the wonderful :syn library. Hence, this bug is more related to the core syncing protocol of :global.

The throw in restart_connect()​

When the :global server attempts to connect to a new node, it will perform a lock to sync the registered names between each node. In the syncing protocol, the :global server will perform a check to verify that the node is not already attempting to perform a sync (indicated by the pending state) within the server. If it is already attempting a sync, it will instead cancel the connection attempt and retry the connection.

Without a throw, it will result in a deadlock situation, where the :global server will wait forever for the node to complete the sync.

prevent_overlapping_partitions to the rescue​

As documented in :global:

As of OTP 25, global will by default prevent overlapping partitions due to network issues by actively disconnecting from nodes that reports that they have lost connections to other nodes. This will cause fully connected partitions to form instead of leaving the network in a state with overlapping partitions.

This means that :global by default will actively disconnect from nodes that report that they have lost connections to other nodes. For small clusters, this is generally a good feature to have so that the cluster can quickly recover from network issues. However, for large clusters, this can cause a lot of unnecessary disconnections and can lead to the above boot loop issue.

As of time of writing, disabling the prevent_overlapping_partitions feature has allowed our cluster to overcome this boot loop issue by preventing a flood of disconnection messages across clusters. However, this flag needs to be used with caution when using the :global server for name registration, as it may result in inconsistencies if there are overlapping partitions and mutliple instances of the same name are registered. Application code needs to be able to handle this case.

Monitoring strategies​

When dealing with large clusters, I would recommend implementing monitoring for:

• global name server message queue length -- the main indicator of the issue
• memory usage of the global name server -- a secondary indicator of long message queues

Tracing the :global server callbacks at runtime is also a good way to debug the issue, though it is usually not easy as the time window before the node goes out-of-memory is usually very short.

I explain this in more detail in my post on understanding Erlang's :global prevent_overlapping_partitions Option.

The prevent_overlapping_partitions option in Erlang is a configuration parameter that affects how the :global module handles network partitions in distributed Erlang systems.

Introduced in Erlang/OTP 25, prevent_overlapping_partitions is a kernel parameter that enforces strict network partition prevention in distributed Erlang systems. When enabled (which is the default in OTP 25+), it ensures that the network remains fully connected and prevents scenarios where network partitions could lead to inconsistent states. when enabled, it essentially prevents :global from performing partitioning to avoid inconsistent states.

Since :global replicates its name/lock tables on every node and tries to keep them consistent, it will try to maintain a fully connected network mesh so updates propagate everywhere. However, an overlapping partition results in a partially connected network—for example, A is connected to B, and B is connected to C, but A and C are unable to communicate directly. In this scenario, B acts as an overlap between the two "sides", since it can reach both, while A and C cannot see each other at all. When different subsets exchange updates inconsistently, this can make :global's internal state inconsistent, and that inconsistency can remain even after the cluster becomes fully connected again.

The Official Warning​

The Erlang documentation provides a strong warning about this feature that's worth examining in detail:

Prevention of overlapping partitions can be disabled using the prevent_overlapping_partitions Kernel parameter, making global behave like it used to do. This is, however, problematic for all applications expecting a fully connected network to be provided, such as for example mnesia, but also for global itself. A network of overlapping partitions might cause the internal state of global to become inconsistent. Such an inconsistency can remain even after such partitions have been brought together to form a fully connected network again. The effect on other applications that expects that a fully connected network is maintained may vary, but they might misbehave in very subtle hard to detect ways during such a partitioning.

Erlang :global documentation

Disabling this feature can lead to subtle and hard-to-detect issues, particularly in applications that expect a fully connected network.

Real-world Examples: CouchDB and Logflare​

CouchDB​

Interestingly, CouchDB has chosen to disable this feature. In a recent commit, they explicitly turned off prevent_overlapping_partitions. Their reasoning is pragmatic:

1. CouchDB doesn't use the :global module
2. They have their own auto-connection module
3. They wanted to avoid potential increased coordination and message overhead during disconnections

Their commit message explains:

# This will toggle to true in Erlang 25+. However since we don't use global
# any longer, and have our own auto-connection module, we can keep the
# existing global behavior to avoid surprises.

Logflare​

For Logflare's situation, we were experiencing instances going out-of-memory with the :global name server going into boot loops, due to flooding of disconnection messages from the syncing protocol. This would lead to certain nodes getting affected, and slowly spreading like an infection as more and more nodes get impacted from the boot loop behaviour. I dive in deeper with this post.

In the end, we were able to fix the issue by disabling prevent_overlapping_partitions and migrating all :global usage over to :syn, an alternative process registry for Erlang. Syn is used across the Supabase stack, in Realtime and now Analytics (Logflare), so it has quite a proven track record.

Conclusion​

From OTP25+, keep prevent_overlapping_partitions enabled. If you have a large cluster with over a hundred nodes with no reliance on :global for name registration, you can (and probably should) disable it to reduce the :global name server's bottleneck.