Skip to main content

Understanding Erlang's :global prevent_overlapping_partitions Option

· 4 min read
👋 I'm a dev at Supabase

I work on logging and analytics, and manage the underlying service that Supabase Logs and Logflare. The service does over 7 billion requests each day with traffic constantly growing, and these devlog posts talk a bit about my day-to-day open source dev work.

It serves as some insight into what one can expect when working on high availability software, with real code snippets and PRs too. Enjoy!😊

The prevent_overlapping_partitions option in Erlang is a configuration parameter that affects how the :global module handles network partitions in distributed Erlang systems.

Introduced in Erlang/OTP 25, prevent_overlapping_partitions is a kernel parameter that enforces strict network partition prevention in distributed Erlang systems. When enabled (which is the default in OTP 25+), it ensures that the network remains fully connected and prevents scenarios where network partitions could lead to inconsistent states. when enabled, it essentially prevents :global from performing partitioning to avoid inconsistent states.

Since :global replicates its name/lock tables on every node and tries to keep them consistent, it will try to maintain a fully connected network mesh so updates propagate everywhere. However, an overlapping partition results in a partially connected network—for example, A is connected to B, and B is connected to C, but A and C are unable to communicate directly. In this scenario, B acts as an overlap between the two "sides", since it can reach both, while A and C cannot see each other at all. When different subsets exchange updates inconsistently, this can make :global's internal state inconsistent, and that inconsistency can remain even after the cluster becomes fully connected again.

The Official Warning​

The Erlang documentation provides a strong warning about this feature that's worth examining in detail:

Prevention of overlapping partitions can be disabled using the prevent_overlapping_partitions Kernel parameter, making global behave like it used to do. This is, however, problematic for all applications expecting a fully connected network to be provided, such as for example mnesia, but also for global itself. A network of overlapping partitions might cause the internal state of global to become inconsistent. Such an inconsistency can remain even after such partitions have been brought together to form a fully connected network again. The effect on other applications that expects that a fully connected network is maintained may vary, but they might misbehave in very subtle hard to detect ways during such a partitioning.

Erlang :global documentation

Disabling this feature can lead to subtle and hard-to-detect issues, particularly in applications that expect a fully connected network.

Real-world Examples: CouchDB and Logflare​

CouchDB​

Interestingly, CouchDB has chosen to disable this feature. In a recent commit, they explicitly turned off prevent_overlapping_partitions. Their reasoning is pragmatic:

  1. CouchDB doesn't use the :global module
  2. They have their own auto-connection module
  3. They wanted to avoid potential increased coordination and message overhead during disconnections

Their commit message explains:

# This will toggle to true in Erlang 25+. However since we don't use global
# any longer, and have our own auto-connection module, we can keep the
# existing global behavior to avoid surprises.

Logflare​

For Logflare's situation, we were experiencing instances going out-of-memory with the :global name server going into boot loops, due to flooding of disconnection messages from the syncing protocol. This would lead to certain nodes getting affected, and slowly spreading like an infection as more and more nodes get impacted from the boot loop behaviour. I dive in deeper with this post.

In the end, we were able to fix the issue by disabling prevent_overlapping_partitions and migrating all :global usage over to :syn, an alternative process registry for Erlang. Syn is used across the Supabase stack, in Realtime and now Analytics (Logflare), so it has quite a proven track record.

Conclusion​

From OTP25+, keep prevent_overlapping_partitions enabled. If you have a large cluster with over a hundred nodes with no reliance on :global for name registration, you can (and probably should) disable it to reduce the :global name server's bottleneck.