config: allow to bootstrap anon replica from RO rs

This commit allows to bootstrap an anonymous replica from a replicaset,
where all the instances are in read-only mode.

The reason of the change is that there are no technical reasons to
forbid this action. An anonymous replica is not registered in `_cluster`
system space, so it can join a replicaset even if there are no writable
instances.

Fixes #9432

@TarantoolBot document
Title: config: anonymous replica is now supported

`replication.anon: true` option is now working.

There are configuration constraints that are related to anonymous
replicas.

* A replicaset must contain at least one non-anonymous instance.
* An anonymous replica can't be configured as writable instance using
  `database.mode` or `<replicaset>.leader` options.
* An anonymous replica can't be configured with
  `replication.election_mode` equals to `candidate`, `voter` or `manual`
  (only `off` is allowed).

A few more nuances about anonymous replicas:

* Anonymous replicas are filtered out from default upstream list.
* A `replication.failover: election` replicaset can contain anonymous
  replicas, but `replication.election_mode` defaults to `off` for them
  (unlike non-anonymous instances, where the default is `candidate`).
* `replication.failover: supervised` skips anonymous replicas, when
  choosing a bootstrap leader.
* A anonymous replica can joined a replicaset, which has all the
  instances in read-only mode (unlike a non-anonymous instance).

See details in [1] and [2].

[1]: https://github.com/tarantool/tarantool/issues/9432
[2]: https://github.com/tarantool/tarantool/pull/9418

changelogs/unreleased/config-anonymous-replica.md

@@ -2,9 +2,3 @@
 
 * Effectively supported `replication.anon` option, which was broken in several
   ways before (gh-9432).
-
-  There are caveats that are not resolved yet:
-
-  * An anonymous replica can't be bootstrapped from a replicaset, where all the
-    instance are in read-only mode, however there are no technical problems
-    there (just too tight validation).

src/box/lua/config/applier/box_cfg.lua

@@ -646,10 +646,9 @@ local function apply(config)
         -- already registered. We should decide what to do in the
         -- case.
         --
-        -- TODO: An anonymous replica is not to be registered in
+        -- Note: an anonymous replica is not to be registered in
         -- _cluster. So, it can subscribe to such a replicaset.
-        -- We should decide what to do in the case.
-        if in_replicaset and not has_snap then
+        if in_replicaset and not has_snap and not is_anon then
             local has_rw = false
             if failover == 'off' then
                 for _, peer_name in ipairs(configdata:peers()) do

test/config-luatest/anonymous_replica_test.lua

@@ -331,3 +331,54 @@ g.test_anonymous_replica_election_mode_off = function(g)
     replicaset['instance-004']:exec(verify_election_mode_off)
     replicaset['instance-005']:exec(verify_election_mode_off)
 end
+
+-- Verify that an anonymous replica can join a replicaset that has
+-- all the instances in read-only mode.
+g.test_join_anonymous_replica_to_all_ro_replicaset = function(g)
+    local config = cbuilder.new()
+        :set_replicaset_option('replication.failover', 'manual')
+        :set_replicaset_option('leader', 'instance-001')
+        :add_instance('instance-001', {})
+        :add_instance('instance-002', {})
+        :add_instance('instance-003', {})
+        :config()
+
+    -- Bootstrap the replicaset.
+    local replicaset = replicaset.new(g, config)
+    replicaset:start()
+
+    -- Unset the leader -- make all the instances read-only.
+    local new_config = cbuilder.new(config)
+        :set_replicaset_option('leader', nil)
+        :config()
+    replicaset:reload(new_config)
+
+    -- Verify that the instances actually enter read-only mode.
+    replicaset:each(function(server)
+        server:exec(function()
+            t.assert_equals(box.info.ro, true)
+        end)
+    end)
+
+    -- Add a new anonymous replica into the config and reflect it
+    -- in the replicaset object. Start the replica.
+    local new_config_2 = cbuilder.new(new_config)
+        :add_instance('instance-004', {
+            replication = {
+                anon = true,
+            },
+        })
+        :config()
+    replicaset:sync(new_config_2)
+    replicaset:start_instance('instance-004')
+
+    -- Verify that the new instance is an anonymous replica and
+    -- that it is synchronized with instance-{001,002,003}.
+    t.helpers.retrying({timeout = 60}, function()
+        replicaset['instance-004']:exec(function()
+            t.assert_equals(box.info.id, 0)
+            t.assert_equals(box.info.status, 'running')
+            t.assert_equals(box.space._cluster:count(), 3)
+        end)
+    end)
+end