Mongo Delayed Node
This was my live experience with Mongo when I felt need of something like Delayed Replica. I had a Mongo ReplicaSet setup for some client with 3 nodes – 1 Primary and 2 Secondaries. It is known that whatever is written to Primary’s Oplog, is replicated to the secondaries immediately without any lag or lag of some milliseconds (as per network capacity). Any disastrous command if issued by mistake on Primary will also impact the data residing in secondaries. That is what happened !! Someone having root rights inadvertently issued a ‘drop’ command on Primary which made complete application to go down since it also wiped out all the data on Secondaries too. To avoid such ‘harmful replication’, we should choose to implement a Delayed Replica Node in our setup.
So, in next paragraphs, I would be focussing on configuring a Delayed Replica node in existing Mongo Replica set which causes a replication delay (lag) between Primary and Delayed Node.
What is Delayed Node
A delayed node is like any secondary member in a Replica Set which have a ‘replication lag’ from Primary. That means this node will fetch group of operations from Primary’s oplog & replay them with some predefined delay. This becomes extremely useful in those conditions where there are high chances of accidental commits on Mongo.
Such Delayed Node can not become Primary while election. Also they are kept hidden from applications to prevent reading lagged data.
How to configure a Delayed Node
This goes in continuation with adding members into a Replica Set on Primary. Click here for more detail. Let’s go for an example:
primary.domain.in -> Primary Mongo Node
secondary1.domain.in -> First Secondary Mongo Node.
secondary2.domain.in -> Second Secondary Mongo Node.
delayed.domain.in -> Third ‘delayed’ Secondary Mongo Node.
Connect to to-be Primary Host:
[js]$ mongo –host primary.domain.in –port 27017 [/js]
Initialize the replica set:
[js]> rs.initiate() [/js]
Adding normal secondary Nodes:
[js]myReplicaset:PRIMARY> rs.add("secondary1.domain.in:27017")[/js]
[js]myReplicaset:PRIMARY> rs.add("secondary2.domain.in:27017")[/js]
Adding Delayed secondary Node:
[js]myReplicaset:PRIMARY> cnf = {_id: ‘myReplicaset’, members: [{_id: 0, host: ‘primary.domain.in:27017’}, {_id: 1, host: ‘secondary1.domain.in:27017’}, {_id: 2, host: ‘secondary2.domain.in:27017’}, {_id: 3, host: ‘delayed.domain.in:27017’, priority: 0, hidden: true, slaveDelay: 600}]}[/js]
Note:
priority: Indicates the relative eligibility of a member to become a primary.
hidden: Replica Set hides this instance and does not include the member in the output of
slaveDelay: Number of seconds “behind” the primary that this replica set member should
“lag”. Here we have choosen 600 seconds (10 minutes).
myReplicaset: Replica Set name.
Reconfiguration of Existing Replica Set
[js]myReplicaset:PRIMARY> rs.reconfig(cnf)
myReplicaset:PRIMARY> rs.conf()[/js]
Check Slave Delay parameters
[js]myReplicaset:PRIMARY> rs.conf()
{
"_id" : "myReplicaset",
"version" : 4,
"members" : [
{
"_id" : 0,
"host" : "primary.domain.in:27018",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : 0,
"votes" : 1
},
{
"_id" : 1,
"host" : "secondary1.domain.in:27019",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : 0,
"votes" : 1
},
{
"_id" : 2,
"host" : "secondary2.domain.in:27020",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : 0,
"votes" : 1
},
{
"_id" : 3,
"host" : "delayed.domain.in:27021",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : true,
"priority" : 0,
"tags" : {
},
"slaveDelay" : 600,
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatTimeoutSecs" : 10,
"getLastErrorModes" : {
},
"getLastErrorDefaults" : {
"w" : 1,
"wtimeout" : 0
}
}
}[/js]
Warning: rs.reconfig() can make Primary to step down which usually causes an election to choose a new Primary. Recommendation is to make these changes during scheduled downtime.
Post these changes, any commit on Primary node will be replicated to the Delayed Node after ‘slaveDelay’ seconds, however rest secondary nodes will immediately apply it.
