Adding a 2nd Secondary Site to VVR

Recently, we've had to add a 2nd secondary VVR site to our RDS (RVG). Basically, this means we wanted to replicate from a single primary out to two secondaries. This is nothing out of the ordinary for Veritas' Volume Replicator, but it was something new to us.

As mentioned in a previous post, there is a VVR site in Orlando (with a two node cluster) and another in South Brunswick (a one node cluster). In this exercise, we added a third site in Secaucus (it, too, is a one node cluster). Below are the steps we took.

In this case, VCS is used, but we wanted to just add the 2nd secondary site in this go around, saving the VCS configuration in Secaucus for the next weekend.


1) Copied the dgids to all nodes in /etc/vx/vras/.rdg

2) Edit /etc/hosts on SEC, SBK and ORL

3) Check DNS on SBK and ORL

4) set routing table on servers for secdamsdbp1 vvr nics
a) /etc/sysconfig/network-scripts/route-bond1

5) add the routes between ORL, SBK and SEC

a) route add -net XXX.XXX.XXX.XXX netmask 255.255.255.0 gw XXX.XXX.XXX.XXX


6) plumb up secdamsdbp-vvr on bond1:0

7) Bring Down DAMSdbSG
a) orldamsdbp2 (vcs primary)

# hagrp -offline DAMSdbSG -sys orldamsdbp2


8) Bring Down VCS
a) orldamsdbp1 (vcs secondary)

# hastop -local

b) orldamsdbp2 (vcs primary, vvr primary)

# hastop -local -force

c) sbkdamsdbp1 (vcs primary, vvr secondary)

# hastop -local -force
# hastop -local -force


9) Check That SRL is clean
a) orldamsdbp2 (vvr primary)

# vradmin -g damsdg repstatus damsrvg


10) Check vxrlink
a) orldamsdbp2 (vvr primary)

# vxrlink -g damsdg status orl_to_sbk


11) Migrate VVR from ORL to SBK
a) orldamsdbp2 (vvr primary)

# vradmin -g damsdg migrate damsrvg sbkdamsdbp-vvr


12) Confirm migration
a) sbkdamsdbp1 (new vvr primary)

# vradmin -g damsdg repstatus damsrvg


13) Add SEC to the RDS, as viewed by SBK
a) sbkdamsdbp1 (vvr primary)

# vradmin -g damsdg addsec damsrvg sbkdamsdbp-vvr secdamsdbp-vvr prlink=sbk_to_sec srlink=sec_to_sbk


14) Confirm New Secondary VVR (it should be stopped and inconsistent)
a) sbkdamsdbp1 (vvr primary)

# vradmin -g damsdg repstatus damsrvg


15) Confirm New Rlink (should be unattached and STALE
a) sbkdamsdbp1 (vvr primary)

# vxrlink -g damsdg status sbk_to_sec


16) Stop replication to SEC (will probably not be replicating) SKIPPED
a) sbkdamsdbp1 (vvr primary)

# vradmin -g damsdg stoprep damsrvg secdamsdbp-vvr


17) Start syncing the data
a) sbkdamsdbp1 (vvr primary)

# vradmin -g damsdg -c friday_night_1 syncrvg damsrvg secdamsdbp-vvr


18) Go watch a movie

19) Verify the data
a) sbkdamsdbp1 (vvr primary) SKIP?

# vradmin -g damsdg -verify syncrvg damsrvg secdamsdbp-vvr


20) Finish the movie

21) Start the replication, with DCM autosync
a) sbkdamsdbp1 (vvr primary)

# vradmin -g damsdg -a startrep damsrvg secdamsdbp-vvr


22) Verify that replication completes, and is consistent and up to date
a) sbkdamsdbp1 (vvr primary)

# vradmin -g damsdg repstatus damsrvg


23) Create Rlink from ORL to SEC
a) orldamsdbp2 (vvr secondary)

# vxmake -g damsdg rlink orl_to_sec local_host=orldamsdbp-vvr remote_host=secdamsdbp-vvr remote_rlink=sec_to_orl remote_dg=damsdg

b) orldamsdbp2 (vvr secondary)

# vxrlink -g damsdg assoc damsrvg orl_to_sec


24) Create Rlink from SEC to ORL
a) secdamsdbp1 (vvr secondary)

# vxmake -g damsdg rlink sec_to_orl local_host=secdamsdbp-vvr remote_host=orldamsdbp-vvr remote_rlink=orl_to_sec remote_dg=damsdg

b) secdamsdbp2 (vvr secondary)

# vxrlink -g damsdg assoc damsrvg sec_to_orl


25) Verify replication status and vxprint -Pl on all three nodes
a) orldamsdbp2 (vvr secondary)

# vradmin -g damsdg repstatus damsrvg
# vxprint -Pl

a) sbkdamsdbp1 (vvr primary)

# vradmin -g damsdg repstatus damsrvg
# vxprint -Pl

a) secdamsdbp1 (vvr secondary)

# vradmin -g damsdg repstatus damsrvg
# vxprint -Pl


26) Test migration from SBK to ORL, orl to sec should be inconsistent
a) sbkdamsdbp1 (vvr primary)

# vradmin -g damsdg migrate damsrvg orldamsdbp-vvr
# vradmin -g damsdg repstatus damsrvg

b) orldamsdbp2 (new vvr primary)

# vradmin -g damsdg -a startrep damsrvg secdamsdbp-vvr


27) Test migration from ORL to SEC, sec to sbk should be consistent and stale
a) orldamsdbp1 (vvr primary)

# vradmin -g damsdg migrate damsrvg secdamsdbp-vvr
# vradmin -g damsdg repstatus damsrvg

b) secdamsdbp1 (new vvr primary)

# vradmin -g damsdg -a startrep damsrvg sbkdamsdbp-vvr


28) Test migration from SEC to ORL, orl to sbk should be consistent and stale
a) secdamsdbp1 (vvr primary)

# vradmin -g damsdg migrate damsrvg orldamsdbp-vvr
# vradmin -g damsdg repstatus damsrvg

b) orldamsdbp2 (new vvr primary)

# vradmin -g damsdg -a startrep damsrvg sbkdamsdbp-vvr


29) Test migration from ORL to SBK
a) orldamsdbp1 (vvr primary)

# vradmin -g damsdg migrate damsrvg sbkdamsdbp-vvr
# vradmin -g damsdg repstatus damsrvg

b) sbkdamsdbp1 (new vvr primary)

# vradmin -g damsdg -a startrep damsrvg secdamsdbp-vvr


30) Migrate from SBK to SEC
a) sbkdamsdbp1 (vvr primary)

# vradmin -g damsdg migrate damsrvg secdamsdbp-vvr
# vradmin -g damsdg repstatus damsrvg

b) secdamsdbp1 (new vvr primary)

# vradmin -g damsdg -a startrep damsrvg orldamsdbp-vvr


31) Migrate from SEC to ORL
a) secdamsdbp1 (vvr primary)

# vradmin -g damsdg migrate damsrvg orldamsdbp-vvr
# vradmin -g damsdg repstatus damsrvg

b) orldamsdbp2 (new vvr primary)

# vradmin -g damsdg -a startrep damsrvg sbkdamsdbp-vvr


32) Bring up VCS
a) orldamsdbp2 (vcs primary, vvr primary)

# hastart

b) orldamsdbp1 (vcs secondary)

# hastart

c) sbkdamsdbp1 (vcs primary, vvr secondary); if it shows as stale, run force the startup.

# hastart -onenode
# hasys -force


33) Start the DAMSdbSG service group
a) orldamsdbp2

# hagrp -online DAMSdbSG -sys orldamsdbp2