In-Band Control Messaging API
This section explains how to use the In-Band Control (IBC) Messaging Application Programming Interface (API). VVR supports a special set of ioctls for accessing the IBC messaging facility. These ioctl commands allow an application to register with the facility, send and receive IBC messages, and unregister from the facility.
The IBC facility enables applications to insert application-defined control messages inband with the Primary RVG update stream being replicated to a Secondary RVG. When an IBC message arrives at the Secondary RVG, replication is frozen until directed to unfreeze by a companion application residing on the Secondary host. In this way, an application can signal a Secondary RVG that some user-defined event has occurred relative to the update stream, such as a point of application-level consistency, and enable the Secondary RVG to take some action while replication is frozen.
VVR provides the following ioctl commands:
IOCTL Commands
Note
The RVG must be started for the IOCTLs to succeed.
RVG devices support five special ioctls: RV_IBC_REGISTER, RV_IBC_UNREGISTER, RV_IBC_SEND, RV_IBC_RECEIVE, and RV_IBC_UNFREEZE. The format for calling each ioctl command is:
For HP-UX
#include <stddef.h>
#include <fcntl.h>
#include <errno.h>
#include <sys/types.h>
typedef uint32_t minor_t;
typedef uint32_t major_t;
#include <vxvm/volmachdep.h>
#include <vxvm/voldefs.h>
#include <vxvm/volioctl.h>
#include <vxvm/volibc.h>
int ioctl(int fd, int cmd, void *arg);
The argument fd is the file descriptor obtained by opening the RVG device using the open (2) system call.
The value of cmd is the ioctl command code, and arg is a pointer to a structure containing the arguments to be passed to the kernel. Definitions of the argument structures for each ioctl are described below.
The return value for all ioctls is 0 if the command was successful, and -1 if it was rejected. If the return value is -1, then errno is set to indicate the cause of the error.
RV_IBC_REGISTER
This ioctl registers an application name for the RVG and returns a key. Only registered application names, using the key, may use the IBC messaging facility on a particular RVG. Multiple application names can be registered for any RVG, up to a maximum of 32.
The ioctl argument structure for the RV_IBC_REGISTER command is:
struct ibc_register_args {
char application_name[NAME_SZ];
int deliver_timeout;
ibc_appid_t application_id;
};
Argument deliver_timeout specifies a time-out value in seconds for delivery of an IBC message after it has arrived at the Secondary RVG. When the time-out expires, the Secondary RVG discards the IBC message and continues replication. See RV_IBC_SEND and RV_IBC_RECEIVE for definition of message delivery. A deliver_timeout of 0 is used to specify no time-out.
Argument application_id is returned by the ioctl. It must be supplied as input argument to all other IBC ioctls.
Use of IBC messages is inherently distributed. A copy or agent of the application is expected to be resident on each participating host, and each participating application must register on its own host. Those resident on the Secondary host must register using an application name identical to the name registered on the Primary host. The returned application_id has a local scope; it can be distributed to any cooperating applications on the same host, but it cannot be used successfully by an application on a remote host.
An IBC message received on the Secondary for an application name that is not registered is discarded after delivery timeout. Registration is not persistent across system reboots. Applications must be registered again after the host reboots. After the Secondary is rebooted, the application must be registered within ten minutes after vxnetd is started if an IBC message has already arrived.
The vxnetd command is started from the system startup script:
/sbin/rc2.d/S994vxnm-vxnetd
On failure, possible values returned in errno are:
EIBC_NOMEM
|
Maximum number of applications (32) already registered.
|
EIBC_DUP_APPLICATION
|
application_name is already registered.
|
RV_IBC_SEND
This ioctl can only be issued against the Primary RVG with a valid key obtained from the RV_IBC_REGISTER ioctl. The ioctl inserts an IBC message into the data update stream of one or all RLINKs attached to the RVG.
If it is desired that an IBC message be inserted at an exact location in the update stream, such as a point of application-level consistency, then there must be no concurrent write activity to the RVG when the RV_IBC_SEND ioctl is issued. Note that writes made to the block device interface of a data volume may be cached, so a disk sync must be done before issuing the ioctl. If there are active writes to the RVG when the ioctl is issued, the insertion point of the IBC message in the RLINK update data stream is arbitrary in relation to that activity.
The ioctl returns using the same semantics as a data write to the RVG; it returns when the IBC message has been committed to the SRL and has also been transferred to all synchronous-mode replicas attached to the RVG.
The ioctl argument structure for the RV_IBC_SEND command is:
struct ibc_send_args { /* IOCTL_STRUCT */
vx_u32_t ibc_magic;
vx_u32_t ibc_version;
ibc_appid_t application_id;
char replica[NAME_SZ];
int flags;
int freeze_timeout;
caddr_t msg_buf;
int msg_len;
};
Argument ibc_magic is used to verify whether the ioctl structure is a valid 4.0 structure. It should be set to NM_IBC_MAGIC.
Argument ibc_version specifies the current IBC version. It should be set to NM_IBC_VERSION.
Argument application_id is the key returned by the RV_IBC_REGISTER ioctl. A registration must be done before the RV_IBC_SEND ioctl can be used.
Argument replica specifies the name of the RLINK to which the IBC message is to be send. The null string specifies a broadcast to all RLINKs currently attached to the Primary RVG.
Argument flags set to RV_IBC_FREEZE causes the secondary replication to freeze for the time-out period specified in freeze_timeout. If replication is not desired to be frozen, then flags should be set to 0.
Argument freeze_timeout specifies a time-out value in seconds between delivery of an IBC message on the Secondary and execution of an RV_IBC_UNFREEZE ioctl against the Secondary RVG. When the time-out expires, replication at the Secondary continues. A time-out value of zero is used to specify no time-out.
Argument msg_buf is a pointer to a buffer containing an IBC message. The content of an IBC message is user-defined and has no restriction except size.
Argument msg_len is the length, in bytes, of the IBC message and can be no greater than 128k bytes.
On failure, possible values returned in errno are:
EIBC_NO_RLINK
|
No RLINK or specified RLINK exists
|
EIO I/O
|
I/O error while logging the IBC message
|
EIBC_MESSAGE_LENGTH
|
Message is greater than maximum allowable length (128K)
|
RV_IBC_RECEIVE
This ioctl can only be issued against a Secondary RVG with a valid key obtained from the RV_IBC_REGISTER ioctl. The ioctl receives an IBC message sent from the Primary RVG. At the time of receipt, Secondary replication is frozen. The state of the data volumes on the Secondary is the same as that on the Primary at the time the IBC message was sent. Secondary replication remains frozen until an RV_IBC_UNFREEZE ioctl is issued against the Secondary RVG, or the freeze_timeout specified when the IBC message was sent expires, or the deliver_timeout specified when the application name was registered for the Primary RVG expires and the receive operation has not been performed.
The ioctl argument structure for the RV_IBC_RECEIVE command is:
struct ibc_receive_args {
ibc_appid_t application_id;
int flags;
ibc_timeout_t timeout;
int drop_count;
caddr_t msg_buf;
size_t buf_len;
size_t msg_len;
};
Argument application_id is the key returned by the RV_IBC_REGISTER ioctl. A registration must be done before the RV_IBC_RECEIVE ioctl can be used.
Argument flags may specify IBC_BLOCK. If this flag is set, the ioctl will block until an IBC message is available to receive. If IBC_BLOCK is not set, the ioctl returns with an error if no IBC message is available.
Argument timeout specifies a time-out value in seconds to block waiting for an IBC message if flag IBC_BLOCK is set. When the time-out has expired, the ioctl returns with an error. A time-out of zero is used to specify no time-out.
Value drop_count is returned by the ioctl. This value contains the number of messages that have been dropped due to delivery time-outs. If drop_count is non-zero, no message is returned and the ioctl returns an error.
Argument msg_buf is a pointer to a buffer to receive the IBC message.
Argument buf_len is the length, in bytes, of the msg_buf.
Value msg_len is returned by the ioctl and specifies the length in bytes of the IBC message. The maximum IBC message length is 128K bytes. If msg_len is greater than buf_len, the IBC message is truncated to buf_len bytes and no error is indicated.
On failure, possible values returned in errno are:
EIBC_NO_APPLICATION
|
Argument application_id is not valid.
|
ENOMSG
|
IBC messages have been dropped due to delivery time-out, or if no IBC message was available.
|
RV_IBC_UNFREEZE
This ioctl can only be issued against a Secondary RVG with a valid key obtained from the RV_IBC_REGISTER ioctl. The ioctl unfreezes replication of the Secondary RVG; that is, it resumes updates of the Secondary volumes.
The ioctl argument structure for the RV_IBC_UNFREEZE command is:
struct ibc_unfreeze_args {
ibc_appid_t application_id;
};
Argument application_id is the key returned by the RV_IBC_REGISTER ioctl. A registration must be done before the RV_IBC_UNFREEZE ioctl can be used.
On failure, the possible values returned in errno are:
EIBC_NO_APPLICATION
|
Argument application_id is not valid.
|
EBUSY
|
There is currently active use of the IBC messaging facility by one or more ioctls using this application_id
|
RV_IBC_UNREGISTER
This ioctl unregisters an application name. This ioctl returns an error if any ioctl is active for the RVG using the registered key.
For a Primary RVG, no RV_IBC_SEND ioctls will be accepted with the registered key after unregistering. Those IBC messages already introduced into the update stream are not affected by a subsequent unregister, even if they have not yet been sent to the Secondary RVG.
For a Secondary RVG, RV_IBC_RECEIVE or RV_IBC_UNFREEZE ioctls using the registered key cannot be successfully executed after the unregister, and any IBC messages arriving for the registered name are discarded.
The ioctl argument structure for the RV_IBC_UNREGISTER command is:
struct ibc_unregister_args {
ibc_appid_t application_id;
};
Argument application_id is the key returned by the RV_IBC_REGISTER ioctl. A registration must be done before the RV_IBC_UNREGISTER ioctl can be used.
On failure, possible values returned in errno are:
EIBC_NO_APPLICATION
|
Application is not registered.
|
EBUSY
|
IBC deliver or unfreeze pending.
|
Using the IBC API
The ioctl command set is intended to be used by a set of daemons, one on the RVG Primary host and one on each Secondary host that is to participate in IBC message retrieval. Each must register under an identical application name and be registered before IBC message generation begins. Because registration does not survive host crashes, but IBC messages once sent do persist beyond host crashes, it is suggested that the Secondary daemons be spawned as a part of system startup.
IBC messages use at-least-once delivery semantics. Retrieval daemons must be tolerant of receiving the same IBC message more than once. It is however guaranteed any duplicate copies of a messages will be delivered before the next new message is delivered.
|
