This documents MQ support for virtio-net Signed-off-by: Michael S. Tsirkin <
mst@redhat.com> --- --- virtio-spec.txt 2013-11-27 20:29:49.117191134 +0200 +++ virtio-spec-mq.txt 2013-11-27 20:29:27.883191885 +0200 @@ -1321,7 +1321,10 @@ features. Subsystem Device ID 1 - Virtqueues 0:receiveq. 1:transmitq. 2:controlq[footnote: + Virtqueues 0:receiveq0. 1:transmitq0. .... 2N[footnote: +N=0 if VIRTIO_NET_F_MQ is not negotiated, otherwise N is derived +from max_virtqueue_pairs control field. +]: receivqN. 2N+1: transmitqN. 2N+2:controlq[footnote: Only if VIRTIO_NET_F_CTRL_VQ set ] @@ -1372,12 +1375,22 @@ were required. VIRTIO_NET_F_GUEST_ANNOUNCE(21) Guest can send gratuitous packets. - Device configuration layout Two configuration fields are + VIRTIO_NET_F_MQ(22) Device supports multiqueue with automatic + receive steering. + + Device configuration layout Three configuration fields are currently defined. The mac address field always exists (though is only valid if VIRTIO_NET_F_MAC is set), and the status field only exists if VIRTIO_NET_F_STATUS is set. Two read-only bits are currently defined for the status field: - VIRTIO_NET_S_LINK_UP and VIRTIO_NET_S_ANNOUNCE. #define VIRTIO_NET_S_LINK_UP 1 + VIRTIO_NET_S_LINK_UP and VIRTIO_NET_S_ANNOUNCE. The following + read-only field, max_virtqueue_pairs only exists if + VIRTIO_NET_F_MQ is set. This field specifies the maximum number + of each of transmit and receive virtqueues + (receiveq0..receiveqN and transmitq0..transmitqN respectively; + N=max_virtqueue_pairs - 1) that can be configured once + VIRTIO_NET_F_MQ is negotiated. Legal values for this field are + 1 to 0x8000. #define VIRTIO_NET_S_LINK_UP 1 #define VIRTIO_NET_S_ANNOUNCE 2 @@ -1389,12 +1402,16 @@ struct virtio_net_config { u16 status; + u16 max_virtqueue_pairs; + }; Device Initialization 1. The initialization routine should identify the receive and - transmission virtqueues. + transmission virtqueues, up to N+1 of each kind. If + VIRTIO_NET_F_MQ feature bit is negotiated, + N=max_virtqueue_pairs-1, otherwise identify N=0. 2. If the VIRTIO_NET_F_MAC feature bit is set, the configuration space “mac” entry indicates the “physical” address of the the @@ -1409,15 +1426,24 @@ struct virtio_net_config { status can be read from the bottom bit of the “status” config field. Otherwise, the link should be assumed active. -5. The receive virtqueue should be filled with receive buffers. +5. Only receiveq0, transmitq0 and controlq are used by default. + To use more queues driver must negotiate the VIRTIO_NET_F_MQ + feature; initialize up to max_virtqueue_pairs of each of + transmit and receive queues; + execute_VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command specifying the + number of the transmit and receive queues that is going to be + used and wait until the device consumes the controlq buffer and + acks this command. + +6. The receive virtqueues should be filled with receive buffers. This is described in detail below in “Setting Up Receive Buffers”. -6. A driver can indicate that it will generate checksumless +7. A driver can indicate that it will generate checksumless packets by negotating the VIRTIO_NET_F_CSUM feature. This “ checksum offload” is a common feature on modern network cards. -7. If that feature is negotiated[footnote: +8. If that feature is negotiated[footnote: ie. VIRTIO_NET_F_HOST_TSO* and VIRTIO_NET_F_HOST_UFO are dependent on VIRTIO_NET_F_CSUM; a dvice which offers the offload features must offer the checksum feature, and a driver which @@ -1434,7 +1460,7 @@ depending on VIRTIO_NET_F_GUEST_CSUM. This is a common restriction in real, older network cards. ] -8. The converse features are also available: a driver can save +9. The converse features are also available: a driver can save the virtual device some work by negotiating these features.[footnote: For example, a network packet transported between two guests on the same system may not require checksumming at all, nor @@ -1448,9 +1474,10 @@ segmentation, if both guests are amenabl Device Operation -Packets are transmitted by placing them in the transmitq, and -buffers for incoming packets are placed in the receiveq. In each -case, the packet itself is preceeded by a header: +Packets are transmitted by placing them in the +transmitq0..transmitqN, and buffers for incoming packets are +placed in the receiveq0..receiveqN. In each case, the packet +itself is preceeded by a header: struct virtio_net_hdr { @@ -1583,6 +1610,9 @@ elements. If VIRTIO_NET_F_MRG_RXBUF is negotiated, each buffer must be at least the size of the struct virtio_net_hdr. +If VIRTIO_NET_F_MQ is negotiated, each of receiveq0...receiveqN +that will be used should be populated with receive buffers. + Packet Receive Interrupt When a packet is copied into a buffer in the receiveq, the @@ -1745,7 +1775,48 @@ Processing this notification involves: 2. Sending VIRTIO_NET_CTRL_ANNOUNCE_ACK command through control vq. -3. . + Automatic receive steering in multiqueue mode + +If the driver negotiates the VIRTIO_NET_F_MQ feature bit (depends +on VIRTIO_NET_F_CTRL_VQ), it can transmit outgoing packets on one +of the multiple transmitq0..transmitqN and ask the device to +queue incoming packets into one the multiple receiveq0..receiveqN +depending on the packet flow. + + struct virtio_net_ctrl_mq { + u16 virtqueue_pairs; + }; + + #define VIRTIO_NET_CTRL_MQ 1 + #define VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET 0 + #define VIRTIO_NET_CTRL_MQ_VQ_PAIRS_MIN 1 + #define VIRTIO_NET_CTRL_MQ_VQ_PAIRS_MAX 0x8000 + +Multiqueue is disabled by default. Driver enables multiqueue by +executing the VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET command, specifying +the number of the transmit and receive queues that will be used; +thus transmitq0..transmitqn and receiveq0..receiveqn where +n=virtqueue_pairs-1 will be used. All these virtqueues must have +been pre-configured in advance. The range of legal values for the +virtqueue_pairs field is between 1 and max_virtqueue_pairs. + +When multiqueue is enabled, device uses automatic receivesteering +based on packet flow.Programming of the receive steering +classificator is implicit. Transmitting a packet of a specific +flow on transmitqX will cause incoming packets for this flow to +be steered to receiveqX. For uni-directional protocols, or where +no packets have been transmitted yet, device will steer a packet +to a random queue out of the specified receiveq0..receiveqn. + +Multiqueue is disabled by setting virtqueue_pairs = 1 (this is +the default). After the command is consumed by the device, the +device will not steer new packets on virtqueues +receveq1..receiveqN (i.e. other than receiveq0) nor read from +transmitq1..transmitqN (i.e. other than transmitq0); accordingly, +driver should not transmit new packets on virtqueues other than +transmitq0. + + Appendix D: Block Device