OASIS Virtual I/O Device (VIRTIO) TC

One line proposal: Let's use new admin command and admin q for all device types to query device config space for new fields. (always). Details below. Query of device capabilities and configuration using DMA interface. Need: Currently device configuration space is exposed as read only registers. It is growing rapidly. Some devices may be even multi-functionality device in coming future such as net + clock + rdma device. For a PCI transport implementing such ever-growing capabilities, configuration is burdensome as plain registers. Hence, it is required for the driver to query capabilities and configuration using a DMA interface. Interface requirements: 1. Maintain backward compatibility for existing defined configuration fields to stay as registers. 2. Any new field added must be accessed via DMA interface, regardless of device implementation (hw/sw etc). Results in single driver code regardless of device implementation. 3. A device must be able to choose, starting from which field driver must query such configuration via DMA interface. This field offset must be greater than currently defined configuration field. 4. Any driver to device query operation must not be mandated to be mediated by the owner device for PCI VFs or SIOV or SF devices. Driver must be able to communicate query capabilities and configuration fields directly to the device regardless of device type being PCI PF, VF, SF/SIOV device uniformly. 5. When having multi-functionality device in future, it is desired to not always query all the configuration but may be able to query per-functionality configurations. For example, query only steering capabilities, query only rdma capabilities or query only clock capabilities. 6. The driver should be able to query config/capabilities without polling for the DMA completion, in other words, the driver should be able to get notification from the device when DMA command completes. 7. The driver should be able to utilize existing interrupt vector and/or virtqueue for query and set operation without demanding additional interrupt vector whenever possible. There are multiple options for DMA interface. Some of these options are listed below that we would like to consider fulfilling above requirements. Option_1: New DMA interface registers. New registers which allows single outstanding DMA command per device. Such as, struct pci_dma_cmd_mmio_registers { le64 cmd_addr; /* rw */ le32 cmd_len; /* rw: ordered write to it after cmd_addr, this triggers cmd to device */ le32 start_offset_cfg_space; /* fields below this offset are not available as register, dma is must */ }; struct pci_config_access_cmd { u8 opcode; /* 0 = get config, 1 to N set dev specific config */ u8 reserved; le16 msix_vector_index; le64 rsp_addr; /* points to struct pci_config_access_rsp */ le32 rsp_len; }; struct pci_config_access_rsp { u8 status; u8 debug_field; u8 cmd_specific_data[]; }; Cons: 1. Duplication of a VQ interface which can do same work and vq is purposed for "bulk data transfer" in spec already like this use case. 2. Some devices already have CVQ that can easily fulfil this role, which is not utilized. 3. Requires per device additional 16 bytes of register space. 4. Requires an extra msix interrupt vector to differentiate from config change interrupts. Option_2: Use admin vq for all the device types regardless of its transport such as PCI PF, PCI VF, PCI SIOV. In this method a new admin command is issued on the admin vq of the device itself. Pros: 1. Requirements 1 to 7 are fulfilled. 2. Driver can reuse the same vector with CVQ that addresses requirement #7. Cons: 1. Requires per device unique admin queue number and count. Still better than dedicated dma interface of #1, as it requires only 4 additional bytes as opposed to 12 bytes. Option_3: Use control VQ for the devices that already has CVQ. In this method an existing CVQ of the device is used to query device config space. Pros: 1. All 7 requirements are fulfilled. 2. Does not need 4 bytes of admin queue registers. 3. Save MSI-X vectors of the AQ. 4. Superior to option_2 as it doesn't require extra AQ. 5. Most efficient of all 3 options. Cons: 1. Those devices which does not have CVQ, may need to add it if at all they are growing. d. Any other option?

On Tue, Aug 01, 2023 at 07:09:14AM +0000, Parav Pandit wrote:
> One line proposal:
> Let's use new admin command and admin q for all device types to query device config space for new fields. (always).
>
> Details below.
>
> Query of device capabilities and configuration using DMA interface.
> Need:
> Currently device configuration space is exposed as read only registers. It is growing rapidly.
> Some devices may be even multi-functionality device in coming future such as net + clock + rdma device.
> For a PCI transport implementing such ever-growing capabilities, configuration is burdensome as plain registers.
> Hence, it is required for the driver to query capabilities and configuration using a DMA interface.
>
> Interface requirements:
> 1. Maintain backward compatibility for existing defined configuration fields to stay as registers.

This is both too specific and too vague. I think you mean "device
should be able to optionally support both config access over DMA and support
existing transitional and non-transitional virtio pci drivers".

> 2. Any new field added must be accessed via DMA interface, regardless of device implementation (hw/sw etc).
> Results in single driver code regardless of device implementation.
> 3. A device must be able to choose, starting from which field driver must query
> such configuration via DMA interface. This field offset must be greater than currently defined configuration field.

2 and 3 look like implementation more than like a requirement.
Try to think what your actual requirement is.

> 4. Any driver to device query operation must not be mandated to be
> mediated by the owner device for PCI VFs or SIOV or SF devices. Driver must be able to
> communicate query capabilities and configuration fields directly to
> the device regardless of device type being PCI PF, VF, SF/SIOV device uniformly.

I don't really know what does it mean to communicate directly with SIOV device.
neither do I know query is and what capabilities are.

> 5. When having multi-functionality device in future, it is desired to not always
> query all the configuration but may be able to query per-functionality configurations.
> For example, query only steering capabilities, query only rdma capabilities or query only clock capabilities.

I don't know what query is and what capabilities are. Do you mean
"access part of config space"?


> 6. The driver should be able to query config/capabilities without
> polling for the DMA completion, in other words, the driver should be able to get
> notification from the device when DMA command completes.
> 7. The driver should be able to utilize existing interrupt vector and/or
> virtqueue for query and set operation without demanding
> additional interrupt vector whenever possible.

additional over what? existing where?

>
> There are multiple options for DMA interface.
> Some of these options are listed below that we would like to consider fulfilling above requirements.


Cc: Zhu Lingshan <lingshan.zhu@intel.com>

Zhu Lingshan are you still interested in a transport for SIOV (what was
called transport vq)? Clearly SIOV needs this as part of the transport.

--
MST

On Tue, Aug 01, 2023 at 07:09:14AM +0000, Parav Pandit wrote: > One line proposal: > Let's use new admin command and admin q for all device types to query device config space for new fields. (always). > > Details below. > > Query of device capabilities and configuration using DMA interface. > Need: > Currently device configuration space is exposed as read only registers. It is growing rapidly. > Some devices may be even multi-functionality device in coming future such as net + clock + rdma device. > For a PCI transport implementing such ever-growing capabilities, configuration is burdensome as plain registers. > Hence, it is required for the driver to query capabilities and configuration using a DMA interface. > > Interface requirements: > 1. Maintain backward compatibility for existing defined configuration fields to stay as registers. This is both too specific and too vague. I think you mean "device should be able to optionally support both config access over DMA and support existing transitional and non-transitional virtio pci drivers". > 2. Any new field added must be accessed via DMA interface, regardless of device implementation (hw/sw etc). > Results in single driver code regardless of device implementation. > 3. A device must be able to choose, starting from which field driver must query > such configuration via DMA interface. This field offset must be greater than currently defined configuration field. 2 and 3 look like implementation more than like a requirement. Try to think what your actual requirement is. > 4. Any driver to device query operation must not be mandated to be > mediated by the owner device for PCI VFs or SIOV or SF devices. Driver must be able to > communicate query capabilities and configuration fields directly to > the device regardless of device type being PCI PF, VF, SF/SIOV device uniformly. I don't really know what does it mean to communicate directly with SIOV device. neither do I know query is and what capabilities are. > 5. When having multi-functionality device in future, it is desired to not always > query all the configuration but may be able to query per-functionality configurations. > For example, query only steering capabilities, query only rdma capabilities or query only clock capabilities. I don't know what query is and what capabilities are. Do you mean "access part of config space"? > 6. The driver should be able to query config/capabilities without > polling for the DMA completion, in other words, the driver should be able to get > notification from the device when DMA command completes. > 7. The driver should be able to utilize existing interrupt vector and/or > virtqueue for query and set operation without demanding > additional interrupt vector whenever possible. additional over what? existing where? > > There are multiple options for DMA interface. > Some of these options are listed below that we would like to consider fulfilling above requirements. Cc: Zhu Lingshan <lingshan.zhu@intel.com> Zhu Lingshan are you still interested in a transport for SIOV (what was called transport vq)? Clearly SIOV needs this as part of the transport. -- MST

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, August 2, 2023 9:50 AM
>
> On Tue, Aug 01, 2023 at 07:09:14AM +0000, Parav Pandit wrote:
> > One line proposal:
> > Let's use new admin command and admin q for all device types to query
> device config space for new fields. (always).
> >
> > Details below.
> >
> > Query of device capabilities and configuration using DMA interface.
> > Need:
> > Currently device configuration space is exposed as read only registers. It is
> growing rapidly.
> > Some devices may be even multi-functionality device in coming future such as
> net + clock + rdma device.
> > For a PCI transport implementing such ever-growing capabilities,
> configuration is burdensome as plain registers.
> > Hence, it is required for the driver to query capabilities and configuration
> using a DMA interface.
> >
> > Interface requirements:
> > 1. Maintain backward compatibility for existing defined configuration fields to
> stay as registers.
>
> This is both too specific and too vague. I think you mean "device should be able
> to optionally support both config access over DMA and support existing
> transitional and non-transitional virtio pci drivers".
>
Not really.
New fields always over DMA,
for example bitmap for virtio-net counters must be via dma interface and upto hash report via register access.

> > 2. Any new field added must be accessed via DMA interface, regardless of
> device implementation (hw/sw etc).
> > Results in single driver code regardless of device implementation.
> > 3. A device must be able to choose, starting from which field driver
> > must query such configuration via DMA interface. This field offset must be
> greater than currently defined configuration field.
>
> 2 and 3 look like implementation more than like a requirement.
> Try to think what your actual requirement is.
>
It is the actual requirement to not have them as register regardless of implementation.


> > 4. Any driver to device query operation must not be mandated to be
> > mediated by the owner device for PCI VFs or SIOV or SF devices. Driver
> > must be able to communicate query capabilities and configuration
> > fields directly to the device regardless of device type being PCI PF, VF, SF/SIOV
> device uniformly.
>
> I don't really know what does it mean to communicate directly with SIOV
> device.
SIOV is not any special beast. It has its own VQ and registers like VFs etc.
Communicate directly means, any mediated PF driver is not involved in communication.
PF delegate some resource as passthrough to the SIOV device and PF role ends there.
PF driver does not get involved in per command, per access transport.

> neither do I know query is and what capabilities are.
>
Capabilities = roughly config space fields.
The reason I split it because dumping everything in a single config space structure is not good as fields gets sparced.
For example, Xuan adding some 64 bits for counter and tomorrow we may need more bits.
And Heng and I will add steering bits after that.
When half year later more counter bits needed, it will be located at some different place.

If one reserved, it may be still not enough, if reserved too much its unwanted wastage.
So capabilities is basically logically clubbed structure per functionality that one can query independently of other.

Like,
struct steering_capability {

};

struct counters_bitmap {

};

> > 5. When having multi-functionality device in future, it is desired to
> > not always query all the configuration but may be able to query per-
> functionality configurations.
> > For example, query only steering capabilities, query only rdma capabilities or
> query only clock capabilities.
>
> I don't know what query is and what capabilities are. Do you mean "access part
> of config space"?
>
Yes, I explained above.

>
> > 6. The driver should be able to query config/capabilities without
> > polling for the DMA completion, in other words, the driver should be
> > able to get notification from the device when DMA command completes.
> > 7. The driver should be able to utilize existing interrupt vector
> > and/or virtqueue for query and set operation without demanding
> > additional interrupt vector whenever possible.
>
> additional over what? existing where?
>
Additional over existing usage.

> >
> > There are multiple options for DMA interface.
> > Some of these options are listed below that we would like to consider fulfilling
> above requirements.
>
>
> Cc: Zhu Lingshan <lingshan.zhu@intel.com>
>
> Zhu Lingshan are you still interested in a transport for SIOV (what was called
> transport vq)? Clearly SIOV needs this as part of the transport.

Its not related to SIOV.
This proposal works for PF, VF, SIOV without any difference.
And possibly for non PCI transport too when done using VQ.

> From: Michael S. Tsirkin <mst@redhat.com> > Sent: Wednesday, August 2, 2023 9:50 AM > > On Tue, Aug 01, 2023 at 07:09:14AM +0000, Parav Pandit wrote: > > One line proposal: > > Let's use new admin command and admin q for all device types to query > device config space for new fields. (always). > > > > Details below. > > > > Query of device capabilities and configuration using DMA interface. > > Need: > > Currently device configuration space is exposed as read only registers. It is > growing rapidly. > > Some devices may be even multi-functionality device in coming future such as > net + clock + rdma device. > > For a PCI transport implementing such ever-growing capabilities, > configuration is burdensome as plain registers. > > Hence, it is required for the driver to query capabilities and configuration > using a DMA interface. > > > > Interface requirements: > > 1. Maintain backward compatibility for existing defined configuration fields to > stay as registers. > > This is both too specific and too vague. I think you mean "device should be able > to optionally support both config access over DMA and support existing > transitional and non-transitional virtio pci drivers". > Not really. New fields always over DMA, for example bitmap for virtio-net counters must be via dma interface and upto hash report via register access. > > 2. Any new field added must be accessed via DMA interface, regardless of > device implementation (hw/sw etc). > > Results in single driver code regardless of device implementation. > > 3. A device must be able to choose, starting from which field driver > > must query such configuration via DMA interface. This field offset must be > greater than currently defined configuration field. > > 2 and 3 look like implementation more than like a requirement. > Try to think what your actual requirement is. > It is the actual requirement to not have them as register regardless of implementation. > > 4. Any driver to device query operation must not be mandated to be > > mediated by the owner device for PCI VFs or SIOV or SF devices. Driver > > must be able to communicate query capabilities and configuration > > fields directly to the device regardless of device type being PCI PF, VF, SF/SIOV > device uniformly. > > I don't really know what does it mean to communicate directly with SIOV > device. SIOV is not any special beast. It has its own VQ and registers like VFs etc. Communicate directly means, any mediated PF driver is not involved in communication. PF delegate some resource as passthrough to the SIOV device and PF role ends there. PF driver does not get involved in per command, per access transport. > neither do I know query is and what capabilities are. > Capabilities = roughly config space fields. The reason I split it because dumping everything in a single config space structure is not good as fields gets sparced. For example, Xuan adding some 64 bits for counter and tomorrow we may need more bits. And Heng and I will add steering bits after that. When half year later more counter bits needed, it will be located at some different place. If one reserved, it may be still not enough, if reserved too much its unwanted wastage. So capabilities is basically logically clubbed structure per functionality that one can query independently of other. Like, struct steering_capability { }; struct counters_bitmap { }; > > 5. When having multi-functionality device in future, it is desired to > > not always query all the configuration but may be able to query per- > functionality configurations. > > For example, query only steering capabilities, query only rdma capabilities or > query only clock capabilities. > > I don't know what query is and what capabilities are. Do you mean "access part > of config space"? > Yes, I explained above. > > > 6. The driver should be able to query config/capabilities without > > polling for the DMA completion, in other words, the driver should be > > able to get notification from the device when DMA command completes. > > 7. The driver should be able to utilize existing interrupt vector > > and/or virtqueue for query and set operation without demanding > > additional interrupt vector whenever possible. > > additional over what? existing where? > Additional over existing usage. > > > > There are multiple options for DMA interface. > > Some of these options are listed below that we would like to consider fulfilling > above requirements. > > > Cc: Zhu Lingshan <lingshan.zhu@intel.com> > > Zhu Lingshan are you still interested in a transport for SIOV (what was called > transport vq)? Clearly SIOV needs this as part of the transport. Its not related to SIOV. This proposal works for PF, VF, SIOV without any difference. And possibly for non PCI transport too when done using VQ.

On 8/2/2023 12:20 PM, Michael S. Tsirkin wrote:
> On Tue, Aug 01, 2023 at 07:09:14AM +0000, Parav Pandit wrote:
>> One line proposal:
>> Let's use new admin command and admin q for all device types to query device config space for new fields. (always).
>>
>> Details below.
>>
>> Query of device capabilities and configuration using DMA interface.
>> Need:
>> Currently device configuration space is exposed as read only registers. It is growing rapidly.
>> Some devices may be even multi-functionality device in coming future such as net + clock + rdma device.
>> For a PCI transport implementing such ever-growing capabilities, configuration is burdensome as plain registers.
>> Hence, it is required for the driver to query capabilities and configuration using a DMA interface.
>>
>> Interface requirements:
>> 1. Maintain backward compatibility for existing defined configuration fields to stay as registers.
> This is both too specific and too vague. I think you mean "device
> should be able to optionally support both config access over DMA and support
> existing transitional and non-transitional virtio pci drivers".
>
>> 2. Any new field added must be accessed via DMA interface, regardless of device implementation (hw/sw etc).
>> Results in single driver code regardless of device implementation.
>> 3. A device must be able to choose, starting from which field driver must query
>> such configuration via DMA interface. This field offset must be greater than currently defined configuration field.
> 2 and 3 look like implementation more than like a requirement.
> Try to think what your actual requirement is.
>
>> 4. Any driver to device query operation must not be mandated to be
>> mediated by the owner device for PCI VFs or SIOV or SF devices. Driver must be able to
>> communicate query capabilities and configuration fields directly to
>> the device regardless of device type being PCI PF, VF, SF/SIOV device uniformly.
> I don't really know what does it mean to communicate directly with SIOV device.
> neither do I know query is and what capabilities are.
>
>> 5. When having multi-functionality device in future, it is desired to not always
>> query all the configuration but may be able to query per-functionality configurations.
>> For example, query only steering capabilities, query only rdma capabilities or query only clock capabilities.
> I don't know what query is and what capabilities are. Do you mean
> "access part of config space"?
>
>
>> 6. The driver should be able to query config/capabilities without
>> polling for the DMA completion, in other words, the driver should be able to get
>> notification from the device when DMA command completes.
>> 7. The driver should be able to utilize existing interrupt vector and/or
>> virtqueue for query and set operation without demanding
>> additional interrupt vector whenever possible.
> additional over what? existing where?
>
>> There are multiple options for DMA interface.
>> Some of these options are listed below that we would like to consider fulfilling above requirements.
>
> Cc: Zhu Lingshan <lingshan.zhu@intel.com>
>
> Zhu Lingshan are you still interested in a transport for SIOV (what was
> called transport vq)? Clearly SIOV needs this as part of the transport.
Hi Michael,

Rebasing "transport vq" on admin vq is still in my pipeline, I am
currently working
on a patch series implementing virtio pci live migration in the
spec(carry on Jason and Eugenio's work).
I will post it next week and back to transport vq task then.

For SIOV, I have implemented the interfaces for querying the
configurations, for example:
1) Access config space:

247 +\begin{lstlisting}
248 +#define VIRTIO_TRANSPTQ_CTRL_CONFIG    6
249 +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_GET    0
250 +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_SET    1
251 +
252 +struct virtio_transportq_ctrl_dev_config_get {
253 +       u32 offset;
254 +       u32 size;
255 +};
256 +
257 +struct virtio_transportq_ctrl_dev_config_set {
258 +       u32 offset;
259 +       u32 size;
260 +       u8  data[];
261 +};
262 +\end{lstlisting}

I think we can re-use the GET command for SR-IOV, I am afraid SET would
be a side-channel attacking surface for SR-IOV.

2) a specific config, e.g., features:

142 +The features negotiation of managed devices is done by the
143 +following commands:
144 +
145 +\begin{lstlisting}
146 +#define VIRTIO_TRANSPTQ_CTRL_FEAT   3
147 + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET        0
148 + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET        1
149 + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET        2
150 +
151 +struct virtio_transportq_ctrl_dev_features {
152 +        u32 features_len;
153 +        u32 features[features_len];
154 +};
155 +\end{lstlisting}

still can re-use the GET commands.

Thanks
Zhu Lingshan
>

On 8/2/2023 12:20 PM, Michael S. Tsirkin wrote: On Tue, Aug 01, 2023 at 07:09:14AM +0000, Parav Pandit wrote: One line proposal: Let's use new admin command and admin q for all device types to query device config space for new fields. (always). Details below. Query of device capabilities and configuration using DMA interface. Need: Currently device configuration space is exposed as read only registers. It is growing rapidly. Some devices may be even multi-functionality device in coming future such as net + clock + rdma device. For a PCI transport implementing such ever-growing capabilities, configuration is burdensome as plain registers. Hence, it is required for the driver to query capabilities and configuration using a DMA interface. Interface requirements: 1. Maintain backward compatibility for existing defined configuration fields to stay as registers. This is both too specific and too vague. I think you mean "device should be able to optionally support both config access over DMA and support existing transitional and non-transitional virtio pci drivers". 2. Any new field added must be accessed via DMA interface, regardless of device implementation (hw/sw etc). Results in single driver code regardless of device implementation. 3. A device must be able to choose, starting from which field driver must query such configuration via DMA interface. This field offset must be greater than currently defined configuration field. 2 and 3 look like implementation more than like a requirement. Try to think what your actual requirement is. 4. Any driver to device query operation must not be mandated to be mediated by the owner device for PCI VFs or SIOV or SF devices. Driver must be able to communicate query capabilities and configuration fields directly to the device regardless of device type being PCI PF, VF, SF/SIOV device uniformly. I don't really know what does it mean to communicate directly with SIOV device. neither do I know query is and what capabilities are. 5. When having multi-functionality device in future, it is desired to not always query all the configuration but may be able to query per-functionality configurations. For example, query only steering capabilities, query only rdma capabilities or query only clock capabilities. I don't know what query is and what capabilities are. Do you mean "access part of config space"? 6. The driver should be able to query config/capabilities without polling for the DMA completion, in other words, the driver should be able to get notification from the device when DMA command completes. 7. The driver should be able to utilize existing interrupt vector and/or virtqueue for query and set operation without demanding additional interrupt vector whenever possible. additional over what? existing where? There are multiple options for DMA interface. Some of these options are listed below that we would like to consider fulfilling above requirements. Cc: Zhu Lingshan <lingshan.zhu@intel.com> Zhu Lingshan are you still interested in a transport for SIOV (what was called transport vq)? Clearly SIOV needs this as part of the transport. Hi Michael, Rebasing "transport vq" on admin vq is still in my pipeline, I am currently working on a patch series implementing virtio pci live migration in the spec(carry on Jason and Eugenio's work). I will post it next week and back to transport vq task then. For SIOV, I have implemented the interfaces for querying the configurations, for example: 1) Access config space: 247 +egin{lstlisting} 248 +#define VIRTIO_TRANSPTQ_CTRL_CONFIGÂÂÂ 6 249 +Â #define VIRTIO_TRANSPTQ_CTRL_CONFIG_GETÂÂÂ 0 250 +Â #define VIRTIO_TRANSPTQ_CTRL_CONFIG_SETÂÂÂ 1 251 + 252 +struct virtio_transportq_ctrl_dev_config_get { 253 +ÂÂÂÂÂÂ u32 offset; 254 +ÂÂÂÂÂÂ u32 size; 255 +}; 256 + 257 +struct virtio_transportq_ctrl_dev_config_set { 258 +ÂÂÂÂÂÂ u32 offset; 259 +ÂÂÂÂÂÂ u32 size; 260 +ÂÂÂÂÂÂ u8Â data[]; 261 +}; 262 +end{lstlisting} I think we can re-use the GET command for SR-IOV, I am afraid SET would be a side-channel attacking surface for SR-IOV. 2) a specific config, e.g., features: 142 +The features negotiation of managed devices is done by the 143 +following commands: 144 + 145 +egin{lstlisting} 146 +#define VIRTIO_TRANSPTQ_CTRL_FEATÂÂ 3 147 + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GETÂÂÂÂÂÂÂ 0 148 + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SETÂÂÂÂÂÂÂ 1 149 + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GETÂÂÂÂÂÂÂ 2 150 + 151 +struct virtio_transportq_ctrl_dev_features { 152 +ÂÂÂÂÂÂÂ u32 features_len; 153 +ÂÂÂÂÂÂÂ u32 features[features_len]; 154 +}; 155 +end{lstlisting} still can re-use the GET commands. Thanks Zhu Lingshan

> From: Zhu, Lingshan <lingshan.zhu@intel.com>
> Sent: Wednesday, August 2, 2023 2:15 PM
> Rebasing "transport vq" on admin vq is still in my pipeline, I am currently
> working on a patch series implementing virtio pci live migration in the
> spec(carry on Jason and Eugenio's work).
> I will post it next week and back to transport vq task then.
>
Awesome and interesting to us as well.
I have draft for member device migration admin commands.
I guess yours will hit the mailing list first, so we can converge on your series.

> For SIOV, I have implemented the interfaces for querying the configurations, for
> example:
> 1) Access config space:
>
> 247 +\begin{lstlisting}
> 248 +#define VIRTIO_TRANSPTQ_CTRL_CONFIG    6
> 249 +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_GET    0
> 250 +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_SET    1
> 251 +
> 252 +struct virtio_transportq_ctrl_dev_config_get {
> 253 +       u32 offset;
> 254 +       u32 size;
> 255 +};
> 256 +
> 257 +struct virtio_transportq_ctrl_dev_config_set {
> 258 +       u32 offset;
> 259 +       u32 size;
> 260 +       u8  data[];
> 261 +};
> 262 +\end{lstlisting}
>
> I think we can re-use the GET command for SR-IOV, I am afraid SET would be a
> side-channel attacking surface for SR-IOV.
>
Get and set both should be inband for PF, VF, SIOV.
I don’t see any need to diverge and create any weird combinations differently for 3 different device types.

> 2) a specific config, e.g., features:
>
> 142 +The features negotiation of managed devices is done by the
> 143 +following commands:
> 144 +
> 145 +\begin{lstlisting}
> 146 +#define VIRTIO_TRANSPTQ_CTRL_FEAT   3
> 147 + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET        0
> 148 + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET        1
> 149 + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET        2
> 150 +
> 151 +struct virtio_transportq_ctrl_dev_features {
> 152 +        u32 features_len;
> 153 +        u32 features[features_len];
> 154 +};
> 155 +\end{lstlisting}
>
> still can re-use the GET commands.
>
> Thanks
> Zhu Lingshan
> >

> From: Zhu, Lingshan <lingshan.zhu@intel.com> > Sent: Wednesday, August 2, 2023 2:15 PM > Rebasing "transport vq" on admin vq is still in my pipeline, I am currently > working on a patch series implementing virtio pci live migration in the > spec(carry on Jason and Eugenio's work). > I will post it next week and back to transport vq task then. > Awesome and interesting to us as well. I have draft for member device migration admin commands. I guess yours will hit the mailing list first, so we can converge on your series. > For SIOV, I have implemented the interfaces for querying the configurations, for > example: > 1) Access config space: > > 247 +egin{lstlisting} > 248 +#define VIRTIO_TRANSPTQ_CTRL_CONFIGÂÂÂ 6 > 249 +Â #define VIRTIO_TRANSPTQ_CTRL_CONFIG_GETÂÂÂ 0 > 250 +Â #define VIRTIO_TRANSPTQ_CTRL_CONFIG_SETÂÂÂ 1 > 251 + > 252 +struct virtio_transportq_ctrl_dev_config_get { > 253 +ÂÂÂÂÂÂ u32 offset; > 254 +ÂÂÂÂÂÂ u32 size; > 255 +}; > 256 + > 257 +struct virtio_transportq_ctrl_dev_config_set { > 258 +ÂÂÂÂÂÂ u32 offset; > 259 +ÂÂÂÂÂÂ u32 size; > 260 +ÂÂÂÂÂÂ u8Â data[]; > 261 +}; > 262 +end{lstlisting} > > I think we can re-use the GET command for SR-IOV, I am afraid SET would be a > side-channel attacking surface for SR-IOV. > Get and set both should be inband for PF, VF, SIOV. I donât see any need to diverge and create any weird combinations differently for 3 different device types. > 2) a specific config, e.g., features: > > 142 +The features negotiation of managed devices is done by the > 143 +following commands: > 144 + > 145 +egin{lstlisting} > 146 +#define VIRTIO_TRANSPTQ_CTRL_FEATÂÂ 3 > 147 + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GETÂÂÂÂÂÂÂ 0 > 148 + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SETÂÂÂÂÂÂÂ 1 > 149 + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GETÂÂÂÂÂÂÂ 2 > 150 + > 151 +struct virtio_transportq_ctrl_dev_features { > 152 +ÂÂÂÂÂÂÂ u32 features_len; > 153 +ÂÂÂÂÂÂÂ u32 features[features_len]; > 154 +}; > 155 +end{lstlisting} > > still can re-use the GET commands. > > Thanks > Zhu Lingshan > >

On 8/2/2023 4:59 PM, Parav Pandit wrote:
>> From: Zhu, Lingshan <lingshan.zhu@intel.com>
>> Sent: Wednesday, August 2, 2023 2:15 PM
>> Rebasing "transport vq" on admin vq is still in my pipeline, I am currently
>> working on a patch series implementing virtio pci live migration in the
>> spec(carry on Jason and Eugenio's work).
>> I will post it next week and back to transport vq task then.
>>
> Awesome and interesting to us as well.
> I have draft for member device migration admin commands.
> I guess yours will hit the mailing list first, so we can converge on your series.
OK
>
>> For SIOV, I have implemented the interfaces for querying the configurations, for
>> example:
>> 1) Access config space:
>>
>> 247 +\begin{lstlisting}
>> 248 +#define VIRTIO_TRANSPTQ_CTRL_CONFIG    6
>> 249 +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_GET    0
>> 250 +  #define VIRTIO_TRANSPTQ_CTRL_CONFIG_SET    1
>> 251 +
>> 252 +struct virtio_transportq_ctrl_dev_config_get {
>> 253 +       u32 offset;
>> 254 +       u32 size;
>> 255 +};
>> 256 +
>> 257 +struct virtio_transportq_ctrl_dev_config_set {
>> 258 +       u32 offset;
>> 259 +       u32 size;
>> 260 +       u8  data[];
>> 261 +};
>> 262 +\end{lstlisting}
>>
>> I think we can re-use the GET command for SR-IOV, I am afraid SET would be a
>> side-channel attacking surface for SR-IOV.
>>
> Get and set both should be inband for PF, VF, SIOV.
> I don’t see any need to diverge and create any weird combinations differently for 3 different device types.
I am not sure, when passthrough a device to a guest, the guest driver
ownes the config space of the device,
if we allow changes against the config space through admin vq from host,
it looks like a risk.
>
>> 2) a specific config, e.g., features:
>>
>> 142 +The features negotiation of managed devices is done by the
>> 143 +following commands:
>> 144 +
>> 145 +\begin{lstlisting}
>> 146 +#define VIRTIO_TRANSPTQ_CTRL_FEAT   3
>> 147 + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GET        0
>> 148 + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SET        1
>> 149 + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GET        2
>> 150 +
>> 151 +struct virtio_transportq_ctrl_dev_features {
>> 152 +        u32 features_len;
>> 153 +        u32 features[features_len];
>> 154 +};
>> 155 +\end{lstlisting}
>>
>> still can re-use the GET commands.
>>
>> Thanks
>> Zhu Lingshan

> From: Zhu, Lingshan <lingshan.zhu@intel.com>
> Sent: Wednesday, August 2, 2023 3:11 PM

> > Get and set both should be inband for PF, VF, SIOV.
> > I don’t see any need to diverge and create any weird combinations differently
> for 3 different device types.
> I am not sure, when passthrough a device to a guest, the guest driver ownes the
> config space of the device, if we allow changes against the config space through
> admin vq from host, it looks like a risk.

I said the opposite above, get and set is INBAND from guest driver to device for PF, VF and SIOV through its own delegated resource.
There is no AQ mediation from host.
Hence, there is no risk.

> From: Zhu, Lingshan <lingshan.zhu@intel.com> > Sent: Wednesday, August 2, 2023 3:11 PM > > Get and set both should be inband for PF, VF, SIOV. > > I donât see any need to diverge and create any weird combinations differently > for 3 different device types. > I am not sure, when passthrough a device to a guest, the guest driver ownes the > config space of the device, if we allow changes against the config space through > admin vq from host, it looks like a risk. I said the opposite above, get and set is INBAND from guest driver to device for PF, VF and SIOV through its own delegated resource. There is no AQ mediation from host. Hence, there is no risk.

On Wed, Aug 02, 2023 at 10:00:09AM +0000, Parav Pandit wrote:
>
>
> > From: Zhu, Lingshan <lingshan.zhu@intel.com>
> > Sent: Wednesday, August 2, 2023 3:11 PM
>
> > > Get and set both should be inband for PF, VF, SIOV.
> > > I don’t see any need to diverge and create any weird combinations differently
> > for 3 different device types.
> > I am not sure, when passthrough a device to a guest, the guest driver ownes the
> > config space of the device, if we allow changes against the config space through
> > admin vq from host, it looks like a risk.
>
> I said the opposite above, get and set is INBAND from guest driver to device for PF, VF and SIOV through its own delegated resource.
> There is no AQ mediation from host.
> Hence, there is no risk.

I think there are different requirements from different people. For
example, consider compatibility. I know it looks like super important
right now, and we should make it possible, but it's just like with
legacy interface - looked very far away when we introduced
non-transitional, to the point where an option in linux to disable
legacy talks about "flying cars", but the flying cars are nowhere to be
seen and, while we are not where everyone can just drop pre-1.0 bits, a
bunch of people are using e.g. vdpa without any legacy, express devices
in qemu also disabled legacy, and so on.


--
MST

On Wed, Aug 02, 2023 at 10:00:09AM +0000, Parav Pandit wrote: > > > > From: Zhu, Lingshan <lingshan.zhu@intel.com> > > Sent: Wednesday, August 2, 2023 3:11 PM > > > > Get and set both should be inband for PF, VF, SIOV. > > > I donât see any need to diverge and create any weird combinations differently > > for 3 different device types. > > I am not sure, when passthrough a device to a guest, the guest driver ownes the > > config space of the device, if we allow changes against the config space through > > admin vq from host, it looks like a risk. > > I said the opposite above, get and set is INBAND from guest driver to device for PF, VF and SIOV through its own delegated resource. > There is no AQ mediation from host. > Hence, there is no risk. I think there are different requirements from different people. For example, consider compatibility. I know it looks like super important right now, and we should make it possible, but it's just like with legacy interface - looked very far away when we introduced non-transitional, to the point where an option in linux to disable legacy talks about "flying cars", but the flying cars are nowhere to be seen and, while we are not where everyone can just drop pre-1.0 bits, a bunch of people are using e.g. vdpa without any legacy, express devices in qemu also disabled legacy, and so on. -- MST

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, August 2, 2023 5:05 PM

> > I said the opposite above, get and set is INBAND from guest driver to device
> for PF, VF and SIOV through its own delegated resource.
> > There is no AQ mediation from host.
> > Hence, there is no risk.
>
> I think there are different requirements from different people.
Yes.
There are two requirements.
1. Build SIOV device that can live in hypervisor for having 1.x based PCI device in guest

2. Build SIOV device ground up that does not need to have existing limitations of 1.x.

The transport VQ proposal 4 patches had heavy inclination for #1 without mentioning the real objective in the cover letter.
I was proposing that when virtio builds SIOV device it should start with #2.
Because #1 is sort of handled with sr-iov or vdpa already.

#2 is more critical to get it right, after SIOV R2 spec is defined.


> For example,
> consider compatibility. I know it looks like super important right now, and we
> should make it possible, but it's just like with legacy interface - looked very far
> away when we introduced non-transitional, to the point where an option in
> linux to disable legacy talks about "flying cars", but the flying cars are nowhere
> to be seen and, while we are not where everyone can just drop pre-1.0 bits, a
> bunch of people are using e.g. vdpa without any legacy, express devices in
> qemu also disabled legacy, and so on.
>
>
> --
> MST

> From: Michael S. Tsirkin <mst@redhat.com> > Sent: Wednesday, August 2, 2023 5:05 PM > > I said the opposite above, get and set is INBAND from guest driver to device > for PF, VF and SIOV through its own delegated resource. > > There is no AQ mediation from host. > > Hence, there is no risk. > > I think there are different requirements from different people. Yes. There are two requirements. 1. Build SIOV device that can live in hypervisor for having 1.x based PCI device in guest 2. Build SIOV device ground up that does not need to have existing limitations of 1.x. The transport VQ proposal 4 patches had heavy inclination for #1 without mentioning the real objective in the cover letter. I was proposing that when virtio builds SIOV device it should start with #2. Because #1 is sort of handled with sr-iov or vdpa already. #2 is more critical to get it right, after SIOV R2 spec is defined. > For example, > consider compatibility. I know it looks like super important right now, and we > should make it possible, but it's just like with legacy interface - looked very far > away when we introduced non-transitional, to the point where an option in > linux to disable legacy talks about "flying cars", but the flying cars are nowhere > to be seen and, while we are not where everyone can just drop pre-1.0 bits, a > bunch of people are using e.g. vdpa without any legacy, express devices in > qemu also disabled legacy, and so on. > > > -- > MST

On Wed, Aug 02, 2023 at 11:51:21AM +0000, Parav Pandit wrote:
>
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, August 2, 2023 5:05 PM
>
> > > I said the opposite above, get and set is INBAND from guest driver to device
> > for PF, VF and SIOV through its own delegated resource.
> > > There is no AQ mediation from host.
> > > Hence, there is no risk.
> >
> > I think there are different requirements from different people.
> Yes.
> There are two requirements.
> 1. Build SIOV device that can live in hypervisor for having 1.x based PCI device in guest
>
> 2. Build SIOV device ground up that does not need to have existing limitations of 1.x.
>
> The transport VQ proposal 4 patches had heavy inclination for #1 without mentioning the real objective in the cover letter.
> I was proposing that when virtio builds SIOV device it should start with #2.
> Because #1 is sort of handled with sr-iov or vdpa already.
>
> #2 is more critical to get it right, after SIOV R2 spec is defined.


Can't we address both? E.g. use a different group type.


>
> > For example,
> > consider compatibility. I know it looks like super important right now, and we
> > should make it possible, but it's just like with legacy interface - looked very far
> > away when we introduced non-transitional, to the point where an option in
> > linux to disable legacy talks about "flying cars", but the flying cars are nowhere
> > to be seen and, while we are not where everyone can just drop pre-1.0 bits, a
> > bunch of people are using e.g. vdpa without any legacy, express devices in
> > qemu also disabled legacy, and so on.
> >
> >
> > --
> > MST
>

On Wed, Aug 02, 2023 at 11:51:21AM +0000, Parav Pandit wrote: > > > > From: Michael S. Tsirkin <mst@redhat.com> > > Sent: Wednesday, August 2, 2023 5:05 PM > > > > I said the opposite above, get and set is INBAND from guest driver to device > > for PF, VF and SIOV through its own delegated resource. > > > There is no AQ mediation from host. > > > Hence, there is no risk. > > > > I think there are different requirements from different people. > Yes. > There are two requirements. > 1. Build SIOV device that can live in hypervisor for having 1.x based PCI device in guest > > 2. Build SIOV device ground up that does not need to have existing limitations of 1.x. > > The transport VQ proposal 4 patches had heavy inclination for #1 without mentioning the real objective in the cover letter. > I was proposing that when virtio builds SIOV device it should start with #2. > Because #1 is sort of handled with sr-iov or vdpa already. > > #2 is more critical to get it right, after SIOV R2 spec is defined. Can't we address both? E.g. use a different group type. > > > For example, > > consider compatibility. I know it looks like super important right now, and we > > should make it possible, but it's just like with legacy interface - looked very far > > away when we introduced non-transitional, to the point where an option in > > linux to disable legacy talks about "flying cars", but the flying cars are nowhere > > to be seen and, while we are not where everyone can just drop pre-1.0 bits, a > > bunch of people are using e.g. vdpa without any legacy, express devices in > > qemu also disabled legacy, and so on. > > > > > > -- > > MST >

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, August 2, 2023 5:24 PM

> > 2. Build SIOV device ground up that does not need to have existing limitations
> of 1.x.
> >
> > The transport VQ proposal 4 patches had heavy inclination for #1 without
> mentioning the real objective in the cover letter.
> > I was proposing that when virtio builds SIOV device it should start with #2.
> > Because #1 is sort of handled with sr-iov or vdpa already.
> >
> > #2 is more critical to get it right, after SIOV R2 spec is defined.
>
>
> Can't we address both? E.g. use a different group type.

SIOV device is just one type, serving two kinds of commands one for supporting backward compat, second for newer interface.

> From: Michael S. Tsirkin <mst@redhat.com> > Sent: Wednesday, August 2, 2023 5:24 PM > > 2. Build SIOV device ground up that does not need to have existing limitations > of 1.x. > > > > The transport VQ proposal 4 patches had heavy inclination for #1 without > mentioning the real objective in the cover letter. > > I was proposing that when virtio builds SIOV device it should start with #2. > > Because #1 is sort of handled with sr-iov or vdpa already. > > > > #2 is more critical to get it right, after SIOV R2 spec is defined. > > > Can't we address both? E.g. use a different group type. SIOV device is just one type, serving two kinds of commands one for supporting backward compat, second for newer interface.

On Wed, Aug 02, 2023 at 11:58:31AM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, August 2, 2023 5:24 PM
>
> > > 2. Build SIOV device ground up that does not need to have existing limitations
> > of 1.x.
> > >
> > > The transport VQ proposal 4 patches had heavy inclination for #1 without
> > mentioning the real objective in the cover letter.
> > > I was proposing that when virtio builds SIOV device it should start with #2.
> > > Because #1 is sort of handled with sr-iov or vdpa already.
> > >
> > > #2 is more critical to get it right, after SIOV R2 spec is defined.
> >
> >
> > Can't we address both? E.g. use a different group type.
>
> SIOV device is just one type, serving two kinds of commands one for supporting backward compat, second for newer interface.

Not sure about specifics but we can have two groups e.g. SIOV-compat and
SIOV-passthrough.

On Wed, Aug 02, 2023 at 11:58:31AM +0000, Parav Pandit wrote: > > > From: Michael S. Tsirkin <mst@redhat.com> > > Sent: Wednesday, August 2, 2023 5:24 PM > > > > 2. Build SIOV device ground up that does not need to have existing limitations > > of 1.x. > > > > > > The transport VQ proposal 4 patches had heavy inclination for #1 without > > mentioning the real objective in the cover letter. > > > I was proposing that when virtio builds SIOV device it should start with #2. > > > Because #1 is sort of handled with sr-iov or vdpa already. > > > > > > #2 is more critical to get it right, after SIOV R2 spec is defined. > > > > > > Can't we address both? E.g. use a different group type. > > SIOV device is just one type, serving two kinds of commands one for supporting backward compat, second for newer interface. Not sure about specifics but we can have two groups e.g. SIOV-compat and SIOV-passthrough.

On 8/2/2023 4:59 PM, Parav Pandit wrote: From: Zhu, Lingshan <lingshan.zhu@intel.com> Sent: Wednesday, August 2, 2023 2:15 PM Rebasing "transport vq" on admin vq is still in my pipeline, I am currently working on a patch series implementing virtio pci live migration in the spec(carry on Jason and Eugenio's work). I will post it next week and back to transport vq task then. Awesome and interesting to us as well. I have draft for member device migration admin commands. I guess yours will hit the mailing list first, so we can converge on your series. OK For SIOV, I have implemented the interfaces for querying the configurations, for example: 1) Access config space: 247 +egin{lstlisting} 248 +#define VIRTIO_TRANSPTQ_CTRL_CONFIGÂÂÂ 6 249 +Â #define VIRTIO_TRANSPTQ_CTRL_CONFIG_GETÂÂÂ 0 250 +Â #define VIRTIO_TRANSPTQ_CTRL_CONFIG_SETÂÂÂ 1 251 + 252 +struct virtio_transportq_ctrl_dev_config_get { 253 +ÂÂÂÂÂÂ u32 offset; 254 +ÂÂÂÂÂÂ u32 size; 255 +}; 256 + 257 +struct virtio_transportq_ctrl_dev_config_set { 258 +ÂÂÂÂÂÂ u32 offset; 259 +ÂÂÂÂÂÂ u32 size; 260 +ÂÂÂÂÂÂ u8Â data[]; 261 +}; 262 +end{lstlisting} I think we can re-use the GET command for SR-IOV, I am afraid SET would be a side-channel attacking surface for SR-IOV. Get and set both should be inband for PF, VF, SIOV. I donât see any need to diverge and create any weird combinations differently for 3 different device types. I am not sure, when passthrough a device to a guest, the guest driver ownes the config space of the device, if we allow changes against the config space through admin vq from host, it looks like a risk. 2) a specific config, e.g., features: 142 +The features negotiation of managed devices is done by the 143 +following commands: 144 + 145 +egin{lstlisting} 146 +#define VIRTIO_TRANSPTQ_CTRL_FEATÂÂ 3 147 + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DEVICE_GETÂÂÂÂÂÂÂ 0 148 + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_SETÂÂÂÂÂÂÂ 1 149 + #define VIRTIO_TRANSPTQ_CTRL_FEAT_DRIVER_GETÂÂÂÂÂÂÂ 2 150 + 151 +struct virtio_transportq_ctrl_dev_features { 152 +ÂÂÂÂÂÂÂ u32 features_len; 153 +ÂÂÂÂÂÂÂ u32 features[features_len]; 154 +}; 155 +end{lstlisting} still can re-use the GET commands. Thanks Zhu Lingshan

On Tue, Aug 1, 2023 at 3:09?PM Parav Pandit <parav@nvidia.com> wrote:
>
> One line proposal:
> Let's use new admin command and admin q for all device types to query device config space for new fields. (always).

Before we mandate anything to admin command, we need to first invent
an admin command over MMIO interface otherwise it would always be an
issue for the nesting.

>
> Details below.
>
> Query of device capabilities and configuration using DMA interface.
> Need:
> Currently device configuration space is exposed as read only registers.

This is wrong:

1) device configuration space is transport independent, some transport
already use DMA to access the device configuration space
2) device configuration space is not read only, we've already had
several examples of using it as write

> It is growing rapidly.
> Some devices may be even multi-functionality device in coming future such as net + clock + rdma device.
> For a PCI transport implementing such ever-growing capabilities, configuration is burdensome as plain registers.

We've already fixed size VIRTIO_PCI_CAP_PCI_CFG. What's wrong with that?

And we have a lot of device specific virtqueues that could be used for
configuration.

> Hence, it is required for the driver to query capabilities and configuration using a DMA interface.
>
> Interface requirements:
> 1. Maintain backward compatibility for existing defined configuration fields to stay as registers.
> 2. Any new field added must be accessed via DMA interface, regardless of device implementation (hw/sw etc).
> Results in single driver code regardless of device implementation.

Virtio is flexible as it decouples transport from the device model.
This breaks this flexibility, and this prevents non-DMA transport from
being developed.

> 3. A device must be able to choose, starting from which field driver must query
> such configuration via DMA interface. This field offset must be greater than currently defined configuration field.
> 4. Any driver to device query operation must not be mandated to be
> mediated by the owner device for PCI VFs or SIOV or SF devices. Driver must be able to
> communicate query capabilities and configuration fields directly to
> the device regardless of device type being PCI PF, VF, SF/SIOV device uniformly.
> 5. When having multi-functionality device in future, it is desired to not always
> query all the configuration but may be able to query per-functionality configurations.
> For example, query only steering capabilities, query only rdma capabilities or query only clock capabilities.
> 6. The driver should be able to query config/capabilities without
> polling for the DMA completion, in other words, the driver should be able to get
> notification from the device when DMA command completes.
> 7. The driver should be able to utilize existing interrupt vector and/or
> virtqueue for query and set operation without demanding
> additional interrupt vector whenever possible.
>
> There are multiple options for DMA interface.
> Some of these options are listed below that we would like to consider fulfilling above requirements.
>
> Option_1:
> New DMA interface registers.
> New registers which allows single outstanding DMA command per device.
> Such as,
> struct pci_dma_cmd_mmio_registers {
> le64 cmd_addr; /* rw */
> le32 cmd_len; /* rw: ordered write to it after cmd_addr, this triggers cmd to device */
> le32 start_offset_cfg_space; /* fields below this offset are not available as register, dma is must */
> };
>
> struct pci_config_access_cmd {
> u8 opcode; /* 0 = get config, 1 to N set dev specific config */
> u8 reserved;
> le16 msix_vector_index;
> le64 rsp_addr; /* points to struct pci_config_access_rsp */
> le32 rsp_len;
> };
> struct pci_config_access_rsp {
> u8 status;
> u8 debug_field;
> u8 cmd_specific_data[];
> };
>
> Cons:
> 1. Duplication of a VQ interface which can do same work and vq is purposed for "bulk data transfer" in spec already like this use case.
> 2. Some devices already have CVQ that can easily fulfil this role, which is not utilized.
> 3. Requires per device additional 16 bytes of register space.
> 4. Requires an extra msix interrupt vector to differentiate from config change interrupts.
>
> Option_2:
> Use admin vq for all the device types regardless of its transport such as PCI PF, PCI VF, PCI SIOV.
> In this method a new admin command is issued on the admin vq of the device itself.
> Pros:
> 1. Requirements 1 to 7 are fulfilled.
> 2. Driver can reuse the same vector with CVQ that addresses requirement #7.
> Cons:
> 1. Requires per device unique admin queue number and count.
> Still better than dedicated dma interface of #1, as it requires only 4 additional bytes as opposed to 12 bytes.
>
> Option_3:
> Use control VQ for the devices that already has CVQ.
> In this method an existing CVQ of the device is used to query device config space.
> Pros:
> 1. All 7 requirements are fulfilled.
> 2. Does not need 4 bytes of admin queue registers.
> 3. Save MSI-X vectors of the AQ.
> 4. Superior to option_2 as it doesn't require extra AQ.
> 5. Most efficient of all 3 options.
> Cons:
> 1. Those devices which does not have CVQ, may need to add it if at all they are growing.
>
> d. Any other option?

Transport virtqueue on top of admin virtqueue will address this seamlessly.

Thanks


>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>

On Tue, Aug 1, 2023 at 3:09âPM Parav Pandit <parav@nvidia.com> wrote: > > One line proposal: > Let's use new admin command and admin q for all device types to query device config space for new fields. (always). Before we mandate anything to admin command, we need to first invent an admin command over MMIO interface otherwise it would always be an issue for the nesting. > > Details below. > > Query of device capabilities and configuration using DMA interface. > Need: > Currently device configuration space is exposed as read only registers. This is wrong: 1) device configuration space is transport independent, some transport already use DMA to access the device configuration space 2) device configuration space is not read only, we've already had several examples of using it as write > It is growing rapidly. > Some devices may be even multi-functionality device in coming future such as net + clock + rdma device. > For a PCI transport implementing such ever-growing capabilities, configuration is burdensome as plain registers. We've already fixed size VIRTIO_PCI_CAP_PCI_CFG. What's wrong with that? And we have a lot of device specific virtqueues that could be used for configuration. > Hence, it is required for the driver to query capabilities and configuration using a DMA interface. > > Interface requirements: > 1. Maintain backward compatibility for existing defined configuration fields to stay as registers. > 2. Any new field added must be accessed via DMA interface, regardless of device implementation (hw/sw etc). > Results in single driver code regardless of device implementation. Virtio is flexible as it decouples transport from the device model. This breaks this flexibility, and this prevents non-DMA transport from being developed. > 3. A device must be able to choose, starting from which field driver must query > such configuration via DMA interface. This field offset must be greater than currently defined configuration field. > 4. Any driver to device query operation must not be mandated to be > mediated by the owner device for PCI VFs or SIOV or SF devices. Driver must be able to > communicate query capabilities and configuration fields directly to > the device regardless of device type being PCI PF, VF, SF/SIOV device uniformly. > 5. When having multi-functionality device in future, it is desired to not always > query all the configuration but may be able to query per-functionality configurations. > For example, query only steering capabilities, query only rdma capabilities or query only clock capabilities. > 6. The driver should be able to query config/capabilities without > polling for the DMA completion, in other words, the driver should be able to get > notification from the device when DMA command completes. > 7. The driver should be able to utilize existing interrupt vector and/or > virtqueue for query and set operation without demanding > additional interrupt vector whenever possible. > > There are multiple options for DMA interface. > Some of these options are listed below that we would like to consider fulfilling above requirements. > > Option_1: > New DMA interface registers. > New registers which allows single outstanding DMA command per device. > Such as, > struct pci_dma_cmd_mmio_registers { > le64 cmd_addr; /* rw */ > le32 cmd_len; /* rw: ordered write to it after cmd_addr, this triggers cmd to device */ > le32 start_offset_cfg_space; /* fields below this offset are not available as register, dma is must */ > }; > > struct pci_config_access_cmd { > u8 opcode; /* 0 = get config, 1 to N set dev specific config */ > u8 reserved; > le16 msix_vector_index; > le64 rsp_addr; /* points to struct pci_config_access_rsp */ > le32 rsp_len; > }; > struct pci_config_access_rsp { > u8 status; > u8 debug_field; > u8 cmd_specific_data[]; > }; > > Cons: > 1. Duplication of a VQ interface which can do same work and vq is purposed for "bulk data transfer" in spec already like this use case. > 2. Some devices already have CVQ that can easily fulfil this role, which is not utilized. > 3. Requires per device additional 16 bytes of register space. > 4. Requires an extra msix interrupt vector to differentiate from config change interrupts. > > Option_2: > Use admin vq for all the device types regardless of its transport such as PCI PF, PCI VF, PCI SIOV. > In this method a new admin command is issued on the admin vq of the device itself. > Pros: > 1. Requirements 1 to 7 are fulfilled. > 2. Driver can reuse the same vector with CVQ that addresses requirement #7. > Cons: > 1. Requires per device unique admin queue number and count. > Still better than dedicated dma interface of #1, as it requires only 4 additional bytes as opposed to 12 bytes. > > Option_3: > Use control VQ for the devices that already has CVQ. > In this method an existing CVQ of the device is used to query device config space. > Pros: > 1. All 7 requirements are fulfilled. > 2. Does not need 4 bytes of admin queue registers. > 3. Save MSI-X vectors of the AQ. > 4. Superior to option_2 as it doesn't require extra AQ. > 5. Most efficient of all 3 options. > Cons: > 1. Those devices which does not have CVQ, may need to add it if at all they are growing. > > d. Any other option? Transport virtqueue on top of admin virtqueue will address this seamlessly. Thanks > > This publicly archived list offers a means to provide input to the > OASIS Virtual I/O Device (VIRTIO) TC. > > In order to verify user consent to the Feedback License terms and > to minimize spam in the list archive, subscription is required > before posting. > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > List help: virtio-comment-help@lists.oasis-open.org > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > Committee: https://www.oasis-open.org/committees/virtio/ > Join OASIS: https://www.oasis-open.org/join/ >