OASIS Virtual I/O Device (VIRTIO) TC

> From: Jason Wang <jasowang@redhat.com> > Sent: Wednesday, August 2, 2023 3:02 PM > > On Wed, Aug 2, 2023 at 5:07âPM Parav Pandit <parav@nvidia.com> wrote: > > > > > > > From: Jason Wang <jasowang@redhat.com> > > > Sent: Wednesday, August 2, 2023 2:23 PM > > > > > > On Tue, Aug 1, 2023 at 3:09âPM Parav Pandit <parav@nvidia.com> wrote: > > > > > > > > One line proposal: > > > > Let's use new admin command and admin q for all device types to > > > > query > > > device config space for new fields. (always). > > > > > > Before we mandate anything to admin command, we need to first invent > > > an admin command over MMIO interface otherwise it would always be an > > > issue for the nesting. > > > > > Nesting can be independent requirement in itself.' > > I don't understand here. If you tie new fields to the DMA interface, it basically > means nesting won't get any new features unless: > > 1) it's a PCI VF > 2) SR-IOV emulation is done > 3) admin virtqueue emulation is done > > If you want differ nesting devices from others, it would be a nightmare to > maintain. > New fields for sure are tied to the DMA interface. And nesting can also get just like how a VQ works in nested mode. > > > > > > 1) device configuration space is transport independent, some > > > transport already use DMA to access the device configuration space > > You can say ccw instead of "some". :) > > Kind of but the transport vq proposal goes in the same way. > We debated many times that the wording transport vq is wrong as it is _not_ going to transport driver notifications. Anyway, there is nothing to discuss here. So focusing on main items below. > > > > > 2) device configuration space is not read only, we've already had > > > several examples of using it as write > > > > > It is even worse to have writable. > > Well, what I meant is that, it's not necessarily read only and not necessarily a > register interface. > I took PCI as being most common interface and took net, blk as devices who are experiencing high growth on features and device specific config space. This isnât really a normative part of the spec. The key takeway to have it, for common things it is read only and its register. > > > > > > It is growing rapidly. > > > > Some devices may be even multi-functionality device in coming > > > > future such as > > > net + clock + rdma device. > > > > For a PCI transport implementing such ever-growing capabilities, > > > configuration is burdensome as plain registers. > > > > > > We've already fixed size VIRTIO_PCI_CAP_PCI_CFG. What's wrong with that? > > > > > The wrong part is: it is still and indirect and slow, sub-optimal register > interface. > > Do we really care about the performance here? When it comes to bulk data transfer, in range of few hundred bytes and looking for 5+ year period, than yes, reading using indirect register is slow. And after all it is still indirect register which is does not have time variability part. So it is not fulfilling the requirement at all. > And if it is one of your major > concerns, it's better to explain it along with the "ever-growing" concern. > I thought it is clear that it is still a register, even slower than current one and still have same issues without it. > > VQ is decoupled from transport already. > > So, there is no flexibility broken. > > And yet you suggested transport dependent VIRTIO_PCI_CAP_PCI_CFG above > > making it further wrong. :) > > > > The context here is that you want to mandate any new fields to be DMA. > DMA is obviously transport specific. There are transports that don't use DMA at > all (e.g the shared memory). > VQ is surprisingly does the DMA without being transport specific. How net device mandate tx pkts via vq, how a console device mandate receive and transmit queue, How a crypto device mandate a control vq, What is proposed here is no different... Shared memory is not for bulk data transfer in virtio spec. We donât see "shared memory" as transport in " Virtio Transport Options" section. > > > > 3. A device must be able to choose, starting from which field > > > > driver must query such configuration via DMA interface. This field > > > > offset must be > > > greater than currently defined configuration field. > > [...] > > > > > > > > > d. Any other option? > > > > > > Transport virtqueue on top of admin virtqueue will address this seamlessly. > > > > > :) > > > > Donât see why one would create few more objects on top of aq when aq or > cvq itself can fulfil the need. > > Can you please elaborate? > > If cvq can work, there's no need for any other methods. Cvq is not present for all the device, at the same time all devices are not experiencing high growth of config space either. Hence the discussion here. Michael raised the concern to use some generic scheme for all the device previously, hence the aq. I think the practical approach is, whichever device has cvq, they should be able to use cvq for config+caps query+set. And rest can burn AQ to transport the same command. > If you care about admin > virtqueue then device configuration space is not the only thing that can be > "ever growing", common_cfg is another one. The idea is to do minimal bootstrap work from the common config space and switch to the queue. So common config shouldnât be growing either other than minimal bootstrap functionality. Hence, common config also to be available via dma command. > Allowing them to be accessed via a > virtqueue (thus DMA in PCI transport) is basically the idea of the transport > virtqueue. Ling shan is rebasing transport virtqueue proposals on top of admin > commands, we can see then. Current proposal doesnt need yet another new transport vq with/without nesting. We just need to make choice as _always_admin command or efficiently use cvq when available.

On Wed, Aug 02, 2023 at 09:57:47AM +0000, Parav Pandit wrote:
> > If you care about admin
> > virtqueue then device configuration space is not the only thing that can be
> > "ever growing", common_cfg is another one.
> The idea is to do minimal bootstrap work from the common config space and switch to the queue.
> So common config shouldn’t be growing either other than minimal bootstrap functionality.
> Hence, common config also to be available via dma command.

That is more like a new transport. I thought this proposal is focusing
on device specific config - let's get a handle on that first?
"common config" is a pci transport specific thing.

On Wed, Aug 02, 2023 at 09:57:47AM +0000, Parav Pandit wrote: > > If you care about admin > > virtqueue then device configuration space is not the only thing that can be > > "ever growing", common_cfg is another one. > The idea is to do minimal bootstrap work from the common config space and switch to the queue. > So common config shouldnât be growing either other than minimal bootstrap functionality. > Hence, common config also to be available via dma command. That is more like a new transport. I thought this proposal is focusing on device specific config - let's get a handle on that first? "common config" is a pci transport specific thing.

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, August 2, 2023 4:56 PM
>
> On Wed, Aug 02, 2023 at 09:57:47AM +0000, Parav Pandit wrote:
> > > If you care about admin
> > > virtqueue then device configuration space is not the only thing that
> > > can be "ever growing", common_cfg is another one.
> > The idea is to do minimal bootstrap work from the common config space and
> switch to the queue.
> > So common config shouldn’t be growing either other than minimal bootstrap
> functionality.
> > Hence, common config also to be available via dma command.
>
> That is more like a new transport. I thought this proposal is focusing on device
> specific config - let's get a handle on that first?
> "common config" is a pci transport specific thing.

Sure, device config is the real pain point we are trying to solve first.

Using cvq for those devices who has it seems the most optimal approach.
If we liberate ourselves from single monolithic config space structure and move to query device capabilities, resources, configuration, at functionality level, life is lot easier.
What are your thoughts?

> From: Michael S. Tsirkin <mst@redhat.com> > Sent: Wednesday, August 2, 2023 4:56 PM > > On Wed, Aug 02, 2023 at 09:57:47AM +0000, Parav Pandit wrote: > > > If you care about admin > > > virtqueue then device configuration space is not the only thing that > > > can be "ever growing", common_cfg is another one. > > The idea is to do minimal bootstrap work from the common config space and > switch to the queue. > > So common config shouldnât be growing either other than minimal bootstrap > functionality. > > Hence, common config also to be available via dma command. > > That is more like a new transport. I thought this proposal is focusing on device > specific config - let's get a handle on that first? > "common config" is a pci transport specific thing. Sure, device config is the real pain point we are trying to solve first. Using cvq for those devices who has it seems the most optimal approach. If we liberate ourselves from single monolithic config space structure and move to query device capabilities, resources, configuration, at functionality level, life is lot easier. What are your thoughts?

On Wed, Aug 02, 2023 at 11:29:53AM +0000, Parav Pandit wrote:
>
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, August 2, 2023 4:56 PM
> >
> > On Wed, Aug 02, 2023 at 09:57:47AM +0000, Parav Pandit wrote:
> > > > If you care about admin
> > > > virtqueue then device configuration space is not the only thing that
> > > > can be "ever growing", common_cfg is another one.
> > > The idea is to do minimal bootstrap work from the common config space and
> > switch to the queue.
> > > So common config shouldn’t be growing either other than minimal bootstrap
> > functionality.
> > > Hence, common config also to be available via dma command.
> >
> > That is more like a new transport. I thought this proposal is focusing on device
> > specific config - let's get a handle on that first?
> > "common config" is a pci transport specific thing.
>
> Sure, device config is the real pain point we are trying to solve first.
>
> Using cvq for those devices who has it seems the most optimal approach.
> If we liberate ourselves from single monolithic config space structure and move to query device capabilities, resources, configuration, at functionality level, life is lot easier.
> What are your thoughts?

Splitting transport and device config is exactly what I'm talking about.
I agree transport should probably be split further -
it only made sense for legacy so we don't need to spend
specification effort on legacy.
splitting device config would require changes to all devices -
I don't see how it's worth the effort.

--
MST

On Wed, Aug 02, 2023 at 11:29:53AM +0000, Parav Pandit wrote: > > > > From: Michael S. Tsirkin <mst@redhat.com> > > Sent: Wednesday, August 2, 2023 4:56 PM > > > > On Wed, Aug 02, 2023 at 09:57:47AM +0000, Parav Pandit wrote: > > > > If you care about admin > > > > virtqueue then device configuration space is not the only thing that > > > > can be "ever growing", common_cfg is another one. > > > The idea is to do minimal bootstrap work from the common config space and > > switch to the queue. > > > So common config shouldnât be growing either other than minimal bootstrap > > functionality. > > > Hence, common config also to be available via dma command. > > > > That is more like a new transport. I thought this proposal is focusing on device > > specific config - let's get a handle on that first? > > "common config" is a pci transport specific thing. > > Sure, device config is the real pain point we are trying to solve first. > > Using cvq for those devices who has it seems the most optimal approach. > If we liberate ourselves from single monolithic config space structure and move to query device capabilities, resources, configuration, at functionality level, life is lot easier. > What are your thoughts? Splitting transport and device config is exactly what I'm talking about. I agree transport should probably be split further - it only made sense for legacy so we don't need to spend specification effort on legacy. splitting device config would require changes to all devices - I don't see how it's worth the effort. -- MST

> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> open.org> On Behalf Of Michael S. Tsirkin
> Sent: Wednesday, August 2, 2023 5:09 PM


> > Sure, device config is the real pain point we are trying to solve first.
> >
> > Using cvq for those devices who has it seems the most optimal approach.
> > If we liberate ourselves from single monolithic config space structure and
> move to query device capabilities, resources, configuration, at functionality
> level, life is lot easier.
> > What are your thoughts?
>
> Splitting transport and device config is exactly what I'm talking about.
> I agree transport should probably be split further - it only made sense for legacy
> so we don't need to spend specification effort on legacy.
> splitting device config would require changes to all devices - I don't see how it's
> worth the effort.

Maybe I was not clear in my idea.
We have canned ourselves as config means _one_ structure.
Due to this thought process, all these transport and things muddy the view.

If one think of functionality-based config, there is no one structure, hence no need to limit ourselves to it.
Taking concrete example,

We have separate commands for,
a. RSS config
b. filters
c. notification coalescing

When you have matching get command, then each functionality grows by their own get command and no need to put in single box of single config structure.

Every device will be able to grow to dynamic need as/if it arise.

> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > open.org> On Behalf Of Michael S. Tsirkin > Sent: Wednesday, August 2, 2023 5:09 PM > > Sure, device config is the real pain point we are trying to solve first. > > > > Using cvq for those devices who has it seems the most optimal approach. > > If we liberate ourselves from single monolithic config space structure and > move to query device capabilities, resources, configuration, at functionality > level, life is lot easier. > > What are your thoughts? > > Splitting transport and device config is exactly what I'm talking about. > I agree transport should probably be split further - it only made sense for legacy > so we don't need to spend specification effort on legacy. > splitting device config would require changes to all devices - I don't see how it's > worth the effort. Maybe I was not clear in my idea. We have canned ourselves as config means _one_ structure. Due to this thought process, all these transport and things muddy the view. If one think of functionality-based config, there is no one structure, hence no need to limit ourselves to it. Taking concrete example, We have separate commands for, a. RSS config b. filters c. notification coalescing When you have matching get command, then each functionality grows by their own get command and no need to put in single box of single config structure. Every device will be able to grow to dynamic need as/if it arise.

On Wed, Aug 02, 2023 at 11:44:46AM +0000, Parav Pandit wrote:
>
>
> > From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > open.org> On Behalf Of Michael S. Tsirkin
> > Sent: Wednesday, August 2, 2023 5:09 PM
>
>
> > > Sure, device config is the real pain point we are trying to solve first.
> > >
> > > Using cvq for those devices who has it seems the most optimal approach.
> > > If we liberate ourselves from single monolithic config space structure and
> > move to query device capabilities, resources, configuration, at functionality
> > level, life is lot easier.
> > > What are your thoughts?
> >
> > Splitting transport and device config is exactly what I'm talking about.
> > I agree transport should probably be split further - it only made sense for legacy
> > so we don't need to spend specification effort on legacy.
> > splitting device config would require changes to all devices - I don't see how it's
> > worth the effort.
>
> Maybe I was not clear in my idea.
> We have canned ourselves as config means _one_ structure.
> Due to this thought process, all these transport and things muddy the view.
>
> If one think of functionality-based config, there is no one structure, hence no need to limit ourselves to it.
> Taking concrete example,
>
> We have separate commands for,
> a. RSS config
> b. filters
> c. notification coalescing
>
> When you have matching get command, then each functionality grows by their own get command and no need to put in single box of single config structure.
>
> Every device will be able to grow to dynamic need as/if it arise.
>
>
>
>

There's one thing to say about putting everything in one place,
and that is that one can then find everything in one place.
For example, at the moment we actually do have a problem with cvq,
and the problem is that it sets internal device state
that is not observable (unlike original pre-1.0 config
space which was writeable).


By the way there's a pci express ECN for relaxed ordering or something
like this. I am yet to look at it, I wonder whether it can be
used to avoid the issues we have with MMIO in a way
that is easier to use than DMA.


--
MST

On Wed, Aug 02, 2023 at 11:44:46AM +0000, Parav Pandit wrote: > > > > From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > > open.org> On Behalf Of Michael S. Tsirkin > > Sent: Wednesday, August 2, 2023 5:09 PM > > > > > Sure, device config is the real pain point we are trying to solve first. > > > > > > Using cvq for those devices who has it seems the most optimal approach. > > > If we liberate ourselves from single monolithic config space structure and > > move to query device capabilities, resources, configuration, at functionality > > level, life is lot easier. > > > What are your thoughts? > > > > Splitting transport and device config is exactly what I'm talking about. > > I agree transport should probably be split further - it only made sense for legacy > > so we don't need to spend specification effort on legacy. > > splitting device config would require changes to all devices - I don't see how it's > > worth the effort. > > Maybe I was not clear in my idea. > We have canned ourselves as config means _one_ structure. > Due to this thought process, all these transport and things muddy the view. > > If one think of functionality-based config, there is no one structure, hence no need to limit ourselves to it. > Taking concrete example, > > We have separate commands for, > a. RSS config > b. filters > c. notification coalescing > > When you have matching get command, then each functionality grows by their own get command and no need to put in single box of single config structure. > > Every device will be able to grow to dynamic need as/if it arise. > > > > There's one thing to say about putting everything in one place, and that is that one can then find everything in one place. For example, at the moment we actually do have a problem with cvq, and the problem is that it sets internal device state that is not observable (unlike original pre-1.0 config space which was writeable). By the way there's a pci express ECN for relaxed ordering or something like this. I am yet to look at it, I wonder whether it can be used to avoid the issues we have with MMIO in a way that is easier to use than DMA. -- MST

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, August 2, 2023 5:21 PM

>
> There's one thing to say about putting everything in one place, and that is that
> one can then find everything in one place.

When a device grows incrementally and each functionality also,
grouping it per functionality gives better pace to keep things self-contained.

Think how messy a set command would be if all 3 functionality in one set command.

> For example, at the moment we actually do have a problem with cvq, and the
> problem is that it sets internal device state that is not observable (unlike original
> pre-1.0 config space which was writeable).
>
Which part is not observable?

>
> By the way there's a pci express ECN for relaxed ordering or something like this.
> I am yet to look at it, I wonder whether it can be used to avoid the issues we
> have with MMIO in a way that is easier to use than DMA.

Waiting to deliver new features on new PCI standard will drastically slow down virtio.
I don't think its good idea, but I would like to know about such ECN, if there is one.

> From: Michael S. Tsirkin <mst@redhat.com> > Sent: Wednesday, August 2, 2023 5:21 PM > > There's one thing to say about putting everything in one place, and that is that > one can then find everything in one place. When a device grows incrementally and each functionality also, grouping it per functionality gives better pace to keep things self-contained. Think how messy a set command would be if all 3 functionality in one set command. > For example, at the moment we actually do have a problem with cvq, and the > problem is that it sets internal device state that is not observable (unlike original > pre-1.0 config space which was writeable). > Which part is not observable? > > By the way there's a pci express ECN for relaxed ordering or something like this. > I am yet to look at it, I wonder whether it can be used to avoid the issues we > have with MMIO in a way that is easier to use than DMA. Waiting to deliver new features on new PCI standard will drastically slow down virtio. I don't think its good idea, but I would like to know about such ECN, if there is one.

On Wed, Aug 02, 2023 at 11:56:11AM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, August 2, 2023 5:21 PM
>
> >
> > There's one thing to say about putting everything in one place, and that is that
> > one can then find everything in one place.
>
> When a device grows incrementally and each functionality also,
> grouping it per functionality gives better pace to keep things self-contained.
>
> Think how messy a set command would be if all 3 functionality in one set command.
>
> > For example, at the moment we actually do have a problem with cvq, and the
> > problem is that it sets internal device state that is not observable (unlike original
> > pre-1.0 config space which was writeable).
> >
> Which part is not observable?

e.g. rss config.

> >
> > By the way there's a pci express ECN for relaxed ordering or something like this.
> > I am yet to look at it, I wonder whether it can be used to avoid the issues we
> > have with MMIO in a way that is easier to use than DMA.
>
> Waiting to deliver new features on new PCI standard will drastically slow down virtio.

features? surely not. but when we are building a new transport then
we are building for the future, it takes years for a new
transport to be deployed.

> I don't think its good idea, but I would like to know about such ECN, if there is one.

I am talking about unordered IO (UIO) ECN.

On Wed, Aug 02, 2023 at 11:56:11AM +0000, Parav Pandit wrote: > > > From: Michael S. Tsirkin <mst@redhat.com> > > Sent: Wednesday, August 2, 2023 5:21 PM > > > > > There's one thing to say about putting everything in one place, and that is that > > one can then find everything in one place. > > When a device grows incrementally and each functionality also, > grouping it per functionality gives better pace to keep things self-contained. > > Think how messy a set command would be if all 3 functionality in one set command. > > > For example, at the moment we actually do have a problem with cvq, and the > > problem is that it sets internal device state that is not observable (unlike original > > pre-1.0 config space which was writeable). > > > Which part is not observable? e.g. rss config. > > > > By the way there's a pci express ECN for relaxed ordering or something like this. > > I am yet to look at it, I wonder whether it can be used to avoid the issues we > > have with MMIO in a way that is easier to use than DMA. > > Waiting to deliver new features on new PCI standard will drastically slow down virtio. features? surely not. but when we are building a new transport then we are building for the future, it takes years for a new transport to be deployed. > I don't think its good idea, but I would like to know about such ECN, if there is one. I am talking about unordered IO (UIO) ECN.

On Wed, Aug 2, 2023 at 5:57?PM Parav Pandit <parav@nvidia.com> wrote:
>
>
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Wednesday, August 2, 2023 3:02 PM
> >
> > On Wed, Aug 2, 2023 at 5:07?PM Parav Pandit <parav@nvidia.com> wrote:
> > >
> > >
> > > > From: Jason Wang <jasowang@redhat.com>
> > > > Sent: Wednesday, August 2, 2023 2:23 PM
> > > >
> > > > On Tue, Aug 1, 2023 at 3:09?PM Parav Pandit <parav@nvidia.com> wrote:
> > > > >
> > > > > One line proposal:
> > > > > Let's use new admin command and admin q for all device types to
> > > > > query
> > > > device config space for new fields. (always).
> > > >
> > > > Before we mandate anything to admin command, we need to first invent
> > > > an admin command over MMIO interface otherwise it would always be an
> > > > issue for the nesting.
> > > >
> > > Nesting can be independent requirement in itself.'
> >
> > I don't understand here. If you tie new fields to the DMA interface, it basically
> > means nesting won't get any new features unless:
> >
> > 1) it's a PCI VF
> > 2) SR-IOV emulation is done
> > 3) admin virtqueue emulation is done
> >
> > If you want differ nesting devices from others, it would be a nightmare to
> > maintain.
> >
> New fields for sure are tied to the DMA interface.

This is different from what you've said above

"Let's use the new admin command and admin q for all device types to ..."

I'm simply replying to your proposal to tie new fields to the admin
command(queue).

> And nesting can also get just like how a VQ works in nested mode.

We should consider reuse the existing one like cvq or inventing
lightwight and self contained methods. Admin virtqueue doesn't fit,
admin command may but we need a MMIO interface for admin command.

>
> > > >
> > > > 1) device configuration space is transport independent, some
> > > > transport already use DMA to access the device configuration space
> > > You can say ccw instead of "some". :)
> >
> > Kind of but the transport vq proposal goes in the same way.
> >
> We debated many times that the wording transport vq is wrong as it is _not_ going to transport driver notifications.
> Anyway, there is nothing to discuss here. So focusing on main items below.
>
> > >
> > > > 2) device configuration space is not read only, we've already had
> > > > several examples of using it as write
> > > >
> > > It is even worse to have writable.
> >
> > Well, what I meant is that, it's not necessarily read only and not necessarily a
> > register interface.
> >
> I took PCI as being most common interface and took net, blk as devices who are experiencing high growth on features and device specific config space.
>
> This isn’t really a normative part of the spec. The key takeway to have it, for common things it is read only and its register.

As mentioned in the past, when developing spec, we should look for
what it can be.

My point is to stick to the device configuration space but invent DMA
interface to access them, then we are all fine.

>
> > >
> > > > > It is growing rapidly.
> > > > > Some devices may be even multi-functionality device in coming
> > > > > future such as
> > > > net + clock + rdma device.
> > > > > For a PCI transport implementing such ever-growing capabilities,
> > > > configuration is burdensome as plain registers.
> > > >
> > > > We've already fixed size VIRTIO_PCI_CAP_PCI_CFG. What's wrong with that?
> > > >
> > > The wrong part is: it is still and indirect and slow, sub-optimal register
> > interface.
> >
> > Do we really care about the performance here?
> When it comes to bulk data transfer, in range of few hundred bytes and looking for 5+ year period, than yes, reading using indirect register is slow.

What kind of configuration requires a few hundred bytes? We should not
duplicate the work of provisioning into device configuration space.

> And after all it is still indirect register which is does not have time variability part.
> So it is not fulfilling the requirement at all.
>
> > And if it is one of your major
> > concerns, it's better to explain it along with the "ever-growing" concern.
> >
> I thought it is clear that it is still a register, even slower than current one and still have same issues without it.
>
> > > VQ is decoupled from transport already.
> > > So, there is no flexibility broken.
> > > And yet you suggested transport dependent VIRTIO_PCI_CAP_PCI_CFG above
> > > making it further wrong. :)
> > >
> >
> > The context here is that you want to mandate any new fields to be DMA.
> > DMA is obviously transport specific. There are transports that don't use DMA at
> > all (e.g the shared memory).
> >
>
> VQ is surprisingly does the DMA without being transport specific.
> How net device mandate tx pkts via vq, how a console device mandate receive and transmit queue,
> How a crypto device mandate a control vq,
>
> What is proposed here is no different...
>
> Shared memory is not for bulk data transfer in virtio spec.
> We don’t see "shared memory" as transport in " Virtio Transport Options" section.

You can see some examples in the kernel drivers. The DMA mandating
excludes any shared memory proposal in the future.

>
> > > > > 3. A device must be able to choose, starting from which field
> > > > > driver must query such configuration via DMA interface. This field
> > > > > offset must be
> > > > greater than currently defined configuration field.
> >
> > [...]
> >
> > > > >
> > > > > d. Any other option?
> > > >
> > > > Transport virtqueue on top of admin virtqueue will address this seamlessly.
> > > >
> > > :)
> > >
> > > Don’t see why one would create few more objects on top of aq when aq or
> > cvq itself can fulfil the need.
> > > Can you please elaborate?
> >
> > If cvq can work, there's no need for any other methods.
> Cvq is not present for all the device, at the same time all devices are not experiencing high growth of config space either.

Adding cvq is much easier than inventing(duplicating) the work of a transport.

Thanks

> Hence the discussion here.
> Michael raised the concern to use some generic scheme for all the device previously, hence the aq.
>
> I think the practical approach is, whichever device has cvq, they should be able to use cvq for config+caps query+set.
> And rest can burn AQ to transport the same command.
>
> > If you care about admin
> > virtqueue then device configuration space is not the only thing that can be
> > "ever growing", common_cfg is another one.
> The idea is to do minimal bootstrap work from the common config space and switch to the queue.
> So common config shouldn’t be growing either other than minimal bootstrap functionality.
> Hence, common config also to be available via dma command.
>
> > Allowing them to be accessed via a
> > virtqueue (thus DMA in PCI transport) is basically the idea of the transport
> > virtqueue. Ling shan is rebasing transport virtqueue proposals on top of admin
> > commands, we can see then.
>
> Current proposal doesnt need yet another new transport vq with/without nesting.
> We just need to make choice as _always_admin command or efficiently use cvq when available.

On Wed, Aug 2, 2023 at 5:57âPM Parav Pandit <parav@nvidia.com> wrote: > > > > > From: Jason Wang <jasowang@redhat.com> > > Sent: Wednesday, August 2, 2023 3:02 PM > > > > On Wed, Aug 2, 2023 at 5:07âPM Parav Pandit <parav@nvidia.com> wrote: > > > > > > > > > > From: Jason Wang <jasowang@redhat.com> > > > > Sent: Wednesday, August 2, 2023 2:23 PM > > > > > > > > On Tue, Aug 1, 2023 at 3:09âPM Parav Pandit <parav@nvidia.com> wrote: > > > > > > > > > > One line proposal: > > > > > Let's use new admin command and admin q for all device types to > > > > > query > > > > device config space for new fields. (always). > > > > > > > > Before we mandate anything to admin command, we need to first invent > > > > an admin command over MMIO interface otherwise it would always be an > > > > issue for the nesting. > > > > > > > Nesting can be independent requirement in itself.' > > > > I don't understand here. If you tie new fields to the DMA interface, it basically > > means nesting won't get any new features unless: > > > > 1) it's a PCI VF > > 2) SR-IOV emulation is done > > 3) admin virtqueue emulation is done > > > > If you want differ nesting devices from others, it would be a nightmare to > > maintain. > > > New fields for sure are tied to the DMA interface. This is different from what you've said above "Let's use the new admin command and admin q for all device types to ..." I'm simply replying to your proposal to tie new fields to the admin command(queue). > And nesting can also get just like how a VQ works in nested mode. We should consider reuse the existing one like cvq or inventing lightwight and self contained methods. Admin virtqueue doesn't fit, admin command may but we need a MMIO interface for admin command. > > > > > > > > > 1) device configuration space is transport independent, some > > > > transport already use DMA to access the device configuration space > > > You can say ccw instead of "some". :) > > > > Kind of but the transport vq proposal goes in the same way. > > > We debated many times that the wording transport vq is wrong as it is _not_ going to transport driver notifications. > Anyway, there is nothing to discuss here. So focusing on main items below. > > > > > > > > 2) device configuration space is not read only, we've already had > > > > several examples of using it as write > > > > > > > It is even worse to have writable. > > > > Well, what I meant is that, it's not necessarily read only and not necessarily a > > register interface. > > > I took PCI as being most common interface and took net, blk as devices who are experiencing high growth on features and device specific config space. > > This isnât really a normative part of the spec. The key takeway to have it, for common things it is read only and its register. As mentioned in the past, when developing spec, we should look for what it can be. My point is to stick to the device configuration space but invent DMA interface to access them, then we are all fine. > > > > > > > > > It is growing rapidly. > > > > > Some devices may be even multi-functionality device in coming > > > > > future such as > > > > net + clock + rdma device. > > > > > For a PCI transport implementing such ever-growing capabilities, > > > > configuration is burdensome as plain registers. > > > > > > > > We've already fixed size VIRTIO_PCI_CAP_PCI_CFG. What's wrong with that? > > > > > > > The wrong part is: it is still and indirect and slow, sub-optimal register > > interface. > > > > Do we really care about the performance here? > When it comes to bulk data transfer, in range of few hundred bytes and looking for 5+ year period, than yes, reading using indirect register is slow. What kind of configuration requires a few hundred bytes? We should not duplicate the work of provisioning into device configuration space. > And after all it is still indirect register which is does not have time variability part. > So it is not fulfilling the requirement at all. > > > And if it is one of your major > > concerns, it's better to explain it along with the "ever-growing" concern. > > > I thought it is clear that it is still a register, even slower than current one and still have same issues without it. > > > > VQ is decoupled from transport already. > > > So, there is no flexibility broken. > > > And yet you suggested transport dependent VIRTIO_PCI_CAP_PCI_CFG above > > > making it further wrong. :) > > > > > > > The context here is that you want to mandate any new fields to be DMA. > > DMA is obviously transport specific. There are transports that don't use DMA at > > all (e.g the shared memory). > > > > VQ is surprisingly does the DMA without being transport specific. > How net device mandate tx pkts via vq, how a console device mandate receive and transmit queue, > How a crypto device mandate a control vq, > > What is proposed here is no different... > > Shared memory is not for bulk data transfer in virtio spec. > We donât see "shared memory" as transport in " Virtio Transport Options" section. You can see some examples in the kernel drivers. The DMA mandating excludes any shared memory proposal in the future. > > > > > > 3. A device must be able to choose, starting from which field > > > > > driver must query such configuration via DMA interface. This field > > > > > offset must be > > > > greater than currently defined configuration field. > > > > [...] > > > > > > > > > > > > d. Any other option? > > > > > > > > Transport virtqueue on top of admin virtqueue will address this seamlessly. > > > > > > > :) > > > > > > Donât see why one would create few more objects on top of aq when aq or > > cvq itself can fulfil the need. > > > Can you please elaborate? > > > > If cvq can work, there's no need for any other methods. > Cvq is not present for all the device, at the same time all devices are not experiencing high growth of config space either. Adding cvq is much easier than inventing(duplicating) the work of a transport. Thanks > Hence the discussion here. > Michael raised the concern to use some generic scheme for all the device previously, hence the aq. > > I think the practical approach is, whichever device has cvq, they should be able to use cvq for config+caps query+set. > And rest can burn AQ to transport the same command. > > > If you care about admin > > virtqueue then device configuration space is not the only thing that can be > > "ever growing", common_cfg is another one. > The idea is to do minimal bootstrap work from the common config space and switch to the queue. > So common config shouldnât be growing either other than minimal bootstrap functionality. > Hence, common config also to be available via dma command. > > > Allowing them to be accessed via a > > virtqueue (thus DMA in PCI transport) is basically the idea of the transport > > virtqueue. Ling shan is rebasing transport virtqueue proposals on top of admin > > commands, we can see then. > > Current proposal doesnt need yet another new transport vq with/without nesting. > We just need to make choice as _always_admin command or efficiently use cvq when available.