On Wed, 2 Aug 2023 15:01:21 +0900, Yui Washizu <
yui.washidu@gmail.com> wrote:
>
> On 2023/07/27 17:28, Jason Wang wrote:
> > On Thu, Jul 27, 2023 at 4:20?PM Xuan Zhuo <
xuanzhuo@linux.alibaba.com> wrote:
> >> On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <
jasowang@redhat.com> wrote:
> >>> On Thu, Jul 27, 2023 at 2:23?PM Xuan Zhuo <
xuanzhuo@linux.alibaba.com> wrote:
> >>>> On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <
lingshan.zhu@intel.com> wrote:
> >>>>>
> >>>>> On 7/27/2023 2:09 PM, Xuan Zhuo wrote:
> >>>>>> On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <
lingshan.zhu@intel.com> wrote:
> >>>>>>> On 7/27/2023 10:30 AM, Xuan Zhuo wrote:
> >>>>>>>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <
lingshan.zhu@intel.com> wrote:
> >>>>>>>>> On 6/30/2023 7:35 PM, Parav Pandit wrote:
> >>>>>>>>>>> From:
virtio-comment@lists.oasis-open.org <
virtio-comment@lists.oasis-
> >>>>>>>>>>> open.org> On Behalf Of Zhu, Lingshan
> >>>>>>>>>>> Sent: Friday, June 30, 2023 6:33 AM
> >>>>>>>>>>>
> >>>>>>>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the
> >>>>>>>>>>> backend?
> >>>>>>>>>> Yes, why not.
> >>>>>>>>>>
> >>>>>>>>>>>>>> The key point is who want to create a new device.
> >>>>>>>>>>>>> DPU can come with a certain number of pre-created ADIs, just make
> >>>>>>>>>>>>> sure the orchestration SW is aware of their device IDs.
> >>>>>>>>>>>>>
> >>>>>>>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> >>>>>>>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> >>>>>>>>>> Xuan partly explained it.
> >>>>>>>>>>
> >>>>>>>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I
> >>>>>>>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
> >>>>>>>>>>>>> wants to create and hot plug in a device to a guest?
> >>>>>>>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
> >>>>>>>>>>>>> machines?
> >>>>>>>>>>>> In your scenario, the supervisor is on the os, which controls the DPU
> >>>>>>>>>>>> to create new devices.
> >>>>>>>>>>>>
> >>>>>>>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the
> >>>>>>>>>>>> entire host is for users. Of course, there are situations where the
> >>>>>>>>>>>> vendor manager are in the HOST. But for bare metal machines, the host
> >>>>>>>>>>>> belongs to the customer, the vendor manager is only in the DPU.
> >>>>>>>>>>>>
> >>>>>>>>>>>> So when the customers buy a new nic for the host, the vendor manager
> >>>>>>>>>>>> will plug a device to the host from the DPU.
> >>>>>>>>>>> I understand once a customer orders a new NIC, you wants to present the NIC
> >>>>>>>>>>> to the host.
> >>>>>>>>>>> However you only owns the DPU and the customer owns the host, that means
> >>>>>>>>>>> this creation and hot plug must be transparent to the host and there may not be
> >>>>>>>>>>> a host driver help handling an interrupt/probe.
> >>>>>>>>>>>
> >>>>>>>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
> >>>>>>>>>>
> >>>>>>>>>>> However this is not PCI which has a tree/switch and can enumerate devices to
> >>>>>>>>>>> the host by spanning the device across the PCI hierarchy.
> >>>>>>>>>>>
> >>>>>>>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
> >>>>>>>>>>
> >>>>>>>>>>> To address an ADI, there is only a device_id.
> >>>>>>>>>>>
> >>>>>>>>>> SIOV device must have a unique identifier at PCI bus level for sure.
> >>>>>>>>>> I cannot speak more about it in this forum due to other logistics issue.
> >>>>>>>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
> >>>>>>>>>>
> >>>>>>>>>>> So, do you mind share how your DPU offload the device model? What kind of
> >>>>>>>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
> >>>>>>>>>>> its own?
> >>>>>>>>>>>
> >>>>>>>>>> It is a virtio nic, blk and other virtio devices for us.
> >>>>>>>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> >>>>>>>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
> >>>>>>>>>>
> >>>>>>>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> >>>>>>>>>> You might find a Linux reference model useful at [1].
> >>>>>>>>>> A usage model already exists in one OS and in use for non virtio devices.
> >>>>>>>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
> >>>>>>>>>>
> >>>>>>>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
> >>>>>>>>>>
> >>>>>>>>>> [1]
https://github.com/Mellanox/scalablefunctions/wiki> >>>>>>>>> well I see SF facing the similar challenge, I can add a command for the
> >>>>>>>>> driver to query all existing SIOV ADIs of a device,
> >>>>>>>>> and the device return ADIs id and status. Looks good? and work for you
> >>>>>>>>> @Xuan?
> >>>>>>>> Could I have your plan for this?
> >>>>>>>>
> >>>>>>>> If you do not mind, I'd like to add a command to query VF's info. Such
> >>>>>>>> as mac, ip, etc.
> >>>>>>> I think the query commands for SIOV is a little more complex, e.g.,
> >>>>>>> need to report device type and its scale(e.g., features, mq).
> >>>>>>> There can be thousands of SIOV ADIs and we don't want output flood.
> >>>>>>>
> >>>>>>> We have discussed implementation a config interrupt to report new
> >>>>>>> created / deleted
> >>>>>>> ADIs on the DPU side, therefore there must be a cap contains related
> >>>>>>> information,
> >>>>>>> my rough approach of the process is:
> >>>>>>> 1) a cap contains the total number of existing ADIs and the max dev id
> >>>>>>> 2) driver queries detailed information of a certain ADI or a bunch of
> >>>>>>> ADIs in a [dev_id....dev_id2] range.
> >>>>>> Yes, Admin Queue can obtain the info of the specific one or more devices.
> >>>>>>
> >>>>>>> I am not sure whether a NIC stores its IP
> >>>>>> IP is the other topic. I want the Admin Queue manage the switch.
> >>>>>> So the switch know about the IP of every device, and the
> >>>>>> Admin Queue will has the ability to config the IP of the device inside the
> >>>>>> switch.
> >>>>> DPU onboard switch? OVS? Does it beyond virtio spec?
> >>>> YES.
> >>> Adding Washizu.
> >>>
> >>> We can have a switch/dpa defined in the networking device for sure.
> >> Yes, I think we should introduce that for the sr-iov. Or for other.
> > This should be a general one as a switch should be transport independent.
> >
> >> I would like to know who is doing this?
> > Washizu, could you confirm if you want to do this or not?
>
>
> Does this mean adding a switch definition to the virtio spec?
>
>
> If so, it will be necessary for the implementation of my plan,
>
> but it may take time (probably several months?) to get started,
>
> as I'm currently working on another task (virtio-net SR-IOV feature in
> qemu).
>
> Anyone is welcome to work on adding the switch definition in the meantime,
>
> it's completely fine with me.
>
> I think I'll work on that if no one has finished the work.
OK, I got.
Because we have a need in this area, I will push the work in this area.
Thanks.
>
>
>
> >
> >> Another question, @Jason are you referring to a new device type or a
> >> new virtio-net feature.
> > Extending virtio-net should be fine, did you see any issues for this?
> >
> >>>> For SIOV, I think this is MUST.
> >>> A learning bridge would be fine as a starter. It's better not to
> >>> couple new scalable capability with any device specific features.
> >>>
> >>>> Maybe you have one simple implementation.
> >>>> But you have to solve the IP steering. So admin queue should has the ability
> >>>> to config the IP steering.
> >>> I think not. Those L2/LN tables/filters are networking specific.
> >> Let us assume that there is a switch/bridge firstly.
> >> The VFs may be passed to different VMs.
> >>
> >> I also think this is the networking specific. But I want to config
> >> the ip for every vf from the pf.
> > What do you mean by ip here (e.g who is the user for this ip?)
> >
> >> Because the user of the vf may be unreliable.
> >> We need a manager to config the ip for every vf.
> > Did you mean you're using a tunnel or not?
> >
> >>
> >>> Control virtqueue is better than admin virtqueue here.
> >> by cq?
> >>
> >> What case?
> > We've already used control virtqueue for steering.
> >
> > Thanks
> >
> >> Thanks.
> >>
> >>> Thanks
> >>>
> >>>> Thanks.
> >>>>
> >>>>>>
> >>>>>> Thanks.
> >>>>>>
> >>>>>>
> >>>>>>> Thanks
> >>>>>>>> Thanks.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Thanks
> >>>>>>>>>
> >>>>>>>> This publicly archived list offers a means to provide input to the
> >>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>>>>>
> >>>>>>>> In order to verify user consent to the Feedback License terms and
> >>>>>>>> to minimize spam in the list archive, subscription is required
> >>>>>>>> before posting.
> >>>>>>>>
> >>>>>>>> Subscribe:
virtio-comment-subscribe@lists.oasis-open.org> >>>>>>>> Unsubscribe:
virtio-comment-unsubscribe@lists.oasis-open.org> >>>>>>>> List help:
virtio-comment-help@lists.oasis-open.org> >>>>>>>> List archive:
https://lists.oasis-open.org/archives/virtio-comment/> >>>>>>>> Feedback License:
https://www.oasis-open.org/who/ipr/feedback_license.pdf> >>>>>>>> List Guidelines:
https://www.oasis-open.org/policies-guidelines/mailing-lists> >>>>>>>> Committee:
https://www.oasis-open.org/committees/virtio/> >>>>>>>> Join OASIS:
https://www.oasis-open.org/join/> >>>>>>>>
> >>>>>
> >>>>> This publicly archived list offers a means to provide input to the
> >>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>>
> >>>>> In order to verify user consent to the Feedback License terms and
> >>>>> to minimize spam in the list archive, subscription is required
> >>>>> before posting.
> >>>>>
> >>>>> Subscribe:
virtio-comment-subscribe@lists.oasis-open.org> >>>>> Unsubscribe:
virtio-comment-unsubscribe@lists.oasis-open.org> >>>>> List help:
virtio-comment-help@lists.oasis-open.org> >>>>> List archive:
https://lists.oasis-open.org/archives/virtio-comment/> >>>>> Feedback License:
https://www.oasis-open.org/who/ipr/feedback_license.pdf> >>>>> List Guidelines:
https://www.oasis-open.org/policies-guidelines/mailing-lists> >>>>> Committee:
https://www.oasis-open.org/committees/virtio/> >>>>> Join OASIS:
https://www.oasis-open.org/join/> >>>>>
> >>>> This publicly archived list offers a means to provide input to the
> >>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>
> >>>> In order to verify user consent to the Feedback License terms and
> >>>> to minimize spam in the list archive, subscription is required
> >>>> before posting.
> >>>>
> >>>> Subscribe:
virtio-comment-subscribe@lists.oasis-open.org> >>>> Unsubscribe:
virtio-comment-unsubscribe@lists.oasis-open.org> >>>> List help:
virtio-comment-help@lists.oasis-open.org> >>>> List archive:
https://lists.oasis-open.org/archives/virtio-comment/> >>>> Feedback License:
https://www.oasis-open.org/who/ipr/feedback_license.pdf> >>>> List Guidelines:
https://www.oasis-open.org/policies-guidelines/mailing-lists> >>>> Committee:
https://www.oasis-open.org/committees/virtio/> >>>> Join OASIS:
https://www.oasis-open.org/join/> >>>>