virtio-comment

 View Only
  • 1.  [RFC PATCH 0/3] virtio-pci: management device

    Posted 08-20-2021 07:59
    Hi all:

    As promised, I post this series to demonstrate the possibility of
    using a transport specific way for implementing the management
    device. The main motivation is to achieve hyper scalability and
    flexibility. This is an alternative way of transporting the
    managed(virtual) device via the admin virtqueue (most of the admin
    virtqueue commands proposed there were just a duplication of the
    PCI capability).

    The idea is simple, introduce a device selector and try to reuse as
    much existing virtio-pci capabilities as possible.

    In order to make this work, two independent capabilities are
    introduced. A new capability for storing MSI-X message is introduced
    for per-device MSI-X configuration for the managed device. PASID
    capability is also introduced for DMA isolation among the managed
    devices.

    To be simple, I don't propose device MMU interface in this series. It
    could be added in the future since it's not a must if PASID is
    supported.

    Sending this for early feedback and comments.

    Thanks


    Jason Wang (3):
    virtio-pci: introduce device specific MSI-X messages store
    virtio-pci: PASID support
    virtio-pci: introduce the management device capability

    content.tex | 202 ++++++++++++++++++++++++++++++++++++++++++++++++++++
    1 file changed, 202 insertions(+)

    --
    2.24.3 (Apple Git-128)




  • 2.  [RFC PATCH 1/3] virtio-pci: introduce device specific MSI-X messages store

    Posted 08-20-2021 07:59
    This patch introduce a new capability that allows the MSI-X messages
    to be stored and setup through a dedicated capability. This will be
    useful for the cases when:

    - the device is required to have more than 2048 MSI-X entries, one
    example is a device have more than 2048 queues
    - the managed device without a transport specific way to configure the
    MSI-X entries

    Signed-off-by: Jason Wang <jasowang@redhat.com>
    ---
    content.tex | 73 +++++++++++++++++++++++++++++++++++++++++++++++++++++
    1 file changed, 73 insertions(+)

    diff --git a/content.tex b/content.tex
    index 620c0e2..50c0523 100644
    --- a/content.tex
    +++ b/content.tex
    @@ -705,6 +705,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
    #define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8
    /* Vendor-specific data */
    #define VIRTIO_PCI_CAP_VENDOR_CFG 9
    +/* MSI-X configuration */
    +#define VIRTIO_CPI_CAP_MSIX_CFG 10
    \end{lstlisting}

    Any other value is reserved for future use.
    @@ -1261,6 +1263,77 @@ \subsubsection{PCI configuration access capability}\label{sec:Virtio Transport O
    specified by some other Virtio Structure PCI Capability
    of type other than \field{VIRTIO_PCI_CAP_PCI_CFG}.

    +\subsubsection{MSI-X configuration capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / MSI-X configuration capability}
    +
    +The VIRTIO_PCI_CAP_MSIX_CFG capability allows the MSI-X vectors to be
    +stored in a virtio-pci device specific store instead of the PCI-E
    +MSI-X table.
    +
    +The capability is immediately followed by an additional field like so:
    +
    +\begin{lstlisting}
    +struct virtio_pci_msix_cap {
    + struct virtio_pci_cap cap;
    + u8 enable; /* read-write */
    + le16 num_vectors; /* read-only for driver */
    + le16 vector_select; /* read-write */
    + le32 msix_address_high; /* read-write */
    + le32 msix_address_low; /* read-write */
    + le32 msix_data; /* read-write */
    + u8 mask; /* read-write */
    +};
    +\end{lstlisting}
    +
    +\begin{description}
    +\item[\field{enable}]
    +The driver uses this to enable (writing 1) or disable (writing 0) the
    +virtio-pci specific MSI-X messages storing. When enabled, the device
    +will use the MSI-X messages that is setup through this
    +capability instead of the PCI-E MSI-X table. When disabled, the
    +device will use the MSI-X messages that is setup through the PCI-E
    +MSI-X table.
    +
    +\item[\field{num_vectors}]
    +The device specifies the maximum number of MSI-X entries supported here.
    +
    +\item[\field{vector_select}]
    +Vector Select. The driver selects which virtqueues the following
    +fields refer to.
    +
    +\item[\field{msix_address_high}]
    +The driver writes the high address of a specific MSI-X entry here.
    +
    +\item[\field{msix_address_lo}]
    +The driver writes the low address of a specific MSI-X entry here.
    +
    +\item[\field{msix_data}]
    +The driver writes the MSI-X data of a specific MSI-X entry here.
    +
    +\item[\field{mask}]
    +The driver use this to mask (write 1) or unmask (write 0) a specific
    +MSI-X entry indexed via \field{vector_select} here.
    +\end{description}
    +
    +\devicenormative{\paragraph}{MSI-X configuration capability}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / MSI-X configuration capability}
    +
    +The device MUST disable virtio-pci device specific MSI-X messages
    +store and presenting 0 to \field{enable} upon reset.
    +
    +The device MUST mask all the vectors and present 0 to \field{mask}
    +upon reset.
    +
    +The device MUST record the pending MSI-X vectors and generate the
    +MSI-X interrupt upon unmasking.
    +
    +The device MUST ignore the value that is wrote to
    +\field{vector_select} which is equal or greater than
    +\field{num_vectors}.
    +
    +\drivernormative{\paragraph}{MSI-X configuration capability}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / MSI-X configuration capability}
    +
    +The driver MUST mask a specific vector before trying to setup the
    +MSI-X addresses and MSI-X data.
    +
    \subsubsection{Legacy Interfaces: A Note on PCI Device Layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}

    Transitional devices MUST present part of configuration
    --
    2.24.3 (Apple Git-128)




  • 3.  [RFC PATCH 2/3] virtio-pci: PASID support

    Posted 08-20-2021 07:59
    This patch introduce a new virtio-pci capability for assigning PASID
    for a specific virtqueue.

    Signed-off-by: Jason Wang <jasowang@redhat.com>
    ---
    content.tex | 54 +++++++++++++++++++++++++++++++++++++++++++++++++++++
    1 file changed, 54 insertions(+)

    diff --git a/content.tex b/content.tex
    index 50c0523..5266965 100644
    --- a/content.tex
    +++ b/content.tex
    @@ -707,6 +707,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
    #define VIRTIO_PCI_CAP_VENDOR_CFG 9
    /* MSI-X configuration */
    #define VIRTIO_CPI_CAP_MSIX_CFG 10
    +/* PASID configuration */
    +#define VIRTIO_PCI_CAP_PASID 11
    \end{lstlisting}

    Any other value is reserved for future use.
    @@ -1334,6 +1336,54 @@ \subsubsection{MSI-X configuration capability}\label{sec:Virtio Transport Option
    The driver MUST mask a specific vector before trying to setup the
    MSI-X addresses and MSI-X data.

    +\subsubsection{PASID configuration}\label{MSI-X configuration capability}{PASID / Virtio Over PCI Bus / PCI Device Layout / PASID configuration}
    +
    +The VIRTIO_PCI_CAP_RING_PASID capability is for assigning PASID for a
    +specific virtqueue.
    +
    +The capability is immediately followed by an additional field like so:
    +
    +\begin{lstlisting}
    +struct virtio_pci_cfg_cap {
    + struct virtio_pci_cap cap;
    + le16 queue_select; /* read-write */
    + le32 {
    + queue_pasid : 20; /* read-write */
    + reserved : 12; /* read-write */
    + };
    + u8 pasid_enable; /* read-write */
    +};
    +\end{lstlisting}
    +
    +\begin{description}
    +\item[\field{queue_select}]
    + Queue Select. The driver selects which virtqueue the following
    + fields refer to.
    +\item[\field{queue_pasid}]
    + PASID for the virtqueue.
    +\item[\field{reserved}]
    + Reserved for the future use.
    +\item[\field{pasid_enable}]
    + The driver use this to enable (write 1) or disable (write 0)
    + PASID for the virtqueue.
    +\end{description}
    +
    +\devicenormative{\paragraph}{PASID configuration}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / PASID configuration}
    +
    +The device MUST present at least one VIRTIO_PCI_CAP_PASID capability
    +if it presents the PASID Extended Capability.
    +
    +The device MUST NOT present VIRTIO_PCI_CAP_PASID capability if PASID
    +Extended Capability is not presented.
    +
    +The device MUST disable PASID for all virtqueues and presenting 0 to
    +\field{enable} upon reset.
    +
    +\drivernormative{\paragraph}{PASID configuration}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / PASID configuration}
    +
    +The driver MUST configure the PASID for each virtqueue if PASID is
    +enabled via PASID Extended Capability.
    +
    \subsubsection{Legacy Interfaces: A Note on PCI Device Layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}

    Transitional devices MUST present part of configuration
    @@ -1590,6 +1640,10 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti
    corresponding to this vector into \field{queue_msix_vector}. Read
    \field{queue_msix_vector}: on success, previously written value is
    returned; on failure, NO_VECTOR value is returned.
    +
    +\item Optionally, if PASID Extended Capability is present and enabled
    + on the device, select a PASID to use for the TLP of a
    + virtqueue. Write the PASID value into \field{pasid}.
    \end{enumerate}

    \subparagraph{Legacy Interface: A Note on Virtqueue Configuration}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Virtqueue Configuration / Legacy Interface: A Note on Virtqueue Configuration}
    --
    2.24.3 (Apple Git-128)




  • 4.  [RFC PATCH 3/3] virtio-pci: introduce the management device capability

    Posted 08-20-2021 07:59
    This patch introduces the management device capability. The device
    that offers this capability is the management device. The management
    device driver can use this capability to create managed device who
    doesn't have a dedicated transport specific configuration interface.

    The idea is simple, introduce a device_select field then we can reuse
    almost all the existing capabilities (except for the shared memory
    one) to configure the managed device.

    A simple managed device provision interface is also proposed.

    To be simple, I tie this capability with the PASID capability instead
    of introducing device MMU functions.

    Signed-off-by: Jason Wang <jasowang@redhat.com>
    ---
    content.tex | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++
    1 file changed, 75 insertions(+)

    diff --git a/content.tex b/content.tex
    index 5266965..d3ebd76 100644
    --- a/content.tex
    +++ b/content.tex
    @@ -709,6 +709,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
    #define VIRTIO_CPI_CAP_MSIX_CFG 10
    /* PASID configuration */
    #define VIRTIO_PCI_CAP_PASID 11
    +/* Managed device configuration */
    +#define VIRTIO_PCI_CAP_MANAGED_DEV_CFG 12
    \end{lstlisting}

    Any other value is reserved for future use.
    @@ -1384,6 +1386,79 @@ \subsubsection{PASID configuration}\label{MSI-X configuration capability}{PASID
    The driver MUST configure the PASID for each virtqueue if PASID is
    enabled via PASID Extended Capability.

    +\subsubsection{Managed device configuration}\label{Managed device configuration}{PASID / Virtio Over PCI Bus / PCI Device Layout / Managed device configuration}
    +
    +The VIRTIO_PCI_CAP_MANAGED_DEV_CFG capability allows the one device to
    +manage other (usually virtual) virtio devices. The device that offers
    +VIRTIO_PCI_CAP_MANAGED_DEV_CFG is called management device. The
    +device that is managed by the management device is called managed
    +device.
    +
    +The capability is immediately followed by an additional field like so:
    +
    +\begin{lstlisting}
    +struct virtio_pci_managed_dev_cap {
    + struct virtio_pci_cap cap;
    + u32 num_devices; /* read-only for driver */
    + le32 device_select; /* read-write */
    + le32 config_select; /* read-write */
    + u8 config[256]; /* read-write */
    + u8 create; /* read-write */
    + u8 create_ok; /* read-only for driver */
    +};
    +
    +/* create_ok values */
    +#define VIRTIO_MGMT_OK 0
    +#define VIRTIO_MGMT_ID_DUP 1
    +#define VIRTIO_MGMT_INVALID_CFG 2
    +\end{lstlisting}
    +
    +\begin{description}
    +\item[\field{num_devices}]
    + The device specifies the maximum number of managed devices
    + here.
    +\item[\field{device_select}]
    + Device Select. The driver selects which device the following
    + fields and other virtio-pci capabilities refer to.
    +\item[\field{config_select}]
    + The driver uses this to select which 256 bytes chunk of
    + \field{config} shows.
    +\item[field{config}]
    + The metadata that is used to create a managed device. The
    + actual format is device specific.
    +\item[\field{create}]
    + The driver use this to create (write 1) or destroy (write 0) a
    + specific managed device.
    +\item[\field{create_ok}]
    + The device specifies whether the managed device is
    + successfully created or destroyed. 0 means success, others
    + means failure.
    +\end{description}
    +
    +\devicenormative{\paragraph}{Managed device configuration}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Managed device configuration}
    +
    +The management device MUST reserved device index 0 as itself.
    +
    +The management device MUST present 0 to \field{device_select} upon
    +reset.
    +
    +The management device MUST fail the write to \field{create} if the
    +\field{device_select} is zero.
    +
    +The management device MUST fail the creation of a managed device if
    +the filed\{device_select} is duplicated with the id of an existing
    +managed device.
    +
    +The device MUST NOT present VIRTIO_PCI_CAP_MANAGED_DEV_CFG if
    +VIRTIO_PCI_CAP_PASID is not presented.
    +
    +\drivernormative{\paragraph}{Managed device configuration}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Managed device configuration}
    +
    +The management driver MUST create a managed device by allocating a
    +non-zero device id for \field{device_select} and use that for the
    +following steps for this managed device if the creating is succeed
    +(read \field{create_ok} as zero).
    +
    \subsubsection{Legacy Interfaces: A Note on PCI Device Layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}

    Transitional devices MUST present part of configuration
    --
    2.24.3 (Apple Git-128)