virtio-comment

 View Only
  • 1.  [RFC PATCH 0/2] Context init virtio-gpu spec

    Posted 08-26-2021 02:13
    Here's the virtio spec change for context init. The first patch
    clarifies existing features, and second patch adds the new features
    outlined here:

    https://lists.freedesktop.org/archives/dri-devel/2021-August/320913.html

    Gurchetan Singh (2):
    virtio-gpu: clarify spec regarding capability sets
    virtio-gpu: add context init support

    virtio-gpu.tex | 133 ++++++++++++++++++++++++++++++++++++++++++++-----
    1 file changed, 120 insertions(+), 13 deletions(-)

    --
    2.31.0




  • 2.  [RFC PATCH 1/2] virtio-gpu: clarify spec regarding capability sets

    Posted 08-26-2021 02:13
    Capability sets will be used as a proxy for the context type,
    so add more detail regarding their use.

    Signed-off-by: Gurchetan Singh <gurchetansingh@chromium.org>
    ---
    virtio-gpu.tex | 58 +++++++++++++++++++++++++++++++++++++++++++++++++-
    1 file changed, 57 insertions(+), 1 deletion(-)

    diff --git a/virtio-gpu.tex b/virtio-gpu.tex
    index 7bfb8dd..35c129f 100644
    --- a/virtio-gpu.tex
    +++ b/virtio-gpu.tex
    @@ -53,7 +53,7 @@ \subsection{Device configuration layout}\label{sec:Device Types / GPU Device / D
    le32 events_read;
    le32 events_clear;
    le32 num_scanouts;
    - le32 reserved;
    + le32 num_capsets;
    };
    \end{lstlisting}

    @@ -67,6 +67,8 @@ \subsubsection{Device configuration fields}
    \field{events_read}, mimicking write-to-clear behavior.
    \item[\field{num_scanouts}] specifies the maximum number of scanouts
    supported by the device. Minimum value is 1, maximum value is 16.
    +\item[\field{num_capsets}] specifies the maximum number of capability
    + sets supported by the device. The minimum value is zero.
    \end{description}

    \subsubsection{Events}
    @@ -498,6 +500,60 @@ \subsubsection{Device Operation: controlq}\label{sec:Device Types / GPU Device /
    This detaches any backing pages from a resource, to be used in case of
    guest swapping or object destruction.

    +\item[VIRTIO_GPU_CMD_GET_CAPSET_INFO] Gets the information associated with
    + a particular \field{capset_index}, which MUST less than \field{num_capsets}
    + defined in the device configuration. Request data is
    + \field{struct virtio_gpu_get_capset_info}. Response type is
    + VIRTIO_GPU_RESP_OK_CAPSET_INFO.
    +
    + On success, \field{struct virtio_gpu_resp_capset_info} contains the
    + \field{capset_id}, \field{capset_max_version}, \field{capset_max_size}
    + associated with capset at the specified {capset_idex}. field{capset_id} MUST
    + be one of the following (see listing for values):
    +
    + \begin{itemize*}
    + \item \href{https://gitlab.freedesktop.org/virgl/virglrenderer/-/blob/master/src/virgl_hw.h#L526}{VIRTIO_GPU_CAPSET_VIRGL} --
    + the first edition of Virgl (Gallium OpenGL) protocol.
    + \item \href{https://gitlab.freedesktop.org/virgl/virglrenderer/-/blob/master/src/virgl_hw.h#L550}{VIRTIO_GPU_CAPSET_VIRGL2} --
    + the second edition of Virgl (Gallium OpenGL) protocol after the capset fix.
    + \end{itemize*}
    +
    +\begin{lstlisting}
    +struct virtio_gpu_get_capset_info {
    + struct virtio_gpu_ctrl_hdr hdr;
    + le32 capset_index;
    + le32 padding;
    +};
    +
    +#define VIRTIO_GPU_CAPSET_VIRGL 1
    +#define VIRTIO_GPU_CAPSET_VIRGL2 2
    +struct virtio_gpu_resp_capset_info {
    + struct virtio_gpu_ctrl_hdr hdr;
    + le32 capset_id;
    + le32 capset_max_version;
    + le32 capset_max_size;
    + le32 padding;
    +};
    +\end{lstlisting}
    +
    +\item[VIRTIO_GPU_CMD_GET_CAPSET] Gets the capset associated with a
    + particular \field{capset_id} and \field{capset_version}. Request data is
    + \field{struct virtio_gpu_get_capset}. Response type is
    + VIRTIO_GPU_RESP_OK_CAPSET.
    +
    +\begin{lstlisting}
    +struct virtio_gpu_get_capset {
    + struct virtio_gpu_ctrl_hdr hdr;
    + le32 capset_id;
    + le32 capset_version;
    +};
    +
    +struct virtio_gpu_resp_capset {
    + struct virtio_gpu_ctrl_hdr hdr;
    + u8 capset_data[];
    +};
    +\end{lstlisting}
    +
    \item[VIRTIO_GPU_CMD_RESOURCE_ASSIGN_UUID] Creates an exported object from
    a resource. Request data is \field{struct
    virtio_gpu_resource_assign_uuid}. Response type is
    --
    2.31.0




  • 3.  [RFC PATCH 2/2] virtio-gpu: add context init support

    Posted 08-26-2021 02:13
    This brings explicit context initialization and different types
    to virtio-gpu.

    In the past, VIRTIO_GPU_F_VIRGL meant the virglrenderer support.
    With VIRTIO_GPU_F_VIRGL + VIRTIO_GPU_F_CONTEXT_INIT, this means
    generic 3D virtualization defined by the context type. It's
    entirely possible the virglrenderer project isn't available on
    the host in this scenario. The VIRTIO_GPU_F_VIRGL naming
    convention is kept since it's easier to redefine the meaning
    rather than changing header files.

    The context type is associated an particular capset id. Virgl
    has two capsets due a prior bug, but for other cases the 1:1
    mapping between context type and capset id is valid.

    In addition, fencing needs to be fixed to accomodate multiple
    context types. In the past, there was one global timeline
    associated witht the OpenGL rendering. Now, there are multiple
    timelines which can be associated with GL, VK or even display
    contexts.

    Signed-off-by: Gurchetan Singh <gurchetansingh@chromium.org>
    ---
    virtio-gpu.tex | 75 ++++++++++++++++++++++++++++++++++++++++++--------
    1 file changed, 63 insertions(+), 12 deletions(-)

    diff --git a/virtio-gpu.tex b/virtio-gpu.tex
    index 35c129f..3357ce5 100644
    --- a/virtio-gpu.tex
    +++ b/virtio-gpu.tex
    @@ -1,14 +1,10 @@
    \section{GPU Device}\label{sec:Device Types / GPU Device}

    virtio-gpu is a virtio based graphics adapter. It can operate in 2D
    -mode and in 3D (virgl) mode. 3D mode will offload rendering ops to
    +mode and in 3D mode. 3D mode will offload rendering ops to
    the host gpu and therefore requires a gpu with 3D support on the host
    machine.

    -3D mode is not covered (yet) in this specification, even though it is
    -mentioned here and there due to some details of the virtual hardware
    -being designed with 3D mode in mind.
    -
    In 2D mode the virtio-gpu device provides support for ARGB Hardware
    cursors and multiple scanouts (aka heads).

    @@ -39,6 +35,8 @@ \subsection{Feature bits}\label{sec:Device Types / GPU Device / Feature bits}
    to other virtio devices is supported.
    \item[VIRTIO_GPU_F_RESOURCE_BLOB (3)] creating and using size-based blob
    resources is supported.
    +\item[VIRTIO_GPU_F_CONTEXT_INIT (4)] multiple context types and
    + synchronization timelines supported. Requires VIRTIO_GPU_F_VIRGL.
    \end{description}

    \subsection{Device configuration layout}\label{sec:Device Types / GPU Device / Device configuration layout}
    @@ -214,7 +212,7 @@ \subsubsection{Device Operation: Request header}\label{sec:Device Types / GPU De
    VIRTIO_GPU_CMD_RESOURCE_CREATE_BLOB,
    VIRTIO_GPU_CMD_SET_SCANOUT_BLOB,

    - /* 3d commands (OpenGL) */
    + /* 3d commands */
    VIRTIO_GPU_CMD_CTX_CREATE = 0x0200,
    VIRTIO_GPU_CMD_CTX_DESTROY,
    VIRTIO_GPU_CMD_CTX_ATTACH_RESOURCE,
    @@ -249,13 +247,16 @@ \subsubsection{Device Operation: Request header}\label{sec:Device Types / GPU De
    };

    #define VIRTIO_GPU_FLAG_FENCE (1 << 0)
    +#define VIRTIO_GPU_FLAG_INFO_RING_IDX (1 << 1)
    +
    +#define VIRTIO_GPU_INFO_RING_IDX_MASK 0x0000001f;

    struct virtio_gpu_ctrl_hdr {
    le32 type;
    le32 flags;
    le64 fence_id;
    le32 ctx_id;
    - le32 padding;
    + le32 info;
    };
    \end{lstlisting}

    @@ -275,6 +276,22 @@ \subsubsection{Device Operation: Request header}\label{sec:Device Types / GPU De
    \item send the response only after command processing is complete.
    \end{itemize*}
    \item[\field{ctx_id}] Rendering context (used in 3D mode only).
    +\item[\field{info}] Information associated with the command. If
    + VIRTIO_GPU_F_CONTEXT_INIT is supported, then the driver MAY set
    + VIRTIO_GPU_FLAG_INFO_RING_IDX bit in the request \field{flags}. In
    + that case:
    + \begin{itemize*}
    + \item VIRTIO_GPU_FLAG_FENCE MUST also be set
    + \item the lower 8 bits of \field{info} indicates the value of a
    + context-specific ring index. The minimum value is 0 and maximum
    + value is 31.
    + \item \field{fence_id} acts as a sequence number on the synchronization
    + timeline defined by \field{ctx_idx} and the ring index.
    + \item when the command associated with \field{fence_id} is complete, the
    + device MUST send a response for all outstanding commands with a sequence
    + number less than or equal to \field{fence_id} on the same
    + synchronization timeline.
    + \end{itemize*}
    \end{description}

    On success the device will return VIRTIO_GPU_RESP_OK_NODATA in
    @@ -516,6 +533,12 @@ \subsubsection{Device Operation: controlq}\label{sec:Device Types / GPU Device /
    the first edition of Virgl (Gallium OpenGL) protocol.
    \item \href{https://gitlab.freedesktop.org/virgl/virglrenderer/-/blob/master/src/virgl_hw.h#L550}{VIRTIO_GPU_CAPSET_VIRGL2} --
    the second edition of Virgl (Gallium OpenGL) protocol after the capset fix.
    + \item \href{https://android.googlesource.com/device/generic/vulkan-cereal/+/refs/heads/master/protocols/}{VIRTIO_GPU_CAPSET_GFXSTREAM} --
    + gfxtream's (mostly) autogenerated GLES and Vulkan streaming protocols.
    + \item \href{https://gitlab.freedesktop.org/olv/venus-protocol}{VIRTIO_GPU_CAPSET_VENUS} --
    + Mesa's (mostly) autogenerated Vulkan protocol.
    + \item \href{https://chromium.googlesource.com/chromiumos/platform/crosvm/+/refs/heads/main/rutabaga_gfx/src/cross_domain/cross_domain_protocol.rs}{VIRTIO_GPU_CAPSET_CROSS_DOMAIN} --
    + protocol for display virtualization via Wayland proxying.
    \end{itemize*}

    \begin{lstlisting}
    @@ -527,6 +550,9 @@ \subsubsection{Device Operation: controlq}\label{sec:Device Types / GPU Device /

    #define VIRTIO_GPU_CAPSET_VIRGL 1
    #define VIRTIO_GPU_CAPSET_VIRGL2 2
    +#define VIRTIO_GPU_CAPSET_GFXSTREAM 3
    +#define VIRTIO_GPU_CAPSET_VENUS 4
    +#define VIRTIO_GPU_CAPSET_CROSS_DOMAIN 5
    struct virtio_gpu_resp_capset_info {
    struct virtio_gpu_ctrl_hdr hdr;
    le32 capset_id;
    @@ -684,21 +710,46 @@ \subsubsection{Device Operation: controlq (3d)}\label{sec:Device Types / GPU Dev

    \begin{description}

    -\item[VIRTIO_GPU_CMD_CTX_CREATE]
    +\item[VIRTIO_GPU_CMD_CTX_CREATE] creates a context for submitting an opaque
    + command stream. Request data is \field{struct virtio_gpu_ctx_create}.
    + Response type is VIRTIO_GPU_RESP_OK_NODATA.
    +
    +\begin{lstlisting}
    +#define VIRTIO_GPU_CONTEXT_INIT_CAPSET_ID_MASK 0x0000003f;
    +struct virtio_gpu_ctx_create {
    + struct virtio_gpu_ctrl_hdr hdr;
    + le32 nlen;
    + le32 context_init;
    + char debug_name[64];
    +};
    +\end{lstlisting}
    +
    +The implementation MUST create a context for the given \field{ctx_id} in
    +the \field{hdr}. For debugging purposes, a \field{debug_name} and it's
    +length \field{nlen} is provided by the driver. If
    +VIRTIO_GPU_F_CONTEXT_INIT is supported, then lower 8 bits of
    +\field{context_init} MAY contain the \field{capset_id} associated with
    +context. In that case, then the device MUST create a context that can
    +handle the specified command stream.
    +
    +If the lower 8-bits of the \field{context_init} are zero, then the type of
    +the context is determined by the device.
    +
    \item[VIRTIO_GPU_CMD_CTX_DESTROY]
    \item[VIRTIO_GPU_CMD_CTX_ATTACH_RESOURCE]
    \item[VIRTIO_GPU_CMD_CTX_DETACH_RESOURCE]
    - Manage OpenGL contexts.
    + Manage virtio-gpu 3d contexts.

    \item[VIRTIO_GPU_CMD_RESOURCE_CREATE_3D]
    - Create OpenGL resources.
    + Create virtio-gpu 3d resources.

    \item[VIRTIO_GPU_CMD_TRANSFER_TO_HOST_3D]
    \item[VIRTIO_GPU_CMD_TRANSFER_FROM_HOST_3D]
    - Transfer data from and to OpenGL resources.
    + Transfer data from and to virtio-gpu 3d resources.

    \item[VIRTIO_GPU_CMD_SUBMIT_3D]
    - Submit rendering commands (mesa gallium command stream).
    + Submit an opaque command stream. The type of the command stream is
    + determined when creating a context.

    \item[VIRTIO_GPU_CMD_RESOURCE_MAP_BLOB] maps a host-only
    blob resource into an offset in the host visible memory region. Request
    --
    2.31.0




  • 4.  [RFC PATCH v2 2/2] virtio-gpu: add context init support

    Posted 08-30-2021 19:42
    This brings explicit context initialization and different types
    to virtio-gpu.

    In the past, VIRTIO_GPU_F_VIRGL meant the virglrenderer support.
    With VIRTIO_GPU_F_VIRGL + VIRTIO_GPU_F_CONTEXT_INIT, this means
    generic 3D virtualization defined by the context type. It's
    entirely possible the virglrenderer project isn't available on
    the host in this scenario. The VIRTIO_GPU_F_VIRGL naming
    convention is kept since it's easier to redefine the meaning
    rather than changing header files.

    The context type is associated an particular capset id. Virgl
    has two capsets due a prior bug, but for other cases the 1:1
    mapping between context type and capset id is valid.

    In addition, fencing needs to be fixed to accomodate multiple
    context types. In the past, there was one global timeline
    associated witht the OpenGL rendering. Now, there are multiple
    timelines which can be associated with GL, VK or even display
    contexts.

    v2: Correct number of bits

    Signed-off-by: Gurchetan Singh <gurchetansingh@chromium.org>
    ---
    virtio-gpu.tex | 75 ++++++++++++++++++++++++++++++++++++++++++--------
    1 file changed, 63 insertions(+), 12 deletions(-)

    diff --git a/virtio-gpu.tex b/virtio-gpu.tex
    index 35c129f..5753466 100644
    --- a/virtio-gpu.tex
    +++ b/virtio-gpu.tex
    @@ -1,14 +1,10 @@
    \section{GPU Device}\label{sec:Device Types / GPU Device}

    virtio-gpu is a virtio based graphics adapter. It can operate in 2D
    -mode and in 3D (virgl) mode. 3D mode will offload rendering ops to
    +mode and in 3D mode. 3D mode will offload rendering ops to
    the host gpu and therefore requires a gpu with 3D support on the host
    machine.

    -3D mode is not covered (yet) in this specification, even though it is
    -mentioned here and there due to some details of the virtual hardware
    -being designed with 3D mode in mind.
    -
    In 2D mode the virtio-gpu device provides support for ARGB Hardware
    cursors and multiple scanouts (aka heads).

    @@ -39,6 +35,8 @@ \subsection{Feature bits}\label{sec:Device Types / GPU Device / Feature bits}
    to other virtio devices is supported.
    \item[VIRTIO_GPU_F_RESOURCE_BLOB (3)] creating and using size-based blob
    resources is supported.
    +\item[VIRTIO_GPU_F_CONTEXT_INIT (4)] multiple context types and
    + synchronization timelines supported. Requires VIRTIO_GPU_F_VIRGL.
    \end{description}

    \subsection{Device configuration layout}\label{sec:Device Types / GPU Device / Device configuration layout}
    @@ -214,7 +212,7 @@ \subsubsection{Device Operation: Request header}\label{sec:Device Types / GPU De
    VIRTIO_GPU_CMD_RESOURCE_CREATE_BLOB,
    VIRTIO_GPU_CMD_SET_SCANOUT_BLOB,

    - /* 3d commands (OpenGL) */
    + /* 3d commands */
    VIRTIO_GPU_CMD_CTX_CREATE = 0x0200,
    VIRTIO_GPU_CMD_CTX_DESTROY,
    VIRTIO_GPU_CMD_CTX_ATTACH_RESOURCE,
    @@ -249,13 +247,16 @@ \subsubsection{Device Operation: Request header}\label{sec:Device Types / GPU De
    };

    #define VIRTIO_GPU_FLAG_FENCE (1 << 0)
    +#define VIRTIO_GPU_FLAG_INFO_RING_IDX (1 << 1)
    +
    +#define VIRTIO_GPU_INFO_RING_IDX_MASK 0x0000001f;

    struct virtio_gpu_ctrl_hdr {
    le32 type;
    le32 flags;
    le64 fence_id;
    le32 ctx_id;
    - le32 padding;
    + le32 info;
    };
    \end{lstlisting}

    @@ -275,6 +276,22 @@ \subsubsection{Device Operation: Request header}\label{sec:Device Types / GPU De
    \item send the response only after command processing is complete.
    \end{itemize*}
    \item[\field{ctx_id}] Rendering context (used in 3D mode only).
    +\item[\field{info}] Information associated with the command. If
    + VIRTIO_GPU_F_CONTEXT_INIT is supported, then the driver MAY set
    + VIRTIO_GPU_FLAG_INFO_RING_IDX bit in the request \field{flags}. In
    + that case:
    + \begin{itemize*}
    + \item VIRTIO_GPU_FLAG_FENCE MUST also be set
    + \item the lower 5 bits of \field{info} indicates the value of a
    + context-specific ring index. The minimum value is 0 and maximum
    + value is 31.
    + \item \field{fence_id} acts as a sequence number on the synchronization
    + timeline defined by \field{ctx_idx} and the ring index.
    + \item when the command associated with \field{fence_id} is complete, the
    + device MUST send a response for all outstanding commands with a sequence
    + number less than or equal to \field{fence_id} on the same
    + synchronization timeline.
    + \end{itemize*}
    \end{description}

    On success the device will return VIRTIO_GPU_RESP_OK_NODATA in
    @@ -516,6 +533,12 @@ \subsubsection{Device Operation: controlq}\label{sec:Device Types / GPU Device /
    the first edition of Virgl (Gallium OpenGL) protocol.
    \item \href{https://gitlab.freedesktop.org/virgl/virglrenderer/-/blob/master/src/virgl_hw.h#L550}{VIRTIO_GPU_CAPSET_VIRGL2} --
    the second edition of Virgl (Gallium OpenGL) protocol after the capset fix.
    + \item \href{https://android.googlesource.com/device/generic/vulkan-cereal/+/refs/heads/master/protocols/}{VIRTIO_GPU_CAPSET_GFXSTREAM} --
    + gfxtream's (mostly) autogenerated GLES and Vulkan streaming protocols.
    + \item \href{https://gitlab.freedesktop.org/olv/venus-protocol}{VIRTIO_GPU_CAPSET_VENUS} --
    + Mesa's (mostly) autogenerated Vulkan protocol.
    + \item \href{https://chromium.googlesource.com/chromiumos/platform/crosvm/+/refs/heads/main/rutabaga_gfx/src/cross_domain/cross_domain_protocol.rs}{VIRTIO_GPU_CAPSET_CROSS_DOMAIN} --
    + protocol for display virtualization via Wayland proxying.
    \end{itemize*}

    \begin{lstlisting}
    @@ -527,6 +550,9 @@ \subsubsection{Device Operation: controlq}\label{sec:Device Types / GPU Device /

    #define VIRTIO_GPU_CAPSET_VIRGL 1
    #define VIRTIO_GPU_CAPSET_VIRGL2 2
    +#define VIRTIO_GPU_CAPSET_GFXSTREAM 3
    +#define VIRTIO_GPU_CAPSET_VENUS 4
    +#define VIRTIO_GPU_CAPSET_CROSS_DOMAIN 5
    struct virtio_gpu_resp_capset_info {
    struct virtio_gpu_ctrl_hdr hdr;
    le32 capset_id;
    @@ -684,21 +710,46 @@ \subsubsection{Device Operation: controlq (3d)}\label{sec:Device Types / GPU Dev

    \begin{description}

    -\item[VIRTIO_GPU_CMD_CTX_CREATE]
    +\item[VIRTIO_GPU_CMD_CTX_CREATE] creates a context for submitting an opaque
    + command stream. Request data is \field{struct virtio_gpu_ctx_create}.
    + Response type is VIRTIO_GPU_RESP_OK_NODATA.
    +
    +\begin{lstlisting}
    +#define VIRTIO_GPU_CONTEXT_INIT_CAPSET_ID_MASK 0x0000003f;
    +struct virtio_gpu_ctx_create {
    + struct virtio_gpu_ctrl_hdr hdr;
    + le32 nlen;
    + le32 context_init;
    + char debug_name[64];
    +};
    +\end{lstlisting}
    +
    +The implementation MUST create a context for the given \field{ctx_id} in
    +the \field{hdr}. For debugging purposes, a \field{debug_name} and it's
    +length \field{nlen} is provided by the driver. If
    +VIRTIO_GPU_F_CONTEXT_INIT is supported, then lower 6 bits of
    +\field{context_init} MAY contain the \field{capset_id} associated with
    +context. In that case, then the device MUST create a context that can
    +handle the specified command stream.
    +
    +If the lower 6-bits of the \field{context_init} are zero, then the type of
    +the context is determined by the device.
    +
    \item[VIRTIO_GPU_CMD_CTX_DESTROY]
    \item[VIRTIO_GPU_CMD_CTX_ATTACH_RESOURCE]
    \item[VIRTIO_GPU_CMD_CTX_DETACH_RESOURCE]
    - Manage OpenGL contexts.
    + Manage virtio-gpu 3d contexts.

    \item[VIRTIO_GPU_CMD_RESOURCE_CREATE_3D]
    - Create OpenGL resources.
    + Create virtio-gpu 3d resources.

    \item[VIRTIO_GPU_CMD_TRANSFER_TO_HOST_3D]
    \item[VIRTIO_GPU_CMD_TRANSFER_FROM_HOST_3D]
    - Transfer data from and to OpenGL resources.
    + Transfer data from and to virtio-gpu 3d resources.

    \item[VIRTIO_GPU_CMD_SUBMIT_3D]
    - Submit rendering commands (mesa gallium command stream).
    + Submit an opaque command stream. The type of the command stream is
    + determined when creating a context.

    \item[VIRTIO_GPU_CMD_RESOURCE_MAP_BLOB] maps a host-only
    blob resource into an offset in the host visible memory region. Request
    --
    2.31.0




  • 5.  Re: [RFC PATCH v2 2/2] virtio-gpu: add context init support

    Posted 09-02-2021 10:32
    Hi,

    > struct virtio_gpu_ctrl_hdr {
    > le32 type;
    > le32 flags;
    > le64 fence_id;
    > le32 ctx_id;
    > - le32 padding;
    > + le32 info;

    How about
    u8 ring_idx;
    u8 padding[3];

    ?

    Otherwise this looks all sane to me.

    take care,
    Gerd




  • 6.  Re: [virtio-comment] [RFC PATCH v2 2/2] virtio-gpu: add context init support

    Posted 09-02-2021 16:59
    On Mon, Aug 30, 2021 at 12:42 PM Gurchetan Singh
    <gurchetansingh@chromium.org> wrote:
    > +\item[\field{info}] Information associated with the command. If
    > + VIRTIO_GPU_F_CONTEXT_INIT is supported, then the driver MAY set
    > + VIRTIO_GPU_FLAG_INFO_RING_IDX bit in the request \field{flags}. In
    > + that case:
    > + \begin{itemize*}
    > + \item VIRTIO_GPU_FLAG_FENCE MUST also be set
    VIRTIO_GPU_FLAG_FENCE is optional currently and has a cost. There
    might be use cases where fencing is not needed even with rings. Why
    is it required when VIRTIO_GPU_FLAG_INFO_RING_IDX is set?

    > + \item the lower 5 bits of \field{info} indicates the value of a
    > + context-specific ring index. The minimum value is 0 and maximum
    > + value is 31.
    For Venus at least, we plan to assign each VkQueue of each logical
    VkDevice a ring index. While 32 for a logical VkDevice seems enough
    (TM), there are apps that require more than one logical device.

    Should we reserve 8 bits and allow up to 63? This way the last two
    bits can be flexible if we run out of bit space in the future.

    > + \item \field{fence_id} acts as a sequence number on the synchronization
    > + timeline defined by \field{ctx_idx} and the ring index.
    > + \item when the command associated with \field{fence_id} is complete, the
    > + device MUST send a response for all outstanding commands with a sequence
    > + number less than or equal to \field{fence_id} on the same
    > + synchronization timeline.
    > + \end{itemize*}
    > \end{description}
    >
    > On success the device will return VIRTIO_GPU_RESP_OK_NODATA in
    > @@ -516,6 +533,12 @@ \subsubsection{Device Operation: controlq}\label{sec:Device Types / GPU Device /
    > the first edition of Virgl (Gallium OpenGL) protocol.
    > \item \href{https://gitlab.freedesktop.org/virgl/virglrenderer/-/blob/master/src/virgl_hw.h#L550}{VIRTIO_GPU_CAPSET_VIRGL2} --
    > the second edition of Virgl (Gallium OpenGL) protocol after the capset fix.
    > + \item \href{https://android.googlesource.com/device/generic/vulkan-cereal/+/refs/heads/master/protocols/}{VIRTIO_GPU_CAPSET_GFXSTREAM} --
    > + gfxtream's (mostly) autogenerated GLES and Vulkan streaming protocols.
    > + \item \href{https://gitlab.freedesktop.org/olv/venus-protocol}{VIRTIO_GPU_CAPSET_VENUS} --
    > + Mesa's (mostly) autogenerated Vulkan protocol.
    > + \item \href{https://chromium.googlesource.com/chromiumos/platform/crosvm/+/refs/heads/main/rutabaga_gfx/src/cross_domain/cross_domain_protocol.rs}{VIRTIO_GPU_CAPSET_CROSS_DOMAIN} --
    > + protocol for display virtualization via Wayland proxying.
    > \end{itemize*}
    >
    > \begin{lstlisting}
    > @@ -527,6 +550,9 @@ \subsubsection{Device Operation: controlq}\label{sec:Device Types / GPU Device /
    >
    > #define VIRTIO_GPU_CAPSET_VIRGL 1
    > #define VIRTIO_GPU_CAPSET_VIRGL2 2
    > +#define VIRTIO_GPU_CAPSET_GFXSTREAM 3
    > +#define VIRTIO_GPU_CAPSET_VENUS 4
    > +#define VIRTIO_GPU_CAPSET_CROSS_DOMAIN 5
    > struct virtio_gpu_resp_capset_info {
    > struct virtio_gpu_ctrl_hdr hdr;
    > le32 capset_id;
    > @@ -684,21 +710,46 @@ \subsubsection{Device Operation: controlq (3d)}\label{sec:Device Types / GPU Dev
    >
    > \begin{description}
    >
    > -\item[VIRTIO_GPU_CMD_CTX_CREATE]
    > +\item[VIRTIO_GPU_CMD_CTX_CREATE] creates a context for submitting an opaque
    > + command stream. Request data is \field{struct virtio_gpu_ctx_create}.
    > + Response type is VIRTIO_GPU_RESP_OK_NODATA.
    > +
    > +\begin{lstlisting}
    > +#define VIRTIO_GPU_CONTEXT_INIT_CAPSET_ID_MASK 0x0000003f;
    > +struct virtio_gpu_ctx_create {
    > + struct virtio_gpu_ctrl_hdr hdr;
    > + le32 nlen;
    > + le32 context_init;
    > + char debug_name[64];
    > +};
    > +\end{lstlisting}
    > +
    > +The implementation MUST create a context for the given \field{ctx_id} in
    > +the \field{hdr}. For debugging purposes, a \field{debug_name} and it's
    > +length \field{nlen} is provided by the driver. If
    > +VIRTIO_GPU_F_CONTEXT_INIT is supported, then lower 6 bits of
    > +\field{context_init} MAY contain the \field{capset_id} associated with
    > +context. In that case, then the device MUST create a context that can
    > +handle the specified command stream.
    > +
    > +If the lower 6-bits of the \field{context_init} are zero, then the type of
    > +the context is determined by the device.
    > +
    > \item[VIRTIO_GPU_CMD_CTX_DESTROY]
    > \item[VIRTIO_GPU_CMD_CTX_ATTACH_RESOURCE]
    > \item[VIRTIO_GPU_CMD_CTX_DETACH_RESOURCE]
    > - Manage OpenGL contexts.
    > + Manage virtio-gpu 3d contexts.
    >
    > \item[VIRTIO_GPU_CMD_RESOURCE_CREATE_3D]
    > - Create OpenGL resources.
    > + Create virtio-gpu 3d resources.
    >
    > \item[VIRTIO_GPU_CMD_TRANSFER_TO_HOST_3D]
    > \item[VIRTIO_GPU_CMD_TRANSFER_FROM_HOST_3D]
    > - Transfer data from and to OpenGL resources.
    > + Transfer data from and to virtio-gpu 3d resources.
    >
    > \item[VIRTIO_GPU_CMD_SUBMIT_3D]
    > - Submit rendering commands (mesa gallium command stream).
    > + Submit an opaque command stream. The type of the command stream is
    > + determined when creating a context.
    >
    > \item[VIRTIO_GPU_CMD_RESOURCE_MAP_BLOB] maps a host-only
    > blob resource into an offset in the host visible memory region. Request
    > --
    > 2.31.0
    >
    >
    > This publicly archived list offers a means to provide input to the
    > OASIS Virtual I/O Device (VIRTIO) TC.
    >
    > In order to verify user consent to the Feedback License terms and
    > to minimize spam in the list archive, subscription is required
    > before posting.
    >
    > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
    > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
    > List help: virtio-comment-help@lists.oasis-open.org
    > List archive: https://lists.oasis-open.org/archives/virtio-comment/
    > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
    > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
    > Committee: https://www.oasis-open.org/committees/virtio/
    > Join OASIS: https://www.oasis-open.org/join/
    >



  • 7.  Re: [virtio-comment] [RFC PATCH v2 2/2] virtio-gpu: add context init support

    Posted 09-09-2021 01:33
    On Thu, Sep 2, 2021 at 9:59 AM Chia-I Wu <olvaffe@gmail.com> wrote:

    > On Mon, Aug 30, 2021 at 12:42 PM Gurchetan Singh
    > <gurchetansingh@chromium.org> wrote:
    > > +\item[\field{info}] Information associated with the command. If
    > > + VIRTIO_GPU_F_CONTEXT_INIT is supported, then the driver MAY set
    > > + VIRTIO_GPU_FLAG_INFO_RING_IDX bit in the request \field{flags}. In
    > > + that case:
    > > + \begin{itemize*}
    > > + \item VIRTIO_GPU_FLAG_FENCE MUST also be set
    > VIRTIO_GPU_FLAG_FENCE is optional currently and has a cost. There
    > might be use cases where fencing is not needed even with rings. Why
    > is it required when VIRTIO_GPU_FLAG_INFO_RING_IDX is set?
    >

    The latest non-RFC spec changes don't require
    VIRTIO_GPU_FLAG_INFO_RING_IDX + VIRTIO_GPU_FLAG_FENCE to go in unison.
    Implementation-wise, command buffer submission always sets
    VIRTIO_GPU_FLAG_FENCE at the moment (prior behavior unchanged with
    multiple-rings). More guest-side changes (say VIRTGPU_EXECBUF_NO_WAIT)
    would be required if one wants to submit a command without waiting for a
    response (or use the ASG technique).


    >
    > > + \item the lower 5 bits of \field{info} indicates the value of a
    > > + context-specific ring index. The minimum value is 0 and maximum
    > > + value is 31.
    > For Venus at least, we plan to assign each VkQueue of each logical
    > VkDevice a ring index. While 32 for a logical VkDevice seems enough
    > (TM), there are apps that require more than one logical device.
    >
    > Should we reserve 8 bits and allow up to 63? This way the last two
    > bits can be flexible if we run out of bit space in the future.
    >

    Done.




    >
    > > + \item \field{fence_id} acts as a sequence number on the
    > synchronization
    > > + timeline defined by \field{ctx_idx} and the ring index.
    > > + \item when the command associated with \field{fence_id} is complete,
    > the
    > > + device MUST send a response for all outstanding commands with a
    > sequence
    > > + number less than or equal to \field{fence_id} on the same
    > > + synchronization timeline.
    > > + \end{itemize*}
    > > \end{description}
    > >
    > > On success the device will return VIRTIO_GPU_RESP_OK_NODATA in
    > > @@ -516,6 +533,12 @@ \subsubsection{Device Operation:
    > controlq}\label{sec:Device Types / GPU Device /
    > > the first edition of Virgl (Gallium OpenGL) protocol.
    > > \item \href{
    > https://gitlab.freedesktop.org/virgl/virglrenderer/-/blob/master/src/virgl_hw.h#L550}{VIRTIO_GPU_CAPSET_VIRGL2}
    > --
    > > the second edition of Virgl (Gallium OpenGL) protocol after the
    > capset fix.
    > > + \item \href{
    > https://android.googlesource.com/device/generic/vulkan-cereal/+/refs/heads/master/protocols/}{VIRTIO_GPU_CAPSET_GFXSTREAM
    > <https://android.googlesource.com/device/generic/vulkan-cereal/+/refs/heads/master/protocols/%7D%7BVIRTIO_GPU_CAPSET_GFXSTREAM>}
    > --
    > > + gfxtream's (mostly) autogenerated GLES and Vulkan streaming
    > protocols.
    > > + \item \href{
    > https://gitlab.freedesktop.org/olv/venus-protocol}{VIRTIO_GPU_CAPSET_VENUS}
    > --
    > > + Mesa's (mostly) autogenerated Vulkan protocol.
    > > + \item \href{
    > https://chromium.googlesource.com/chromiumos/platform/crosvm/+/refs/heads/main/rutabaga_gfx/src/cross_domain/cross_domain_protocol.rs}{VIRTIO_GPU_CAPSET_CROSS_DOMAIN
    > <https://chromium.googlesource.com/chromiumos/platform/crosvm/+/refs/heads/main/rutabaga_gfx/src/cross_domain/cross_domain_protocol.rs%7D%7BVIRTIO_GPU_CAPSET_CROSS_DOMAIN>}
    > --
    > > + protocol for display virtualization via Wayland proxying.
    > > \end{itemize*}
    > >
    > > \begin{lstlisting}
    > > @@ -527,6 +550,9 @@ \subsubsection{Device Operation:
    > controlq}\label{sec:Device Types / GPU Device /
    > >
    > > #define VIRTIO_GPU_CAPSET_VIRGL 1
    > > #define VIRTIO_GPU_CAPSET_VIRGL2 2
    > > +#define VIRTIO_GPU_CAPSET_GFXSTREAM 3
    > > +#define VIRTIO_GPU_CAPSET_VENUS 4
    > > +#define VIRTIO_GPU_CAPSET_CROSS_DOMAIN 5
    > > struct virtio_gpu_resp_capset_info {
    > > struct virtio_gpu_ctrl_hdr hdr;
    > > le32 capset_id;
    > > @@ -684,21 +710,46 @@ \subsubsection{Device Operation: controlq
    > (3d)}\label{sec:Device Types / GPU Dev
    > >
    > > \begin{description}
    > >
    > > -\item[VIRTIO_GPU_CMD_CTX_CREATE]
    > > +\item[VIRTIO_GPU_CMD_CTX_CREATE] creates a context for submitting an
    > opaque
    > > + command stream. Request data is \field{struct virtio_gpu_ctx_create}.
    > > + Response type is VIRTIO_GPU_RESP_OK_NODATA.
    > > +
    > > +\begin{lstlisting}
    > > +#define VIRTIO_GPU_CONTEXT_INIT_CAPSET_ID_MASK 0x0000003f;
    > > +struct virtio_gpu_ctx_create {
    > > + struct virtio_gpu_ctrl_hdr hdr;
    > > + le32 nlen;
    > > + le32 context_init;
    > > + char debug_name[64];
    > > +};
    > > +\end{lstlisting}
    > > +
    > > +The implementation MUST create a context for the given \field{ctx_id} in
    > > +the \field{hdr}. For debugging purposes, a \field{debug_name} and it's
    > > +length \field{nlen} is provided by the driver. If
    > > +VIRTIO_GPU_F_CONTEXT_INIT is supported, then lower 6 bits of
    > > +\field{context_init} MAY contain the \field{capset_id} associated with
    > > +context. In that case, then the device MUST create a context that can
    > > +handle the specified command stream.
    > > +
    > > +If the lower 6-bits of the \field{context_init} are zero, then the type
    > of
    > > +the context is determined by the device.
    > > +
    > > \item[VIRTIO_GPU_CMD_CTX_DESTROY]
    > > \item[VIRTIO_GPU_CMD_CTX_ATTACH_RESOURCE]
    > > \item[VIRTIO_GPU_CMD_CTX_DETACH_RESOURCE]
    > > - Manage OpenGL contexts.
    > > + Manage virtio-gpu 3d contexts.
    > >
    > > \item[VIRTIO_GPU_CMD_RESOURCE_CREATE_3D]
    > > - Create OpenGL resources.
    > > + Create virtio-gpu 3d resources.
    > >
    > > \item[VIRTIO_GPU_CMD_TRANSFER_TO_HOST_3D]
    > > \item[VIRTIO_GPU_CMD_TRANSFER_FROM_HOST_3D]
    > > - Transfer data from and to OpenGL resources.
    > > + Transfer data from and to virtio-gpu 3d resources.
    > >
    > > \item[VIRTIO_GPU_CMD_SUBMIT_3D]
    > > - Submit rendering commands (mesa gallium command stream).
    > > + Submit an opaque command stream. The type of the command stream is
    > > + determined when creating a context.
    > >
    > > \item[VIRTIO_GPU_CMD_RESOURCE_MAP_BLOB] maps a host-only
    > > blob resource into an offset in the host visible memory region.
    > Request
    > > --
    > > 2.31.0
    > >
    > >
    > > This publicly archived list offers a means to provide input to the
    > > OASIS Virtual I/O Device (VIRTIO) TC.
    > >
    > > In order to verify user consent to the Feedback License terms and
    > > to minimize spam in the list archive, subscription is required
    > > before posting.
    > >
    > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
    > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
    > > List help: virtio-comment-help@lists.oasis-open.org
    > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
    > > Feedback License:
    > https://www.oasis-open.org/who/ipr/feedback_license.pdf
    > > List Guidelines:
    > https://www.oasis-open.org/policies-guidelines/mailing-lists
    > > Committee: https://www.oasis-open.org/committees/virtio/
    > > Join OASIS: https://www.oasis-open.org/join/
    > >
    >



  • 8.  Re: [virtio-comment] [RFC PATCH v2 2/2] virtio-gpu: add context init support

    Posted 09-09-2021 22:45
    Thanks. v1 looks good to me.

    On Wed, Sep 8, 2021 at 6:33 PM Gurchetan Singh
    <gurchetansingh@chromium.org> wrote:
    >
    >
    >
    > On Thu, Sep 2, 2021 at 9:59 AM Chia-I Wu <olvaffe@gmail.com> wrote:
    >>
    >> On Mon, Aug 30, 2021 at 12:42 PM Gurchetan Singh
    >> <gurchetansingh@chromium.org> wrote:
    >> > +\item[\field{info}] Information associated with the command. If
    >> > + VIRTIO_GPU_F_CONTEXT_INIT is supported, then the driver MAY set
    >> > + VIRTIO_GPU_FLAG_INFO_RING_IDX bit in the request \field{flags}. In
    >> > + that case:
    >> > + \begin{itemize*}
    >> > + \item VIRTIO_GPU_FLAG_FENCE MUST also be set
    >> VIRTIO_GPU_FLAG_FENCE is optional currently and has a cost. There
    >> might be use cases where fencing is not needed even with rings. Why
    >> is it required when VIRTIO_GPU_FLAG_INFO_RING_IDX is set?
    >
    >
    > The latest non-RFC spec changes don't require VIRTIO_GPU_FLAG_INFO_RING_IDX + VIRTIO_GPU_FLAG_FENCE to go in unison. Implementation-wise, command buffer submission always sets VIRTIO_GPU_FLAG_FENCE at the moment (prior behavior unchanged with multiple-rings). More guest-side changes (say VIRTGPU_EXECBUF_NO_WAIT) would be required if one wants to submit a command without waiting for a response (or use the ASG technique).
    >
    >>
    >>
    >> > + \item the lower 5 bits of \field{info} indicates the value of a
    >> > + context-specific ring index. The minimum value is 0 and maximum
    >> > + value is 31.
    >> For Venus at least, we plan to assign each VkQueue of each logical
    >> VkDevice a ring index. While 32 for a logical VkDevice seems enough
    >> (TM), there are apps that require more than one logical device.
    >>
    >> Should we reserve 8 bits and allow up to 63? This way the last two
    >> bits can be flexible if we run out of bit space in the future.
    >
    >
    > Done.
    >
    >
    >
    >>
    >>
    >> > + \item \field{fence_id} acts as a sequence number on the synchronization
    >> > + timeline defined by \field{ctx_idx} and the ring index.
    >> > + \item when the command associated with \field{fence_id} is complete, the
    >> > + device MUST send a response for all outstanding commands with a sequence
    >> > + number less than or equal to \field{fence_id} on the same
    >> > + synchronization timeline.
    >> > + \end{itemize*}
    >> > \end{description}
    >> >
    >> > On success the device will return VIRTIO_GPU_RESP_OK_NODATA in
    >> > @@ -516,6 +533,12 @@ \subsubsection{Device Operation: controlq}\label{sec:Device Types / GPU Device /
    >> > the first edition of Virgl (Gallium OpenGL) protocol.
    >> > \item \href{https://gitlab.freedesktop.org/virgl/virglrenderer/-/blob/master/src/virgl_hw.h#L550}{VIRTIO_GPU_CAPSET_VIRGL2} --
    >> > the second edition of Virgl (Gallium OpenGL) protocol after the capset fix.
    >> > + \item \href{https://android.googlesource.com/device/generic/vulkan-cereal/+/refs/heads/master/protocols/}{VIRTIO_GPU_CAPSET_GFXSTREAM} --
    >> > + gfxtream's (mostly) autogenerated GLES and Vulkan streaming protocols.
    >> > + \item \href{https://gitlab.freedesktop.org/olv/venus-protocol}{VIRTIO_GPU_CAPSET_VENUS} --
    >> > + Mesa's (mostly) autogenerated Vulkan protocol.
    >> > + \item \href{https://chromium.googlesource.com/chromiumos/platform/crosvm/+/refs/heads/main/rutabaga_gfx/src/cross_domain/cross_domain_protocol.rs}{VIRTIO_GPU_CAPSET_CROSS_DOMAIN} --
    >> > + protocol for display virtualization via Wayland proxying.
    >> > \end{itemize*}
    >> >
    >> > \begin{lstlisting}
    >> > @@ -527,6 +550,9 @@ \subsubsection{Device Operation: controlq}\label{sec:Device Types / GPU Device /
    >> >
    >> > #define VIRTIO_GPU_CAPSET_VIRGL 1
    >> > #define VIRTIO_GPU_CAPSET_VIRGL2 2
    >> > +#define VIRTIO_GPU_CAPSET_GFXSTREAM 3
    >> > +#define VIRTIO_GPU_CAPSET_VENUS 4
    >> > +#define VIRTIO_GPU_CAPSET_CROSS_DOMAIN 5
    >> > struct virtio_gpu_resp_capset_info {
    >> > struct virtio_gpu_ctrl_hdr hdr;
    >> > le32 capset_id;
    >> > @@ -684,21 +710,46 @@ \subsubsection{Device Operation: controlq (3d)}\label{sec:Device Types / GPU Dev
    >> >
    >> > \begin{description}
    >> >
    >> > -\item[VIRTIO_GPU_CMD_CTX_CREATE]
    >> > +\item[VIRTIO_GPU_CMD_CTX_CREATE] creates a context for submitting an opaque
    >> > + command stream. Request data is \field{struct virtio_gpu_ctx_create}.
    >> > + Response type is VIRTIO_GPU_RESP_OK_NODATA.
    >> > +
    >> > +\begin{lstlisting}
    >> > +#define VIRTIO_GPU_CONTEXT_INIT_CAPSET_ID_MASK 0x0000003f;
    >> > +struct virtio_gpu_ctx_create {
    >> > + struct virtio_gpu_ctrl_hdr hdr;
    >> > + le32 nlen;
    >> > + le32 context_init;
    >> > + char debug_name[64];
    >> > +};
    >> > +\end{lstlisting}
    >> > +
    >> > +The implementation MUST create a context for the given \field{ctx_id} in
    >> > +the \field{hdr}. For debugging purposes, a \field{debug_name} and it's
    >> > +length \field{nlen} is provided by the driver. If
    >> > +VIRTIO_GPU_F_CONTEXT_INIT is supported, then lower 6 bits of
    >> > +\field{context_init} MAY contain the \field{capset_id} associated with
    >> > +context. In that case, then the device MUST create a context that can
    >> > +handle the specified command stream.
    >> > +
    >> > +If the lower 6-bits of the \field{context_init} are zero, then the type of
    >> > +the context is determined by the device.
    >> > +
    >> > \item[VIRTIO_GPU_CMD_CTX_DESTROY]
    >> > \item[VIRTIO_GPU_CMD_CTX_ATTACH_RESOURCE]
    >> > \item[VIRTIO_GPU_CMD_CTX_DETACH_RESOURCE]
    >> > - Manage OpenGL contexts.
    >> > + Manage virtio-gpu 3d contexts.
    >> >
    >> > \item[VIRTIO_GPU_CMD_RESOURCE_CREATE_3D]
    >> > - Create OpenGL resources.
    >> > + Create virtio-gpu 3d resources.
    >> >
    >> > \item[VIRTIO_GPU_CMD_TRANSFER_TO_HOST_3D]
    >> > \item[VIRTIO_GPU_CMD_TRANSFER_FROM_HOST_3D]
    >> > - Transfer data from and to OpenGL resources.
    >> > + Transfer data from and to virtio-gpu 3d resources.
    >> >
    >> > \item[VIRTIO_GPU_CMD_SUBMIT_3D]
    >> > - Submit rendering commands (mesa gallium command stream).
    >> > + Submit an opaque command stream. The type of the command stream is
    >> > + determined when creating a context.
    >> >
    >> > \item[VIRTIO_GPU_CMD_RESOURCE_MAP_BLOB] maps a host-only
    >> > blob resource into an offset in the host visible memory region. Request
    >> > --
    >> > 2.31.0
    >> >
    >> >
    >> > This publicly archived list offers a means to provide input to the
    >> > OASIS Virtual I/O Device (VIRTIO) TC.
    >> >
    >> > In order to verify user consent to the Feedback License terms and
    >> > to minimize spam in the list archive, subscription is required
    >> > before posting.
    >> >
    >> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
    >> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
    >> > List help: virtio-comment-help@lists.oasis-open.org
    >> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
    >> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
    >> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
    >> > Committee: https://www.oasis-open.org/committees/virtio/
    >> > Join OASIS: https://www.oasis-open.org/join/
    >> >