Package com.google.android.filament

Class Stream

  • public class Stream
extends java.lang.Object
    Stream is used to attach a native video stream to a filament Texture. Stream supports three different configurations:
    TEXTURE_ID
    takes an OpenGL texture ID and incurs a copy
    ACQUIRED
    connects to an Android AHardwareBuffer
    NATIVE
    connects to an Android SurfaceTexture

    Before explaining these different configurations, let's review the high-level structure of an AR or video application that uses Filament. 

     while (true) {

     // Misc application work occurs here, such as:
     // - Writing the image data for a video frame into a Stream
     // - Moving the Filament Camera

     if (renderer.beginFrame(swapChain)) {
         renderer.render(view);
         renderer.endFrame();
     }
 }

    Let's say that the video image data at the time of a particular invocation of beginFrame becomes visible to users at time A. The 3D scene state (including the camera) at the time of that same invocation becomes apparent to users at time B.

    • If time A matches time B, we say that the stream is synchronized.
    • Filament invokes low-level graphics commands on the driver thread.
    • The thread that calls beginFrame is called the main thread.

    The TEXTURE_ID configuration achieves synchronization automatically. In this mode, Filament performs a copy on the main thread during beginFrame by blitting the external image into an internal round-robin queue of images. This copy has a run-time cost.

    For ACQUIRED streams, there is no need to perform the copy because Filament explictly acquires the stream, then releases it later via a callback function. This configuration is especially useful when the Vulkan backend is enabled.

    For NATIVE streams, Filament does not make any synchronization guarantee. However they are simple to use and do not incur a copy. These are often appropriate in video applications.

    Please see sample-stream-test and sample-hello-camera for usage examples.

    See Also:
    Texture.setExternalStream(com.google.android.filament.Engine, com.google.android.filament.Stream), Engine.destroyStream(com.google.android.filament.Stream)

    • Method Detail

      • getStreamType

        public Stream.StreamType getStreamType()
        Indicates whether this Stream is NATIVE, TEXTURE_ID, or ACQUIRED.

      • setAcquiredImage

        public void setAcquiredImage​(java.lang.Object hwbuffer,
                             java.lang.Object handler,
                             java.lang.Runnable callback)
        Updates an 
        ACQUIRED
        stream with an image that is guaranteed to be used in the next frame. This method tells Filament to immediately "acquire" the image and trigger a callback when it is done with it. This should be called by the user outside of beginFrame / endFrame, and should be called only once per frame. If the user pushes images to the same stream multiple times in a single frame, only the final image is honored, but all callbacks are invoked. This method should be called on the same thread that calls Renderer.beginFrame(com.google.android.filament.SwapChain, long), which is also where the callback is invoked. This method can only be used for streams that were constructed without calling the Builder.stream method. See Stream for more information about NATIVE, TEXTURE_ID, and ACQUIRED configurations.
        Parameters:
        hwbuffer - HardwareBuffer (requires API level 26)
        handler - Executor or Handler.
        callback - a callback invoked by handler when the hwbuffer can be released.

      • setDimensions

        public void setDimensions​(@IntRange(from=0L)
                          int width,
                          @IntRange(from=0L)
                          int height)
        Updates the size of the incoming stream. Whether this value is used is stream dependent. On Android, it must be set when using Stream.Builder.stream(long)
        Parameters:
        width - new width of the incoming stream
        height - new height of the incoming stream

      • readPixels

        public void readPixels​(@IntRange(from=0L)
                       int xoffset,
                       @IntRange(from=0L)
                       int yoffset,
                       @IntRange(from=0L)
                       int width,
                       @IntRange(from=0L)
                       int height,
                       @NonNull
                       Texture.PixelBufferDescriptor buffer)
        Reads back the content of the last frame of a Stream since the last call to Renderer.beginFrame(com.google.android.filament.SwapChain, long).

        The Stream must be a copy stream, which can be checked with getStreamType(). This function is a no-op otherwise.

        
  Stream buffer                  User buffer (PixelBufferDescriptor)
  +--------------------+
  |                    |                .stride         .alignment
  |                    |         ----------------------->-->
  |                    |         O----------------------+--+   low addresses
  |                    |         |          |           |  |
  |             w      |         |          | .top      |  |
  |       <--------->  |         |          V           |  |
  |       +---------+  |         |     +---------+      |  |
  |       |     ^   |  | ======> |     |         |      |  |
  |   x   |    h|   |  |         |.left|         |      |  |
  +------>|     v   |  |         +---->|         |      |  |
  |       +.........+  |         |     +.........+      |  |
  |            ^       |         |                      |  |
  |          y |       |         +----------------------+--+  high addresses
  O------------+-------+

        Typically readPixels() will be called after Renderer.beginFrame(com.google.android.filament.SwapChain, long).

        After calling this method, the callback associated with buffer will be invoked on the main thread, indicating that the read-back has completed. Typically, this will happen after multiple calls to Renderer.beginFrame(com.google.android.filament.SwapChain, long), Renderer.render(com.google.android.filament.View), Renderer.endFrame().

        readPixels is intended for debugging and testing. It will impact performance significantly.

        Parameters:
        xoffset - left offset of the sub-region to read back
        yoffset - bottom offset of the sub-region to read back
        width - width of the sub-region to read back
        height - height of the sub-region to read back
        buffer - client-side buffer where the read-back will be written

        The following format are always supported:

      • Texture.Format.RGBA
      • Texture.Format.RGBA_INTEGER

        • The following types are always supported:

      • Texture.Type.UBYTE
      • Texture.Type.UINT
      • Texture.Type.INT
      • Texture.Type.FLOAT

        • Other combination of format/type may be supported. If a combination is not supported, this operation may fail silently. Use a DEBUG build to get some logs about the failure.

        Throws:
        java.nio.BufferOverflowException - if the specified parameters would result in reading outside of buffer.

      • getTimestamp

        public long getTimestamp()
        Returns the presentation time of the currently displayed frame in nanosecond. This value can change at any time.
        Returns:
        timestamp in nanosecond.

      • getNativeObject

        public long getNativeObject()