Why depth map is all black when I use ffmpeg to extract depth map?

Question

Today, I use kinect to record a video. And then I use Azure Kinect Viewer to check the depth(like figure in the below).

Snipaste_2023-09-02_14-54-35

But when I use ffmpeg to extract all the depth map, I found that all of that are black. I don't know why?

Snipaste_2023-09-02_14-53-23

This is my cmd log:

PS D:\下载的内容\ffmpeg-master-latest-win64-gpl-shared\ffmpeg-master-latest-win64-gpl-shared\bin> .\ffmpeg -i D:\recordDk\output_Imu_10s.mkv -map 0:1 -vsync 0 depth%04d.png ffmpeg version N-111873-gfba4546175-20230901 Copyright (c) 2000-2023 the FFmpeg developers   built with gcc 13.1.0 (crosstool-NG 1.25.0.196_227d99d)   configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-w64-mingw32- --arch=x86_64 --target-os=mingw32 --enable-gpl --enable-version3 --disable-debug --enable-shared --disable-static --disable-w32threads --enable-pthreads --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --disable-libpulse --enable-libvmaf --disable-libxcb --disable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-chromaprint --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libkvazaar --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librist --enable-libssh --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libvpl --enable-openal --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --enable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --disable-libdrm --disable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags= --extra-libs=-lgomp --extra-version=20230901   libavutil      58. 17.100 / 58. 17.100   libavcodec     60. 24.100 / 60. 24.100   libavformat    60. 11.100 / 60. 11.100   libavdevice    60.  2.101 / 60.  2.101   libavfilter     9. 11.100 /  9. 11.100   libswscale      7.  3.100 /  7.  3.100   libswresample   4. 11.100 /  4. 11.100   libpostproc    57.  2.100 / 57.  2.100 -vsync is deprecated. Use -fps_mode Passing a number to -vsync is deprecated, use a string argument as described in the manual. [matroska,webm @ 00000200d8cfa9c0] Unknown/unsupported AVCodecID S_K4A/IMU. [matroska,webm @ 00000200d8cfa9c0] Could not find codec parameters for stream 0 (Video: mjpeg (MJPG / 0x47504A4D), none(bt470bg/unknown/unknown), 1280x720): unspecified pixel format Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options [matroska,webm @ 00000200d8cfa9c0] Could not find codec parameters for stream 3 (Subtitle: none): unknown codec Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options [matroska,webm @ 00000200d8cfa9c0] Could not find codec parameters for stream 4 (Attachment: none): unknown codec Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options Input #0, matroska,webm, from 'D:\recordDk\output_Imu_10s.mkv':   Metadata:     title           : Azure Kinect     encoder         : libmatroska-1.4.9     creation_time   : 2023-09-02T01:53:51.000000Z     K4A_DEPTH_DELAY_NS: 0     K4A_WIRED_SYNC_MODE: STANDALONE     K4A_COLOR_FIRMWARE_VERSION: 1.6.110     K4A_DEPTH_FIRMWARE_VERSION: 1.6.80     K4A_DEVICE_SERIAL_NUMBER: 000531513312     K4A_START_OFFSET_NS: 200011000   Duration: 00:00:10.27, start: 0.000000, bitrate: 523161 kb/s   Stream #0:0(eng): Video: mjpeg (MJPG / 0x47504A4D), none(bt470bg/unknown/unknown), 1280x720, SAR 1:1 DAR 16:9, 15 fps, 15 tbr, 1000k tbn (default)     Metadata:       title           : COLOR       K4A_COLOR_TRACK : 19076185248593246       K4A_COLOR_MODE  : MJPG_720P   Stream #0:1(eng): Video: rawvideo (b16g / 0x67363162), gray16be, 1024x1024, SAR 1:1 DAR 1:1, 15 fps, 15 tbr, 1000k tbn (default)     Metadata:       title           : DEPTH       K4A_DEPTH_TRACK : 391323405650020169       K4A_DEPTH_MODE  : WFOV_UNBINNED   Stream #0:2(eng): Video: rawvideo (b16g / 0x67363162), gray16be, 1024x1024, SAR 1:1 DAR 1:1, 15 fps, 15 tbr, 1000k tbn (default)     Metadata:       title           : IR       K4A_IR_TRACK    : 436026307751124134       K4A_IR_MODE     : ACTIVE   Stream #0:3(eng): Subtitle: none (default)     Metadata:       title           : IMU       K4A_IMU_TRACK   : 740337297503450518       K4A_IMU_MODE    : ON   Stream #0:4: Attachment: none     Metadata:       filename        : calibration.json       mimetype        : application/octet-stream       K4A_CALIBRATION_FILE: calibration.json Stream mapping:   Stream #0:1 -> #0:0 (rawvideo (native) -> png (native)) Press [q] to stop, [?] for help Output #0, image2, to 'depth%04d.png':   Metadata:     title           : Azure Kinect     K4A_DEVICE_SERIAL_NUMBER: 000531513312     K4A_START_OFFSET_NS: 200011000     K4A_DEPTH_DELAY_NS: 0     K4A_WIRED_SYNC_MODE: STANDALONE     K4A_COLOR_FIRMWARE_VERSION: 1.6.110     K4A_DEPTH_FIRMWARE_VERSION: 1.6.80     encoder         : Lavf60.11.100   Stream #0:0(eng): Video: png, gray16be(progressive), 1024x1024 [SAR 1:1 DAR 1:1], q=2-31, 200 kb/s, 15 fps, 15 tbn (default)     Metadata:       title           : DEPTH       K4A_DEPTH_TRACK : 391323405650020169       K4A_DEPTH_MODE  : WFOV_UNBINNED       encoder         : Lavc60.24.100 png [out#0/image2 @ 00000200d8d3cb80] video:19102kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown frame=  153 fps=0.0 q=-0.0 Lsize=N/A time=00:00:10.26 bitrate=N/A speed=19.7x

Answer

Jianxin xiao Greetings!

The viewer normalizes the depth based on depth modes min and max depth so that the entire 16bit depth range is used. Then it uses the following code to colorize.

static inline BgraPixel ColorizeBlueToRed(const DepthPixel &depthPixel,
                                              const DepthPixel &min,
                                              const DepthPixel &max)
    {
        constexpr uint8_t PixelMax = std::numeric_limits::max();

        // Default to opaque black.
        //
        BgraPixel result = { 0, 0, 0, PixelMax };

        // If the pixel is actual zero and not just below the min value, make it black
        //
        if (depthPixel == 0)
        {
            return result;
        }

        uint16_t clampedValue = depthPixel;
        clampedValue = std::min(clampedValue, max);
        clampedValue = std::max(clampedValue, min);

        // Normalize to [0, 1]
        //
        float hue = (clampedValue - min) / static_cast(max - min);

        // The 'hue' coordinate in HSV is a polar coordinate, so it 'wraps'.
        // Purple starts after blue and is close enough to red to be a bit unclear,
        // so we want to go from blue to red.  Purple starts around .6666667,
        // so we want to normalize to [0, .6666667].
        //
        constexpr float range = 2.f / 3.f;
        hue *= range;

        // We want blue to be close and red to be far, so we need to reflect the
        // hue across the middle of the range.
        //
        hue = range - hue;

        float fRed = 0.f;
        float fGreen = 0.f;
        float fBlue = 0.f;
        ImGui::ColorConvertHSVtoRGB(hue, 1.f, 1.f, fRed, fGreen, fBlue);

        result.Red = static_cast(fRed * PixelMax);
        result.Green = static_cast(fGreen * PixelMax);
        result.Blue = static_cast(fBlue * PixelMax);

        return result;
    }

Reference :

https://github.com/microsoft/Azure-Kinect-Sensor-SDK/blob/95f1d95f1f335b57a350a80a3a62e98e1ee4258d/tools/k4aviewer/k4adepthpixelcolorizer.h#L35

Also, see Using third-party tools and let me know if that helps.

Share via

Why depth map is all black when I use ffmpeg to extract depth map?

1 answer

Your answer