Generating depth map from point cloud

I am trying to generate a depth map from the point cloud. I know that I can project the point cloud to the image plane, however there is already a function (ScreenCoordinateToWorldNearestNeighbor) in the TangoSupport script that finds the XYZ point given a screen coordinate.

I am unable to get this support function to work, and it seems that one or more of my inputs are invalid. I am updating my depthmap texture in the OnTangoDepthAvailable event.

    public void OnTangoDepthAvailable(TangoUnityDepth tangoDepth)
{
    _depthAvailable = true;

    Matrix4x4 ccWorld = _Camera.transform.localToWorldMatrix;
    bool isValid = false;
    Vector3 colorCameraPoint;
    for (int i = 0; i < _depthMapSize; i++)
    {
        for (int j = 0; j < _depthMapSize; j++)
        {
            if (TangoSupport.ScreenCoordinateToWorldNearestNeighbor(
                _PointCloud.m_points, _PointCloud.m_pointsCount,
                tangoDepth.m_timestamp, 
                _ccIntrinsics,
                ref ccWorld, 
                new Vector2(i / (float)_depthMapSize, j / (float)_depthMapSize),
                out colorCameraPoint, out isValid) == Common.ErrorType.TANGO_INVALID)
            {
                _depthTexture.SetPixel(i, j, Color.red);
                continue;
            }

            if (isValid)
            {
                //_depthTexture.SetPixel(i, j, new Color(colorCameraPoint.z, colorCameraPoint.z, colorCameraPoint.z));
                _depthTexture.SetPixel(i, j,
                    new Color(UnityEngine.Random.value,UnityEngine.Random.value,UnityEngine.Random.value));
            }
        }
    }
    _depthTexture.Apply();
    _DepthMapQuad.material.mainTexture = _depthTexture;
}

If I had to guess, I would say that I am passing in the wrong matrix (ccWorld). Here is what it says in the documents for the matrix param:

Transformation matrix of the color camera with respect to the Unity world frame.

I know that the support function doesn't retu successfully because the depthTexture is always red.

Any ideas? Also I noticed that the performance is pretty bad, even when my depth map is 8x8. Should I not be updating the depthmap when ever new depth data is available (inside OnTangoDepthAvailable)?