HTML can't reach element with mouse/Can't click through parent divs

Ok, I have the following html code

<div id="absoluteWrapper"  style = "z-index:-11; pointer-events: none;">
    <div id="wrapper"  style = "z-index:-11; pointer-events: none;">
        <div id="leftWrap"  style = "z-index:-11; pointer-events: none;">
            <div id="miniWrapL"  style = "z-index:-11; pointer-events: none;">
                <div id="aboveMapPic"   style = "pointer-events: none;">
                    <img src="images/aboveMap.png" alt="" />
                </div>
                <div id="map"></div>
            </div>
        </div>



        <div id="rightWrap">

            <div id="miniWrapR">

            </div>

        </div>

    </div>
</div>

One of the "iermost" divs here, the one with the id of "map" harbors a map from googles map API and after extensive testing using javascript and the console I've realized my "click" events don't reach it (and thus I can't zoom in and out, which is a problem)

I have set the z-index of all its (some of them absolutely positioned) parents to absurdly low values and I have even tried setting pointer-events on them equal to none... still not working.

Do note that when I remove the div from this "stack" of parents clicking on the map, dragging it and zooming in and out works just fine.

I've tested this in both chromium and firefox.

Help me Obi-One Kenobi... etc

But seriously... do you guys have any suggestions as to what is causing this or how can I fix it ? I could add an overly over the div, but considering the fact that I need to click the actual div, since it communicates with the google mpas API... that would be a rather hard thing to do.