OnPointerEnter and OnPointerExit not registering

I am working on an inventory system and at the moment I am looking into swapping inventory spots. Currently I have a hierarchy as :

Now what I have noticed is when I am swapping it only works if I was to swap say for example Inventory Slot 1 with Inventory Slot 4. BUT if I am to swap it back for example Inventory Slot 4 with Inventory Slot 1 my OnPointerEnter and OnPointerExit do not register, is this supposed to be right? This seems to be a hierarchy issue and I don't know how to solve this.

GIF of the swapping and a screenshot of my Inventory Slot GameObject :

http://imgur.com/kS1C1TL - GIF

Code of my OnPointers :

public void OnPointerEnter(PointerEventData data){
    itemCurrentlyOn = gameObject;
}

public void OnPointerExit(PointerEventData data){
    itemCurrentlyOn = null;
}

public void OnPointerUp(PointerEventData data){
    // IF we are dragging an item.
    if(itemBeingDragged != null){
        print (itemCurrentlyOn);
        // IF we release the mouse button on an actual inventory slot.
        if(itemCurrentlyOn != null && itemCurrentlyOn != gameObject){
            // Swap the information in the 2 inventory slots.
            GetComponentInParent<Inventory> ().SwapInventorySlots (itemBeingDragged.GetComponent<Inventory_Slot> (), itemCurrentlyOn.GetComponent<Inventory_Slot> ());
        }

        itemBeingDragged = null;
        // retu this gameobject to its original location.
        rectTrans.localPosition = localRectTrans;
    }
}

public void OnBeginDrag(PointerEventData data){
    // IF we have an item to drag.
    if(isItem){
        // Set the itemBeingDragged to this gameobject.
        itemBeingDragged = gameObject;
        // Set the current item we are hovered on to null.
        //itemCurrentlyOn = null;
    }
}