Plotly assigning colors based on label

I'm sure there's a really easy way to do this but I can't for the life of me figure it out. I'm doing a scatter graph in Plot_ly. Each point that I'm plotting has been assigned a label e.g. Pass, High, Low, Sigma high etc...

My problem is that I can't seem to fix what color plot_ly assigns to each label. I want Pass to always be White, High to always be Green and so on, but it seems to randomly assign colors each time.

Here is the code I'm using to plot

      p<-plot_ly(PT, 
             type='scatter', 
             x=XAVG, 
             y=YAVG, 
             mode= "markers",
             marker=list(size=3,bgcolor="#e5e5e5"),
             color = label,
             colors=c("yellow","red","cyan","magenta",'blue',"green3","white"),

             zeroline=F,
             text=paste("MEAS = ",PT$MEAS,"Head_ID = ",substr(PT$SHD_ID,2,5)))

Any help would be much appreciated as I've been stuck on this for ages!

Thanks!