I'm currently the developer behind the site Twitch Links (NSFW). I'm the sole creator and I'm starting to feel conceed about a simple query that is the back bone of the web site (I'm by no means a DBA).
To quickly summarize, it's collecting URL's from Twitch.TV chat. It's grabbed around 700k in 3 weeks. The site visually displays the most recent Imgur's and Youtube videos. I have a single "links" table, with a "favourites" table where I stored user favourited links (Id, LinkId, UserId).
Here's the query:
SELECT
Id,
URL,
CapturedOn,
Chael,
(SELECT COUNT(*) FROM favourites WHERE LinkId = links.Id) AS NumFavourites, # Is this bad per row?
Type,
Data,
CapturedBy
FROM links
WHERE
Chael LIKE "%%" AND # Can sometimes be populated with a single value, e.g. "Chael like '%riotgames%'"
Type IN ('Imgur', 'YouTube') AND # Can sometimes be "Type LIKE '%Imgur%'" or "Type LIKE '%Facebook%'" - there are about 20 different types.
Deleted = 0 AND # Out of 500k rows, about 100 will be deleted.
Id > 0 AND # Will be set to a high ID after first view to only retu latest rows.
Data IS NOT NULL # Exclude badly parsed links.
ORDER BY
Id DESC LIMIT 40;
Here's the explain plan:
Here are the keys:
PRIMARY KEY (`ID`),
KEY `idx_links_Chael` (`Chael`),
KEY `idx_links_Type` (`Type`),
KEY `idx_links_CapturedOn` (`CapturedOn`)
The "Chael LIKE" statement can change when requesting links for a particular users favourite chaels only. This becomes:
Chael IN (SELECT CONCAT('#', ChaelName) FROM chaelfavourites WHERE UserId = X) AND
There is an index on the "favourites" table on its Id, UserId, LinkId columns.
There is an index on "chaelfavourites" for column "UserId".
Here are my questions about it:
-
Why is it showing that 382k rows are being scaed? Shouldn't the order by ID DESC LIMIT 40 always limit it to much less, ie, find 40 rows matching the WHERE then stop?
-
When the sites been ruing a year, and it has say 12m rows, will this query still scale? Will MySQL be clever enough to realize with the "ORDER BY Id DESC" to start at the end of the data set on disk and work backwards?
I've been thinking to flatten this query completely to a more view style table "SELECT l.* FROM validlinks v JOIN links l ON l.Id = v.LinkId ORDER BY Id DESC LIMIT 40". But will that simple join be worth it?
Any input would be appreciated.

