Like the others have said, color matched grille, no door ding guards are a sign of an XE. That is, if the previous owner has not changed out parts.
If before 07, look at the door jam and check what size tire is listed. If it lists an 18 inch wheel, it is either an SE or LE. If it lists and 17 inch wheel that is not a 285 series, than good chances it is an XE.
Things changed 08+, so im not sure how to be totally sure with those years.