A lot of what makes something casual is nuance and detail.  Structure and material, to me, are the deciding factors.  A suit can be too casual for the office if the jacket is unstructured.  A pair of trousers, even if they are wool, can be casual if they are uncreased and are not hemmed at an appropriate length. A knit or skinnier tie can be casual.  Sidebar: Despite what many men think, a skinnier tie is not “old-fashioned” or “from the 50s” – it can be quite modern – it is just more casual.

At the same time, clothing that is too casual for Monday through Thursday can be worn on casual Friday.  That unstructured cotton suit from J. Crew? Perfect for casual Friday.  Ignore coworkers jabbing you and asking you why you are so dressed up—you know better.  That pair of uncreased khakis that hits the top of your shoe?  Casual Friday.  Knit tie? Casual Friday.