Broken is an informal term. Cards that don't work, or are bad for the game, are broken. This can mean the card is unplayable or literally does not work: Decipher infamously errataed Frenzy of Arrows (18U79) to spot an Orc archer, but there are no Orc archers in the entire game to spot. More often, it means that something is intolerably unpleasant to play against, either because it is overpowered or because it is an NPE. Broken isn't just applied to cards: it can also include particular combos or entire decks.
Decipher attempted to errata or X-List all of the broken cards, but the last sets, Rise of Saruman and Treachery & Deceit were only covered hastily, and very last set, Ages End was not covered at all.
Unofficial formats often X-List or errata broken cards. Fixing broken cards is one of the main goals of the PC Errata project.
Broken is also occasionally used to refer to cards that are so weak that they are useless: an infamous example is The Weight of a Legacy (1R282) . This usage is much more common in the specific than in the general: people may talk about how a nerf broke a deck, but when talking about "broken decks" in general, the implied meaning is overpowered or NPE decks.
This meaning of broken to mean nonfunctional, overpowered, unpleasant, and/or occasionally underpowered things is common to online game discussion in general.