Sleep is a 1st-level spell that affects 5d8 (+2d8/level if upcast) hit points' worth of creatures within a 20-foot radius burst anywhere within 90 feet of the caster, giving said creatures the Unconscious condition (which itself is a combination of Incapacitated and Prone). It requires no save, lasts a whole minute with no further actions spent, and drops the affected creatures prone giving attacks against them Advantage and thus more likely to be struck.
Hold Person is a 2nd-level spell that, at the cost of one action, completely paralyzes (again, Incapacitated with extra flavor) any humanoid target within 60 feet of the caster if they fail their Wisdom saving throw. You need not do anything additional to maintain effect besides concentrate on the spell; attacks against the target also have Advantage and, better still, automatically suffer a critical hit when successfully struck.
By comparison, Maddening Whispers is on its face actually not that great. It has a diminished range (30 feet) compared to the alternatives; you need to continually use your action every turn to maintain the effect (which is also broken when they take damage), basically removing you and the target from combat; and it can't be upcast, so it only ever targets one creature at a time.
But it has a handful of perks to its benefit, such as:
- ...it requires a Charisma saving throw, which isn't uncommon, but almost certainly a lower stat for most creatures.
- ...it doesn't specifically target any particular demographic or creature type, in the way Hold Person predictably targets humanoids.
- ...it has no material components, so it can be cast at almost any time.
- ...it only requires one save, where something like Hold Person allows for saves every turn.
Taking an objective look at it... it's a speech-dependent Hold Monster alternative that you can acquire three spell levels lower than you'd normally be able. Yeah, I would have to agree, that's pretty insanely overpowered... provided, of course, that no one is stupid or reckless enough to attack the incapacitated monster before any of its partners and/or minions are defeated first. But in that rare eventuality, at least it can be recast again immediately.
If it is overpowered, what might be some reasonable adjustments a willing GM could offer to make it more reasonable?
Any changes to the spell's effects fundamentally make it a different spell all together. You certainly can't change the "whispering" action every turn, or it stops being "Maddening Whispers". If you limit the creature types it can affect, it becomes a more niche Hold Person variant. If you limit its range any further, you practically have to be on top of your target to make effective use of it. And it's also getting dangerously close to the nature of a "save-or-suck" spell, so if you add more saves, you increase the chance of an already fragile spell being more easily broken.
Honestly, aside from just barring it at the table for how grossly powerful its effect is, my only real suggestions for any attempt at "balancing" this spell would be to either require some sort of material component or focus (maybe a coin on a string to mimic a pendulum for invoking a trance-like hypnosis), or to increase its spell level to maybe 3rd. Beyond that, though... I don't have many ideas. It seems like it was designed to be broken from the onset.