The more important thing for Spectres now is really their passive. I think most of us can agree that the Spectre passive is utterly useless. Giving Spectres a dex debuff would make the passive even harder to trigger. (Although I seriously don't see how it can be worse) So before giving this to Spectres, I would highly suggest gamigo rework Spectre passive. Something to do more damage or gather souls quicker will be nice. Would help with the whole Glad superiority thing.
My suggestion was to give spectres ability to ignore certain % of opponent's defense whenever using a soul consuming attack, it would increase their damage by a lot.