Since I've realized this is a struggle a lot of people have but don't know how to fix, I though I'd share my solution. I don't look at this as a permanent fix because it is a complicated process and takes far too many steps. But, until there is a better solution, I thought I'd share this for anyone who might need it.
Using this process you can remove CS's from your Universe. This will not delete the character. They will still be viewable and editable by the creator, but it will remove them from the characters tab in the universe. They might also still show up when tagging characters in posts but it at least removes them from the Universe itself.
Anyway...
You have to go to the roleplay management page. Which, there is no direct link to this anymore (which I asked for it to be brought back on the suggestions forum because it's super useful). How I get to this page is I got to my private messages then I click the tab roleplays which will be in the navigation bar that says "Overview, profile, board preferences, private messages, characters, roleplays, manage groups"
So you go to the roleplays tab. It'll open up a page that looks something like this...
If you scroll down you'll get a list of all the RPs you've ever joined and/or created. If it's one you joined it'll basically just have links to those RPs, settings, characters, etc. with an option to abandon. But if it's one you've created it'll list every character in that RP with options next to those characters' names to Approve or Remove those characters that looks like this...
From there just choose remove next to the character you wish to remove. You're also able to add characters back if someone returns or whatever else. They won't be removed from the RP forever, but for everyone but the GM much of their presence will be gone.
I tested this earlier with one of my own characters in an RP and this function still works. If this is confusing or you have any issues, feel free to reach out and I'll help you to the best of my ability! At least it's a work around until a more permanent structure gets added. And hopefully there will be an easier way to reach the roleplay management page in the future.
Hope that helps :)