...specials can operate on non-imperials inhabitants, while a building does nothing for minor species.
As far as I know, a building should work on any object that meets its effectsgroups' scope conditions while its activation condition is met. Being player or non-player owned shouldn't matter.
Perhaps we can think of them this way: Specials are intrinsic to the planet, and remain unless the planet is utterly destroyed
Special can be applied to any object, not just planets; buildings could have a special attached, for example, or systems or ships or fleets. But yes, in practice, specials just add some effects groups to the object they're attached to. They could also be checked for by conditions, to affect scope / activation of other effects, though I don't think that's done much yet.
buildings are subject to destruction via bombardment.
Sounds like you might want to add a structure meter to buildings, to track their progress toward destruction?