The Silent Librarians are one of many groups that Hristo Tiktoffen was working for —feeding information to— as he worked on repairing Castle Heterodyne.
Other groups Tiktoffen mentions ✣ include major powers such as Wulfenbach, the Master of Paris, and Queen Albia of England; but also underground organizations such as the Order, who sent him Zola Malfeazium before he was ready for her (he may never have admitted to being ready until had control of the Castle and no longer had a use for them).
It's possible that Silent Librarians are another major force otherwise unmentioned, or that they are associated with the sovereign organization we do know about, the Incorruptible Republic of the Immortal Library of the Grand Architect.