If your custom protocol is implemented using a C or C++  library, you can interface with the library using Node.js addons. 

See https://nodejs.org/api/addons.html for more details.