pass data to a stream without serializing it

[mderazon]

Right now I have two streams on the same logger - stdout and another one that sends logs to a third party service (opbeat.com).

Opbeat agent expects a standard js error object (or subclass) to be passed so it can show it with all the context in their dashboard.

Problem is that if I use Bunyan's std error serializer, error object gets transformed into a plain object with some properties from the original error. I then have to try and recreate the error from that plain object just so it could be sent to Opbeat correctly (it actually proved to be more difficult than I thought - mderazon/bunyan-opbeat#1).

Using Bunyan's std serializer is nice because then you get all kind of cool formatting on stdout, but at the same time we lose the ability for other streams to process the original data.

I don't know if this need has come up before, I couldn't find it in any other issue but I don't think it's limited to this specific stream.

Possible solutions for this:

1. As a Bunyan user, create another separate logger that doesn't use any serializers and use it in addition to the stdout logger.
2. Add a serialize boolean flag to the stream that if set to 'false', tells Bunayn to pass data as is and not use serializers. (default should be 'true' so maybe a better name is needed)
3. Add the ability to define serializers on the stream level in addition to the global level.

Is there any nice and clean solution I can use that doesn't require any change to Bunyan ?

[jbcpollak]

I have a similar issue. Our bunyan-logsene stream deserializes the data, manipulates it, and then passes it to the lower-level logsene library (see line 36 of https://github.com/6RiverSystems/bunyan-logsene/blob/master/index.js)

I would personally like to have the req/res/err serializers do their thing automatically, but I would like my 'stream' to receive the Javascript object containing the entry, rather than the serialized entry. I'm not sure if that is practical or not.

[jbcpollak]

After digging more, I think you can achieve this by specifying type: raw on your stream definition, which will pass the wrap object to the stream:

streams: [
    {
        stream: MyStream,
        type: 'raw'
    }
]

I do think the stream itself should be able to declare it requires raw data, rather than having to declare it at the configuration level.

[pmcnr-hx]

I can confirm that type: 'raw' does what you're looking for. I've successfully created a TransformStream where I manipulate raw Bunyan log records.

[jbcpollak]

It seems to me like type: 'raw' should be an attribute of the Stream class itself, the end user probably shouldn't have to configure it themselves.

[kayjtea]

I have the exact same problem integrating with Azure Application Insights, which also needs the Error object in order to log an exception.

Specifying type: 'raw' passes the JSON record to the stream, but it appears the properties of the record are still run through the serializers, and consequently the err property arrives as a plain object. Even when you don't specify any serializers (which is itself problematic because serializers are specified at the logger level, meaning you can't bypass serializers on a stream-by-stream basis for multi-stream loggers), sending in an Error object as the first parameter to any of the logging methods will still run the Error through the standard error serializer before it gets to a type: 'raw' stream.

This hack will give the actual Error object to a type: 'raw' stream, but it breaks the serializer contract: serializers: {err: function(input) {return input;}}

Is there another way for the stream to consistently get the actual Error object that was logged without breaking the serializer contract?

[mderazon]

@kayjtea If it's any consolation, this is still a problem for me as well.
I ended up doing the following:

1. Define stream as raw
2. Not define any serializer for err
3. Pass err inside an object instead of first argument.

This way, err doesn't get touched at all.

I still think that serializers should be defined on the stream level instead of globally, not sure why it wasn't designed like that.
But like you said, even that wouldn't solve it completely, because it still gets treated when you pass it as first argument...

[pwmckenna]

@trentm I would be up for taking a first pass at adding stream specific serializers if you would be in favor of that feature. Any potential complications to look out for?

[pwmckenna]

@mderazon is this the type of api you'd be looking for? #501

[kayjtea]

@pwmckenna If the stream-specific serializers exactly specify the set of serializers to use for the stream AND this set is allowed to be empty then this would address "pass data to a stream without serializing".

[matthewadams]

FYI, I entered #553 to support serializers per stream.