Capturereplay should share single file handle during playback

[fquinner]

Capturereplay currently opens a new filehandle for each subscribed-to symbol. This can get out of hand really fast and blow O/S limits. It should open a single filehandle and use that instead.

[dmagOM]

I've been giving this a bit of consideration, and there's really two options for how to proceed.

First of all, just to explain the current behaviour - each time capturereplay receives a new subscription (handled by subscriptionHandlerOnNewRequestCb), it opens a new instance of the playback file (effectively a new file handle). It then begins iterating the playback looking for messages which match the symbol which was subscribed. Each time the timer interval fires, a new message for the instrument is found and published, until the file ends or gets rewound. The net result is the playback plays through out of sync for different instruments to which

So the options:

• We can keep the existing behaviour by implementing file position tracking in the playback API, then substitute the new file handle for each instrument for this position.
• We can modify the behaviour to play all data back in order - on each timer interval, we iterate a set number of messages, publishing those for instruments which have been subscribed, and applying all to a basic instrument cache (for either all the instruments in the file (populated by the readSymbolsFromFile function.
  • This can be either done on a 'batch' (taking a set number of messages each interval) or 'real-time' (taking messages which would have been triggered in the past interval) basis.
  • Granularity of the timer for the event trigger could be increased, making near real-time playback possible.

I suspect the second option is probably more desirable at this stage.

Other notes:

• The playback header format is a bit light - it doesn't have a version or anything similar. Now might be a good time to expand it so we can more easily enhance without breaking in the future.
• It might be nice to implement some sort of file position tracking API anyway.