Details
-
Enabler
-
Could have
-
None
-
Data Processing
-
-
-
-
Description
In SP-3626 we developed the infrastructure to easily benchmark and profile the visibility receive pipeline, as a single instance in a single repository, and to do so with hardcoded data dimensions.
While some preliminary analysis was done, the parameter space is quite large and there are many moving parts that impact each other so an in-depth analysis was deferred for a future ticket, this one.
There is already some low hanging fruit in the receiver, such as the minimisation of memory copies and data type conversions, but once they've been addressed there may be more. YAN-1418 encompasses some of this, but probably isn't straight forward to implement.
Analysis may want to explore (though not limited to this):
- How aggregation responds to the memory store size and state. e.g. We don't want a single aggregated payload to be more than half the memory store size and we can only have a single payload in flight in that case which doesn't allow for any buffering.
- If processors can't keep up with reception, then the plasma store will fill up. For our existing processors, does this occur, and at what rate? Think about how we might handle this case gracefully
(These ideas are taken from the initial testing, see YAN-1574, for more details)