In-Process vs RPC Model Inference
Two ways to call a trained model from a trading or research system — loading it directly into the calling process, or calling it over the network as a separate service — and the latency, isolation, and deployment trade-offs between them.
Calling a colleague at the next desk is instant, but if they're out sick the task stalls with them; calling a separate specialist team over email is slower per request, but that team can be scaled or upgraded independently without touching your desk. Serving a model's predictions faces the same choice between in-process inference, where the model is loaded directly into the calling application's own memory, and RPC (remote procedure call) inference, where the model lives in a separate service reached over the network.
In-process inference has essentially no communication overhead — a function call within the same memory space typically costs nanoseconds to low microseconds, which matters for a trading system reacting to ticks on a strict latency budget. Its cost is coupling: every application that wants the model must load its own copy, and updating the model means redeploying every one of those applications.
RPC inference decouples the model into its own service, reachable by any caller over the network, commonly gRPC or REST. A single upgrade is deployed once, and the service scales independently of any caller — but a fast local network round trip typically adds somewhere around 0.1 to 1 millisecond, and a request crossing availability zones can add several milliseconds more, often the dominant cost for models cheap enough to evaluate in microseconds themselves.
In-process inference trades deployment flexibility for near-zero call latency; RPC inference trades a network round trip, typically fractions of a millisecond to several milliseconds, for centralized, independently scalable model deployment.
Further reading
- Kleppmann, Designing Data-Intensive Applications, ch. on service communication