Design an inference batching system for a single GPU that can handle up to 100 inputs per batch while users wait synchronously, maximizing utilization under compute constraints.

To design an inference batching system for a single GPU with synchronous user waits and a maximum of 100 inputs per batch, we need to prioritize GPU utilization and minimize latency. The system will accept individual requests, queue them, and group them into batches of up to 100. Once a batch is full or a timeout is reached, it will be processed by the GPU. Results will be returned to the originating users synchronously. Key considerations include efficient batching logic to reduce idle time between batches, robust error handling for failed batches, and a design that can scale to multiple GPUs by distributing batches across them.