MediumDataSystem design

Design a document processing pipeline.

Asked at

Databricks

1Give yourself 5 minutes

2Answer out loud, not in your head

3Then compare with the answer below

Reference answer

A robust document processing pipeline can be designed with distinct layers for ingestion, processing, and storage. The data flow typically starts with a client uploading a document via an API gateway, which handles authentication, authorization, and generates a pre-signed URL for direct upload to object storage. This object storage serves as the repository for raw, unstructured documents. To decouple ingestion from processing, a message queue is employed. Worker services consume messages from the queue, retrieve documents from object storage, perform necessary processing (e.g., OCR, data extraction, classification), and then store the processed data or metadata in a database. The API gateway also exposes endpoints for clients to query the status of their document processing.

Hiring now

Databricks has 857 open roles right now

landed applies to roles like these for you, on each company’s own site, with answers written for the role.

Get 5x interviews10 free auto-applies. No card.

Director, Enterprise (NSW)
Sydney, Australia
Engineering Manager - Notebook Dataplane
Mountain View, California
Sr. Research Scientist - Agent Flywheel
New York City, New York

Or practice this question first

Next question