SYS · 01 · INGEST backend & distributed systems · 4+ years · Kolkata, India

I move
terabytes, safely.

Senior Software Engineer

$ who   senior software engineer · backend & distributed systems · entiovi
$ build  java · spring boot · kafka/rabbitmq · redis · k8s · airflow
$ ship   20k+ vehicles live · 3 tb+ pipelines · 99% accuracy

See the work Download résumé
metrics · production updated · prod
M·01
20k+
Vehicles tracked live — fleet scaled 2× on Spring Boot + Redis
M·02
99%
Real-time telemetry accuracy under bulk load
M·03
3 TB+
Processed through middleware-controlled ETL
M·04
20+
Global sources unified — World Bank, IMF, GDELT…

SYS · 02 · VALIDATE

The readme.

What I build, the backend discipline underneath it, and the rules I build by.

In [1]: whoami

Rohan Singh — backend engineer at the core: 4+ years designing Java / Spring Boot microservices, event-driven middleware on Kafka, RabbitMQ and ActiveMQ, and Redis-backed real-time systems — the kind that scaled a live fleet platform from 10,000 to 20,000+ vehicles at 99% accuracy. That backend discipline is what makes my data platforms work: middleware-controlled ETL processing 3 TB+ across 20+ global sources on GCP and Azure, stateless services that scale horizontally behind the load balancer, and privacy enforced at the column level for GDPR/CCPA. Most recently I've carried the same system-design instincts into GenAI — RAG over self-hosted LLaMA 70B, multi-agent platforms. New stack, same engineering.

In [2]: head(principles)
  • Backend first — every "data platform" I ship is a distributed system underneath: control planes, queues, idempotent workers, stateless services that scale horizontally.
  • Resilience by design — systems must be recoverable and stateful; Parquet checkpoint staging eliminates data loss on failure.
  • Abstraction over repetition — build middleware that kills manual configuration and handles unbounded scale gracefully.
  • Security at the core — privacy is enforced dynamically at the column level, never treated as an afterthought.
  • Stacks change, fundamentals don't — strong DSA, LLD/HLD and design patterns are why picking up Airflow, then vector search, then agentic AI was fast. New tools land on old foundations.

SYS · 03 · TRANSFORM

Case studies.

Five platforms, five hard constraints — real-time scale, privacy, fragmentation, and data residency. All backend systems underneath.

stream: fleet-telemetry Dec 2022 — 2025

Real-Time Fleet & Transport Platform

Java microservices tracking 20,000+ vehicles live — capacity doubled with zero loss in accuracy.

Entiovi Technologies · Backend engineer → senior

JavaSpring Boot ActiveMQRedis SSE + Redis Pub/SubKubernetes MySQL
expand · problem & impact

Problem → Approach

A live fleet platform had to double capacity — from 10,000 to 20,000+ vehicles streaming telemetry — without dropping accuracy or adding latency under bulk load.

  • Re-architected ingestion, messaging and caching paths — partitioned workloads and tuned thread/connection pools to double fleet capacity.
  • Asynchronous Spring Boot + ActiveMQ middleware with Redis cache optimization for reliability, latency and throughput under bulk fleet load.
  • Real-time status streaming over SSE backed by Redis Pub/Sub — services stay stateless and scale horizontally behind the load balancer.
  • Docker + Kubernetes with replica scaling, health probes and rolling deployments for zero-downtime releases.

Impact

Fleet capacity2×
Data accuracy99%
Deploy downtime0
transform: privacy-etl Aug 2024 — Present

Privacy-First ETL Platform

Middleware-controlled ETL platform with built-in differential privacy at terabyte scale.

Entiovi Technologies · Sole architect & senior data engineer

Apache AirflowPython Spring BootParquet MySQLDifferential Privacy
expand · problem & impact

Problem → Approach

Customers needed a privacy-preserving ETL platform that could ingest unbounded data volumes, run resumable jobs without losing state on failure, and enforce GDPR/CCPA at the column level — all without code changes per source.

  • Spring Boot middleware drives Airflow runtime config — bulk vs incremental, source allowlists, schedules — so onboarding a new source needs zero code.
  • Parquet-based checkpoint staging: peak memory dropped 40%+, jobs resume from the last successful checkpoint after failure.
  • Differential Privacy engine with dynamic PII masking and column-level encryption enforced per-policy.

Impact

Peak memory−40%
ComplianceGDPR / CCPA
Data loss on failure0%
transform: semantic-intel 2024

Multi-Source RAG Platform

GCP-hosted platform unifying 3 TB+ of US legislative, Senate and court data behind a RAG layer.

Entiovi Technologies · Sole architect & data-engineering team lead

Apache AirflowGCP MinIO (S3)Qdrant RabbitMQDocling LLaMA 70B
expand · problem & impact

Problem → Approach

Researchers needed unified, semantically searchable access to fragmented public records — lobbyist filings, Senate disclosures, legislative codes, and court documents — across rate-limited APIs and incompatible schemas.

  • Two-phase Airflow pipeline: raw sync (bulk / incremental / skip-step) into MinIO, then normalization, metadata enrichment and DB sync.
  • RabbitMQ-decoupled RAG pipeline — Docling chunks PDFs into dense + sparse vectors with sequence tracking for high-fidelity retrieval from Qdrant.
  • GCP infrastructure architected for sustained large-volume processing with graceful API throttling.

Impact

Data volume3 TB+
Sync time−40%
RetrievalDense + sparse
transform: resilience-intel 2024

Resilience Intelligence Platform

World Bank × Stanford platform unifying 20+ global data sources into a canonical schema.

Entiovi · World Bank · Stanford — Lead data engineer

Apache AirflowPython AzureMinIO OpenTelemetryApache Superset
expand · problem & impact

Problem → Approach

Macroeconomic and policy researchers were stitching together 20+ heterogeneous sources (World Bank, IMF, GDELT, WTO, Meltwater) by hand — slow, brittle, no observability, no DS-model integration path.

  • Parallelized DAGs per source family with canonical-schema normalization downstream, cutting sync time 30%.
  • DS model execution integrated inline so analysts run scoring jobs against fresh data without leaving the platform.
  • Full OpenTelemetry observability — every stage emits traces, metrics and logs to a unified backend.

Impact

Sources unified20+
Sync time−30%
ObservabilityEnd-to-end OTel
transform: on-prem-rag 2024

On-Prem RAG Platform

Self-hosted GenAI platform — RAG over 3,000+ institutional documents on on-prem LLaMA 70B.

Entiovi Technologies · Backend & GenAI engineer

LLaMA 70B (Ollama)LangChain QdrantRabbitMQ Python
expand · problem & impact

Problem → Approach

An institution wanted conversational access to 3,000+ internal documents but could not send sensitive content to hosted LLM APIs — data residency and privacy were hard requirements.

  • Self-hosted LLaMA 70B via Ollama for fully on-prem inference.
  • RabbitMQ-decoupled microservices keep ingestion, embedding, retrieval and generation independently scalable.
  • RAG over Qdrant with chunk-level provenance so every answer cites its sources.

Impact

Documents indexed3,000+
HostingFully on-prem
PrivacyZero data egress

SYS · 04 · ORCHESTRATE

One pipeline, zero leaks.

The reference architecture behind the case studies — a Spring Boot control plane driving Airflow, stateless services scaling behind the load balancer. Scroll to run it.

⌁ apis ▤ files ≋ streams ⇥ ingest ▦ stage · parquet ⛨ mask · pii ◫ store ➤ serve
~/pipeline ⟶ inspector beat 1/4 · ingest
$ airflow dags trigger ingest_all [ok] worldbank · 214,332 rows [ok] imf · 88,102 rows [ok] gdelt · 1.2M events [ok] courtlistener · 44,910 docs [..] meltwater · syncing…
trace 8f3a · pipeline.run ├─ extract ▇▇▇▇▇▇▇▇ 612ms ├─ normalize ▇▇▇▇ 341ms ├─ mask_pii ▇▇ 118ms └─ load ▇▇▇▇▇▇ 455ms # p99 1.9s · 0 errors · otel → grafana
$ docling chunk court_doc_114.pdf chunks: 482 · dense + sparse qdrant upsert · sequence ids ✓ retrieval top-k=8 · 96ms # every answer cites its chunks
$ pipeline status checkpoints ✓ parquet staged pii masking ✓ column-level compliance ✓ gdpr / ccpa data loss 0 bytes
C·01 · INGESTED
3 TB+
C·02 · SOURCES
20+
C·03 · SYNC TIME
−40%

SYS · 05 · COMPILE

The toolchain.

Backend core first — everything else compiles on top of it. Flip a card for the receipts behind each tool.

SYS · 06 · SHIP

Shipping since 2021.

From fintech intern to backend lead — real-time fleet telemetry, terabyte-scale privacy infrastructure, and the platforms in between.

2024

Senior Data Engineer

Xafe.ai (Entiovi Technologies) · Kolkata, India

Aug 2024 — Present

Sole architect of a middleware-controlled ETL platform designed for unbounded data scale — a Spring Boot control plane driving Airflow, with built-in differential privacy and column-level enforcement.

  • Runtime-configurable bulk / incremental modes via Spring Boot middleware — onboarding new sources requires zero code changes.
  • Real-time pipeline & job status streamed to clients over SSE backed by Redis Pub/Sub — services stay stateless and scale horizontally behind the load balancer.
  • Parquet-based checkpoint staging eliminates data loss on failure and reduced peak memory by 40%+.
  • Differential privacy engine with dynamic PII masking and column-level encryption for full GDPR / CCPA compliance.
  • Containerized on Docker + Kubernetes with replica scaling, health probes and rolling deployments — zero-downtime releases.
Spring BootApache Airflow Redis Pub/SubKubernetes PythonParquet MySQL
2024

Senior Software Engineer

Entiovi Technologies · Kolkata, India

Apr 2024 — Present

Architected and led the data-engineering team behind multiple GCP / Azure platforms — the Semantic Intelligence Platform, the Resilience Intelligence Platform, and Namaiste.ai.

  • Two-phase Airflow pipeline ingesting 3 TB+ of US legislative and court data into MinIO with normalization & DB sync — sync time improved 40%.
  • RabbitMQ-decoupled RAG pipeline using Docling for dense / sparse chunking and Qdrant for vector retrieval.
  • Unified 20+ global sources into a canonical schema — parallelized DAGs cut sync time 30%, full OpenTelemetry observability.
  • Built self-hosted GenAI platform — RAG over 3,000+ docs via on-prem LLaMA 70B + LangChain + Qdrant.
Apache AirflowGCP AzureQdrant RabbitMQLLaMA 70B LangChain
↳ promoted after 16 months
2022

Software Engineer

Entiovi Technologies · Kolkata, India

Dec 2022 — Apr 2024

Built real-time vehicle tracking and async middleware that significantly improved reliability and throughput across bulk fleet operations — scaling the platform from 10,000 to 20,000+ vehicles.

  • Doubled fleet capacity by re-architecting ingestion, messaging and caching paths — partitioned workloads and tuned thread/connection pools.
  • Achieved 99% data accuracy on real-time telemetry across complex bulk fleet ops.
  • Designed async middleware in Spring Boot + ActiveMQ with Redis cache optimization.
Spring BootActiveMQ RedisMySQL
show earlier ↓
2021

Software Development Intern

HighRadius · Bhubaneswar, India

May 2021 — May 2022

Full-stack feature development in Java EE and React for enterprise fintech clients — production engineering and client delivery.

Java EEReact REST APIs
2018–22

B.Tech, Electronics & Instrumentation Engineering

Kalinga Institute of Industrial Technology (KIIT) · Bhubaneswar

2018 — 2022

CGPA 8.6 / 10. Engineering fundamentals plus self-driven specialization into software and data systems.

SYS · 07 · SINK

Always up for a good problem.

Pick a channel. If you're building something at the intersection of data, privacy, and AI — I want to hear about it.

~/contact ⟶ available channels online
$ mail --to rhnsngh1999@gmail.com via Email $ open --github github.com/rohan1769 via GitHub $ open --linkedin linkedin.com/in/rohan-singh via LinkedIn
$ say-hello
Say hello
✓ copied to buffer