An operations engineer clicks “allocate VLAN” for a new customer service, and the system behind that click has a fraction of a second to answer a question that, in our architecture, has no single authoritative source: is this number actually free? No one database knows for sure. There are three of them — each owned by a different team, none transactionally consistent with the others — and before the system can say “yes, go ahead,” it has to ask all three at once.
Three registries, zero shared transaction
The system we build tracks IP addressing and network resources — VLANs, services, node identifiers — in a large telco operator’s metropolitan access network. When allocating a resource for a service (fixed-line internet, TV, or mobile), checking our own database isn’t enough. The same VLAN has to be free in three places at the same time: our own registry, the operator’s inventory system, and the equipment vendor’s network controller, which manages the actual device configuration. Each of these systems has a different owner, a different update cycle, and a different sync lag — anywhere from a few seconds to a few hours.
This isn’t a problem you solve with one database transaction. The systems belong to different teams; some integrations are read-only database links, others SOAP/REST calls to the vendor’s controller. None of us can lock someone else’s database for the duration of an allocation. Instead of an architectural fix for consistency — event sourcing or change data capture into one authoritative stream — we inherited a cascade of verifiers and a cache with a TTL. It has run in production against real subscribers for years, so before anyone proposes “rewrite this as event sourcing,” it’s worth understanding why the approach survived.
Parallel verification and a Redis reservation
Allocating a VLAN itself is a multi-step algorithm with business rules that depend on the service type and network segment — different ranges for fixed-line internet, different ones for TV, different ones per mobile operator sharing the infrastructure. Once the algorithm picks a candidate, three independent components query the three sources in parallel, and the result is a logical OR: if any system reports the candidate as occupied, it’s rejected.
public VlanAllocationResult checkAvailability(int vlanCandidate, ServiceContext ctx) {
// Three independent sources checked in parallel — none of them
// is transactionally consistent with the others.
var localRegistry = CompletableFuture.supplyAsync(
() -> localRegistryVerifier.isOccupied(vlanCandidate, ctx));
var operatorInventory = CompletableFuture.supplyAsync(
() -> inventoryVerifier.isOccupied(vlanCandidate, ctx));
var networkController = CompletableFuture.supplyAsync(
() -> controllerVerifier.isOccupied(vlanCandidate, ctx));
CompletableFuture.allOf(localRegistry, operatorInventory, networkController)
.orTimeout(3, TimeUnit.SECONDS)
.join();
boolean occupied = localRegistry.join() || operatorInventory.join()
|| networkController.join();
if (occupied) {
return VlanAllocationResult.rejected(vlanCandidate);
}
// Short-lived reservation in Redis — protects against two
// engineers filling out the form for the same VLAN at once.
boolean reserved = redisTemplate.opsForValue()
.setIfAbsent("vlan:reservation:" + vlanCandidate,
ctx.sessionId(), Duration.ofMinutes(5));
return reserved
? VlanAllocationResult.reserved(vlanCandidate)
: VlanAllocationResult.rejected(vlanCandidate);
}Redis solves a different problem than the verifiers do: it doesn’t check whether a VLAN is occupied on the network, only whether someone else is reserving it right now. Without it, two engineers filling out the form at the same moment could both get handed the same “free” number — none of the three databases would know anything yet, since the commit happens only at the end of the process. The reservation TTL is short on purpose; it guards the time spent filling out a form, not a long-term lock.
Migrating a database means migrating its whole ecosystem
We hit the same lack of shortcuts migrating from Oracle to PostgreSQL. Depending on an externally maintained Oracle instance limited our control over query performance, IP address lookups among them. But before production could switch over, we had to close out functional debt: two features left behind in an earlier version of the system — VLAN profile editing and mobile-network interface editing — had to land in the new version, or someone would end up hand-editing the production database. That was a deliberately accepted, bounded, counted risk for the transition period.
The cutover itself required a coordinated shutdown of three independent services on one machine, in a specific order, plus a prepared list of SQL transformations — swapping aggregate functions, removing Oracle-specific constructs, and restarting PostgreSQL sequences to the maximum existing value while accounting for tables where triggers fire on update, not just on insert. Migrating a live database holding a telecom network’s state turned out to mean migrating the whole ecosystem around it, not just the database.
When you don’t fix the algorithm, you monitor it
Not every bug you find needs an immediate fix. The algorithm computing a DSLAM identifier derived it from the VLAN number assigned to a video service via a simple modulo operation. For certain VLAN ranges, this caused drift between two independently maintained identifiers for the same device. We considered changing the formula, but chose something else: leave the algorithm untouched, and let a separate monitoring query catch the drift by comparing the stored value against the computed one and flagging mismatches.
The reasoning is simple: the algorithm has run for years against a production network serving real subscribers, and changing the allocation formula carries risk harder to predict than the cost of occasional, detectable data drift. A known algorithm bug we monitor is cheaper than a data migration bug we didn’t see coming.
Takeaway. When you don’t have a single source of truth, you don’t pretend you do — you design the system to treat divergence between sources as the normal state, not the exceptional one, and invest in detecting inconsistency instead of chasing the illusion of one transaction.
None of these three stories ends with “we rewrote it properly.” A cascade of verifiers, a Redis reservation, a cautious database migration, and monitoring instead of fixing — none of that is a lack of ambition. It’s recognizing that in infrastructure carrying millions of real connections, pragmatism is often more engineering than elegance is.