“We believe the issue is resolved as we have expanded the database threshold to more than 1 trillion records. In the meantime, we are working with Microsoft to develop a warning system on database thresholds so we can anticipate these issues in the future.”
This is the key phrase regarding the recent event where BI stopped sending out alerts for the criminals it was tracking on behalf of police departments around the country. A company like this should do everything it can to design it’s systems for tracking so an eventuality like this doesn’t happen. How long before they bump up against the 1 Trillion record limit? I ask you. Let’s go back to the original article as it was posted on the BBC Online:
Thousands of US sex offenders, prisoners on parole and other convicts were left unmonitored after an electronic tagging system shut down because of data overload.
BI Incorporated, which runs the system, reached its data threshold – more than two billion records – on Tuesday.
This left authorities across 49 states unaware of offenders’ movement for about 12 hours.
BI increased its data storage capacity to avoid a repeat of the problem.
Prisons and other corrections agencies were blocked from getting notifications on about 16,000 people, BI Incorporated spokesman Jock Waldo said on Wednesday.
So the question I have a question as to how 16,000 people results in 2 Billion records in the database? Is that really all they are doing? How much old junk data are they keeping for legal purposes or just because they can keep it for potential future use? And how is it that a company depends on Microsoft to bail them out of such a critical situation. These seems like a very amateurish mistake. And could have been avoided by anyone with the title of Database Administrator who monitors the server on a regular basis. They should have known this thing was hitting an upper limit months ago and started rolling out a new database and moving records into it. This also shows the fundamental flaw in using SQL based record keeping for so-called real time data. Facebook gave up on it long ago as did Google. Rows and Tables and real time updates, doesn’t scale well. And if you cannot employ a Database Administrator to tell you when you are hitting a critical limit, but are dumping it off on the vendor, well good luck with that one guys.