We’re back online, sorry for the downtime – Here’s why it took so long

After Jorge had dockerized the entire platform and it had been running for a few months, I noticed some dubious stats that didn’t make sense (more wallets than addresses with positive balance, some wallets with a negative balance). I guess our updateBalanceTables code is still not perfect. This is really hard to get right, because it directly works with the databases in order to avoid having to recalculate all the balances every block (which takes about half an hour nowadays!), and there are quite some edge cases. I was actually thinking about rewriting this part in Haskell, but now I’m not sure if BGE is still worth investing so much time in, with alternatives like BlockSci proliferating.

Anyway, unfortunately while trying to debug I noticed an unwanted side effect of docker: the common way of killing docker containers is absolutely brutal and even though we had already spent considerable time making BGE pretty much safe to kill in any state, this brutality was just too much: our databases became corrupted.

That’s a pity because rebuilding everything from scratch takes a few days ATM with the blockchain growing and growing. But what’s really frustrating is when you have just waited 3 days for FastBlockReader to finish and then an assert throws an exception, because somehow some blocks are missing!

Now Jorge had similar issues in the winter when dockerizing and switching servers, and the culprit seems to be somewhere between bitcoind and bitcoinJ. BitcoinJ’s master version is still not being segwit compatible, forcing us to use a rather unmaintained segwit branch. In addition, our assumption that it was safe to use bitcoind’s on-disk block files through a bitcoinJ loader seems to have been unwarranted. We’re still not sure why, but for now the only way to avoid this problem was bootstrapping with the PeerSource giving us the blocks one by one over the internal network. This process takes a bit longer, more like 5 days ATM.

So we really ought to implement saner process killing in our docker containers, completely rewrite BGE in a microservice architecture and finally get away from BitcoinJ. Or just try BlockSci. Which is a little hard as long as our systems are running, because it needs 50G RAM 😉 What do you think?

May 2, 2018

Posted In: Code

Tags: , , ,

Leave a Comment

Charts with the BGE-API and dimplejs

Finally it is out! For almost two months we promised it. Due to some open issues in the real life, it was not easy to find spare time to work on it, but finally it is done!

We are proud to present you the first examples of charts using Bitcoin-Graph-Explorer API and dimple.js. Dimple is a Library that uses d3 and exposes a simplified usage of d3.

First of all we need to include d3 and dimple in our page, and jquery for simplicity of the ajax calls:

<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.1/jquery.min.js">
<script src="http://d3js.org/d3.v3.min.js"/>
<script src="http://dimplejs.org/dist/dimple.v2.1.6.min.js"/>

With all that included, now we can start using the BGE server:

Continue reading Charts with the BGE-API and dimplejs

November 30, 2016

Posted In: Code

Leave a Comment

Bitcoin Graph Explorer 3.1 is here

As you probably know, our site is based on — and actually a kind of demo site for — our own blockchain analysis server Bitcoin Graph Explorer (BGE). The source code for this has always been available, but to be honest it would have been hard for anyone but us (well, including us) to actually use it.

So we have spent a few weeks thinking about how we could make the code more accessible, adding such niceties as a license, Readme, wrapper scripts, nix packages … in short, we are proud to be finally able to really tell you that Bitcoin Graph Explorer is open source software that we invite you to try out, use and improve with us.

This also means you have less excuses for trusting our website with your data when you don’t want to. Just run your own copy of BGE! Admittedly, it still requires some commitment: Storing and analysing the blockchain takes a lot of memory and time. Blockchains come with a cost, you know. We recommend at least a 512GB SSD (the DB takes about 300G at the moment and grows quickly at about triple the rate of the pure bitcoind blockchain). Populating the DB takes about a day on our server. On a hard disk, that is probably more like a week. You should, however, be able to work with relatively little RAM since we have rebuilt BGE to replace RAM with LMDB where possible.

To summarize, there are now at least three options of using BGE: Directly on bitcoinprivacy.net, the public API (blog post forthcoming, overview in the github readme) at http://api.bitcoinprivacy.net, and on your own server (preferably via the API though you could also work with the DB directly). We hope you find this useful for your bitcoin projects.

September 26, 2016

Posted In: Code, News

Tags:

Leave a Comment