Que significa minar bitcoins

Que significa minar bitcoins, W128 bitcoin, Eclipsemc bitcoin

Que significa minar bitcoins

Blocks the latest viruses, Win Anti Virus 2004 comparison with Norton AntiVirus & Mcafee Virus Scan. Download Win Anti Virus immediately for $39.95. Winantivirus.com compares features, price and benefits. Win Anti Virus offers 24/7 customer support. Kursentwicklung bitcoin Que significa minar bitcoins

Ideas to start a lucrative new business... or add more revenue streams to an existing business!...One of the biggest stumbling blocks for people who want to start their own, profitable Internet business is a lack of business ideas...visit us today! Bitcoin genproclimit kiev doors

W128 bitcoin Build a great Link Exchange Directory. Have ZEUS, the Internet Marketing Robot, work for you! You train him and teach him how to work. Then he goes out on the web & finds sites with similar interests as yours. A+++ Rated by Total Harmony.Com Eclipsemc bitcoin

Total Merchant Services offers a comprehensive line of payment solutions for your business credit card, debit card and check acceptance for all types of businesses including retail stores, restaurants, service, and internet businesses. Blockchain algorithm

Guerrillamail bitcoin Link to Free Secrets of Internet Millionaires. Our free package shows you how and includes: Free book, Free audio cassette, Free CD ROM, 6 Free Special Reports, and much more. Total value is over $585. Order Today No obligation. Bitcoin value prediction chart Buy sell bitcoins usa

W128 bitcoin

Learn Texas Holdem Poker, 7 Cards Stud and Omaha Poker Rules. Join Poker Tables or Tournaments to Play Against Others. Play for free or Get up to $100 on your First Deposit. Chat Options. 24/7 Reliable Support. Free Download! Bitcoin value prediction Bitcoin value prediction chart

This company has cut $10,000+ checks for people who worked its plan for only 30 days. Profit from the Upcoming Real Internet Gold Rush. Discover the fastest growing, most Lucrative Income Opportunity ever. Join for free. Buy sell bitcoins usa http://acert-payments135.5055050.ru

Blockchain algorithm Blue Advantage: PPO for North Carolinians under age 65. Blue Cross Blue Shield of North Carolina. Visit to learn more about Blue Advantage, download a FREE enrollment application and more. North Carolina Health Insurance Bitcoin exchange rate usd

Paying too much for music, movie and software downloads? MP3 Download Review is a free service to Internet downloaders that compares the best places to download movies, music, games, and more. Download more and spend less. Multipool bitcoin payout

Bitcoin genproclimit Blue Advantage: PPO for North Carolinians under age 65. Blue Cross Blue Shield of North Carolina. Visit to learn more about Blue Advantage, download a FREE enrollment application and more. Small Group Health Insurance Bitcoin ewallets Kursentwicklung bitcoin

Eclipsemc bitcoin

 
Find, compare and buy Internet and Communication Applications and other Computer Software products at Shopping.com.  Read product reviews and compare prices with tax and shipping from thousands of online stores. Cryptocurrency mining software Portafoglio bitcoin anonimo

Factory Direct You Save $$$ Lowest Prices on Internet! Introducing Revolutionary New Mattress Technology. Sleep in Oxygen Rich Envirnment Increase Longevity Pressure Relief Correct Support. Enter to Win International blockchain summit elets 2k

Bridge 21 bitcoin Copy Any DVD to DVD, DVD to CD, Copy PS1, PS2 and XBox Games. Download Full Length Movies & MP3 music files. 90 Day Guaranteed. Instant Download. Perfect Quality DVD. Bonus Free Full Movie Downloads Brain wallet bitcoin

Bonus bitcoin faucet Are you traveling ? Is your passport out of date?, Did you change your name? Do you need to add visa pages ? Call toll free or visit our website to download US passport applications & requirements. One millionth of a bitcoin

Perfect money to bitcoin Own the #1 rated Internet Business. $39,700 U.S. investment. Complete training and support. Home or office based. Proven at home job opportunity. 700 franchises in 87 countries. Serious inquiries. Keyhunter bitcoin Bitcoin wisdom okcoin ltc

Guerrillamail bitcoin

 
An travel affiliate program center dedicated to travel and asia hotels. Become part of one of the internet's fastest growing segments and increase your web site's revenue and profit potential. Bitcoin block size increase Mt gox lost bitcoins

Free Trial! PPCbidtracker manages keywords and bids on the internet's premier Pay Per Click search engines. Easy setup and account management make your PPC campaigns even more cost effective. Fpga bitcoin aldabergenov.stavknife.ru

100 bitcoins free Find Out Anything About Anyone! Start finding people, build your family tree, run your credit report for free and much more with this database of internet resources. Unlimited access $19.99! Guerrillamail bitcoin

R/Girlsgonebitcoin Download new McAfee Personal Firewall Plus 6.0 (2005) now. Get comprehensive, easy to use firewall protection and security. Authorized McAfee Partner. 100% satisfaction guaranteed. Save $10. Invest in bitcoin stock Brute force blockchain

Download Touch Typing tutor software today and learn how to touch type. 30 day money back guarantee. Buy the Deluxe CD Touch Typing version for only $39.95 + $5.95 for shipping. Affiliate. Sentix bitcoins sentiment Trezor bitcoin wallet

Buy sell bitcoins usa

 
VitalityMedical.com   an Internet healthcare wholesale superstore. Find a wide selection and great prices for shower chairs, assisted living, diabetics, urologicals, respiratory and more. Reporterii realitatii bitcoin Worldcore bitcoin

Our directory contains over 140 listings all especially for large sized women. Get info on clothing, magazines, support groups, health, exercise videos, internet resources & more. Double-Spending fast payments in bitcoin costco caviar

Voxelus bitcointalk Amazing greeting cards for your family and friends free. Download directly to your inbox and personalize for special occasions. Birthday, holiday, thank you, romantic or just to say hi. Blockchain hack

Tien ao bitcoin la gi Own #1 rated Internet Business. $39,700 U.S. investment. Complete training and support. Home or office based. Proven internet business. 700 franchises in 87 countries. Serious inquiries. Namecoin bitcoin chart Bitcoin machine stockton ca

This turnkey internet based system allows me the freedom to earn income from home. Free Training and Support. Learn how to earn like me. View our FREE online presentation. Click here. Bitcoin machine stockton ca Namecoin bitcoin chart

newbigsearch:

b16d0d36c49845317c6af84cfc7bd617
http://ico9.allcrypto.company Couchbase, the NoSQL Database Thu, 29 Mar 2018 18:17:59 +0000 en-US hourly 1 https://wordpress.org/?v=4.9.4 http://ico9.allcrypto.company/wp-content/uploads/2017/04/cropped-Couchbase-Logo-50x50-32x32.png http://ico9.allcrypto.company 32 32 124124012 http://ico9.allcrypto.company/developing-bitcoin-cryptocurrency-application-nodejs-nosql/ http://ico9.allcrypto.company/developing-bitcoin-cryptocurrency-application-nodejs-nosql/#comments Wed, 28 Mar 2018 14:00:44 +0000 http://ico9.allcrypto.company/?p=4892 I’ve been following cryptocurrency related subjects such as Bitcoin for a few months now and I’m very fascinated with everything that has been going on. As a web application developer, one topic that I’ve been particularly interested in learning more […]

The post Developing a Bitcoin Cryptocurrency Application with Node.js and NoSQL appeared first on The Couchbase Blog.

]]>
I’ve been following cryptocurrency related subjects such as Bitcoin for a few months now and I’m very fascinated with everything that has been going on.

As a web application developer, one topic that I’ve been particularly interested in learning more about is around cryptocurrency exchanges and how to make them. From the front, these applications seem to be tools for managing accounts, converting Bitcoin to a fiat currency like USD and back, and transferring Bitcoin to other people, but are they more?

We’re going to look at some examples with Node.js and the NoSQL database, Couchbase, that covers topics modeled around cryptocurrency exchanges.

Disclaimer: I am not a cryptocurrency expert, nor have I taken part in any development around financial services or exchanges. I am an enthusiast of the subject material and anything obtained from this article should be properly tested and used at your own risk.

The Take-Away

There are a few things that you will and won’t get from this particular article. For example, let’s start with the things you won’t get out of this article:

  • We won’t configure any banking or credit card services to transfer fiat currencies such as USD.
  • We won’t be broadcasting any signed transactions to the Bitcoin network, finalizing a transfer.

That said, here are some things you can look forward to learning about in this article:

  • We’ll create a hierarchical deterministic (HD) wallet that can generate an unlimited amount of keys for a given seed, each representing a user wallet.
  • We’ll create user accounts each with wallets based on the master seed.
  • We’ll create transactions that represent deposits, withdrawals, and transfers of funds from the exchange, without actually working with a fiat currency.
  • We’ll lookup balances from the Bitcoin network.
  • We’ll create signed transactions to be broadcasted on the Bitcoin network.

There are many things that we’ll see in this article that can be done way better. If you found something that could be improved, definitely share it in the comments. Like I said, I’m not an expert on the topic, just a fan.

The Project Requirements

There are a few requirements that must be met to be successful with this project:

  • You must have Node.js 6+ installed and configured.
  • You must have Couchbase 5.1+ installed and configured with a Bucket and RBAC profile ready to go.

The main point is that I won’t be going through how to get Couchbase up and running. It isn’t a difficult process, but you’ll need a Bucket set up with an application account and an index for querying with N1QL.

Creating a Node.js Application with Dependencies

We’re going to create a new Node.js application and download the dependencies before we start adding any logic. Create a project directory somewhere on your computer and execute the following commands from a CLI within that directory:

npm init -y
npm install couchbase --save
npm install express --save
npm install body-parser --save
npm install joi --save
npm install request request-promise --save
npm install uuid --save
npm install bitcore-lib --save
npm install bitcore-mnemonic --save

I know I could have done all of the dependency installations in a single line, but I wanted to make them clear to read. So what are we doing in the above commands?

First, we’re initializing a new Node.js project by creating a package.json file. Then we’re downloading our dependencies and adding them to the package.json file via the --save flag.

For this example we’ll be using Express Framework. The express, body-parser, and joi packages are all relevant towards accepting and validating request data. Because we’ll be communicating with public Bitcoin nodes, we will be using the request and request-promise promise wrapping package. The very popular bitcore-lib package will allow us to create wallets and sign transactions while the bitcore-mnemonic package will allow us to generate a seed that can be used for our HD wallet keys. Finally, couchbase and uuid will be used for working with our database.

Now we probably want to better structure our project. Add the following directories and files within your project directory if they do not already exist:

package.json
config.json
app.js
routes
    account.js
    transaction.js
    utility.js
classes
    helper.js

All of our API endpoints will be split into categories and placed in each appropriate routing file. We don’t have to do this, but it makes our project just a little bit cleaner. To remove a ton of Bitcoin and database logic out of our routes, we’ll be adding everything that isn’t data validation into our classes/helper.js file. The config.json file will have all of our database information as well as our mnemonic seed. In a realistic scenario, this file should be treated like gold and receive as much protection as it could possibly receive. The app.js file will have all of our configuration and bootstrapping logic towards connecting our routes, connecting to the database, etc.

As a convenience, we’re going to add one more dependency to our project and set it up:

npm install nodemon --save-dev

The nodemon package will allow us to hot-reload our project every time we change a file. It is not a requirement, but can save us some time as we’re building.

Open the package.json file and add the following script to make it happen:

...
"scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "./node_modules/nodemon/bin/nodemon.js app.js"
},
...

We can start the development process of our application at this point.

Developing the Database and Bitcoin Logic

When it comes to developing our application, before we start worrying about API endpoints, we want to create our database and Bitcoin related logic.

We’re going to be spending our time in the project’s classes/helper.js file. Open it and include the following:

const Couchbase = require("couchbase");
const Request = require("request-promise");
const UUID = require("uuid");
const Bitcore = require("bitcore-lib");

class Helper {

    constructor(host, bucket, username, password, seed) {
        this.cluster = new Couchbase.Cluster("couchbase://" + host);
        this.cluster.authenticate(username, password);
        this.bucket = this.cluster.openBucket(bucket);
        this.master = seed;
    }

    createKeyPair(account) { }

    getWalletBalance(addresses) { }

    getAddressBalance(address) { }

    getAddressUtxo(address) { }

    insert(data, id = UUID.v4()) { }

    createAccount(data) { }

    addAddress(account) { }

    getAccountBalance(account) { }

    getMasterAddresses() { }

    getMasterKeyPairs() { }

    getMasterAddressWithMinimum(addresses, amount) { }

    getMasterChangeAddress() { }

    getAddresses(account) { }

    getPrivateKeyFromAddress(account, address) { }

    createTransactionFromAccount(account, source, destination, amount) { }

    createTransactionFromMaster(account, destination, amount) { }

}

module.exports = Helper;

We’re going to be passing this class around as a singleton for our application. In the constructor method, we are establishing a connection to our database cluster, opening a Bucket, and authenticating. The open Bucket will be used throughout this helper class.

Let’s knock out the Bitcoin logic before the database logic.

If you’re not familiar with HD wallets, they are essentially a wallet derived of a single seed. Using the seed you can derive children and those children can have children, and so on and so forth.

createKeyPair(account) {
    var account = this.master.deriveChild(account);
    var key = account.deriveChild(Math.random() * 10000 + 1);
    return { "secret": key.privateKey.toWIF().toString(), "address": key.privateKey.toAddress().toString() }
}

The master variable in the createKeyPair function represents the top level seed key. Each user account will be a direct child of that key, hence we are deriving a child based on an account value. The account value is a person number and every account created will get an incremental number. However, we’re not going to generate account keys and call it a day. Instead, each account key will have 10,000 possible private and public keys in case they don’t want to use the same key more than once. Once we’ve generated a key at random, we return it.

Similarly, we have a getMasterChangeAddress function like the following:

getMasterChangeAddress() {
    var account = this.master.deriveChild(0);
    var key = account.deriveChild(Math.random() * 10 + 1);
    return { "secret": key.privateKey.toWIF().toString(), "address": key.privateKey.toAddress().toString() }
}

When we start creating accounts, they will start at one, leaving zero for the exchange or the web application, or whatever you want to call it. We are also allocating 10 possible addresses to this account. These addresses will do two possible things. The first is that they will hold Bitcoin to transfer to other accounts and the second is that they will receive remainder payments, otherwise known as the change. Remember, in a Bitcoin transaction, all unspent transaction output (UTXO) must be spent, even if it is less than the desired amount. This means that the desired amount gets sent to the destination and the remainder gets sent back into one of these 10 addresses.

Are there other ways or better ways to do this? Absolutely, but this one will work for the example.

To get a balance for any address we use or generate using the HD seed, we can use a public Bitcoin explorer:

getAddressBalance(address) {
    return Request("https://insight.bitpay.com/api/addr/" + address);
}

The above function will take an address and get the balance in decimal format as well as satoshis. Going forward, the satoshi value is the only relevant value to us. If we have X number of addresses for a given account, we can get the total balance using a function like this:

getWalletBalance(addresses) {
    var promises = [];
    for(var i = 0; i < addresses.length; i++) {
        promises.push(Request("https://insight.bitpay.com/api/addr/" + addresses[i]));
    }
    return Promise.all(promises).then(result => {
        var balance = result.reduce((a, b) => a + JSON.parse(b).balanceSat, 0);
        return new Promise((resolve, reject) => {
            resolve({ "balance": balance });
        });
    });
}

In the above getWalletBalance function we are making a request for each address and when they’ve all completed, we can add the balances and return them.

It takes a little more than just an address balance to be able to transfer cryptocurrency. Instead we need to know the unspent transaction output (UTXO) for a given address. This can be found using the same API from BitPay:

getAddressUtxo(address) {
    return Request("https://insight.bitpay.com/api/addr/" + address + "/utxo").then(utxo => {
        return new Promise((resolve, reject) => {
            if(JSON.parse(utxo).length == 0) {
                reject({ "message": "There are no unspent transactions available." });
            }
            resolve(JSON.parse(utxo));
        });
    });
}

If there is no unspent transaction output, it means there is nothing we can transfer and we should throw an error instead. Having enough to transfer is a different story.

For example, we could do something like this:

getMasterAddressWithMinimum(addresses, amount) {
    var promises = [];
    for(var i = 0; i < addresses.length; i++) {
        promises.push(Request("https://insight.bitpay.com/api/addr/" + addresses[i]));
    }
    return Promise.all(promises).then(result => {
        for(var i = 0; i < result.length; i++) {
            if(result[i].balanceSat >= amount) {
                return resolve({ "address": result[i].addrStr });
            }
        }
        reject({ "message": "Not enough funds in exchange" });
    });
}

In the above function, we are taking a list of addresses and checking to see which one holds an amount greater than the threshold that we provide. If none of them have enough balance, we should probably relay that message.

The final utility related function, is something we’ve kind of already seen:

getMasterKeyPairs() {
    var keypairs = [];
    var key;
    var account = this.master.deriveChild(0);
    for(var i = 1; i <= 10; i++) {
        key = account.deriveChild(i);
        keypairs.push({ "secret": key.privateKey.toWIF().toString(), "address": key.privateKey.toAddress().toString() });
    }
    return keypairs;
}

The above function will get us all master keys, which will be useful for signing and checking value.

Just to reiterate, I’m using a finite value to how many keys are generated. You may or may not want to do the same, it is up to you.

Now let’s dive into some NoSQL logic for storing our application data.

As of right now there is no data in our database. The first logic step might be to create some data. While it isn’t particularly difficult standalone, we can create a function like this:

insert(data, id = UUID.v4()) {
    return new Promise((resolve, reject) => {
        this.bucket.insert(id, data, (error, result) => {
            if(error) {
                reject({ "code": error.code, "message": error.message });
            }
            data.id = id;
            resolve(data);
        });
    });
}

Essentially, we’re accepting an object and an id to be used as a document key. If a document key isn’t provided, we’ll automatically generate it. When all said and done, we’ll return what was created including the id in the response.

So let’s say we want to create a user account. We can do the following:

createAccount(data) {
    return new Promise((resolve, reject) => {
        this.bucket.counter("accounts::total", 1, { "initial": 1 }, (error, result) => {
            if(error) {
                reject({ "code": error.code, "message": error.message });
            }
            data.account = result.value;
            this.insert(data).then(result => {
                resolve(result);
            }, error => {
                reject(error);
            });
        });
    });
}

Remember, accounts are driven by an auto incrementing numeric value for this example. We can create incrementing values by using a counter in Couchbase. If the counter doesn’t exist, we’ll initialize it at 1 and increment it every next call. Remember, 0 is reserved for application keys.

After we get our counter value, we add it to the object that was passed and call our insert function, which in this case generates a unique id for us.

We haven’t seen it yet because we don’t have any endpoints, but lets assume that when we create an account, it has no address information, only an account identifier. We might want to add an address for the user:

addAddress(account) {
    return new Promise((resolve, reject) => {
        this.bucket.get(account, (error, result) => {
            if(error) {
                reject({ "code": error.code, "message": error.message });
            }
            var keypair = this.createKeyPair(result.value.account);
            this.bucket.mutateIn(account).arrayAppend("addresses", keypair, true).execute((error, result) => {
                if(error) {
                    reject({ "code": error.code, "message": error.message });
                }
                resolve({ "address": keypair.address });
            });
        });
    });
}

When adding an address, we first get the user by the document id. When the document is retrieved, we get the numeric account value and create a fresh keypair of our 10,000 options. Using a sub-document operation, we can add the keypair to the user document without ever having to download the document or manipulating it.

Something very serious to note about what we’ve just done.

I am storing the unencrypted private key and public address in the user document. This is a big no-no for production. Remember all those stories you read about where people got their keys stolen? In reality, we’d want to encrypt the data before inserting it. We can do that by using the Node.js crypto library, or if we’re using Couchbase Server 5.5, the Node.js SDK for Couchbase offers encryption. We won’t explore it here though.

Okay, we’ve got account data and addresses in the database now. Let’s query for that data:

getAddresses(account) {
    var statement, params;
    if(account) {
        statement = "SELECT VALUE addresses.address FROM " + this.bucket._name + " AS account USE KEYS $id UNNEST account.addresses as addresses";
        params = { "id": account };
    } else {
        statement = "SELECT VALUE addresses.address FROM " + this.bucket._name + " AS account UNNEST account.addresses as addresses WHERE account.type = 'account'";
    }
    var query = Couchbase.N1qlQuery.fromString(statement);
    return new Promise((resolve, reject) => {
        this.bucket.query(query, params, (error, result) => {
            if(error) {
                reject({ "code": error.code, "message": error.message });
            }
            resolve(result);
        });
    });
}

The above getAddresses function can do one of two things. If an account was provided, we’ll use a N1QL query to get all addresses for that particular account. If no account was provided, we’ll get all addresses for every account in the database. In both scenarios, we’re only getting public addresses, nothing sensitive. Using a parameterized N1QL query, we can return the database results back to the client.

Something to note in our query.

We’re storing our addresses in an array in the user documents. Using an UNNEST operator, we can flatten those addresses and make the response more attractive.

Now let’s say we have an address and we want to get the corresponding private key. We might do the following:

getPrivateKeyFromAddress(account, address) {
    var statement = "SELECT VALUE keypairs.secret FROM " + this.bucket._name + " AS account USE KEYS $account UNNEST account.addresses AS keypairs WHERE keypairs.address = $address";
    var query = Couchbase.N1qlQuery.fromString(statement);
    return new Promise((resolve, reject) => {
        this.bucket.query(query, { "account": account, "address": address }, (error, result) => {
            if(error) {
                reject({ "code": error.code, "message": error.message });
            }
            resolve({ "secret": result[0] });
        });
    });
}

Given a particular account, we create a query similar to what we saw previously. This time, after we UNNEST, we do a WHERE condition to give us results only for the matching address. If we wanted to we could have done an array operation instead. With Couchbase and N1QL, there are numerous ways to solve a problem.

We’re going to change gears a bit here. Up until now we’ve done account oriented operations in our NoSQL database. Another important aspect is transactions. For example, maybe user X deposits some USD currency for BTC and user Y does a withdrawal. We need to store and query for that transaction information.

The API endpoint functions will save the transaction data, but we can still query for it.

getAccountBalance(account) {
    var statement = "SELECT SUM(tx.satoshis) AS balance FROM " + this.bucket._name + " AS tx WHERE tx.type = 'transaction' AND tx.account = $account";
    var query = Couchbase.N1qlQuery.fromString(statement);
    return new Promise((resolve, reject) => {
        this.bucket.query(query, { "account": account }, (error, result) => {
            if(error) {
                reject({ "code": error.code, "message": error.message });
            }
            resolve({ "balance": result[0].balance });
        });
    });
}

Given an account, we want to get an account balance for a particular user.

Wait a second, let’s take a step back because didn’t we already create some account balance functions? Technically we did, but those functions were for checking the wallet balance, not the account balance.

This is where some of my experience turns into gray area. Every time you transfer Bitcoin, there is a fee involved, and sometimes it is quite expensive. When you make a deposit, it isn’t cost effective to transfer money into your wallet because it would be charged a miner fee. Then you would be charged to withdraw and even transfer again. You’ve already lost most of your Bitcoin at that point.

Instead, I believe exchanges have a holding account similar to a stock exchange money market account. There is record of the money that you should have in your account, but it isn’t technically in a wallet. When you want transfer, you’re transferring from the application address, not your user address. When you withdraw, it is just being subtracted.

Again, I don’t know if that is truly how it works, but that is how I’d do it in order to avoid fees everywhere.

Going back to our getAccountBalance function. We’re taking a sum of every transaction. Deposits have a positive value while transfers and withdrawals have a negative value. Aggregating this information together should give you an accurate number, excluding your wallet balance. Later we’ll be getting an account with wallet balance.

Given the little that we know about account balances, we can try to create a transaction from our wallet:

createTransactionFromAccount(account, source, destination, amount) {
    return new Promise((resolve, reject) => {
        this.getAddressBalance(source).then(sourceAddress => {
            if(sourceAddress.balanceSat < amount) {
                return reject({ "message": "Not enough funds in account." });
            }
            this.getPrivateKeyFromAddress(account, source).then(keypair => {
                this.getAddressUtxo(source).then(utxo => {
                    var transaction = new Bitcore.Transaction();
                    for(var i = 0; i < utxo.length; i++) {
                        transaction.from(utxo[i]);
                    }
                    transaction.to(destination, amount);
                    this.addAddress(account).then(change => {
                        transaction.change(change.address);
                        transaction.sign(keypair.secret);
                        resolve(transaction);
                    }, error => reject(error));
                }, error => reject(error));
            }, error => reject(error));
        }, error => reject(error));
    });
}

If provided a source address, destination address, and amount, we can create and sign a transaction to be later broadcasted on the Bitcoin network.

First we get the balance for the source address in question. We need to make sure it has enough UTXO to meet the send amount expectation. Note that in this example, we’re doing single address transactions. If you wanted to get complicated, you could send from multiple addresses in a single transaction. We’re not going to do that here. If our single address has enough funds, we get the private key for it and the UTXO data. With the UTXO data we can create a Bitcoin transaction, apply the destination address and a change address, then sign the transaction using our private key. The response can be broadcasted.

Similarly let’s say we wanted to transfer Bitcoin from our holding account:

createTransactionFromMaster(account, destination, amount) {
    return new Promise((resolve, reject) => {
        this.getAccountBalance(account).then(accountBalance => {
            if(accountBalance.balance < amount) {
                reject({ "message": "Not enough funds in account." });
            }
            var mKeyPairs = this.getMasterKeyPairs();
            var masterAddresses = mKeyPairs.map(a => a.address);
            this.getMasterAddressWithMinimum(masterAddresses, amount).then(funds => {
                this.getAddressUtxo(funds.address).then(utxo => {
                    var transaction = new Bitcore.Transaction();
                    for(var i = 0; i < utxo.length; i++) {
                        transaction.from(utxo[i]);
                    }
                    transaction.to(destination, amount);
                    var change = helper.getMasterChangeAddress();
                    transaction.change(change.address);
                    for(var j = 0; j < mKeyPairs.length; j ++) {
                        if(mKeyPairs[j].address == funds.address) {
                            transaction.sign(mKeyPairs[j].secret);
                        }
                    }
                    var tx = {
                        account: account,
                        satoshis: (amount * -1),
                        timestamp: (new Date()).getTime(),
                        status: "transfer",
                        type: "transaction"
                    };
                    this.insert(tx).then(result => {
                        resolve(transaction);
                    }, error => reject(error));
                }, error => reject(error));
            }, error => reject(error));
        }, error => reject(error));
    });
}

We’re assuming that our exchange addresses have been loaded with an insane amount of Bitcoin to meet demand.

The first step is to make sure we have funding in our holding account. We can execute that query that sums up each of our transactions to get a valid number. If we have enough, we can get all 10 of our master key pairs and the addresses. We need to check which address has enough funds to send. Remember, single address transactions here, when there could be more.

If an address has enough funds, we get the UTXO data and start making a transaction. This time instead of our wallet as the source address, we use the wallet of the exchange. After we get a signed transaction, we want to create a transaction in our database to subtract the value that we’re transferring.

Before we move onto the API endpoints, I want to re-iterate a few things:

  • I’m assuming that popular exchanges have a holding account to avoid fees imposed on wallet addresses.
  • We are using single address transactions in this example, rather than aggregating what we have.
  • I’m not encrypting the key data in the account documents, when I should be.
  • I’m not broadcasting any transactions, only creating them.

Now let’s focus on our API endpoints, the simple part.

Designing RESTful API Endpoints with Express Framework

Remember, as we configured in the beginning, our endpoints will be split across three files which act as groupings. We’ll start with the smallest and simplest group of endpoints, which are more for utility than anything else.

Open the project’s routes/utility.js file and include the following:

const Bitcore = require("bitcore-lib");
const Mnemonic = require("bitcore-mnemonic");

module.exports = (app) => {

    app.get("/mnemonic", (request, response) => {
        response.send({
            "mnemonic": (new Mnemonic(Mnemonic.Words.ENGLISH)).toString()
        });
    });

    app.get("/balance/value", (request, response) => {
        Request("https://api.coinmarketcap.com/v1/ticker/bitcoin/").then(market => {
            response.send({ "value": "$" + (JSON.parse(market)[0].price_usd * request.query.balance).toFixed(2) });
        }, error => {
            response.status(500).send(error);
        });
    });

}

Here we have two endpoints, one for generating mnemonic seeds and the other for getting the fiat value of a Bitcoin balance. Neither are truly necessary, but on first launch, it might be nice to generate a seed value to later save in our config file.

Now open the project’s routes/account.js file so we can handle account information:

const Request = require("request-promise");
const Joi = require("joi");
const helper = require("../app").helper;

module.exports = (app) => {

    app.post("/account", (request, response) => { });

    app.put("/account/address/:id", (request, response) => { });

    app.get("/account/addresses/:id", (request, response) => { });

    app.get("/addresses", (request, response) => { });

    app.get("/account/balance/:id", (request, response) => { });

    app.get("/address/balance/:id", (request, response) => { });

}

Notice that we’re pulling the helper class from the app.js file that we haven’t started yet. Just go with it for now and it will make sense later, although it isn’t anything special.

When it comes to creating accounts, we have the following:

app.post("/account", (request, response) => {
    var model = Joi.object().keys({
        firstname: Joi.string().required(),
        lastname: Joi.string().required(),
        type: Joi.string().forbidden().default("account")
    });
    Joi.validate(request.body, model, { stripUnknown: true }, (error, value) => {
        if(error) {
            return response.status(500).send(error);
        }
        helper.createAccount(value).then(result => {
            response.send(value);
        }, error => {
            response.status(500).send(error);
        });
    });
});

Using Joi we can validate the request body and throw errors if it isn’t correct. Assuming the request body is correct, we can call our createAccount function to save a new account in the database.

With an account created, we can add some addresses:

app.put("/account/address/:id", (request, response) => {
    helper.addAddress(request.params.id).then(result => {
        response.send(result);
    }, error => {
        return response.status(500).send(error);
    });
});

Using the account id that was passed, we can call our addAddress function to use a subdocument operation on our document.

Not so bad right?

To get all addresses for a particular account, we might have something like the following:

app.get("/account/addresses/:id", (request, response) => {
    helper.getAddresses(request.params.id).then(result => {
        response.send(result);
    }, error => {
        response.status(500).send(error);
    });
});

Alternatively, if we don’t provide an id, we can get all addresses from all accounts using the following endpoint function:

app.get("/addresses", (request, response) => {
    helper.getAddresses().then(result => {
        response.send(result);
    }, error => {
        response.status(500).send(error);
    });
});

Now for probably the trickiest endpoint function. Let’s say we want to get the balance of our account, which includes the holding account as well as each of our wallet addresses. We can do the following:

app.get("/account/balance/:id", (request, response) => {
    helper.getAddresses(request.params.id).then(addresses => helper.getWalletBalance(addresses)).then(balance => {
        helper.getAccountBalance(request.params.id).then(result => {
            response.send({ "balance": balance.balance + result.balance });
        }, error => {
            response.status(500).send({ "code": error.code, "message": error.message });
        });
    }, error => {
        response.status(500).send({ "code": error.code, "message": error.message });
    });
});

The above will call both of our functions for obtaining balance, and add the results together to get one massive balance.

The account endpoints weren’t particularly interesting. Creating transactions is a bit more exciting.

Open the project’s routes/transaction.js file and include the following:

const Request = require("request-promise");
const Joi = require("joi");
const Bitcore = require("bitcore-lib");
const helper = require("../app").helper;

module.exports = (app) => {

    app.post("/withdraw", (request, response) => { });

    app.post("/deposit", (request, response) => { });

    app.post("/transfer", (request, response) => { });

}

We have three different types of transaction. We can deposit fiat currency for Bitcoin, withdraw Bitcoin for fiat currency, and transfer Bitcoin to new wallet addresses.

Let’s take a look at the deposit endpoint:

app.post("/deposit", (request, response) => {
    var model = Joi.object().keys({
        usd: Joi.number().required(),
        id: Joi.string().required()
    });
    Joi.validate(request.body, model, { stripUnknown: true }, (error, value) => {
        if(error) {
            return response.status(500).send(error);
        }
        Request("https://api.coinmarketcap.com/v1/ticker/bitcoin/").then(market => {
            var btc = value.usd / JSON.parse(market)[0].price_usd;
            var transaction = {
                account: value.id,
                usd: value.usd,
                satoshis: Bitcore.Unit.fromBTC(btc).toSatoshis(),
                timestamp: (new Date()).getTime(),
                status: "deposit",
                type: "transaction"
            };
            helper.insert(transaction).then(result => {
                response.send(result);
            }, error => {
                response.status(500).send(error);
            });
        }, error => {
            response.status(500).send(error);
        });
    });
});

After we validate the input, we check the current value of Bitcoin in USD with CoinMarketCap. Using the data in the response, we can calculate how many Bitcoin we should get based on the USD amount deposited.

After creating a database transaction, we can save it and since it is a positive number, it will come back as positive balance when querying.

Now let’s say we want to withdraw money from our Bitcoin:

app.post("/withdraw", (request, response) => {
    var model = Joi.object().keys({
        satoshis: Joi.number().required(),
        id: Joi.string().required()
    });
    Joi.validate(request.body, model, { stripUnknown: true }, (error, value) => {
        if(error) {
            return response.status(500).send(error);
        }
        helper.getAccountBalance(value.id).then(result => {
            if(result.balance == null || (result.balance - value.satoshis) < 0) {
                return response.status(500).send({ "message": "There are not `" + value.satoshis + "` satoshis available for withdrawal" });
            }
            Request("https://api.coinmarketcap.com/v1/ticker/bitcoin/").then(market => {
                var usd = (Bitcore.Unit.fromSatoshis(value.satoshis).toBTC() * JSON.parse(market)[0].price_usd).toFixed(2);
                var transaction = {
                    account: value.id,
                    satoshis: (value.satoshis * -1),
                    usd: parseFloat(usd),
                    timestamp: (new Date()).getTime(),
                    status: "withdrawal",
                    type: "transaction"
                };
                helper.insert(transaction).then(result => {
                    response.send(result);
                }, error => {
                    response.status(500).send(error);
                });
            }, error => {
                response.status(500).send(error);
            });
        }, error => {
            return response.status(500).send(error);
        });
    });
});

Similar events are happening here. After validating the request body, we get our account balance and make sure the amount we’re withdrawing is less than or equal to our balance. If it is, we can do another conversion based on the current price from CoinMarketCap. We’ll create a transaction using a negative value and save it to the database.

In both these instances we are relying on CoinMarketCap which has had negative controversy in the past. You may want to pick a different resource for conversions.

Finally we have transfers:

app.post("/transfer", (request, response) => {
    var model = Joi.object().keys({
        amount: Joi.number().required(),
        sourceaddress: Joi.string().optional(),
        destinationaddress: Joi.string().required(),
        id: Joi.string().required()
    });
    Joi.validate(request.body, model, { stripUnknown: true }, (error, value) => {
        if(error) {
            return response.status(500).send(error);
        }
        if(value.sourceaddress) {
            helper.createTransactionFromAccount(value.id, value.sourceaddress, value.destinationaddress, value.amount).then(result => {
                response.send(result);
            }, error => {
                response.status(500).send(error);
            });
        } else {
            helper.createTransactionFromMaster(value.id, value.destinationaddress, value.amount).then(result => {
                response.send(result);
            }, error => {
                response.status(500).send(error);
            });
        }
    });
});

If the request contains a source address, we are going to transfer from our own wallet, otherwise we are going to transfer from the wallet the exchange manages.

All of this is based on functions that we had previously created.

With the endpoints out of the way, we can focus on bootstrapping our application and come to a conclusion.

Bootstrapping the Express Framework Application

Right now we have two files that remain untouched by the example. We haven’t added a configuration or driving logic to bootstrap our endpoints.

Open the project’s config.json file and include something like the following:

{
    "mnemonic": "manage inspire agent october potato thought hospital trim shoulder round tired kangaroo",
    "host": "localhost",
    "bucket": "bitbase",
    "username": "bitbase",
    "password": "123456"
}

Remember this file is incredibly sensitive. Consider locking it down or even using a different approach. If the seed is exposed, every private key for all user accounts and the exchange account can be obtained with zero effort.

Now open the project’s app.js file and include the following:

const Express = require("express");
const BodyParser = require("body-parser");
const Bitcore = require("bitcore-lib");
const Mnemonic = require("bitcore-mnemonic");
const Config = require("./config");
const Helper = require("./classes/helper");

var app = Express();

app.use(BodyParser.json());
app.use(BodyParser.urlencoded({ extended: true }));

var mnemonic = new Mnemonic(Config.mnemonic);
var master = new Bitcore.HDPrivateKey(mnemonic.toHDPrivateKey());

module.exports.helper = new Helper(Config.host, Config.bucket, Config.username, Config.password, master);

require("./routes/account.js")(app);
require("./routes/transaction.js")(app);
require("./routes/utility.js")(app);

var server = app.listen(3000, () => {
    console.log("Listening at :" + server.address().port + "...");
});

What we’re doing is initializing Express, loading the configuration information, and linking up our routes. The module.exports.helper variable is our singleton that will be used in every other JavaScript file.

Conclusion

You just saw how to get started with building your own cryptocurrency exchange using Node.js and Couchbase as the NoSQL database. We covered a lot, from generating HD wallets to creating endpoints with complex database logic.

I cannot stress this enough though. I am a cryptocurrency enthusiast and have no true experience in the financial space. The things I shared should work, but can be done a whole lot better. Don’t forget to encrypt your keys and keep your seed safe. Test your work and know what you’re getting yourself into.

If you’d like to download this project, check it out on GitHub. If you’d like to share insight, experience, ect., on the topic, please share it in the comments. The community can work towards creating something great!

If you’re a fan of Golang, I created a similar project in a previous tutorial.

The post Developing a Bitcoin Cryptocurrency Application with Node.js and NoSQL appeared first on The Couchbase Blog.

]]>
http://ico9.allcrypto.company/developing-bitcoin-cryptocurrency-application-nodejs-nosql/feed/ 3 4892
http://ico9.allcrypto.company/condemn-microservices-architecture-fail-even-start/ http://ico9.allcrypto.company/condemn-microservices-architecture-fail-even-start/#respond Tue, 27 Mar 2018 14:27:51 +0000 http://ico9.allcrypto.company/?p=4885 A lot has already been said about microservices over the last few years, but I commonly see new distributed systems being developed with the old mindset of monoliths. The side effect of building something new without the understanding of some […]

The post How to Condemn Your Microservices Architecture to Fail Before You Even Start appeared first on The Couchbase Blog.

]]>
A lot has already been said about microservices over the last few years, but I commonly see new distributed systems being developed with the old mindset of monoliths. The side effect of building something new without the understanding of some key concepts is that you will end up with more problems than before, which is definitely not the goal you had in mind.

In this article, I would like to tackle some concepts that we historically take for granted and which might lead to poor architecture when applied to microservices:

 

Make Synchronous Calls Only

In a monolith architecture, we are used to All-Or-Nothing availability, we can always call any service anytime. However, in a microservices world, there are no guarantees that a service on which we depend will be online. It could be even worse, as a slow service can potentially slow down the whole system as each thread will be locked for a certain amount of time while waiting for the response of the external service.

How locked/slow threads eventually consumes your entire thread pool

We are still learning how to properly implement communications between services, but the rule of thumb is to make everything asynchronous, and that is where one of the first challenges emerges, because historically we haven’t exercised enough our ability to transform a synchronous flow into asynchronous.

Hopefully, most of the use cases can be implemented asynchronously with the right amount of effort. Amazon, for instance, implemented its whole order system that way, and you barely feel it. They will almost certainly let you place an order successfully, but if there is any problem with the payment or if the product is out of stock, you will receive an email notification a few minutes or hours later telling you about it and which actions need to be taken. The advantage of this approach is clear; even if the payment or stock service is down, it won’t block users from placing an order. That is the beauty of asynchronous communication.

Of course, not everything in your system can be async, and to deal with the common problems of synchronous calls like network instability, high latency or temporary unavailability of services, we had to come up with a set of patterns to avoid cascading failures such as local caches, timeouts, retries, circuit breakers, bulkheads, etc. There are many frameworks out there implementing those concepts, but the Netflix Hystrix is currently the most well-known library.

There is nothing essentially wrong with this approach; it works pretty well for several companies. The only downside is that we are pushing an extra responsibility for each service, which makes your microservice even less “micro”. Some options have been proposed in the last two years to address this issue. The Service Mesh pattern, for instance, tries to externalize this complexity in a form of a sidecar container : 

I personally like this approach, especially because it is language agnostic, which means that It will work for all of your microservices regardless of the language in which it is written. Another advantage is the standardized metrics, as different libraries/frameworks might use a slightly different algorithm for retries, timeouts, circuit breaks, etc. Those small differences might impact significantly the generated metrics, making impossible to have a reliable vision of the system’s behavior. 

UPDATE:  If you want to read more about the Service Mesh pattern, check out this excellent presentation.

In Summary, thinking about how services will communicate with each other is essential for a successful architecture, and should be planned ahead to avoid a chain of dependencies. The less the services know about each other, the better the architecture is.

 

Just Use RDBMS for Everything

 

After a 30-year monopoly of RDBMS, it is understandable why many people still think that way. However, nowadays, virtually every day a new database is born (and two JavaScript frameworks). If in the old days, picking a database was a matter of choosing one among five, today, the same task demands much more attention.

Source: db-engines.com – initial release dates

As Martin Fowler said 12 years ago, there are a lot of benefits on choosing a specialized storage, such as: higher performance, lower cost, etc. I won’t spend your time going through all the downsides of RDBMS (slow reads, sparse data, impedance mismatch, joins, etc). Rather, I would just like to highlight again how the database plays a major role in the overall performance of the system, and an inappropriate choice will eventually cost you much more money.

The benefits of polyglot persistence are crystal clear, and the maturity of the solutions have been proven by numerous successful critical use cases, just to name a few: Pokémon Go, AirBnB, Viber, eBay, Sky, Amadeus, Amazon, Google, LinkedIn and Netflix

I would have agreed five years ago with the “learning curve argument” of why you have not started with NoSQL yet. Nonetheless, since then, a lot has changed, and some companies have put a lot of effort on making it really easy for developers and DBAs, like the Couchbase Spring Boot/Spring Data support or the recently launched  Kubernetes Operator which aims to automate most of the DBAs work.

 

Don’t think about Debugging and Observability

 

One day, you will eventually have a distributed bug which spreads inconsistencies on your whole system. Then, you realize that there is no easy way to understand where things are failing: Was it a bug? Was it a network issue? Was the service temporarily unavailable?

That’s is why you have to plan in advance how are you going debug your system. Hopefully, for networking a Service Mesh might be a quick fix, and for distributed logging, tools like FluentD or Logstash are handy. But, when we talk about understanding how an entity reached a specific state, or even how to correlate data between services, there is no easy tool out there.  

To address this issue, you can use Event Sourcing/Logging. In this pattern, each service stores (and validates) all changes to the state of the application in an event object, and naturally, whenever you need to check what happened with a particular entity, all you need to do is navigate through all the log of events related to it. 

If you also add versioning to your state, fixing inconsistencies will be even easier now, as you will have the ability to fix the inconsistent messages by just setting the state of the object to what it was before, and then replay all messages received from the problematic one onwards.

Both versioning and logging can be done asynchronously, and you probably won’t query this data often, which makes it a cheap in-house solution for debugging/auditing your system. I will post next week a deep dive on this pattern, so hold on for a week.

There are many other frameworks/patterns to help you debug your microservices, but they all usually demand a single distributed strategy in order to work. Unfortunately, the moment you realize that you need such a thing, it is already too late and you will need to spend a significant amount of time refactoring the whole thing. That is one of the main reasons why you need to define how are you going to observe/debug your system before even start.

If you have any questions, feel free to tweet me at @deniswsrosa

The post How to Condemn Your Microservices Architecture to Fail Before You Even Start appeared first on The Couchbase Blog.

]]>
http://ico9.allcrypto.company/condemn-microservices-architecture-fail-even-start/feed/ 0 4885
http://ico9.allcrypto.company/geospatial-aspnet-aurelia-google-maps/ http://ico9.allcrypto.company/geospatial-aspnet-aurelia-google-maps/#respond Fri, 23 Mar 2018 17:26:01 +0000 http://ico9.allcrypto.company/?p=4870 Geospatial search is now fully supported in Couchbase Server 5.5. Check out the Couchbase Server 5.5 announcement, and download the developer build for free right now. In this post, I’m going to demonstrate the geospatial capabilities of Couchbase Full Text […]

The post Geospatial Search with ASP.NET Core, Aurelia, and Google Maps appeared first on The Couchbase Blog.

]]>

Geospatial search is now fully supported in Couchbase Server 5.5. Check out the Couchbase Server 5.5 announcement, and download the developer build for free right now.

In this post, I’m going to demonstrate the geospatial capabilities of Couchbase Full Text Search by creating a web-based UI that performs searches. Whenever I think of geospatial searches, I think about Yelp, which is great at helping me find restaurants in a specific area.

So I’m going to have a little fun and create a very bare-bones version of Yelp, but only for hotels.

If you want to follow along, the full source code is available on Github.

Getting set up

Here are the steps I took to create a new project before I started writing code.

  1. At the command line: dotnet new aurelia. This assumes that you have .NET Core installed. Note that Geospatial Search is not a .NET-only feature: you can use it with the other Couchbase SDKs like Node.js, Java, etc. It also assumes that you’ve installed a SPA template for Aurelia. You can also go with Angular or React if you’d like, but I really like Aurelia, and I think you should give it a chance.
  2. The above command will create a shell of an ASP.NET Core project. In this blog post, I’m not going to use Razor. I’m just using ASP.NET as a backend for REST API endpoints.
  3. npm install aurelia-google-maps. You don’t have to use this, but the aurelia-google-maps plugin will make it easy for me to interact with Google Maps in my app.
  4. I opened this project in Visual Studio 2017. I added Couchbase.Extensions.DependencyInjection with NuGet. You don’t have to use this extension but it makes things easier.
  5. I installed Couchbase Server 5.5, including the Full Text Search service. I setup the travel-sample bucket. I created a user “matt” with full access to that bucket.

Create a Geospatial Index

Before building the ASP.NET backend, we need to create a geospatial index in Couchbase Server. Once you log in, click “Search” on the menu (it’s under “Workbench”). Click “Add Index” to get started.

Create Geospatial index

I named my index “mygeoindex”. I selected travel-sample as the bucket to index.

In “Type Mappings”, I uncheck the default. I add a new type mapping with a type name of “hotel”. Every hotel document in “travel-sample” has a type with a value of “hotel”. Check the “only index specified fields” box.

I’m going to add two child fields. One is “geo”, which contains the geospatial coordinates inside a hotel document. Make sure to select “geopoint” as the type. The other is “name”, which will be the name of the hotel. I choose to “store” each of these: it will make the index larger, but I can avoid a secondary lookup if I store the information in the index.

Important Note: There is a bug (NCBC-1651) in the current release of the .NET SDK that will cause an error if you try to read from a geopoint field. In the code samples, I’ve created a workaround: I don’t actually get the geo & name fields from the search index. I instead use the document key returned by search to make a secondary “get” call and get the full document. Keep in mind this is still a technique you may want to consider if you want to keep the size of your index down. This bug has already been fixed and will be in a future release. Such is the peril of being on the cutting edge!

That’s all there is to it. Click “Create Index”. Watch the “indexing progress” on the next screen until it gets to 100% (it should not take very long, assuming you remembered to uncheck “default”).

ASP.NET Core REST Endpoints

Next, let’s move over to ASP.NET. I’ll create two endpoints. One endpoint will demonstrate the bounding box search method, and the other will demonstrate the distance search method.

I’ll need a Couchbase bucket object to execute the queries. Follow the examples in my blog post about dependency injection or check out the source code on Github if you’ve never done this before.

Bounding Box

A “bounding box” search means that you define a box on a map, and you want to search for points of interest that are inside of that box. You only need two points to define a box: the top right corner coordinates and the bottom left corner coordinates. (Coordinates are latitude and longitude).

public class BoxSearch
{
    public double LatitudeTopLeft { get; set; }
    public double LongitudeTopLeft { get; set; }
    public double LatitudeBottomRight { get; set; }
    public double LongitudeBottomRight { get; set; }
}

To create a bounding box geospatial query, use the GeoBoundingBoxQuery class available in the .NET SDK. I’ll do this inside of a POST method with the above BoxSearch class as a parameter.

[Route("api/Box")]
        [HttpPost]
        public IActionResult Box([FromBody] BoxSearch box)
        {
            var query = new GeoBoundingBoxQuery();
            query.TopLeft(box.LongitudeTopLeft, box.LatitudeTopLeft);
            query.BottomRight(box.LongitudeBottomRight, box.LatitudeBottomRight);
            var searchParams = new SearchParams()
                // .Fields("geo", "name") // omitting because of bug NCBC-1651
                .Limit(10)
                .Timeout(TimeSpan.FromMilliseconds(10000));
            var searchQuery = new SearchQuery
            {
                Query = query,
                Index = "mygeoindex",
                SearchParams = searchParams
            };
            var results = _bucket.Query(searchQuery);

// ... snip ...

All I need to return from this endpoint is a list of the results: each hotel’s coordinates and the hotel’s name & location. I created a GeoSearchResult class for this.

public class GeoSearchResult
{
    public double Latitude { get; set; }
    public double Longitude { get; set; }
    public InfoWindow InfoWindow { get; set; }
}

public class InfoWindow
{
    public string Content { get; set; }
}

I’ve constructed this class to match the Google Maps plugin that I’ll be using later.

Finally, I’ll use this class to return some results from the endpoint.

// ... snip ...

            var list = new List<GeoSearchResult>();
            foreach (var hit in results.Hits)
            {
                // *** this part shouldn't be necessary
                // the geo and name should come with the search results
                // but there's an SDK bug NCBC-1651
                var doc = _bucket.Get<dynamic>(hit.Id).Value;
                // ****************
                list.Add(new GeoSearchResult
                {
                    Latitude = doc.geo.lat,
                    Longitude = doc.geo.lon,
                    InfoWindow = new InfoWindow
                    {
                        Content = doc.name + "<br />" +
                            doc.city + ", " +
                            doc.state + " " +
                            doc.country
                    }
                });
            }
            return Ok(list);
        }

A “distance” search is another way to perform geospatial queries. This time, instead of a box, it will be more like a circle. You supply a single coordinate, and a distance. The distance will be the radius from that point.

public class PointSearch
{
    public double Latitude { get; set; }
    public double Longitude { get; set; }
    public int Distance { get; set; }
    // miles is being assumed as the unit
    public string DistanceWithUnits => Distance + "mi";
}

I’m defaulting it to miles, but certainly you can use kilometers instead, or present the option in the UI.

The endpoint will be very similar to the bounding box endpoint, except that it uses GeoDistanceQuery.

[Route("api/Point")]
[HttpPost]
public IActionResult Point([FromBody] PointSearch point)
{
    var query = new GeoDistanceQuery();
    query.Latitude(point.Latitude);
    query.Longitude(point.Longitude);
    query.Distance(point.DistanceWithUnits);
    var searchParams = new SearchParams()
        // .Fields("geo", "name") // omitting because of bug NCBC-1651
        .Limit(10)
        .Timeout(TimeSpan.FromMilliseconds(10000));
    var searchQuery = new SearchQuery
    {
        Query = query,
        Index = "mygeoindex",
        SearchParams = searchParams
    };
    var results = _bucket.Query(searchQuery);

    var list = new List<GeoSearchResult>();
    foreach (var hit in results.Hits)
    {
        // *** this part shouldn't be necessary
        // the geo and name should come with the search results
        // but there's an SDK bug NCBC-1651
        var doc = _bucket.Get<dynamic>(hit.Id).Value;
        // ****************
        list.Add(new GeoSearchResult
        {
            Latitude = doc.geo.lat,
            Longitude = doc.geo.lon,
            InfoWindow = new InfoWindow
            {
                Content = doc.name + "<br />" +
                          doc.city + ", " +
                          doc.state + " " +
                          doc.country
            }
        });
    }
    return Ok(list);
}

At this point, you can start testing these endpoint with Postman or Fiddler if you’d like. But it will be so much nice to see this on a map.

Auerlia and Google Maps

In Aurelia, I’ve created two components: geosearchbox and geosearchpoint.

Auerlia components

Each of them will have a Google Maps component that the user can interact with. These maps will be centered on San Francisco, because that’s where a lot of the hotels in “travel-sample” are located.

Bounding Box search component

The google-map` component has a map-click.delegate that will will fire whenever the users clicks on the map. In geosearchbox.html:

<google-map
    if.bind="markers"
    map-click.delegate="clickMap($event)"
    latitude="37.780986253433895"
    longitude="-122.45291600632277"
    zoom="12"
    markers.bind="markers">
</google-map>

markers is simply an array containing coordinates of search results that should appear on the map. Initially it will be empty.

When the user first clicks the map, this will set the first coordinate (top left) in the form. In geosearchbox.ts:

public clickMap(event : any) {
    var latLng = event.detail.latLng,
        lat = latLng.lat(),
        lng = latLng.lng();

    // only update top left if it hasn't been set yet
    // or if bottom right is already set
    if (!this.longitudeTopLeft || this.longitudeBottomRight) {
        this.longitudeTopLeft = lng;
        this.latitudeTopLeft = lat;
        this.longitudeBottomRight = null;
        this.latitudeBottomRight = null;
    } else {
        this.longitudeBottomRight = lng;
        this.latitudeBottomRight = lat;
    }
}

Then, click another spot on the map. This will set the second coordinate (bottom right).

My implementation is very bare bones. No fancy graphics and no validation of the second coordinate being to the bottom right of the first. The fields on a form will simply be populated with the latitude and longitude. In geosearchbox.html:

<p>
    Bounding box search:
    <br />
    Latitude (top left):
        <input type="text" value="${ latitudeTopLeft }" />
    Longitude (top left):
        <input type="text" value="${ longitudeTopLeft }" />
    <br />
    Latitude (bottom right):
        <input type="text" value="${ latitudeBottomRight }" />
    Longitude (bottom right):
        <input type="text" value="${ longitudeBottomRight }" />
    <br />
    <input
        if.bind="latitudeTopLeft && latitudeBottomRight"
        click.trigger="searchClick()"
        type="button"
        name="search"
        value="Search" />
</p>

Once you’ve selected two coordinates, a search button will appear. Click that to post these coordinates to the endpoint created earlier, and it will trigger the searchClick() method as seen in geosearchbox.ts:

public searchClick() {
    let boxSearch = {
        latitudeTopLeft: this.latitudeTopLeft,
        longitudeTopLeft: this.longitudeTopLeft,
        latitudeBottomRight: this.latitudeBottomRight,
        longitudeBottomRight: this.longitudeBottomRight
    };

    console.log("POSTing to api/Box: " + JSON.stringify(boxSearch));

    this.http.fetch('api/Box', { method: "POST", body: json(boxSearch) })
        .then(result => result.json() as Promise<any[]>)
        .then(data => {
            this.markers = data;
        });
}

When Aurelia, Google Maps, ASP.NET Core, and Couchbase all work together, it looks like this:

Geospatial bounding box

Distance Search

Implementing the “distance” geostatial query will be similar to the bounding box UI. This time, you only need to click a single point on the map. But, you will need to type in a distance (in miles).

The google-map component will look identical. The clickMap function is different:

public clickMap(event: any) {
    var latLng = event.detail.latLng,
        lat = latLng.lat(),
        lng = latLng.lng();

    this.longitude = lng;
    this.latitude = lat;
}

Specify a distance (in miles), and then click ‘search’ to make a POST request to the endpoint we wrote earlier.

geosearchbox.html:

    <p>
        Distance search:
        <br />
        Latitude: <input type="text" value="${ latitude }" />
        Longitude: <input type="text" value="${ longitude }" />
        <br />
        Distance (miles): <input type="text" value="${ distance }" />
        <br />
        <input if.bind="latitude" click.trigger="searchClick()" type="button" name="search" value="Search" />
    </p>

geosearchbox.ts:

    public searchClick() {
        let pointSearch = {
            latitude: this.latitude,
            longitude: this.longitude,
            distance: this.distance
        };

        console.log("POSTing to api/Point: " + JSON.stringify(pointSearch));

        this.http.fetch('api/Point', { method: "POST", body: json(pointSearch) })
            .then(result => result.json() as Promise<any[]>)
            .then(data => {
                this.markers = data;
            });
    }
}

Below is a clip of the search in motion. Note how the results change as I move the coordinate around.

Geospatial distance point search query

Summary

With Couchbase’s built-in geospatial indexing and search feature, all the math and the searching is delegated to the Couchbase Data Platform. So you can focus on building a killer UI (better than mine anyway) and rock-solid business logic.

Be sure to check out the documentation for a complete overview of the geospatial capabilities of Couchbase.

If you need help or have questions, please check out the Couchbase Server forums, and if you have any questions about the Couchbase .NET SDK, check out the .NET SDK forums.

If you’d like to get in touch with me, please leave a comment or find me on Twitter @mgroves.

The post Geospatial Search with ASP.NET Core, Aurelia, and Google Maps appeared first on The Couchbase Blog.

]]>
http://ico9.allcrypto.company/geospatial-aspnet-aurelia-google-maps/feed/ 0 4870
http://ico9.allcrypto.company/ipv6-galore-couchbase/ http://ico9.allcrypto.company/ipv6-galore-couchbase/#respond Thu, 22 Mar 2018 16:33:07 +0000 http://ico9.allcrypto.company/?p=4827 Every single device on the internet, whether it be a laptop computer, cell phone, camera, or even your TV has an IP address. The total number of Internet Protocol version 4 (IPv4) addresses is over 4 billion, which is far […]

The post IPv6 Galore With Couchbase appeared first on The Couchbase Blog.

]]>
Every single device on the internet, whether it be a laptop computer, cell phone, camera, or even your TV has an IP address. The total number of Internet Protocol version 4 (IPv4) addresses is over 4 billion, which is far less than the total number of mobile devices in the world. As more and more appliances and devices come online everyday (as shown in the figure below), connecting them to the internet using IPv4 addresses might not be an option, because these addresses have already run out. To support the growing needs of our customers, we are happy to announce IPv6 support in Couchbase Server 5.5.

Figure 1 : Growth of devices and IPv6

First defined in 1996, Internet Protocol Version 6 (IPv6) is the most current version of the Internet Protocol (IP), and incorporates many features that address several IPv4 challenges. Among the many challenges, it can support accommodating over 340 undecillion addresses — more than enough to handle the world’s current IP needs.

Enabling IPv6 in Couchbase

By default, when Couchbase Server starts up it uses IPv4. To enable IPv6 in Couchbase Server 5.5 and above, here are a few steps for the linux platform. (Note that based on your OS platform, the steps might be slightly different.)

Repeat the following steps for each node of the cluster :

  1. Install Couchbase Server
  2. Stop the couchbase-server service.
  3. In /opt/couchbase/etc/couchbase/static_config, set ipv6 to true.
  4. Delete /opt/couchbase/var/lib/couchbase/config/config.dat.
  5. Restart the couchbase-server service: this starts Couchbase Server in IPv6 mode. To check, point the browser at http://[::1]:8091, which is the IPv6 address and port number on which Couchbase Server should be running.
Note: In a clustered setup, IPv6 is enabled only when IPv6 is enabled on all the nodes

How do you know if IPv6 is working?

For now, you can check the logs and see if IPv6 addresses are there:

As shown in the figure above, you will see machine addresses in IPv6 format.

Upgrading an IPv4 cluster to IPv6

Given the complex nature of managing both an IPv4 and an IPv6 environment in general, most users are going to start a-new instead of upgrading in-place.  In the event that you need to upgrade/migrate your data from an existing cluster, we recommend using our Cross Datacenter Replication feature:

  1. The source cluster must have dual stack support at the OS-level
  2. Upgrade the source cluster to Couchbase Server 5.5
  3. Create a new cluster with IPv6 enabled and configured, as above.  Note that this cluster needs to be sized appropriately for the workload, but does not need to be identical to the source cluster.
  4. Create a cluster reference and replication stream from the source to the destination cluster. 
  5. Monitor the XDCR queue from the source until all mutations are replicated to the destination cluster.
  6. Reconfigure the application to start accessing the destination cluster.
  7. Once all applications are moved, the source cluster can be decommissioned.
Note: IPv4 and IPv6 clusters cannot be paired for bi-directional XDCR replication. For bi-directional replication, both clusters need to be IPv4 or IPv6.

Get going with IPv6

It’s time to get going with IPv6!  Try the IPv6 feature out in Couchbase Server 5.5 developer build and give us your feedback. What are you waiting for? Head over to the Couchbase Downloads page and try out Couchbase 5.5 Developer Build – along with all the other new features in Couchbase Server 5.5 Developer Build. Review the documentation for Couchbase IPv6 feature here. Let us know what you think, we want to hear from you.

Note:  To use 5.5 Developer Build, you need to do a fresh install. The developer build release must only be used in a non-production setting. We consider Developer Build releases to have some rough edges and bugs.

The post IPv6 Galore With Couchbase appeared first on The Couchbase Blog.

]]>
http://ico9.allcrypto.company/ipv6-galore-couchbase/feed/ 0 4827
http://ico9.allcrypto.company/geospatial-queries-using-python-search-cities/ http://ico9.allcrypto.company/geospatial-queries-using-python-search-cities/#respond Thu, 22 Mar 2018 13:37:43 +0000 http://ico9.allcrypto.company/?p=4864 Using Python we are creating a tool that will use geospatial queries with Couchbase REST API and Couchbase Full Text Search, which will help us in searching a database of cities.

The post Geospatial queries: Using Python to search cities appeared first on The Couchbase Blog.

]]>
16665891 10211247083188500 2291386332147570222 o

Daniel Ancuta is a software engineer with several years of experience using different technologies. He’s a big fan of “The Zen of Python,” which he tries to apply not only in his code but in his private life as well. You can find him on Twitter: @daniel_ancuta

Geospatial queries: Using Python to search cities

Geolocation information is used every day in almost every aspect of our interaction with computers. Either it’s a website that wants to send us personalized notifications based on location, maps that show us the shortest possible route, or just tasks running in the background that checks the places we’ve visited.

Today, I’d like to introduce you to geospatial queries that are used in Couchbase. Geospatial queries allow you to search documents based on their geographical location.

Together, we will write a tool in Python that uses geospatial queries with Couchbase REST API and Couchbase Full Text Search, which will help us in searching a database of cities.

Prerequisites

Dependencies

In this article I used Couchbase Enterprise Edition 5.1.0 build 5552 and Python 3.6.4.

To run snippets from this article you should install Couchbase 2.3 (I am using 2.3.4) via pip.

Couchbase

  1. Create a cities bucket
  2. Create a cities search with geo field of type geopoint type. You can read about it in the Inserting a Child Field part of the documentation.

It should look like the image below:

image 1

Populating Couchbase with data

First of all, we need to have data for our exercise. For that, we will use a database of cities from geonames.org.

GeoNames contains two main databases: list of cities and list of postal codes.

All are grouped by country with corresponding information like name, coordinates, population, time zone, country code, and so on. Both are in CSV format.

For the purpose of this exercise, we will use the list of cities. I’ve used PL.zip but feel free to choose whichever you prefer from the list of cities.

Data model

City class will be our representation of a single city that we will use across the whole application. By encapsulating it in a model, we unify the API and don’t need to rely on third-party data sources (e.g., CSV file) which might change.

Most of our snippets are located (until said otherwise) in the core.py file. So just remember to update it (especially when adding new imports) and not override the whole content.

# core.py
class City:
   def __init__(self, geonameid, feature_class, name, population, lat, lon):
       self.geonameid = geonameid
       self.feature_class = feature_class
       self.name = name
       self.population = population
       self.lat = lat
       self.lon = lon

   @classmethod
   def from_csv_row(cls, row):
       return cls(row[0], row[7], row[1], row[12], row[4], row[5])

 

CSV iterator to process cities

As we have a model class, it’s time to prepare an iterator that will help us to read the cities from the CSV file.

# core.py
import csv
from collections import Iterator

class CitiesCsvIterator(Iterator):
   def __init__(self, path):
       self._path = path
       self._fp = None
       self._csv_reader = None

   def __enter__(self):
       self._fp = open(self._path, 'r')
       self._csv_reader = csv.reader(self._fp, delimiter='\t')

       return self

   def __exit__(self, exc_type, exc_val, exc_tb):
       self._fp.close()

   def __next__(self):
       return City.from_csv_row(next(self._csv_reader))

Insert cities to Couchbase bucket

We have unified the way to represent a city, and we have an iterator that would read those from csv file.

It’s time to put this data into our main data source, Couchbase.

# core.py
import logging
import sys
from couchbase.cluster import Cluster, PasswordAuthenticator


logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler(sys.stdout))


def get_bucket(username, password, connection_string='couchbase://localhost'):
   cluster = Cluster(connection_string)
   authenticator = PasswordAuthenticator(username, password)
   cluster.authenticate(authenticator)

   return cluster.open_bucket('cities')


class CitiesService:
   def __init__(self, bucket):
       self._bucket = bucket

   def load_from_csv(self, path):
       with CitiesCsvIterator(path) as cities_iterator:
           for city in cities_iterator:
               if city.feature_class not in ('PPL', 'PPLA', 'PPLA2', 'PPLA3',
                                             'PPLA4', 'PPLC'):
                   continue

               logger.info(f'Inserting {city.geonameid}')
               self._bucket.upsert(
                   city.geonameid,
                   {
                       'name': city.name,
                       'feature_class': city.feature_class,
                       'population': city.population,
                       'geo': {'lat': float(city.lat),
                               'lon': float(city.lon)}
                   }
               )

 

To check if everything we wrote so far is working, let’s load CSV content into Couchbase.

# core.py

bucket = get_bucket('admin', 'test123456')
cities_service = CitiesService(bucket)
cities_service.load_from_csv('~/directory-with-cities/PL/PL.txt', bucket)

 

At this point you should have cities loaded into your Couchbase bucket. The time it takes depends on the country you have chosen.

Search cities

We have our bucket ready with data, so it’s time to come back to CitiesService and prepare a few methods that would help us in searching cities.

But before we start, we need to modify the City class slightly, by adding the following method:

# core.py

@classmethod
def from_couchbase_dict(cls, row):
fields = row['fields']

return cls(row['id'],
fields['feature_class'],
fields['name'],
fields['population'],
fields['geo'][1],
fields['geo'][0])

That’s a list of methods we will implement in CitiesService:

  • get_by_name(name, limit=10), returns cities by their names
  • get_by_coordinates(lat, lon), returns city by coordinates
  • get_nearest_to_city(city, distance=’10’, unit=’km’, limit=10), returns nearest city

get_by_name

# core.py
from couchbase.fulltext import TermQuery

INDEX_NAME = 'cities'

def get_by_name(self, name, limit=10):
result = self._bucket.search(self.INDEX_NAME,
TermQuery(name.lower(), field='name'),
limit=limit,
fields='*')

for c_city in result:
yield City.from_couchbase_dict(c_city)

 

get_by_coordinates

# core.py
from couchbase.fulltext import GeoDistanceQuery

INDEX_NAME = 'cities'

def get_by_coordinates(self, lat, lon):
result = self._bucket.search(self.INDEX_NAME,
GeoDistanceQuery('1km', (lon, lat)),
fields='*')

for c_city in result:
yield City.from_couchbase_dict(c_city)

 

get_nearest_to_city

# core.py
from couchbase.fulltext import RawQuery, SortRaw

INDEX_NAME = 'cities'

def get_nearest_to_city(self, city, distance='10', unit='km', limit=10):
query = RawQuery({
'location': {
'lon': city.lon,
'lat': city.lat
},
'distance': str(distance) + unit,
'field': 'geo'

})
sort = SortRaw([{
'by': 'geo_distance',
'field': 'geo',
'unit': unit,
'location': {
'lon': city.lon,
'lat': city.lat
}
}])

result = self._bucket.search(self.INDEX_NAME,
query,
sort=sort,
fields='*',
limit=limit)

for c_city in result:
yield City.from_couchbase_dict(c_city)

As you might notice in this example, we used RawQuery and SortRaw classes. Sadly, couchbase-python-client API does not work correctly with the newest Couchbase and geo searches.

Call methods

As we now have all methods ready, we can call it!

# core.py

bucket = get_bucket('admin', 'test123456')

cities_service = CitiesService(bucket)
# cities_service.load_from_csv('/my-path/PL/PL.txt')

print('get_by_name')
cities = cities_service.get_by_name('Poznań')
for city in cities:
print(city.__dict__)

print('get_by_coordinates')
cities = cities_service.get_by_coordinates(52.40691997632544,
16.929929926276657)
for city in cities:
print(city.__dict__)

print('get_nearest_to_city')
cities = cities_service.get_nearest_to_city(city)
for city in cities:
print(city.__dict__)

Where to go from here?

I believe this introduction will enable you to work on something more advanced.

There are a few things that you could do:

  • Maybe use a CLI tool or REST API to serve this data?Improve the way we load data, because it might not be super performant if we want to load ALL cities from ALL countries.

You can find the whole code of core.py in github gist.

If you have any questions, don’t hesitate to tweet me @daniel_ancuta.

 

This post is part of the Community Writing program

The post Geospatial queries: Using Python to search cities appeared first on The Couchbase Blog.

]]>
http://ico9.allcrypto.company/geospatial-queries-using-python-search-cities/feed/ 0 4864
http://ico9.allcrypto.company/index-partitioning-couchbase-server/ http://ico9.allcrypto.company/index-partitioning-couchbase-server/#respond Mon, 19 Mar 2018 20:17:25 +0000 http://ico9.allcrypto.company/?p=4804 Index partitioning is a new feature available in Couchbase Server 5.5. Check out the Couchbase Server 5.5 Developer Build announcement and download the developer build for free right now. In this post, I’m going to cover: Why you might want to […]

The post Index Partitioning in Couchbase Server 5.5 appeared first on The Couchbase Blog.

]]>

Index partitioning is a new feature available in Couchbase Server 5.5.

In this post, I’m going to cover:

  • Why you might want to use index partitioning
  • An example of how to do it
  • Partition elimination
  • Some caveats to be careful with

Index partitioning

When developing your application, you may want to take advantage of Couchbase Server’s ease of scaling to give more resources to indexing. With multi-dimensional scaling (MDS), one option is that you can add multiple high-end machines to the cluster with index capabilities as you need them.

To take advantage of multiple nodes with index services, you would have to create index replicas. This is still possible, and if this is working for you, it’s not going away.

Index replicas

Couchbase Server 5.5 is introducing another way to spread the index load around: index partitioning. Instead of replicating an index, you can now split up the index amongst the nodes with hashing.

Index partitioning

And, you can use partitioning and replicas together in concert. Index partition replicas will be used automatically, and with no interruption if a node goes down.

Index partitioning with replicas

The primary benefits to index partitioning:

  • The index scan load is now balanced amongst all the index nodes. This leads to a more even distribution of work and better performance.
  • A query that uses aggregation (e.g. SUM + GROUP BY) can be run in parallel on each partition.

How to use index partitioning

The syntax for creating an index partition is PARTITION BY HASH(<field>). For example, if I wanted to create a compound index on the “travel-sample” bucket for fields airline, flight, source_airport, and destination_airport:

CREATE INDEX ix_route ON `travel-sample` (airline, flight, source_airport, destination_airport) PARTITION BY HASH(airline);

When you create this index, it will show as “partitioned” in the Couchbase Console.

Index partitioning in Couchbase Console

Partition elimination

Partition elimination is one of the benefits of using index partitioning. This is a feature that is unique in the NoSQL market to Couchbase.

Let’s say you have a partition on the airline field, as above. Next, write a query that uses that index and specifies the airline value:

SELECT t.*
FROM `travel-sample` t
WHERE airline IN ["UA", "AA"]
AND source_airport = "SFO"

Then, the indexing service will only scan the matching partitions (“UA” and “AA”). This leads to a faster range query response, and the same latency as a non-partitioned index, regardless of cluster size. More on this later.

Index partitioning caveats

You may have noticed that “airline” was used in the above CREATE INDEX command. When using index partitioning, you must specify one (or more) fields to give to the hash to use for partitioning. This hash will determine how to divide up the index.

The simplest thing you can do is use the document key in the hash:

CREATE INDEX ix_route2 ON `travel-sample` (airline, flight, source_airport, destination_airport) 
PARTITION BY HASH(META().Id);

But unlike Couchbase’s key-value engine, you can use whatever fields you like. But you must keep in mind that these fields should be considered immutable. That is, you shouldn’t change the value of the fields. So, if you have a field with a value that does not typically change (many Couchbase users create a “type” field, for instance), that would be a good candidate for partitioning.

If you choose to index on an “immutable” field, be warned that this may also cause some skewing of the partition (using META().Id will minimize the amount of skew). If you partition on the “type” field, where 10% of documents have a type of “order” and 90% of documents have a type of “invoice”, then the partition is likely to look similar. Index partitioning uses an optimization algorithm to balance RAM, CPU and data size during rebalancing, but skewing is still going to be a possibility.

So why wouldn’t you use META().Id to reduce skewing? Recall the partition elimination section above. If your queries fall along the same lines as your index partitions, then you are minimizing the “scatter+gather” operations having to check all partitions, and you can further reduce latency.

One more caveat: index partitioning is an Enterprise Edition only feature.

Summary

Index partitioning allows you to more easily and automatically scale out your indexing capabilities. If you are using a lot of N1QL queries in your project, then this will come in very handy and make your job much easier.

If you have any questions about index partitioning or anything else with indexing, please check out the Couchbase Server Forums or the N1QL forums if you have questions about creating indexes and queries. Be sure to download the developer build of Couchbase Server 5.5 today, and try it out. We’d love to hear your feedback.

You can reach me by leaving a comment below or finding me on Twitter @mgroves.

The post Index Partitioning in Couchbase Server 5.5 appeared first on The Couchbase Blog.

]]>
http://ico9.allcrypto.company/index-partitioning-couchbase-server/feed/ 0 4804
http://ico9.allcrypto.company/timed-tasks-using-couchbase-go/ http://ico9.allcrypto.company/timed-tasks-using-couchbase-go/#respond Mon, 19 Mar 2018 18:50:44 +0000 http://ico9.allcrypto.company/?p=4852 Alberto Marchetti is a full-stack developer, and author of “RenderScript: parallel computing on Android, the easy way.” He is always living on the edge by constantly jumping into the discovery of modern languages and technologies.   Timed tasks using Couchbase and Go In […]

The post Timed tasks using Couchbase and Go appeared first on The Couchbase Blog.

]]>
Alberto Marchetti is a full-stack developer, and author of “RenderScript: parallel computing on Android, the easy way.” He is always living on the edge by constantly jumping into the discovery of modern languages and technologies.

 

Timed tasks using Couchbase and Go

In this post I’m going to show how you can exploit the Couchbase indexing system to create a timed-tasks distributed system. The example code for the project, together with its run instructions, can be found at https://github.com/cmaster11/cb-blog-timed-tasks.
Disclaimer: Because of the complexity of the topic, only relevant code extracts are posted in this page.

The concept

Let’s try to define the requirements of such a system:

  • The main feature of a timed tasks system is to be able to specify when a certain task will be executed in time. This can be achieved by using an ExecuteAt field, which will contain the desired execution time (Unix time, milliseconds-based).
  • A modern software requirement is that it must support a multi-node environment, which means it needs to be a distributed system. We must then ensure that multiple workers will NOT process the same tasks! We can use a nice Couchbase feature here, pessimistic locking, which will let a worker fetch a document and lock it down, so that no other workers can process it.

The following is a possible structure to represent our task:

type Task struct {
    Id string

    // The desired task execution time
    ExecuteAt int64

    // Task-specific content
    Content string
}

 

Couchbase features

First, here’s an overview of the Couchbase features we’ll be using:

META()

Every document in a Couchbase bucket has an associated META()-document, which contains document entity-specific information like:

  • id – the document key inside the bucket.
  • cas – an int64 number, used by Couchbase to prevent race conditions while editing documents.
  • expiration – when a document is meant to expire, or 0 if it will never expire.

Hint: These fields (e.g., META().cas) can be indexed (starting from Couchbase 5.0).

CAS (Check and Set)

When fetching a document, its CAS value is returned too, and subsequent calls to alter the document can specify this value to make sure they’re going to edit the desired version of the document.

Example:

  • Client A fetches a document and its current CAS value is 1234.
  • Client B edits the document, which alters the CAS value to 5678.
  1. If A tries to edit the document without providing the CAS value, the edit will be successful, but changes made by B will be lost.
  2. If A tries to edit the document providing the CAS value (1234), an error will be returned because the current one (5678) is different. Client A will then need to fetch the document again and re-execute the process.

The CAS value is an extremely useful tool to ensure we’re not replacing or altering a wrong/newer version of a document, losing its changes.

Pessimistic locking

Couchbase lets us “lock” a document, so that it can only be read and written by one client at a time, using gocb.GetAndLock Go SDK function.

// Lock the document
lockTime := 10 // seconds
lockedCAS, err := bucket.GetAndLock(documentKey, lockTime, &outStruct)

// Unlock it
_, err = bucket.Unlock(documentKey, lockedCAS)

 

When a document is locked, every other request to lock/mutate/unlock it will throw an error (it’s still possible to simply get the document), unless the correct CAS value is used.

Note: The maximum lock time of a document is 15 seconds, and using a lockTime value of 0 will cause the maximum time to be set. This creates a limitation on how long a task can run before being automatically marked as available (by locking timeout).

Hint: While a document is locked, its returned CAS value is -1.

Indexing and querying

Of note two hints put together tell us that we can index a field (META().cas), which turns to -1 when a document is locked. It also means  that we can query documents based on this condition!

The query

Let’s try to define a query to match the requirements:

  • We want to get a task id, which can be used later to get-and-lock the document: SELECT Id.
  • The task should not be already locked: WHERE META().cas <> -1.
  • The task needs to be executed now: WHERE ExecuteAt <= NOW_MILLIS() (NOW_MILLIS returns the current Unix time in milliseconds).
  • We need to fetch the closest task in time, so we want to sort tasks by their execution time: ORDER BY ExecuteAt ASC.
  • Let’s say for now (!!!) that a worker will want to get only one task to process at a time: LIMIT 1.

The result should be similar to this query:

SELECT `Id`
FROM `timed_tasks` // Our bucket
WHERE META().`cas` <> -1
AND `ExecuteAt` <= NOW_MILLIS()
ORDER BY `ExecuteAt` ASC
LIMIT 1

Its execution will return an array similar to:

[{
    "Id": "task_id_goes_here"
}]

 

The index

We can now plan a query-specific index, optimized for the execution of the query we just thought about. Query-specific indexes are a must to improve NoSQL database query performance.

  • The query is checking that a document is not currently locked:
    WHERE META().cas <> -1.
  • Also, it’s directly asking the execution time to be in the past. We then need to index the ExecuteAt field.
    The index query could then be the following:

CREATE INDEX `idx_timed_task`
ON `timed_tasks`
(`ExecuteAt` ASC)
WHERE META().`cas` <> -1

 

Optimizing the query

We can now further optimize the query:

  • We can tell the query to use our index by providing a hint to it: USE INDEX (idx_timed_task USING GSI).
  • We can ask Couchbase to wait for the index to be up to date (usually indexing is an asynchronous process) before executing the query, so that our results will for sure contain unlocked tasks, by providing a consistency requirement at SDK level: query.Consistency(gocb.RequestPlus).

The flow

A possible flow for the timed task consumer worker loop is:

  1. Query for an available task id.
  2. Get and lock the task.
  3. Process the task.
  4. Delete the task.

Multiple nodes

Let’s think for a second about how a multi-node setup can alter this flow.

If multiple workers are going to query for available tasks concurrently, they’d probably find the same task, and only one of them would be able to process it successfully, while the other workers will have to repeat the loop (execute a new query) to get new tasks.

We can implement then another approach:

  1. Query for available tasks ids, limiting the amount of ids to the number of workers.
  2. For each task id, try to lock the task. At first successful lock, go to 4.
  3. If no tasks have been successfully locked, repeat loop.
  4. Process the task.
  5. Delete the task.

At its best, every worker will be able to successfully lock one task at first try. At its worse, workers will need to try to lock multiple documents unsuccessfully. The average execution will see workers successfully locking tasks, maybe after trying to lock a few others.

We have to make a compromise between how frequently we want to query the database, and how many failed lock attempts we can support. Generally speaking, trying to lock documents will be much faster than executing N1QL queries.

The code

Let’s take a look at some relevant code examples:

The producer

The generation of the task can be summed up in this function:

func NewTask(executeAt time.Time, content string) (*Task, error) {
    if executeAt.IsZero() {
        return nil, errors.New("executeAt must not be a zero time")
    }

    taskUUID, err := uuid.NewV1() // github.com/satori/go.uuid
    if err != nil {
        return nil, err
    }

    // Convert time.Time to int64 milliseconds
    executeAtMillis := executeAt.UnixNano() / int64(time.Millisecond)

    task := Task{
        Id:        taskUUID.String(),
        ExecuteAt: executeAtMillis,
        Content:   content,
    }

    return &task, nil
}

 

Once we generate a valid task object, we can simply insert it in our bucket with:

_, err := controller.bucket.Insert(task.Id, task, 0)

 

The consumer

We can get and lock a document by id, using this code:

// Using zero values for lock time will set the maximum time available.
task := new(Task)
lockedCAS, err := controller.bucket.GetAndLock(taskId, 0, &task)

A task can be removed using this code:

_, err := controller.bucket.Remove(taskId, lockedCAS)

The main consumer code can be summed up with the following snippet:

taskIds, err := couchbaseController.QueryNextTaskIds(consumersCount)
...

if len(taskIds) == 0 {
    ...
    // No tasks have been found, restart the loop
}

var taskId string
var task *internal.Task
var lockedCAS gocb.Cas

for _, taskId = range taskIds {
    // Lock and get the task, so that only this consumer will process it
    task, lockedCAS, err = couchbaseController.GetAndLockTask(taskId)
    if err != nil {
        ...
        // Error getting the task, proceed to next one in list
        continue
    }

    // Successfully locked task!
    // Move out to process it
    break
}

if task == nil {
    ...
    // No tasks could be locked, restart loop
}

// Actual processing of the task
// Improvement: could also return an error, which would let the task be
// processed by another worker later.
processTask(task)

/*
Remove the task from Couchbase.
The task will be currently locked, which means we need to provide the
current CAS value, so that the producer is authorized to remove it.
 */
err = couchbaseController.RemoveTask(taskId, lockedCAS)
...

 

Conclusion

In this post we’ve seen a way to create a reliable distributed timed tasks system using Couchbase and Go.

This system could be further developed by:

  • Support for processing errors.
  • Implementing a retry feature (if processing fails, reschedule the task in the future).
  • Improving the locking logic by:
    • Tuning the maximum number of returned task ids (instead of the default workers count).
    • Supporting a task processing duration of more than 15 seconds (the maximum lock time of a document in Couchbase).

Thank you for your time, and happy developing!

This post is part of the Community Writing program

The post Timed tasks using Couchbase and Go appeared first on The Couchbase Blog.

]]>
http://ico9.allcrypto.company/timed-tasks-using-couchbase-go/feed/ 0 4852
http://ico9.allcrypto.company/kubernetes-operators-game-changer/ http://ico9.allcrypto.company/kubernetes-operators-game-changer/#comments Mon, 19 Mar 2018 16:17:21 +0000 http://ico9.allcrypto.company/?p=4848 The whole web developer community is excited about Kubernetes (K8s). No wonder it is the hottest topic at the conferences and developer events that I have been to the last year. It is not just a tool for managing containers, in […]

The post Why Kubernetes Operators are a game changer appeared first on The Couchbase Blog.

]]>
The whole web developer community is excited about Kubernetes (K8s). No wonder it is the hottest topic at the conferences and developer events that I have been to the last year.

It is not just a tool for managing containers, in fact, K8s allows you to easily add load balancing and avoids the need of a service discovery layer (you no longer need eureka for instance).  K8s also automate application deployments and updates, and most importantly, let you plug/write custom controller for your infrastructure.

Fantastic, right? But managing stateless containers isn’t that complicated, after all, they are essentially ephemeral and you can kill and spin up instances as you wish without any considerable side effects.

But this is half of the story, in general, applications can’t be entirely stateless, and in most cases, we simply push the state to the layers down below. So, how do we deal with stateful applications in K8s? Luckily, since version 1.5 there is something called StatefulSets.

 

Stateful Containers

Kubernetes StatefulSets gives you a set of resources to deal with stateful containers, such as: volumes, stable network ids, ordinal indexes from 0 to N, etc. Volumes are one of the key features which allow us to run stateful applications on top of Kubernetes, let’s see the two main types currently supported:

 

Ephemeral storages volumes

The behavior of ephemeral storages is different than what you are used to in Docker, in Kubernetes the volume outlives any containers that run within the Pod and the data is preserved across container restarts. But if the Pod get killed, the volume is automatically removed.

 

Persistent storage volumes

In a persistent storage, as the name suggests, the data lifetime is independent of the Pod’s lifetime. So, even when the Pod dies or is moved to another node, that data will still persists until it is explicit deleted by the user. In those kind of volumes, the data is stored typically remotely.

 

We are looking forward for Kubernetes support to Local Persistent Storages as it will definitely be the best fit for running databases, but in the meantime, we use ephemeral storages by default. At this point, you might wonder why we are using ephemeral storages instead of the persistent ones. Not surprisingly, there are many reasons for that:

  • Ephemeral storages are faster and cheaper than persistent, it would require more infrastructure/networking to use persistent storages as you need to send the data back and forth
  • K8s 1.9 introduced Raw Block Support, which allows you to access physical disks in your VM instance to use it in your application.
  • Maintain networked storage systems is not trivial
  • You can always try to reboot the container first instead of killing the whole Pod:  
    kubectl exec POD_NAME -c CONTAINER_NAME reboot
  • You can configure Couchbase to automatically replicate your data, so even if N Pods dies, no data will be lost.
  • Part of K8s job is to run Pods in different racks to avoid massive failures.

However, there are a few scenarios where using Remote Persistent Storages would worth the extra latency cost, like in massive databases for instance, when the rebalancing process take several minutes to finish. That is why we also will add support for Remote Persistent Storages.

One of the downsides of Statefulsets is the limited management, that is why we decided to extend the Kubernetes API through the use of a Custom Resource Definition (CRD), which allows us to create a custom native resource in Kubernetes similar to a StatefulSet or a Deployment, but designed specifically for managing Couchbase instances.

Great! So, with StatefulSets/CRDs we have all the hardware operations arranged, there is just a “small” thing missing here, what about the state of the application itself? In a database, for instance, adding a new node to the cluster is not nearly enough, you would still be required to trigger some processes, such as rebalancing to move/replicate some of the data to the newly added node to make it fully operational. That is exactly why K8s Operators joined the game.

 

 

Kubernetes Operators

 

Kubernetes 1.7 has added an important feature called Custom Controllers.  In summary, it enables developers to extend and add new functionalities, replace existent ones (like replacing kube-proxy for instance), and of course, automate administration tasks as if they were a native Kubernetes component.

An Operator is nothing more than a set of application-specific custom controllers. So, why is it a game changer? Well, controllers have direct access to Kubernetes API, which means they can monitor the cluster, change pods/services, scale up/down and call endpoints of the running applications, all according to custom rules written inside those controllers.

To illustrate this behavior, let’s see how Couchbase’s Operator works when a Pod gets killed:

You can find out more about operators here

As you can see in the figure above, the Operator monitors and analyzes the cluster, and based on a set of parameters, trigger a series of actions to achieve the desired state. This reconciliation process is all over the place in K8s. But not all actions are equal, in our example, we have two distinct categories:

  • Infrastructure – add a new node: The operator requests via Kubernetes API to launch a new Pod running Couchbase Server.
  • Domain Specific – Add node to the cluster/ Trigger data rebalancing: The operator knows how Couchbase works and calls the correct rest endpoint to add the new node to the cluster and trigger data rebalancing.

That is the real power of Operators, they allow you to write an application to fully manage another, and guess which kind stateful applications are the hardest to manage? You are right: Databases.

Developers have always expected databases to work out-of-the-box, when in fact, they historically are exactly the opposite. We even have a specific name for the person responsible for taking care of the database, our beloved DBAs.

Couchbase’s Operator was created as an effort change this scenario and make databases easy to manage without locking you to a specific cloud vendor. Currently, it supports automated cluster provisioning, elastic scalability, auto recovery, logging and access to the web console, but many more features are coming in the future. If you want to read more about it, please check out this article or refer to Couchbase’s official documentation here.

I also have to mention that It is the very first official operator launched for a database, and there are some small community projects already trying to build operators for MySQL, Postrgres and other databases.

The Operator’s ecosystem is growing quickly, rook for instance, let you deploy something very similar to AWS S3. The Apache Kafka operator is coming soon and there are many other initiatives out there, we expect a major boost in the number of operators in the upcoming months now that all major cloud providers support K8s.

Finally, Kubernetes provides a cloud-agnostic application deployment and management. It is so powerful that might lead us to treat cloud providers almost like a commodity, as you will be able to migrate freely between them.

In the future, choosing a cloud provider could be just a matter of which one offers the best performance/cost. The market impact of this radical shift is still unclear, but as developers, we are certainly the biggest winners.

 

If you have any questions, feel free to tweet me at @deniswsrosa

The post Why Kubernetes Operators are a game changer appeared first on The Couchbase Blog.

]]>
http://ico9.allcrypto.company/kubernetes-operators-game-changer/feed/ 2 4848
http://ico9.allcrypto.company/auditing-couchbase-n1ql-statements/ http://ico9.allcrypto.company/auditing-couchbase-n1ql-statements/#respond Mon, 19 Mar 2018 15:41:48 +0000 http://ico9.allcrypto.company/?p=4841 Couchbase Server 5.5 includes the ability to keep a record of all N1QL actions taken by users. This is part of Couchbase’s more general audit functionality, introduced in 5.0. Auditing is only available in Enterprise edition. Auditing lets the administrators […]

The post Auditing Couchbase N1QL Statements appeared first on The Couchbase Blog.

]]>
Couchbase Server 5.5 includes the ability to keep a record of all N1QL actions taken by users. This is part of Couchbase’s more general audit functionality, introduced in 5.0. Auditing is only available in Enterprise edition.

Auditing lets the administrators of the system track who is accessing what data in the system. This is important when the data being stored is sensitive in some way, such as information about users. Couchbase Server 5.5 supports auditing of N1QL statements, and lets the administrator specify what types of statements (SELECTs? INSERTs?) should actually be audited.

It is important to understand what Couchbase Server 5.5 does not do. In particular, it does not allow record-level auditing. If an UPDATE statement is run and modifies five records, the audit record will include the whole statement that ran, including any parameters passed in, and it will say that five records were updated. It will not say what specific records were updated, or what their values were before or after the operation. Fundamentally, N1QL auditing audits statements, not records.

To configure audit, log in to the Couchbase Admin console. Navigate to the Security tab (on the side) and to the Audit tab (at the top of the screen). You should now see a screen like this:

This tab lets you configure auditing in general. The checkbox at the top indicates whether auditing should be done at all. “Target Log Directory” shows where to put the audit log records. The records appear in a file named “audit.log” in the target log directory. The next set of text boxes control log rotation by size and time interval.

Next are three dropdowns for various types of events, giving you fine control over what sorts of activities should be logged. Generally speaking, audit only what you must. The actual throughput cost of auditing depends on how much is audited, and the type of statements being audited. Ten percent throughput loss due to auditing is a reasonable off-the-cuff estimate, but you should definitely test the actual effect before rolling out a new system.

Finally, you can whitelist users in the “Ignore Events From These Users” box. These are users who are trusted so completely their actions do not need to be logged. For example, you may have an automated script that inserts new data. You trust this script completely. Creating a whitelisted user and having the script use that user’s credentials may be useful to avoid generating too many audit records.

Toggle the “N1QL Events” dropdown, to see the types of events available for N1QL.

There are two general types. First are events corresponding to N1QL statement types. For example, you can choose to audit all INSERT events, or all DELETE events. It might for example be reasonable to audit all events that modify data (INSERT/DELETE/UPDATE/UPSERT), but ignore statements that only retrieve data (SELECT).

Second are events corresponding to APIs exposed by the query engine. The N1QL query engine makes a number of APIs available, typically for monitoring the system. Each of these API endpoints is a separate event type. For example, there is one for the /admin/stats endpoint, and another for the /admin/ping endpoint. You have separate control over whether to audit accesses to these APIs.

Plain Query

We’ll start by auditing a simple SELECT statement.

Go to the “Buckets” page of the admin console, and create a bucket named “test” (no quotes). Memory quota 100 MB is fine for our purposes. Then go to the Query and create a primary index on the new bucket, to allow us to run N1QL queries on it.

create primary index on test

The go back to the audit configuration screen and select “Audit events & write them to a log” at the top, and the “SELECT statement” option under “N1QL Events”. Then press “Save” at the bottom of the screen.

Then run a query like this.

curl http://localhost:8093/query/service -d "statement=select * from test" -u Administrator:password

And let’s have a look at the audit log. The “Target Log Directory” field of the audit configuration screen has the directory where the audit log is stored. We’ll use the “tail” command to show the last few records of the audit log in this directory. On Mac systems, this command works:

tail ~/Library/Application\ Support/Couchbase/var/lib/couchbase/logs/audit.log

You should see several long lines of JSON text. Each line is one audit record. The last one is the record for the statement we sent. Reformatted, it looks like this:

{
  "timestamp": "2018-03-14T05:53:34.976-07:00",
  "real_userid": {
    "source": "local",
    "user": "Administrator"
  },
  "requestId": "d0554df3-fd99-40f5-b911-b3e4f0faf050",
  "statement": "select * from test",
  "isAdHoc": true,
  "userAgent": "curl\/7.43.0",
  "node": "127.0.0.1:8091",
  "status": "success",
  "metrics": {
    "elapsedTime": "822.147\u00b5s",
    "executionTime": "785.755\u00b5s",
    "resultCount": 0,
    "resultSize": 0
  },
  "id": 28672,
  "name": "SELECT statement",
  "description": "A N1QL SELECT statement was executed"
}

Let’s go through these field by field:

  • “timestamp” shows the time from the query node.
  • “real_userid” shows what user credential was supplied with the request. In this case it is the build-in user, “Administrator”.
  • “requestId” is the UUID the query engine generates for every request. These IDs are unique with very high probability.
  • “statement” is the actual statement we executed.
  • “isAdHoc” is true in this case, showing that we sent an actual statement for execution, rather than running a prepared statement.
  • “userAgent” is the User-Agent string from the original request. This is useful for distinguishing whether the request came from an SDK, or the CBQ shell, or the Query WorkBench.
  • “node” is the IP address from which the request was received.
  • “status” shows what happened to the request. In this case, it succeeded.
  • “metrics” is a set of statistics about the result. This matches the metrics that were sent with the result of the original request.
  • “id” is the event type ID. The audit records for all SELECT queries have the same id, 28672.
  • “name” is the short name of the event type. This will be the same for all SELECT queries.
  • “description” is the long name of the event type. This is also the same for all SELECT queries.

Note that the audit record provides for only one user, although the query engine allows for multiple credentials per request. This is by design. N1QL allowed multiple credentials for queries back when our credentials were per-bucket, and multiple credentials were therefore necessary for multi-bucket joins. But as of 5.0, with RBAC, multiple credentials are no longer necessary. We support them for backward compatibility, but the right way to handle such cases is to create users with credentials for multiple buckets, and use one such user for each query. If you insist on using multiple credentials for an audited query, the query will get audited, but there will be a separate audit record for every credential supplied. That’s a bit awkward, so we strongly suggest updating the permissions model to use RBAC permissions in such cases.

Prepare Statement

Now let’s consider a more sophisticated case, with a prepared statement. First, go back to the audit configuration screen, and turn on auditing of SELECT and PREPARE statements. Remember to hit “Save” at the bottom of the screen.

Now, we’ll first prepare a statement. Here we are preparing a SELECT statement, with name “example”. Note that the statement has an unnamed parameter.

curl http://localhost:8093/query/service -d "statement=prepare example as select * from test where one=?" -u Administrator:password

Then, we’ll execute the statement, supplying an argument for the statement. In this case, the statement will run, but return no results.

curl http://localhost:8093/query/service -d 'prepared="example"&args=["bar"]'

Now let’s have a look at the audit log again.

tail ~/Library/Application\ Support/Couchbase/var/lib/couchbase/logs/audit.log

The log will show two events, one for the PREPARE, and one for the SELECT executed from the prepared statement:

{
  "timestamp": "2018-03-14T06:27:39.884-07:00",
  "real_userid": {
    "source": "local",
    "user": "Administrator"
  },
  "requestId": "9f76b8c2-ed9f-42f8-bc5c-31fb3326a661",
  "statement": "prepare example as select * from test where one=?",
  "isAdHoc": true,
  "userAgent": "curl\/7.43.0",
  "node": "127.0.0.1:8091",
  "status": "success",
  "metrics": {
    "elapsedTime": "6.591126ms",
    "executionTime": "6.515079ms",
    "resultCount": 1,
    "resultSize": 1279
  },
  "id": 28674,
  "name": "PREPARE statement",
  "description": "A N1QL PREPARE statement was executed"
}
{
  "timestamp": "2018-03-14T06:27:52.992-07:00",
  "real_userid": {
    "source": "internal",
    "user": "unknown"
  },
  "requestId": "56c5278b-5842-45a9-8549-5c7f52f109a7",
  "statement": "",
  "positionalArgs": [
    "\"bar\""
  ],
  "isAdHoc": false,
  "userAgent": "curl\/7.43.0",
  "node": "127.0.0.1:8091",
  "status": "success",
  "metrics": {
    "elapsedTime": "1.363373ms",
    "executionTime": "1.334763ms",
    "resultCount": 0,
    "resultSize": 0
  },
  "id": 28672,
  "name": "SELECT statement",
  "description": "A N1QL SELECT statement was executed"
}

The fields of the audit records are similar to the earlier execution of a SELECT statements, but two fields bear notice:

  • “positionalArgs” contains the argument supplied with the query.
  • “isAdHoc” is in this case false, because the SELECT was executed from a prepared statement that was sent earlier.

API Request

Next, let’s try auditing one of the query engine APIs. Go to the audit configuration page, and turn on the “/admin/ping API request” event type. Don’t forget to save the configuration at the bottom of the page.

Now send a ping:

curl -v http://localhost:8093/admin/ping

Don’t expect much, the “{}” at the bottom is the entire result:

*   Trying ::1...
* Connected to localhost (::1) port 8093 (#0)
&gt; GET /admin/ping HTTP/1.1
&gt; Host: localhost:8093
&gt; User-Agent: curl/7.43.0
&gt; Accept: */*
&gt; 
&lt; HTTP/1.1 200 OK
&lt; Date: Wed, 14 Mar 2018 13:54:24 GMT
&lt; Content-Length: 2
&lt; Content-Type: text/plain; charset=utf-8
&lt; 
* Connection #0 to host localhost left intact
{}

Then let’s have a look at the audit log (again, using the location on Macs):

tail ~/Library/Application\ Support/Couchbase/var/lib/couchbase/logs/audit.log

The resulting audit log message, formatted, looks like this:

{
  "timestamp": "2018-03-14T06:54:24.887-07:00",
  "real_userid": {
    "source": "internal",
    "user": "unknown"
  },
  "httpMethod": "GET",
  "httpResultCode": 200,
  "errorMessage": "",
  "id": 28697,
  "name": "/admin/ping API request",
  "description": "An HTTP request was made to the API at /admin/ping."
}

Here “timestamp” and “real_userid” fields work as before, in the SELECT example. “httpMethod” is the type of HTTP request. “httpResultCode” and “errorMessage” indicate what happened with the request. “Id”, “name” and “description” are specific to the audit event; these fields will be identical for all audit records created for /admin/ping events.

Forward Filtering

(This is an advanced topic. You don’t need to know the material in this section to use N1QL auditing effectively. But a look under the hood may be of interest to advanced users.)

Auditing is controlled in each server by an executable called the audit demon. The audit demon creates all records in the audit log. In 5.0, the audit demon was responsible for all filtering of events; clients sent records for all auditable events, and the audit demon would create audit records in the log, or not, depending on the filtering configuration. Unfortunately this would be very inefficient when auditing is highly filtered an clients are doing a lot of potentially auditable work. A client such as the query engine might generate millions of records only to have them thrown away by the audit demon when they arrived.

To alleviate this problem, in 5.5 Couchbase supports forward filtering. The query engine is aware of the current audit configuration, and sends only the currently audited records to the audit demon. It also sends a special audit record to indicate that it has received the new configuration and is aware of it.

This dual filtering is why you may see two types of configuration records in the audit log. A record like this indicates the audit demon has received a new configuration:

{"rotate_size":20971520,"log_path":"/Users/johanlarson/Library/Application Support/Couchbase/var/lib/couchbase/logs","rotate_interval":86400,
"disabled_userids":[],"auditd_enabled":true,
"disabled":[20485,20488,20489,20490,20491,28673,28675,28676,28677,28678,28679,28680,28681,28682,
 28683,28684,28685,28686,28687,28688,28689,28690,28691,28692,28693,28694,28695,28697,28698,28699,
28700,28701,28702,32770,32771,32772,32780],
"enabled:[20480,20482,20483,28672,28674,32768,32769,32773,32774,32775,32776,32777,32778,32779,32781,32782],
"real_userid":{"source":"ns_server","user":"Administrator"},"sessionid":"8b3d16bffa8444ce596b64a78c0185f7",
"remote":{"ip":"127.0.0.1","port":52153},
"timestamp":"2018-03-14T06:25:30.370-07:00","id":8240,"name":"configured audit daemon",
"description":"loaded configuration file for audit daemon"}

And a record like this indicates that the query engine has received a new configuration:

{"timestamp":"2018-03-14T06:25:30.427-07:00",
"real_userid":{"source":"","user":""},"uuid":"26571424","id":28703,
"name":"N1QL configuration","description":"States that N1QL is using audit configuration with specified uuid"}

Note the UUID that identifies the configuration. You can get this UUID from the configuration, like this:

curl http://localhost:8091/pools/default -u Administrator:password

Look for the “auditUid” field.

You can get the complete audit configuration like this:

curl http://localhost:8091/settings/audit -u Administrator:password

{"disabled":[20485,20488,20489,20490,20491,28673,28675,28676,28677,28678,
28679,28680,28681,28682,28683,28684,28685,28686,28687,28688,28689,
28690,28691,28692,28693,28694,28695,28698,28699,28700,28701,28702,
32770,32771,32772,32780],
"uid":"18635804","auditdEnabled":true,"disabledUsers":[],
"logPath":"/Users/johanlarson/Library/Application Support/Couchbase/var/lib/couchbase/logs",
"rotateInterval":86400,"rotateSize":20971520}

Loading the Audit Log

Couchbase Server currently only supports one destination for audit records: a file on the server. But sometimes it would be useful to get the audit records into the database itself. This is not difficult, since the audit records are JSON. But loading the log does require use of a utility, cbimport.

Assuming you have the audit log created in the standard location on a Mac, and you have created the “test” bucket, this incantation loads the audit.log file into the “test” bucket:

/Applications/Couchbase\ Server.app/Contents/Resources/couchbase-core/bin/cbimport json -c http://localhost:8091 -u Administrator -p password -b test -g "#UUID#" -d file:///Users/johanlarson/Library/Application\ Support/Couchbase/var/lib/couchbase/logs/audit.log -f lines

That’s rather a lot to take it, and you would need slightly different variations on other systems, so let’s go through this step by step.

  • /Applications/Couchbase\ Server.app/Contents/Resources/couchbase-core/bin/cbimport is the full path to the cbimport command on a Mac. For other systems, the utilities are located elsewhere. See this document.
  • -c http://localhost:8091 is the URL of the server where Couchbase is running
  • -u Administrator -p password is the username and password of the user we are uploading the data as (in this case the default administrator.)
  • -b test is the name of the bucket we are uploading the data into.
  • -g “#UUID#” is the type of key to generate for each document entered into the bucket. In this case, we are using a UUID, but there are many other options. Check the cbimport documentation for more information.
  • -d file:///Users/johanlarson/Library/Application\ Support/Couchbase/var/lib/couchbase/logs/audit.log is a file URL pointing to the location of the audit log. Note the three forward slashes and the backslash to allow a space in the URL path. The logs, including the audit log, are placed in standard directories that vary from system to system. See this document for more information.

Once the audit records are in the system, you can query them just like any other data.Go to the Query WorkBench to try it out.

This query shows how many audit records you have:

select count(*) as num from test

And this query breaks down the count by audit record type:

select name, count(*) as num from test group by name

Summary

  • Requests to query engine are auditable as of 5.5 EE.
  • Auditing in general supports filtering by event type and user whitelisting.
  • Requests are marked as events by query type and API endpoint.
  • Additional documentation about auditing of N1QL statements is available here.

The post Auditing Couchbase N1QL Statements appeared first on The Couchbase Blog.

]]>
http://ico9.allcrypto.company/auditing-couchbase-n1ql-statements/feed/ 0 4841
http://ico9.allcrypto.company/using-couchbase-analytics-node-js-javascript/ http://ico9.allcrypto.company/using-couchbase-analytics-node-js-javascript/#respond Mon, 19 Mar 2018 15:00:59 +0000 http://ico9.allcrypto.company/?p=4681 The Couchbase Analytics Service, sometimes referred to as CBAS, is a great thing for Couchbase and your NoSQL data needs because it allows you to create and run potentially complex queries against massive amounts of data efficiently using a familiar […]

The post Using Couchbase Analytics with Node.js and JavaScript appeared first on The Couchbase Blog.

]]>
The Couchbase Analytics Service, sometimes referred to as CBAS, is a great thing for Couchbase and your NoSQL data needs because it allows you to create and run potentially complex queries against massive amounts of data efficiently using a familiar SQL dialect.

We’re going to see how to use this analytics service, new as of Couchbase 5.5, with the Node.js SDK for Couchbase.

As of right now, Couchbase 5.5 is available as Developer Build, however, code and concepts are likely to remain similar, if not the same, when a stable release is pushed.

For this example, we’re going to be working with the travel-sample example Bucket that can be optionally installed with Couchbase Server. It isn’t considered a huge dataset, but it will work with this example.

Configuring a Couchbase Analytics Dataset

Within the Analytics tab of the Couchbase Administrative Dashboard, execute the following query:

CREATE BUCKET travel WITH { "name":"travel-sample" };

The above query will create a travel Analytics Bucket based of the Couchbase travel-sample Bucket.

With the bucket created, we can create a shadow dataset to work with. Execute the following queries:

CREATE SHADOW DATASET airlines ON travel WHERE `type` = "airline";
CREATE SHADOW DATASET airports ON travel WHERE `type` = "airport";

The above queries will create two shadow datasets based on data that exists in the travel-sample bucket. While the shadow datasets are created, they need to be initialized:

CONNECT BUCKET travel;

Once initialized, the Analytics service will begin making its copy of the documents and actively monitor for changes. We can test the data by executing a query like the following:

SELECT * FROM airports;

The above query will find all documents from our shadow dataset. In other words, we’ll get all documents that have a type property of airline.

This is all great, but as of now, we’ve only used the Couchbase Administrative Dashboard. We want to be able to run queries against our Analytics dataset from Node.js.

Querying CBAS from a Node.js Application

Assuming you have Node.js installed, create a new project directory and execute the following command:

npm init -y

The above command will create a new package.json file at your CLI path. In order to communicate with Couchbase, we’re going to need the SDK. It can be installed by executing the following from the CLI:

npm install couchbase --save

Now we can start developing our application. Create an app.js file in your project with the following:

const Couchbase = require("couchbase");

var cluster = new Couchbase.Cluster("couchbase://localhost");
cluster.authenticate("travel", "123456");
cluster.enableCbas(["localhost:8095"]);

var statement = "SELECT * FROM airports";
var query = Couchbase.CbasQuery.fromString(statement);

cluster.query(query, (error, result) => {
    if(error) {
        throw error;
    }
    console.log(result);
});

Not too much is happening in the above code, but we’ll break it down anyways.

var cluster = new Couchbase.Cluster("couchbase://localhost");
cluster.authenticate("travel", "123456");
cluster.enableCbas(["localhost:8095"]);

Before we can start executing queries, we need to connect to the cluster and enable interactions with the Analytics service. We also need to authenticate to the cluster with a user that has permission to use Analytics. In this example, Analytics is active on the node that we’re connecting to.

Notice that we’re not connecting to a Bucket. Had we wanted to use N1QL, we’d open a Bucket, but we’re not using N1QL even if it might look similar.

var statement = "SELECT * FROM airports";
var query = Couchbase.CbasQuery.fromString(statement);

cluster.query(query, (error, result) => {
    if(error) {
        throw error;
    }
    console.log(result);
});

Using a simple Analytics query, we can parse it and execute it. The results of the query will be returned and printed to the logs.

Not too bad right?

Since Analytics uses SQL++, you can make your queries much more complicated using other operators such as WHERE, subqueries, and operations on collections.

Conclusion

You just saw how to create a simple Node.js application that uses the Couchbase Analytics Service (CBAS) rather than N1QL. Analytics and SQL++ is not a replacement to N1QL, but instead a compliment for much larger or complicated datasets, where as N1QL shines on smaller datasets.

To learn more about Analytics or the Couchbase Node.js SDK, visit the Couchbase Developer Portal.  Learn about other new features available in the Couchbase Server 5.5 Developer Build.

The post Using Couchbase Analytics with Node.js and JavaScript appeared first on The Couchbase Blog.

]]>
http://ico9.allcrypto.company/using-couchbase-analytics-node-js-javascript/feed/ 0 4681