Wednesday, 26 April 2017

Announcing: JSONPath.sh

Announcing: A JSONPath implementation written in Bash - JSONPath.sh.

It implements the JSONPath standard as written at http://goessner.net/ and was based on the original JSON.sh code from https://github.com/dominictarr/JSON.sh.

Note that JSON output is not arrays as with other JSONPath processors.

The script and documentation can be found at https://github.com/mclarkson/JSONPath.sh

Tuesday, 7 March 2017

CI for Obdi Plugins

A Plugin I was Working On
I've been writing Obdi plugins and I wanted the plugin I was currently working on to auto-update itself when I do a git push. I started looking at adding an option to the admin interface, something like an 'auto-update' checkbox in the Plugins page but after looking at the problem for a while it made sense not to add any bloat to Obdi, especially since online GIT sites each do things in slightly different ways, and instead make simple scripts to auto-update the Obdi plugin for me.

On this page you will find one solution for auto-updating a plugin immediately after a Git Push. I tested using Stash (now BitBucket) at work using the 'Http Request Post Receive Hook', which works well.

Theory

The workflow goes like this:
  • Make a code change.
  • Do a 'git commit' and 'git push'.
  • The GIT provider immediately opens a user-provided Web URL.
  • This URL is a simple Web Server and:
    • The Web Server runs an Obdi plugin update script.
    • The update script logs into Obdi and updates the plugin.
NOTE that this workflow is for a development box, and is just a temporary feature whilst developing. Extending this to a more permanent solution would be fairly straight-forward but would be tuned for each environment.

Code

The following code snippet is the full Web Server written in Google Go (golang).

package main

import (
        "bytes"
        "io"
        "log"
        "net/http"
        "os/exec"
)

func main() {

        http.HandleFunc("/post", PostOnly(HandlePost))

        log.Fatal(http.ListenAndServe(":8988", nil))
}

type handler func(w http.ResponseWriter, r *http.Request)

func HandlePost(w http.ResponseWriter, r *http.Request) {

        defer r.Body.Close()

        cmd := exec.Command("bin/kick_off_build.sh")

        var out bytes.Buffer
        cmd.Stdout = &out

        err := cmd.Run()
        if err != nil {
                io.WriteString(w, "ERROR\n"+out.String()+"\n")
                return
        }

        io.WriteString(w, out.String()+"\n")
}

func PostOnly(h handler) handler {

        return func(w http.ResponseWriter, r *http.Request) {
                if r.Method == "POST" {
                        h(w, r)
                        return
                }
                http.Error(w, "post only", http.StatusMethodNotAllowed)
        }
}


So, the above code, compiled with 'go build FILENAME' will create a server that:
  • Listens on port 8988.
  • Only accepts a POST request - and it discards any data sent to it.
  • Runs a script, './bin/kick_off_build.sh'.
And the script is as follows:

#!/bin/bash

ipport="1.2.3.4:443"
plugin="myplugin"
adminpass="password"

# Login

guid=`curl -ks -d \
    '{"Login":"admin","Password":"'"$adminpass"'"}' \
    https://$ipport/api/login | grep -o "[a-z0-9][^\"]*"`

# Find plugin

declare -i id

id=$(curl -sk \
    "https://$ipport/api/admin/$guid/plugins?name=$plugin" \
    | sed -n 's/^ *"Id": \([0-9]\+\).*/\1/p')

[[ $id -le 0 ]] && {
  echo "Plugin, $plugin, not found. Aborting update"
  exit 1
}

# Remove plugin

curl -sk -X DELETE "https://$ipport/api/admin/$guid/plugins/$id"

# Install plugin

curl -sk -d '{"Name":"'"$plugin"'"}' \
    "https://$ipport/api/admin/$guid/repoplugins"

exit 0

 

Three variables will need changing:
  • ipport - the address and port of the Obdi master.
  • plugin - the name of the plugin.
  • adminpass - Admin's password.

Done

That's it! It's quite simple but it works pretty well.

For a plugin with three script-running REST end-points it takes about 13 seconds after 'git push'ing for the plugin to be fully updated, end-points compiled, and ready for use.

To force the plugin to reinstall, maybe because the auto-update failed, which could happen if you logged into the admin interface at just the wrong point, then run, 'curl -X POST http://WebServerIP:8988/post'.



Thursday, 26 January 2017

Obdi Rsync Backup Lift-and-Shift Stats

Hi! I've just done a fairly large lift-and-shift from a physical server to AWS using Obdi Rsync Backup and thought I'd share a couple stats.

First off, what's this lift and shift? Well, our Obdi server with the Rsync Backup plugin lives in AWS on a single instance. It does daily backups over VPN of all the physical servers. We can choose backup files and get rsyncbackup to turn them into AWS instances.

Yesterday I did this for a server with tons of small files and around 460 GB of data.

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      493G  462G   26G  95% /
tmpfs           3.9G     0  3.9G   0% /dev/shm

I started it around midday yesterday and it produced the following output.


So, I was pretty happy that it worked! Looking at the following screen shot you can see that initially, just getting the size of the files took 32 minutes (A), copying the files took 10.8 hours (B), and making the filesystem modifications (running grub etc.) took about 1 minute.


Thought I'd share that so you can get an idea of times involved. Incidentally, the obdi AWS instance, which also contains the backup files, is an m4.2xlarge instance.

Cheers!