technology on RomanT.net

New Year's Start

Wed, 02 Jan 2019 15:03:09 +1100

According to the blog’s first post - I began in 2007. Vast majority of the published ( and those that died in draft form ) were a result of trying to fix things I came upon, and the associated learnings: be it automating WebSphere deployment 😜 or a GPS logger’s integration with Lightroom.

Here are a few new things I learnt week.

Wordpress Migration

I killed my WP hosting sometime in early ‘18, and just archived the content. Upon a number of conversations early on with Grant - I contemplated a conversion to Jekyll. But did not have the drive at the time to make the leap. So the WP archive to Jekyll ‘.md zip’ sat there for ~6 months.

A few days ago over lunch I had a quick perusal of static generators, and their associated ecosystem trajectories. Picked Hugo as the path forward.

The Good

Hugo’s build, and validation is ⚡️ FAST, and Go’s Templating Language reminds me of HandlebarsJS. Fact that Hugo has limited theming & customisations is a boon to focus and productivity
Markdown everywhere 😍

The ~~Broken~~ Still In Limbo

Hugo’s Customisation
- Alebit limited in themes and out of the box iintegrations ( a matter of time ) means in some instances like SEO, when you hit roadblocks - you have to design or add them yourself. Plus I really should have got the redirects sorted out prior to changing the structure
Authoring Experience
- Should not have match the quality, and fidelity of what Medium.com has created - but it has set the standard of what is possible in the browser for content creation and curation
Decisions on Image optimisation
- ie. LQIP, Progressive JPEG re-encoding
- CDNs vs DAMs vs localised imagemagick
LD-JSON content extension and integration
Figuring out when to stop.. do I really need to write a custom Hugo shortcode to display Apple Watch HR data ?

Netlify

Immediately upon stumbling across Netlify, they piqued my interest. They have some notable pedigree from Github, Docker, Serverless but most importantly I love the user experience of their product: a streamlined barebones application platform. They have thought through the landscape of customer demands, competitors’ ability, and are now navigating accordingly. Just when you started to think that application hosting, and delivery is a done deal with same set of tools throughout, Netlify re-applies the pattern atop a highly optimising storage, and service coupling back-end to provide a very differentiated out of the box experience.

tmux & Kubernetes

After an immersive week of Kubernetes deployments, and some Wavefront, it was starting to be a pain to easily monitor the K8s state whilst trying to script several scaling and failover scenarios. Constant Terminal.app switching was inevitable. Jetisoning my love of screen it was time to adjust.

After a little bit of prodding tmux with panels for logs, hosts, deployments, services .. ugh ..

or I could have just used kubebox 🤷‍♂️

Visualising Apple Health Data

TL;DR Version

Add chart.html to your layouts/shortcodes folder.

Within your content use:

{{< chart id="meh" width="800" height="500" title="keh" data="heart.csv" >}}

The Step-by-Step

After a quite poor run with James, I decided to see how I went on the trusty phone, since Apple ⌚ tracks all movement. Quickly realised this is an opportunity for a post-dinner mini project.

Lets walk through how to create the headline graphic with ChartJS in Hugo.

Get The Data

The team at Test-Driven Data Analysis does an excellent job of showing how you can get the data off the watch, but most importantly convert it from XML to something more usable.

The fun starts once you have the CSVs.

Listing the generated apple_health_export at this point should present you with :

1.3G export.xml
385M export_cda.xml
318M ActiveEnergyBurned.csv
189M BasalEnergyBurned.csv
140M DistanceWalkingRunning.csv
89M HeartRate.csv
72M StepCount.csv
17M AppleExerciseTime.csv
7.9M DistanceCycling.csv
4.1M FlightsClimbed.csv
397K HeartRateVariabilitySDNN.csv
79K SwimmingStrokeCount.csv
77K DistanceSwimming.csv
63K RestingHeartRate.csv
62K WalkingHeartRateAverage.csv
12K VO2Max.csv
5.3K convert.py
971B BodyMass.csv
940B UVExposure.csv
497B Height.csv

Create the Hugo Chart JS Layout Shortcode

The inside of HeartRate.csv looks like this the following, multipled by hundreds of thousands of rows. So pick a subsection.

sourceName	endDate	more	value
Roman’s Apple Watch	2015-05-22 13:05:21 +1100	..	81
Roman’s Apple Watch	2015-05-22 13:43:56 +1100	..	65

Charts themselves are quite straight forward, but you will need to map the data first

Recommended some dirty cut or awk magic, and cleanup if you’re going to be sharing / exporting, as naturally it does have your Watch’s identifier included.

Using lodash, map it to distinct dataPoints, and labels Arrays for injestion by ChartJS.

data = $.csv.toObjects(response);
labels = _.map(data, 'endDate');
dataPoints = _.map(data, 'value');

How AWS taught SSD's to sprint the marathon with DynamoDB

Mon, 02 Jul 2012 19:18:45 +0000

Graham spends a lot of his time tuning and finding the cause of performance issues on some of the largest WebSphere Portal installations. Inevitably during our recent conversation we turned the topic to SSD’s, and performance considerations of database workloads atop.

Interestingly the most visible SSD proponent and database service is none other than Amazons AWS service DynamoDB. Launched earlier this year, a database service which can be scaled at the turn of the knob, whilst offering performance, availability and the NoSQL feature-set. Considering the whole premature klonking-out with small-block, highly-randomised IO that hits the SSD controllers, you’d think a very ambitious undertaking.

SSD’s are great, but not a panacea for all workloads, in their raw state; especially for write-skewed database workloads. One of the largest causes of headaches for SSD Controller manufacturers and those looking to optimize workloads on SSD’s is Write Amplification, otherwise known as the variable amount of operations or re-writes of the original size that have to occur before a write is actually processed / stored by the disk.

The conversation reminded me of a post by Todd Hoff who attended an AWS session of “DynamoDB for Developers”, and wrote :

Initially they ran into performance problems due to SSD garbage collections cycles

This is inevitable. In my VMUG talk from last week, I refer to the Controller, and rather its FTL (Flash Translation Layer) as the black box of the device, where all the secret sauce of the unit lives. This isn’t by mistake, as all the IP around the type of algorithms to use for wear levelling, garbage collection or any internal process resides within.

• SSD controller

A single SSD within a single host, can cope with a lot of write-abuse, as given even a few minutes of ‘idle’ time, can effectively “defragment” its block + page layouts, and thus provide a buffer for the next onslaught. Although en-masse, where AWS operates, this simply will not work. As you will have many thousands of users and transactions in the hundreds of millions (+speculation), the drive will not have time to rest and carry out the vital to its survival, levelling and Garbage Collection.

As the folk over at Tintri have shown, when SSD’s are pummelled to a point of saturation OR effectively outside the boundary of when the internal algorithms perform cleanup, the performance or rather the predictability of said performance is significantly diminished. Below you can see latencies of nearly 300ms for an SSD device.

• tintri latency - source: tintri blog

When you’re operating a on a significant scale, you can’t have “unknown” hold you back, so that’s why (+speculative mode) Amazon worked directly with the manufactures of their SSD suppliers to have a say in:

When and how GC kicks in
RAID design across individual cells (similar to vRAID within Violin)
Granularity of block mapping
- How blocks are read+cached+written, and aligned DIRECTLY for DynamoDB’s block-sized, in order to not necessarily alleviate, but minimise re-writes
Locality of writes

The last being most interesting, and I believe an absolute must for the future of SSD to have an enduring life in the enterprise. Applications, or rather file-systems and thus API’s should allow for the ability to direct or at least specify the inter-relationship of data being written. Simply dumping blocks and forgetting about them is simple, but has dire consequences for the life of the storage medium in this case. Surrounding writes, or data operations with meta about them allows for solutions such as DynamoDB to exist and flourish. This requires a very tight integration between the software and hardware. There’s a reason Apple is one of the largest NAND / SSD suppliers (direct to consumer) out there, and why they bought Anobit, the Flash-controller firm last year. Go ponder.

Measuring NAND Longevity

Sat, 03 Mar 2012 10:13:11 +0000

No secret that NAND/Flash/SSD drives are fundamentally different to their spinning cousins. Yet given the relative lack of high-visibility maturity of NAND technologies within the enterprise, adoption of standards has yet to proliferate. Specifically around the determination of failure rate.

When we start looking at availability of mechanical drives or most components inside a server, we first refer to the MTBF.

• My helpful screenshot

The consumer-grade disks are generally rated at several thousand hours, whereas enterprise-grade drives typically see 1.3million+ hours of operation.

Due to the differences between spinning and SSD disk, it isn’t always appropriate to use MTBF as a measurement of the drives availability.

The NAND memory cell within the SSD doesn’t act in the same fashion as a spinning-disk block-abstraction. It must first be erased, prior to a write being carried out; unlike the spinning medium where a simple overwrite is possible.

The bits within a NAND cell are registered by electrons moving through a membrane/tunnel-oxide. It is responsible for keeping those electrons in check.

Gradually, over many program/erase operations the oxide starts to wear out. It is this event that should be avoided, as it will slowly start to cause single-bit errors, and cell locations that expect a 0 will be met with a friendly 1.

For this reason, the Flash Transformation Layer, responsible for all the internals of NAND management & carries out wear-leveling operations which evenly spread the load across hopefully all NAND cells.

Whilst looking at SSDs, there are SLC, MLC, and recently eMLC based flash memory. Difference being the amount of bits that are stored in the individual NAND cells. SLC has a single, whilst MLC/eMLC store two bits. In a lot of cases they’re actually exactly the same physical die, and what separates SLC from MLC/eMLC is the aforementioned FTL, amount of reserved space for the load balancing, and capability of the onboard ECC. This results in tunnel oxide in an SLC-rated drive having a typical life of around 100,000 program/erase cycles and 10,000 for MLC.

Now that you can see, that -most- of the wear on the drive occurs during cells actually being written to, and not idle time, given lack of moving components. With nothing changing, the larger the drive coupled with better FLT, the longer the drive will endure.

The reality check comes in the form of manufacturing processes, and as AnandTech points out, with the reduction in die size, results in extra sensitivity required to read each bit - there’s an inevitable hit in individual endurance of the cells. Yet the performance has been clearly increasing with each generation of die-size reduction, which is a result of the gain achieved in the controller power and thus any ECC coupled with greater reserved capacity on the disk has a net effect of - higher performance, and total better longevity.

NAND Endurance

Instead of looking at a mechanical, time-driven metric, the best way to calculate endurance of the drive is stress the oxide tunnel, and see how many times it can be trespassed, which translates to the amount of bits that can be written.

A standard that only several manufacturers have started to adopt is the JESD218 which gives an Endurance Rating in Terabytes Written (TBW). It also nicely classes client vs enterprise workloads where something as simple as the Power on state is measured (among others… see references).

Client

40ºC
8hrs/day

Enterprise

55ºC
24hrs/day

based on a use scenario in which the SSDs are actively used for some period of time during which the SSDs are written to their endurance ratings

Most importantly the standard is not using just time as a function by which to measure, but the actual endurance level. Therefore when you see a 20PB “TBW Enterprise” rating on a drive, you can quickly determine that the endurance of the drive over a 5 year period offers an ability to write 11 Terabytes per day [20*1024/(5*365)].

Next time you’re picking a drive, be weary of the correct metric being used to show you how long the unit will actually last. Knowing that each application is different, with its own workload profile - you will be able to easily figure out how long the said drive will last.

References

A prediction . . .

Sat, 02 Apr 2011 18:34:37 +0000

Amazon did a great job of not calling it ‘amazon cloud services’ when it first launched, this way they stay away from having to define it [cloud]. Once we remember that ‘cloud’ is really an abstraction of abstractions (even from definitions), we soon realise that apps written for tomorrows consumption don’t have to be tied to processor architecture or even a specific OS to run.

Now AWS are going up the stack and developing their platform play atop the IaaS they already have. Most importantly not concentrating purely on a single facet as performance (like the freshly announced Australian startup OrionVM), but useability and management through an extensive set of API’s. The differentiator is a little further up from a mere hypervizor.

Along the same path of execution in the past year we’ve seen VMware reach … up, and look to fill the gaps in Virtual Infrastructure Management+awareness solutions through vSM/Request-Manager for vCD, and more recently vCenter Operations.

The wonderful thing about technology is that by focusing on the value-add, and services further up, you can actually replace the foundation without the need to take down and rebuild the whole offering; as Apple did with Rosetta in their migration away from PowerPC to Intel, shows us you can provide a platform which negates the tight-coupling of CPU > OS > App of the yesteryear. In the current incarnation of cloud deployments across all worthwhile platforms, we can see a single constant - base compute.

… just throwing this out there, but it is not only possible, but most probable that within 24 months, not only AWS but other players will launch an Atom or even ARM-based service publicly, which will do away with offering infrastructure services and concentrate from the get-go on platform; all through the same management interface.

PGP WDE vs. Lion Disk Encryption

Sat, 05 Mar 2011 20:54:19 +0000

Beyond the veil of security, and sense of calm I get when leaving my laptop in the car or airport lobbies; the next most important thing to me is performance (after all, running my VMware lab on a laptop with a single drive isn’t exactly cheap). Thus mid last year I invested in a 240 GB OCZ Vertex 2 SSD drive, to run inside my i7 MBP. Encrypting your drive absolutely kills performance, so I did some very un-scientific tests today upon my upgrade to Lion that I wish to share. Below are several Xbench results for HDD performance only, covering ‘fresh’ HDD, and encrypted. Within Snow Leopard (10.6) with PGP WDE, and in Lion (10.7) with the inbuilt Disk Encryption.

First up as you can see we have the fresh install, and nicely enough, with just an OS upgrade, there’s an increase in the disk throughput. More importantly though, is the encrypted performance. Averaging out PGP vs Lion, we witness a doubling of the MB/s capability within the latter. On the one hand, its a shame I have a license for PGP that I spent ~$175AUD, on the other, it served its purpose, and is now going to be replaced by something that comes free with the OS, and performs better. Symantec has a talent for buying dying horses.

Extending vShield

Sun, 13 Feb 2011 11:45:28 +0000

What

At its core vShield Edge is a very lightweight appliance mostly based on LinuxFromScratch and the rest comes from VMware’s BlueLane acquisition.

If you want to skip the verbosity, skip straight to How

We are hearing more about vShield recently, because it is a prerequisite component of vCloud Director. It facilitates the critical piece that makes virtual datacenters work; network connectivity. Unfortunately it is still a young product, and as such feature-poor.

In the last two posts I covered two preludes. The first is how to bake your own NAT forwarding router and use it inplace of vShield within the App Network. Second being how to get inside the vSE appliance.

… which is where we’ll start today.

Why

The unfortunate repercussion of ‘rolling-your-own’ is that it simply doesn’t scale, and that’s what vCD + Cloud is all about. I simply can’t deploy my own NAT solution with every vApp, and have it integrate with vCloud API’s. So I started at the other end. Customizing vShield Edge itself.

For this piece, I wanted to add only two extra features

DNS resolution Simply put, this will allow for local DNS resolution within each vApp/Organization. More importantly can be made to work with the currently unused DNS Suffix, when creating networks.
Dynamic DNS Registration I have a dream… that one day, when you deploy a resource, it will be able to resolve and ‘see’ others within the same vApp/Organization or vSE ‘Network Boundary’.

How

Summary

Get inside vShield Manager
Search for vShieldEdge.ovf
Modify it until you’re satisfied

You’re done

/common/em/components/zones/vShield.ovf
/common/em/components/ni/ovf/vshield-ni-infra.ovf
/common/em/components/edge/vShieldEdge.ovf

Note that the other vShield products are also available to you.

Am not going to go through every command required to get this to work. If you’re stuck at any point whilst attempting this, feel free to drop me a line.

First, lets fix the dhcpd.conf generating file - /root/vSEdge/vse_dhcp_config.pl

Search for "ad-hoc" and replace that line with:

$cmd = "include \"/etc/rndc.key\";\nddns-update-style interim;\nddns-updates on;\nddns-domainname \"wcm.lab\";\nddns-rev-domainname \"in-addr.arpa\";\n";

This will do several things for us:

Setup trust between the dhcp and named daemons via the rndc.key
Set the zone names and allow dhcpd to update them

For bind/named itself - since we’re dealing with LinuxFromScratch, you can get the code from the LFS project page

Compiling named is done via:

./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --enable-threads --with-libtool --disable-shared && make

You can then find the executable in ./bin/named/named

Unfortunately I haven’t been able to figure out how to get it to work with the current libraries present within vShield, so you’ll also have to copy some libs across as well.

On vShield:

$ mkdir -p /var/named /named_libs

On your compiling machine:

$ scp /lib/{ld-2.5.so,ld-linux.so.2,libc.so.6,libnsl.so.1,libpthread.so.0} root@vse:/named_libraries

Once that is all set and done, you’re just left with having to configure named like on any other machine, which means you will require at the very least:

named.conf
named.ca
rndc.conf + key
zone files

I will be using the ones I’ve prepared earlier for other projects.

… and execute it using:

$ /named_libs/ld-2.5.so --library-path /named_libs/ /usr/sbin/named -g -c /etc/named.conf

I wrapped the above and placed it into /etc/inittab.

After you’re done, you overwrite the vShieldEdge.ovf within the manager VM, and have vSE Manager provision your own, custom vShield Edge appliance.

The above is meant for those that like to get their fingers a little dirty trying to improve things, this sometimes will break expected functionality.

Some gotchas, and debugging tips:

If for some reason vSE isn’t starting, and by looking at the console, you can see that ifconfig doesn’t return any interfaces with an IP, run /root/vSEdge/edge_configure.pl - this will nearly always tell you where you went wrong in your vse_ script modifications
To find linked libraries for any other packages you wish to inject, use export LD_DEBUG=<type> before running a binary where the <type> is referring to [versions | files | libs]
Don’t forget to disable/modify iptables, otherwise you’ll have trouble getting inside vSE
For anything else, there’s my contact page

By Example: Enabling SSH in vShield Edge

Fri, 07 Jan 2011 21:54:10 +0000

I wanted to learn more about how vShield Edge + Manager appliance works under the covers. By default, no inbound SSH capability exists. In the video I show you how you can enable it, so that you can extract the innards of the appliance, and hack away to create something beautiful.

Initially started writing this as a static blog-post, unfortunately too many screenshots + and explanations would be required. Hence I simply recorded some of the steps.

Caveats

There is no possibility that this is in any way supported by our VMware friends, please don’t raise SR’s on a vSE that you’ve Roman’ed up!
In the last step, I replace the root user password. Be sure to restore it from the backup immediately upon ssh’ing back in. As vCloud Director uses that password to interact with vSE, and will be unable to do so.

Exception: Error creating Edge VM

Thu, 06 Jan 2011 11:20:35 +0000

Upon trying to create a NAT-Routed network within vCD, you can be faced with a

“Error creating Edge VM”

If you look within the debug logs, the excerpt of the stack-trace will look something like this:

com.vmware.ssdc.util.LMException: Error creating Shield network appliance.
...
at java.lang.Thread.run(Thread.java:619)
Caused by: com.vmware.ssdc.util.LMException: Error creating Edge VM.
... 17 more
Caused by: java.lang.NullPointerException
...

The cause is a misconfigured ESX Host that is sitting in the same resource pool as others connected to the dVS against which vCD is trying to create the network group.

Solution being to either remove the host, or alternatively verify that all hosts are connected EQUALLY to the dVS. This is where Host Profiles come in very handy.

I apologise for the lack of a vCD screenshot, I fixed the problem before realising that it was one; at least there will always be logs. Once you do see this again, feel free to send in a screenie.

Related Local Files - logs/vcloud-container-debug.log

By Example - Networks in vCloud Director

Wed, 05 Jan 2011 17:31:08 +0000

In order to demonstrate to some customers, partners and colleagues that haven’t had a chance to enjoy vCloud Director yet, I’ve recently prepared some videos which demonstrate how you setup a network within an Organization on top of vCD. They’re designed to be completely self-sufficient, so the ‘intro’ is re-used.

Direct Network

This is by far the most … straight forward way of providing your VM’s with an outside connection.

Routed Network

The elegance of vShield Edge allows you to configure DHCP, Firewall, and NAT IP Mapping through an easy to click-through UI. Be it for your Organization or vApp. Here I’ll show you how to setup a NAT-Routed vShield backed internal network.

Custom’ish vApp Network

There are environments that have already invested a lot in customizing their own DHCP server rules, Dynamic DNS registrations, reverse lookups, firewalls… the list goes on. In essence, here I can show you how you can re-use your current soft-router, and have it used within the vCloud realm.

vCloud Director Not Starting

Tue, 21 Dec 2010 14:26:23 +0000

Attempted to bring up vCD and saw the following in vcloud-container-debug.log

| ERROR | Start Level Event Dispatcher | StartupUtils | Error starting application: Could not bind network port: 80 on host address: 172.16.225.131 |
java.net.BindException: Cannot assign requested address
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:365)
...

One of these days I’ll allocate some static IP’s to my whole Fusion-bound vLab. Until then - I will put up with DHCP assigned IP’s.

To fix, first stop vCD with vmware-vcd stop, then modify cloud-director/etc/global.properties to reflect the IP changes for:

vcloud.cell.ip.primary
consoleproxy.host.https
vcloud.cell.ips

.. or anywhere you have the old address[es]. Some love from sed will do the trick here

Unsupported element 'VirtualSystemCollection' + how to split an OVA

Sun, 11 Jul 2010 04:23:38 +0000

A colleague recently stumbled upon this beautiful error when trying to import an OVA to an ESX and ESXi host:

Unsupported element 'VirtualSystemCollection'

The cause is that the packaged OVA is actually a vApp extract from vCenter, and standalone hosts (not under management of a vCenter) are not able to accomodate vApp.

From VMware

vApps: Ensuring seamless application movement and choice between clouds

It seems a single host isn’t “cloud”-enough. 😉

An OVA that is extracted from vCenter which contains only a single VM is extremely close in its structure to the OVF apart from a tuple called … *drum roll* … <VirtualSystem ovf:id="cake"> in the single, and <VirtualSystemCollection ovf:id="cake"> in one that encompasses multiple machines. So a hosts ability is limited to not being able to parse this collection of machines and discriminate between their properties, that’s why you need to spend $ on vCenter!

This is all great, but my colleague is now stuck half-way around the world with a set of machines that need to be deployed on a single host, and all of them are saved within this OVA.

An OVA is just a wrapper for an OVF (think VMX) and some VMDK’s {anyone VM/OVconfused yet?}.

$ tar tf cake.ova
cake.ovf
cake-disk1.vmdk
cake-disk2.vmdk
cake-disk3.vmdk
cake-disk4.vmdk
cake-disk5.vmdk
cake.mf

We can see there’s a collection of VMDK’s, the manifest which inside looks like this:

SHA1(cake.ovf)= 03AD765EC45B90E13BC22D0115088F08021F2AE5
SHA1(cake-disk1.vmdk)= 92BB519FD1926D4F3C170C727C037E2D5D79775B

… and more importantly the OVF, which describes the OVA motherload.

My thoughts were at first to just create a VMX for each of the VMDKs and then I can go to bed. Unfortunately one of the machines has two disks, so the first issue is with figuring out how the OVF nests the rasd - Resource Allocation Setting Data (as per the DMTF OVF Specification V1.1).

Thanks to Mr. Pekka for writing xmlindent or you could just use the online version

I was quickly able to see to which VM disks 3+4 belonged to, and create the appropriate VMX skeleton. Now if only there was a parser…

Configuring NAT on ESX and ESXi

Sun, 11 Jul 2010 02:35:22 +0000

ESX doesn’t have NAT inbuilt, so here’s how to configure it with the help of a VMware appliance called pfSense (an Open Source firewall/router).

There are three components in this setup:

Host
router/pfSense
NAT Client

Host

A great read for beginners and those refreshing is the VMware Virtual Networking Concepts whitepaper

Now lets create a network that our NAT’ed VMs will be using.

When prompted under Connection Type, select a ‘VM Network’, as this is for the typical traffic within the Virtual Machine (not IO or management of your machines).

Lets create a vSwitch that doesn’t connect to anything, a dud, a blankie. This will be our NAT’ed environment. It’s quite important that you DON’T connect a network card to this vSwitch to prevent any inadvertent DHCP leakage. Make sure you have nothing selected.

Give it a name to differentiate.

Once you’re done, click finish and you will have something two network available to your VMs:

VM Network
NAT Network

Time to setup pfSense. Once you’ve downloaded and extracted it. You have the option of either copying it directly to your datastore and then adding directly to inventory, or importing via the Standalone Converter. I find the latter is always faster.

router/pfSense

Incase you’re converting pfSense first (like I did whilst re-doing it for this post), I recommend you disable the network interfaces until you’ve finished setting up the host networks. We’ll enable these in a later step.

Disabled interfaces

Once the conversion is complete, time to configure our virtual router. pfSense is provided with two NICs out of the box. One for the WAN interface (which is your internal LAN), and one for ‘its’ LAN - the one on which it will be servicing DHCP requests.

Mark down the last 4 digits of the MAC address, these will help to validate the following step.

Configuring pfSense NICs

Start the pfSense VM. You will be guided through the mapping of the interfaces, and just to make sure - check to see the MAC addresses matching to the VM Network (in my case 67:3c) and NAT Network (67:46).

Upon following the wizard, and if you’ve followed everything accordingly (or rather I documented the steps properly) you will be shown the interfaces within pfSense, their mapping (WAN vs LAN) and IP addresses.

NAT Client[s]

You are now ready to assign clients on this host to the NAT Network and have them pick up addresses dished out by your shiny new appliance.

The whole setup takes just under 5 minutes from start to finish to complete.

VSI_NODE_net_tcpip_plumb when adding Ports

Sat, 05 Jun 2010 08:50:32 +0000

Exception when adding a network

Adding a new port (i.e. a vMotion interface) to a vSwitch on vSphere/ESX leads to this lovely error message. If you check your vpxd.log you’ll see something the image verbalised.

[2010-06-04 15:30:52.411 03152 info 'App'] [VpxLRO]; ERROR task-54417; host-36589; vim.host.NetworkSystem.updateNetworkConfig: vim.fault.PlatformConfigFault:
(vim.fault.PlatformConfigFault) {
dynamicType = <unset>,
faultCause = (vmodl.MethodFault) null,
text = "SysinfoException: Node (VSI_NODE_net_tcpip_plumb) ; Status(bad0017)= Out of resources; Message= Instance(0): Input(4) if=0 portset=VMkernel macaddr=00:50:56:76:16:67 tsomss=65535 ",
msg = "Error during the configuration of the host: SysinfoException: Node (VSI_NODE_net_tcpip_plumb) ; Status(bad0017)= Out of resources; Message= Instance(0): Input(4) if=0 portset=VMkernel macaddr=00:50:56:76:16:67 tsomss=65535 ",
}

Key here is the Out of resources; message. The reason for this is none other than the default number of ports for the vSwitch on ESX is 24, and if you have VM’s or other interfaces using up ports (such as AppSpeed probes), you will quickly run out. Switch this easily by going to Configuration -> Networking -> Properties [for the vSwitch in question] and up the value up to and including your growth requirements for the future.

Note

After you setup a new host, set it to more ports than you require, as you’ll need to restart the host for the ports to be provisioned; best to do this immediately after installation.

I think a better message would be to explicitly say “Out of available ports on vSwitch - [blah]”; instead of the semi-cryptic one presented.

Creating bootable ISO : Linux, Solaris, Windows

Mon, 31 May 2010 12:44:54 +0000

Whenever you’re automating OS deployments, you will at one point require to build a custom ISO, with your own kickstart, jumpstart or an OEM folder with apps and instructions to boot.

Here are three mkisofs commands that you can use for each OS, and to ease your search a little.

You will require to be in the root of each ISO directory Remember the lone period “.” at end of each statement

Linux

mkisofs -q -V VOLUME_NAME -b isolinux.bin -c boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -R -J -T -o /LinuxISO.iso .<br />

Solaris

mkisofs -b boot/grub/stage2_eltorito -c .catalog -no-emul-boot -boot-load-size 4 -boot-info-table -relaxed-filenames -l -ldots -r -N -d -D -V VOLUME_NAME -o /SolarisISO.iso .<br />

Windows

The Bootable_NoEmulation.img normally lives in the [BOOT] folder. I tend to move it.

mkisofs -q -b Bootable_NoEmulation.img -no-emul-boot -boot-load-seg 1984 -boot-load-size 4 -iso-level 2 -J -joliet-long -l -D -relaxed-filenames -N -V VOLUME_NAME -o /WindowsISO.iso .<br />

Slipstream VMware Video Drivers into Windows

Sun, 09 May 2010 18:21:45 +0000

Am currently working on automating WebSphere Portal install on Windows. I do not want Windows restarting the first time it comes up, whether its for installing VMware Tools, or any of its drivers.

What I require is to add the VMware Video drivers to the base Windows installation. This will work for XP/Windows 2003. You can easily integrate the drivers once you’ve found them with the use of nLite. The trick is in finding them.

First thing you’ll need is the VMware Tools ISO that we discovered a couple posts back.

Exploded VMware Tools ISO

NOTE: I rename VMware Tools[64].msi to vtools[64].msi

The next step is to extract the vtools64.msi which you can accomplish with:

msiexec /a vtools64.msi /qb TARGETDIR=c:\driver\

Extracting VMware Tools

After a few seconds, you’re done. By navigating to the Common64\VMware\Drivers\Video folder, you will find the vmx_svga.inf that you need for nLite as well as the vmx_svga.sys which is the driver. When it comes to nLite - just point it to the directory.

VMware video directory listing

vmwogl32.dll
vmwogl64.dll
vmx_fb.dll
vmx_mode.dll
vmx_svga.cat
vmx_svga.inf
vmx_svga.sys

ESX 4.x kickstart and booting from SAN

Sat, 08 May 2010 20:39:04 +0000

Kickstarts are wonderful. Simple and effective. The ability to automate the mundane installations for *nix all about. In my case, I need to automate the installation of hosts that will be booting from SAN.

We have an admin that once installed a couple of ESX hosts atop other host’s boot LUNs. It’s pointless blaming the admin, as there’s a basic rule that must be followed at all times that will prevent this from happening.

Note - Don’t let the host see boot LUNs for any other host than itself.

ESX Dedicated Boot LUNs

This can be accomplished with:

LUN Masking (on the host HBA)
SAN Zoning (on your fabric)
Port Grouping (on your storage)

Horror story aside, lets have a look at a basic kickstart for the ESX. When you install ESX by hand, it creates a ks.cfg file in root’s home. Inside you will find the host configuration at the time of install.

ASSUMPTION: You have already provisioned the necessary space on your array for your host, and have mapped the host access to that LUN.

After a few modifications and some cleanup, I ended up with the following

accepteula
keyboard us
auth --enablemd5 --enableshadow
#this will work on all 'fresh' LUNs. If you want to an
#existing vmfs store, then add --overwritevmfs
clearpart --firstdisk
install cdrom
rootpw --iscrypted $1$MoIxblff$td1t5Oe4Kq8KZqyByh5Bc1
timezone --utc 'Australia/Sydney'
network --addvmportgroup=true --device=vmnic0 --bootproto=dhcp
part '/boot' --fstype=ext3 --size=1100 --onfirstdisk
part 'none' --fstype=vmkcore --size=110 --onfirstdisk
part 'HOSTNAME_boot' --fstype=vmfs3 --size=9232 --grow --onfirstdisk
virtualdisk 'esxconsole' --size=8232 --onvmfs='HOSTNAME_boot'
part 'swap' --fstype=swap --size=1228 --onvirtualdisk='esxconsole'
part '/var/log' --fstype=ext3 --size=2000 --onvirtualdisk='esxconsole'
part '/' --fstype=ext3 --size=5000 --grow --onvirtualdisk='esxconsole'
%post --interpreter=bash
# ntp settings
esxcfg-firewall --enableService ntpClient
chkconfig ntpd on
cat > /etc/ntp.conf <<EOF
# ---- ntp.conf ----
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery

#replace with your ntp server address
server x.x.x.x

hostname HOSTNAME
sed -i -e "s/PermitRootLogin.*/PermitRootLogin yes/g" /etc/ssh/sshd_config

EOF

I set out to install onto the FIRST available drive. In my case, thats always LUN 0, and if by some accident there’s already a VMFS on it - installation will cease (you can accomplish a force through by adding --overwritevmf to the clearpart line.

ESX Kickstart Installation

Once the setup completes you will be greeted with a “Press <enter> to reboot” - in a normal RedHat install you can get around this by adding reboot --eject just before the %post statement. Unfortunately this doesn’t seem to work for ESX. I’ll update the post once I find a way.

You may notice that I keep referring to HOSTNAME - that’s because the ks.cfg is part of another script which puts together a custom ISO for each of our blades.

Where to find VMware tools on the ESX host

Wed, 28 Apr 2010 14:06:10 +0000

Whenever you update your host, there’s a chance that VMware tools have been updated.

To get them, head over to “/vmimages/tools-isoimages” on the host and you’ll be greeted with:

If you’re just after the rpm then rip open the ISO and you’re done.

Migrating vCenter as a Virtual Machine

Mon, 25 Jan 2010 21:40:57 +0000

The caveats of running vCenter within a VM has been written about to death. So I will mention only what I never found in those posts that is relevant to the environment within which I work:

greater virtual performance over physical

Not every environment is large enough to dedicate a whole 8/12/40/96 GB of RAM Blade to a vCenter. Its simply a waste; the premise on which people dive into virtualisation is that green feeling, saving the environment or the saving the USD.

So businesses tend to ‘donate’ a physical box that will be the nucleus of their virtual offering. Yet this tends to be an underpowered, not-far from decommission door stopper.

As a Virtual Machine, you’re able to utilize the benefit of prompt resizing of the hardware, and at the ability to easily place it on higher-powered machines.

That’s the benefit, although what REALLY happens when you decide to move your vCenter around?

vMotion Mission Statement:

Storage VMotion lets you relocate virtual machine disk files between and across shared storage locations while maintaining continuous service availability and complete transaction integrity.

To me, that reads - “stuff will happen while we move your data”.

Unfortunately that isn’t always the case. Here’s the scenario.

Host

vCenter was living on DS1 (slow-LUN), and I wanted to put it across to DS2. Since vCenter is very important, it’s continuous service availability is crucial. Read: {perfect candidate for svMotion}.

The message that never went away

What happened next is counterintuitive to the credo. The vSphere client freezes, and no new connections can be made to vCenter. If you start monitoring the datastores (kudos to VMware) the VM is actually being transferred.

Once the migration completed, the first time I had to reconnect the vSphere Client; I wasn’t happy with this outcome. So I attempted a DS2->DS3 (both fast-LUNs). Not only did my connection to vCenter not drop this time, but the whole process was seamless and worked as advertised.

vCenter Migration

I ended up moving the VM in which I house SQL Server (vCenterDB) in the same fashion to the faster LUNS, and everything proceeded to operate as designed.

Lesson learnt: get decent storage.

Inconsistency in ESX consuming large LUNs

Sun, 24 Jan 2010 14:40:46 +0000

… go with me here.

You are going to have black caviar (highly recommended). You were provided with 50 grams; you figure you can fit at most 45 grams onto the piece of that delicious dark-rye. So what do you do? Do you disregard and thus throwout the 5 grams? … Well it’s not important what YOU would do, its important what VMware does!

Mr. ESX tends to look at the excruciatingly expensive, sustenance-providing caviar, and throws out the majority that it can’t handle, and opts for the remains in the hard-to-reach crevices of the jar.

What does this all mean for the geek?

There’s a decrepit limit of 2TB minus 512 bytes for each LUN that you can present to ESX. Anything larger, it has no love for. So if you were to present it with a 4TB LUN, you would naïvely assume that you would get the bastardised version of 2TB and the rest would be lost in the ether. I guess that would be somewhat logical.

Lets try it:

Capacity vs. Available Space

There you have it. Instead of actually using up as much as ESX’ly possible (~2TB) from a LUN that has been allocated, VMware chose to only pick up the left-overs (~500GB).

Re-Installing vCenter with new DSN credentials

Wed, 13 Jan 2010 12:57:56 +0000

If you’re ever in a situation where you’ve moved the vCenter database, and have actually changed the login details for the database that the DSN points to, and you’re using SQL Server. Then upon re-installing vCenter, you will be greeted with:

“Database job [Past day stats rollupvcenter] was created by another user. Please use the same user to setup your DSN or remove the job. ODBC Error: [Microsoft][SQL Native Client][SQL Server]The specified @job_name (‘Past Day stats rollupvcenter’) does not exist.

Source: IDC 2010 SMO Analysis | repurposed

When vCenter is first installed, it schedules jobs with the help of the system DB - MSDB. What’s left to do, is just remove the jobs created by the previous dbo of your vCenter data. You achieve this by first listing what jobs are created on the SQL Server.

SELECT [job_id],
[originating_server_id],
[name],
[enabled],
[description],
[start_step_id],
[category_id],
[owner_sid],
[notify_level_eventlog],
[notify_level_email],
[notify_level_netsend],
[notify_level_page],
[notify_email_operator_id],
[notify_netsend_operator_id],
[notify_page_operator_id],
[delete_level],
[date_created],
[date_modified],
[version_number]
FROM [msdb].[dbo].[sysjobs]

As you can see, four scheduled operations exist. Thankfully you don’t have to worry about just clearing this table, as there’s a stored procedure that comes within MSDB → sp_delete_job

Run it for each of the jobs, and you’ll be ready to continue installing vCenter.

SOAP wrapper for LabManager written in PHP

Wed, 30 Sep 2009 23:49:30 +0000

LabManager is a great product for when you have to deal with provisioning of images in a hands-off fashion from the IT team. If you’re already a vSphere shop, users don’t need access to a vSphere client and can manage the lifecycle of their machines through the browser (Windows only though for the console).

For what I’m building, I needed access to the API provided by VMware, and unfortunately, it is only provided as a SOAP interface. The documentation is a given, although it is of quite a poor standard. Since case-sensitivity is a must, not to mention correct parameters.

So how do we go about creating an easy to use PHP page which would consume the .NET WebService?

<?php
$namespace = "http://vmware.com/labmanager";
$soap_dat["username"] = "user";
$soap_dat["password"] = "password";
$soap_dat["organizationname"] = "org";
$soap_dat["workspacename"] = "cake";
$client = new SoapClient("https://server_name/LabManager/SOAP/LabManager.asmx?wsdl");
$header = new SOAPHeader($namespace, 'AuthenticationHeader', $soap_dat);
$client->__setSoapHeaders($header);
$config["configurationType"] = "1";
$result = $client->ListConfigurations($config);
print_r($result);
?>

After using some of the methods, I decided to write a wrapper for the LabManager’s API, so I could use it within other applications we develop inhouse. You can then call this directly from the command line, and integrate it with your bash/perl - and not have to re-invent the wheel.

As such - I present to you phLabManager - “Simple, lightweight wrapper for LabManager 4.0 SOAP API” written in PHP.

Once you’ve grabbed the labmanager.php, you can write quite succinct calls directly to the methods, without worrying about the implementation. There’s an example on usage available.

I haven’t finished implementing all the methods, and in the coming days will endeavour to do so, alternatively - you can do it, and commit into the tree. It would be great to get some help in optimizing my rough implementation.

vSphere Client - Unexpected end of file has occurred

Sun, 23 Aug 2009 17:10:17 +0000

In the unfortunate scenario that the machine you use to run your vSphere client ever crashes (unfortunately VMware doesn’t make an OSX Client), you may wish to start up the client, and click the performance tab. You may be greeted with the following when selecting the “Performance” tab:

Unexpected end of file has occurred. The following elements are not closed

This is due to a file which hosts all the chart settings (an XML-based key value pair collection) becomes corrupt, and is actually cut off.

Its quite easily found at : c:\Documents and Settings\user_name\Application Data\VMware\server_name-charts.xml

From here, you have several choices:

revert to a backup
delete it
clear it (just place <ChartSettings /> inside of it)

That’s all there is to it. Happy vSphere’ing

CLI Wizard for WebSphere Portal CumulativeFix Management

Sat, 08 Aug 2009 22:40:00 +0000

The unfortunate side-effect of any enterprise-grade software is that it is difficult to manage. There are at least two reasons that I see for this:

Training + certification revenue
Tools that are clobbered together over years of being driven by big-customer requests/demands, and not a defined architectural vision.

One such example is installing and uninstalling cumulative fixes for WebSphere. My first step was to download the PUI, followed by its extraction and then I had to load the extremely cumbersome and slow Java GUI. Then to use it, I either dropped into X11 forwarding the session from my Mac, or for my unfortunate colleagues - they have to start a VNC session first and then use the client. Stupid (not the colleagues).

So I came up with something that wrapped the updatePortal.sh in between a purpose built interface, accessible from the command line. I bring you, the CLI Wizard for Portal CF’s.

Lets disassemble the necessary steps first, and start with uninstalling.

Portal places all the CF definitions into PortalServer/version/PK{number}.efix, these are just xml files which describe the PK. What we need is the PK number and the description.

After a few lines of bash, on a server with several CF’s installed the output will look similar to this:

Not very useful though, yet. So we now allow for actually selecting one of the CF’s through the great bash select statement.

Having the PK number, we are easily able to cut/awk and pass this to the trusty updatePortal.sh to do its uninstalling deed.

The installation is much the same, although you must first obtain the PK jar. In the office we have a build server that spits them out in (in zips with the readme). So the script first of all gives a listing of available CF’s to install, and then when you choose the appropriate zip, it will download it and install.

You can download the complete script, and let me know if you have any issues. I thought of distributing this on Google Code, so others that deal with Portal can extend it, as Portal’s Java GUI leaves much to be desired.

How to virtualize your office - without their knowledge

Sat, 25 Jul 2009 19:46:29 +0000

People complain that they don’t have enough machines, and when they need servers, they’re available. Two of the problems that were prevalent in my workplace. I set out to attempt to rectify this by firstly measuring the utilization rate on some of our busiest servers.

Instead of installing and managing software that monitored CPU/Memory/HDD/Network utilization, I merely converted the whole machine to a virtual one - and offered it as a replacement under the same IP to all the users. This was done with a quick P2V conversion of the server into a VMware image pushed to a dedicated IBM Blade server running ESX 4.0 - which had far more grunt than the originating server. A few tweaks on the DHCP server to assign the right IP to the new born, and we were away. NOTE: the usage shown is of the machine running on the Blade, not on its original hardware; I’ll come back to this later in the post.

CPU Usage - 1 Machine

Disk Usage - 1 Machine

Memory Usage - 1 Machine

Before I continue, I’ll just tell you more about the software stack that is deployed on this specific machine. There is WebSphere + AppServer as well as a local instance of DB2. Mostly as you can see, its Java based, so memory usage will be fairly straight forward as it all gets eaten up on app start, and the variations that we see are from DB2. (I could use a remote DB and alleviate the memory variation all-together, which is something I will be doing with all deployments from now on).

Firstly now, the CPU Usage, throughout the day with constant use of the machine, averaged out to ~5%. Am certain this isn’t unheard of, for those that actual look at their data centres and the individual/un-optimized servers. Disk usage, aligns quite nicely with the Memory footprint, which once again points to DB2 doing its job.

So what can I improve here? - I can do several things:

remove local DB, and deploy into a contained VM cluster, or just point it at a dedicated machine (which will also be a VM)
if I offload the DB, I can increase the heap size within WebSphere for increased startup time, and overall performance
put more machines onto this single Blade

Realising the first two points only after looking at the graphs (to write this post), I still haven’t implemented them. I did though - do the third.

CPU Usage - 3 Machines

Disk Usage - 3 Machines

Memory Usage - 3 Machines

Very quickly you realise that although there’s an abundance of processing power, there’s just not enough RAM on the host blade. The ration of Ghz to Gbyte’s of RAM is about 3:1, instead it should be at the very least 1:1. Keep this a consideration next time have to put a new purchase order for more machines.

In the beginning of the post, I mentioned that the current blade host although more powerful than the original home of this software stack, it is far more scalable, and that is one of the reasons why I wanted to transfer and consolidate the numerous x345’s we have onto several blades. Most of the x345’s that were hosting servers are now turned off, the remaining few, I’m building a storage cluster with, to act as a iSCSI target server for the ESX’ed Blades.

What was satisfying was some people commenting on how fast the servers responded, little did they know that they were virtualized. Lesson: don’t buy 20 cheap and nasty servers, but a blade centre and start populating it as your budget permits.

VMWare Converter fails to publish a split-sparse image to ESX

Mon, 20 Apr 2009 14:38:05 +0000

"FAILED: The object or item referred to could not be found"

.. is the extremely helpful message that VMWare converter displays when it fails.

Digging deeper, within the logs we can see that there are multiple instances of

"Warning: failed to create directory"

and

"Warning: failed to clone directory tree".

The simple work-around is to convert the vmdk disk to a monolithic-sparse.

You can do this by issuing:

$ vmware-vdiskmanager -r original.vmdk -t 0 destination.vmdk

This will clone the disk image as well as modify it from being composed of 2GB files for the entirety of your VM to a single vmdk referred to as a ‘monolithic-sparse’ (merely referring to the fact that it will increase in size automatically to encompass the the VM partition).

After completing the cloning process, you should have no problems in restarting the conversion process, and it should complete as advertised.

Empty iPhone Emails - Solved

Mon, 25 Aug 2008 21:51:28 +0000

iPhones Firmware update 2.1, seems to have gotten rid of the problem. If now only Apple would fix the damn calendar bug

Have you recently sent an email from your beloved iPhone and have it delivered - empty? Then when you look at the sent on your phone you get the lovely: “This message has no content”.

It was simple - the problem was my signature. Which constituted of a simple ‘•’. An elevated full-stop, also known as bullet-point.

There appears to be a bug in the iPhone that is as follows:

If you send an email with a • as the LAST character, your sending bar at the bottom of Mail.app will continually say ‘Sending’, and the email will not even show up in your sent
If you send an email with words, then • and then some more words - not a problem - it will be delivered

If I didn’t know better, I’d say that upon Unicode conversion upon sending, the email becomes corrupt through a rogue parser.

Quite an oversight on Apple’s behalf, and has only appeared since the 2.0.2 update.

Moral of the story - don’t have a bullet point as the last character in your emails, let alone your signature.

Have you noticed any strangeness in your iPhones behaviour?

•

Paperless office - the story of the S510M

Sun, 24 Aug 2008 01:28:39 +0000

Going green? Or going neat? Get rid of that paper!

Paper stack. Source: flickr{bookgrl}

Think of all the bills, invoices, warranty cards, tax refunds and bank statements that you file away, day after day. The problem here is that paper is bulky, and takes up a lot of physical space, especially if you’re an owner of a filing cabinet.

My hope for this task was to achieve a state where I can be light on my feet, so should I require to move places or send off a tax returns, I can now do so within a few minutes and not hours, and perhaps even to save myself a bit of sanity

Goals

quick - no one wants to spend 5 minutes processing each document that comes in the mail
searchable - putting in an account number of any kind, should bring up ALL the relevant document. So OCR here is a must
manageable and easy - I don’t want to wade through hundred of folders to get the document that I want. Plus should I wish to group a set of files together for a submission, that should also be easily achieved

What will you need:

document scanner
OCR software
some time to plan-out your storage strategy

Quick

Speed is an important factor when deciding on picking the right document scanner. I looked at several brands, but being an Apple user, am unfortunately limited to a certain domain of equipment with drivers that will work with OSX. I went with a Fujitsu ScanSnap S510M. The ‘M’ n the model denoting ‘Mac’, so naturally the only difference between that and the non-mac version is that its white and comes with apple drivers. Fortunately for me and you, it competes very well against the competition such as Canon DR 2050C

Searchable

Yep Devon Papers software Comparison

The ability to find a scanned document by varying characteristics such as tags and creation dates and not merely by the hierarchy of parent folders, I find quite a necessity. Several applications that I looked at, had each their good points and down-right annoying.

DEVONthink Pro Office apart from the dated interface, lack of tagging ability {think folders}, $150 dollar price tag - is quite a lot like Lightroom for your documents. It has a lot of nice features, such as word occurrence calculators, which helps you pick out the most unique descriptions for each document. Devon keeps a database that you must also backup, of all the structure and data that you enter about each of the documents, so not the most agile of concepts.

Yep by Ironic Software is one of the more popular solutions. Thanks in-part to their clean design and ability to tag each document. It too relies on an database {XML-based plist} for the storage of its own meta-data. After spending quite a bit of time with Yep, I very much like the speed and almost ‘freshness’ of the app. Its drawbacks are also its strong points, it doesn’t have an OCR engine, which is great - since it leaves it up to you the user to figure out which one you wish to use. I tried both ABBY, and Acrobat; prefer the latter due to speed, and ability to do some compression with help of AppleScript before spitting out the resulting PDF.

I did notice a few bugs on first use

by setting the ‘original document created’ field, the date doesn’t get saved to the file itself, and thus isn’t reflected in Finder’s ‘Info’.
ability to create multiple smart collections of documents with the same name {screenshot below}
when searching for tag occurrence within documents and you aggregate several tags together, an empty result-set is generated

I sent the above-mentioned issues to the developers, and I have yet to hear back from them. Will update this section when I get a reply.

Despite the technical flaws, am a big fan of the workflow that Yep presents, yet their testing practices leave much to be desired.

Last but not least, Papers, an application that specializes in the storage and research of scientific papers. I would leave it at that, I didn’t find it useful for the task at hand, that’s not to say it isn’t good at what its designed for.

Manageable

Although tightly related with the previous section, manageability of your documents is something that will underpin how well you’re able to search for items once you get all those documents into some sort of a repository.

Unfortunately all the applications I’ve looked at, keep their own database of the meta-data. This is unfortunate for two reasons:

Switching cost - between applications is increased, as you’re more likely to stay with an app once you’ve imported a copious number of documents and succumbed to the apps way of thinking.
Backup - of not only the documents but also the database has to be thought of. This is where Devon makes it easy, by allowing you to place its database file into any location you desire, such as with your documents. What’s worse is that Yep doesn’t give you that freedom, and keeps its library within your Application Support folder. I received an email from ‘Ironic Support’ informs me that the only way to keep documents and data-store in one location, is if you keep all documents in your Application Support/Yep folder.

The Scanner

An important consideration for this whole task is the scanner itself. So what are you looking for in a document scanner? It doesn’t really matter what I say here, as price inevitably dictates peoples choice. Personally I saw the time it takes me to collate, organise, and file my .. err files extremely wasteful. It ended up being about how much my time is worth to me in the long run.

After doing some research the decision was between two models, the Fujitsu S310M and the S510M. The baby {given its diminutive size} of the two, S310M has one of the most attractive features in a document scanner, it can be powered merely off the USB ports on your laptop. So if you travel a lot, and its important to scan research papers {if not available in PDF-form already} or contacts, then you can’t go past the S310M. I chose to go with the S510M hoping that the extra $100 will result in a better quality scanning ‘head’ if I may call it that; so your choice should also depend on how many documents you’re going to be scanning. If you’re merely going to be doing a couple of pages a day, then grab the S310M alternatively if you are swamped with papers on a daily basis, or are going to be re-scanning your complete paper filing cabinet it might be an idea to go for the big brother S510M.

The shots below are to allow you to see comparatively how big or small this document scanner really is. I didn’t want to use the metric system, nor the imperial system - to prevent confusion. So I picked a unit everyone is familiar with. So the scanner is about 1.3 iPhones tall, with a footprint of about 1.1 iPhones, with the length of just under 2.3 iPhones.

Both scanners do duplex scanning, thus double sided documents are not a problem. What impressed me most is how well the paper feeder actually works. I’ve had the experience of working with a Canon DR 2050C and if you mis-align a page, or put something of obscure size, such as a shopping docket through - you were bound to get it jammed, not so with the S510M. I threw about a hundred sheets of A4, A5, dockets, scrunched receipts from the corner PC store, throughout the duration of the day and without a single twitch it generated a double-sided PDF for me; with the help of Acrobat 8.0 Professional and some AppleScript it OCR’ed and knocked out backgrounds for me too. In instances where I was a little eager and mis-aligned the paper, the software automatically corrected any skewing and rotated the page upright - nice!

Conclusion

When I first embarked on setting myself free from dependence on the myriad of documents, I was very hesitant to lay down the $’s for a scanner. Albeit since it arrived, it has been a blessing. As soon as I scan a document, back it up {on that in the next post}, I shred it. Hence to anyone that values their time, and ability to streamline some of the mundane tasks of paper management I couldn’t recommend the idea of a paperless office more.

Have you made your office paperless?

Fantastic GPS Logger, a Field Report for BT-1000P

Fri, 15 Aug 2008 22:00:40 +0000

To add to the ever increasing arsenal of gadgets to never leave at home, comes the BT-1000P GPS Logger.

Having recently purchased the unit, especially for the purpose tagging all the photos for an upcoming trip, I flew away extremely happy, witnessing the outstanding performance of this little device. You can see a screenshot of Google Earth displaying the track below; its the takeoff from Sydney towards Tokyo.

Generated using Google Earth and BT-1000P

What sets this unit apart from other loggers and GPS units on the market, is its ability to not only acquire a signal within the boundaries of an airplanes fuselage, but keep that sync as the airplane is banking. This latter achievement is the result of the -158dBm sensitivity rating of the unit.

Geotagging

In order to couple the data off the unit with your images, first of all you must make sure that the date+time is the same on both the unit, and your camera. Then based on the time of each point recorded and the time a photo was taken, the closest data point becomes the location of where the photo was taken.

There’s nothing you can do about the GPS, since its sync’ed with the satellites and stored in UTC. What is important though is to adjust the cameras time to the Time Zone of your location, this will save you much heartache later on when you have to adjust the times.

For my needs, I thought I’d try out HoudahGeo, and GPSPhotoLinker. I very quickly dismissed HoudahGeo, because I was unable to try it ‘properly’. By which I mean use it unrestricted for a period of my journey. I don’t believe its possible to evaluate certain software unless you can actually try your complete workflow with the software, and see if it fits, HoudahGeo unfortunately provides you with a handicapped version.

In comes GPSPhotoLinker, which is actually a free app, and gets the job done surprisingly well. What you’ll notice is that once you combine your photos with the exported data from the GPS, it will actually look up the City and Suburb data as well and embed that into the EXIF along with the GPS location. This is extremely useful for my Lightroom catalog where I can pick things based on the City as looked up by GPSPhotoLinker - brilliant! This is extremely helpful when you’re island-hopping, as is inevitable in Greece, or are merely in a new place.

Lightroom Screenshot Locations

As it stands, I think its a wonderful little unit that has served its purpose extremely well. I would highly recommend it, and will not be leaving the house without. It now lives at the top of my camera bag.

BT-1000P mini-review on a mac

Thu, 12 Jun 2008 14:36:53 +0000

Am soon going away on a little trip, over to Europe, through Tokyo. Wishing to record my trip, as well as have some data for geocoding the photos, I went on a quest to find a GPS unit that allowed me to:

log data to internal memory
make sure there’s LOTS of memory
and lots of battery

After looking at the Sony GPS-CS1, Wintec WBT-100 and GlobalSat DG-100, I concluded that the BT-1000P is by far the best unit for my needs.

First of all, the GPS chip supports up to 51-channels, with sensitivity down to -158dBm, which allows me to get a satellite lock even at the university concrete monstrosity of a car-park.
It has integrated bluetooth, so I can use it as a replacement for my ageing Holux-236.
Most importantly, it can log around 200,000 points, with my current configuration of strings, only 122,398 (@5 second intervals, that’s 7 days of 24 hour logging before I have to clear it!)

What pushed me to purchasing after looking at the specifications though is the battery.

The same as you find in most Nokia phones [BL-5C], which can be picked up cheaply off eBay
Delivers a claimed 32 hrs of logging, although I was able to get 26 hours without any problems, before the little red battery starts to flash. Mind you - I didn’t fully charge it, and just started running it soon after receiving it in the post.

Now I needed the unit to work on a mac, which the BT-1000P doesn’t do out of the box. The good news is that there’s a project called bt747 that lets you interact with the BT-1000P.

Upon connecting the unit via USB the rest of the steps are quite simple:

Download and install the USB->Serial driver
Download bt747
Switch the BT-1000P to ‘LOG’ mode
Copy RXTXcomm.jar and librxtxSerial.jnilib into /Library/Java/Extensions
Run run_rxtx.sh

You should get an extremely ugly GUI

Just hit ‘Connect Port Nbr’ and you’re ready to go. You should get something similar to: After this, all you have to do, is run around the block a few times with your unit in LOG mode. Come back home, run the app again then: Under the ‘File’ tab, set the Output directory to a location that has world write access, then hit ‘Apply & Set the above values’ Alternatively create a directory in your /var and sudo chmod 777 <directory>.

Now -> ‘Log’ tab
Hit ‘Get Log’ button {this might take a while, depending on how long your jog was}
After its complete, feel free to select the output format, either CSV, GPX or KML.

If you chose to export a KML, and you have Google Earth installed, you should get a nice looking track. Unfortunately it is evident that the BT-1000P doesn’t filter its points prior to writing them out, resulting in data that isn’t representative being recorded, hence you can observe the spike.

BT-1000P Google Earth

Now to get ready for my last exam tomorrow morning, and then to start packing for the trip. Once I’m back, will report on how the unit goes in the field, and I’ll include some geocoded images.

UPDATE

Upon the comment below by Benn, I remembered a feature that this unit has, that every time you take a photo on your camera - you can also hit the big red button on the tom of the unit. This marks the POI in the data stream. Merely allows you to come back for any manual intervention and find interesting parts easier.

Get Nightly WebKit without SVN

Sun, 13 Apr 2008 18:59:22 +0000

Everyone agrees that WebKit is the faster browser engine. Plus it’s my favourite browser/engine in OSX

Problem is that it doesn’t update itself automatically - and every night there’s a beautiful new build that comes out - which I simply must have.

So in order to save you time - here’s a little script that I put together this afternoon that will download the latest build at the push of a button.

Initially I wrote it with wget, but realised not everyone will have it installed - hence here’s a version with curl

#!/bin/bash
# getWebKit.sh - Obtain the latest WebKit build
#
# Created by Roman Tarnavski on 2008-04-13.
# For an updated version head to www.romant.net
#
#
# Distributed under a Creative Commons Attribution 3.0 License
# For more details on the license: http://creativecommons.org/licenses/by/3.0/
var_dir="/var/tmp/"
echo -n "Downloading WebPage ..."
location="http://nightly.webkit.org"
curl http://nightly.webkit.org/index.html -o ${var_dir}index.html > .get_webkit.log 2>&1
nightly=`cat ${var_dir}index.html | grep ".dmg" | cut -d '"' -f 2 | uniq`
echo "Done"
filename=`echo $nightly | cut -d '/' -f 5`
echo -n "Getting WebKit ..."
curl $location$nightly -o $var_dir$filename >> .get_webkit.log 2>&1
echo "Done"
if [ $? ]; then
echo -n "Installing WebKit ..."
rm -rf /Applications/WebKit.app
hdiutil attach -quiet "$var_dir$filename"
cp -R /Volumes/WebKit/WebKit.app /Applications/
hdiutil detach -quiet /Volumes/WebKit
echo "Done"
exit 0
fi
#cleanup
if (( $? )); then rm ${var_dir}index.html && rm $var_dir$filename ; fi

Let me know how you go

File Backup Strategy

Tue, 15 Jan 2008 14:56:04 +0000

Photos are the most precious material possession anyone owns, and should be the first to be saved in the case of a fire.

So how does one provide the appropriate backup solution to safeguard their cherished memories? My general approach to backing up the photos is by no means simple, yet it does the job.

Just as a disclaimer - the main application that I use is unfortunately Windows only - although a similar setup can be achieved using rsync.

Here goes:

Download the images off the camera using Lightroom, which simply stores them in the yyyy-mm-dd format
SyncBackSE then automatically mirrors the newly added images onto a separate hard drive in my T60’s expansion slot, for a pseudo RAID-1 solution
When I’m at home there are two states for my laptop, either docked or free and about. In either case, my network drives get mounted. Once this occurs, the local SyncBackSE copies the images straight onto my server.
The server’s two main backup drives are configured in a RAID-1, using Adaptec 1210SA adapter
I then have another copy of SyncBackSE installed on the server, which once a week copies all the files from the RAID-1 drive, onto a third drive within the server. This gives me some time to recover shots if I did delete any.
Once a month or there about – I also have an external drive which I merely copy manually and then store at a friends house.
As mentioned in the previous step, a third drive is given 1 week of lead time prior to being updated. Now in order to solve the geographic problem of having all the backups under the one roof - I have a Site-to-Site VPN with a relative, for whom I purchased a 500GB drive, onto which I then backup once a week.

The great thing about this setup is that as soon as a file is added by me or I modify tags within Lightroom of a photo, the change is automatically replicated across 4 drives - just in case, and overall – each image exists in 8 places.

Pros

Quick and automated backup solution
When I’m travelling – I still utilize the complete backup cycle – by connecting via a VPN directly to the server using my laptop in an internet cafe – and provided they have enough UP bandwidth – I’m done uploading in 20-30 minutes, and my server then takes care of everything.

Cons

This occurs within a one physical domain (provided my laptop is at home at the time) - so should a fire occur, the complete setup is futile - as everything is burnt.
There’s a finite amount of space available on my laptop, and although it wasn’t a problem when I shot JPEG with the 300D, now with the ease of RAW processing within Lightroom and ability to shoot it easily with my 40D - one days outing ends up being about 1-2 GB.

Solution to the first problem is in the next step of my backup workflow, and to the second problem I’ll cover in a later post.

In order to help distinguish between my drives, I have a naming scheme for them. I grab the names out of Greek mythology, sometimes even attempting to match the name to the task of the drive.

Here’s a diagram of a solution that works for me

Backup Strategy

If anyone finds this useful/too-much – am interested to hear your opinions. Feel free to share how you do it – and any suggestions you may have.

Renaming a Project in TFS

Fri, 21 Dec 2007 11:11:33 +0000

I’ve recently decided to try out Microsoft’s latest incarnation of Team Foundation Server; a document and code versioning, as well as defect tracking system for developers.

Since TFS manages source code, as well as documents, it is an extremely convenient repository of data for your projects; yet - by putting all your eggs in the one basket, you may be unpleasantly surprised when you decide to rename your project. In Team Foundation Server 2005, this ‘feature’ was absent, if you were hoping to find it in 2008 – stop looking, there are too many DB references to update. We can only hope that Microsoft will bring about a tool to do this as part of a service pack.

Music Everywhere

Wed, 25 Jul 2007 00:54:31 +0000

As with everything, it’s always about the amount of time and money you’re willing to put in, in order to get what you want.

I like music, and as someone that has a few computers around the house, coupled with a Vista Media Centre - I want to be able to have access to mood altering music throughout.

My music application of choice happens to be iTunes™. Understandably it isn’t everyone’s favourite, and I must admit - it does seem very sluggish on Windows™ compared to its OSX™ counterpart.

The point of this post is to setup streaming of music to iTunes™ utilising mt-daapd [aka Firefly], Ubuntu, VMWare and obviously the mp3’s themselves - hosted on a Windows™ share.

Many people will have a ‘server’ of some sort, be it for the hundred of gigabytes of family photos and videos, a common repository for house mates to exchange files or simply in my case - all of the above.

I have a server that runs Windows™ 2003 Server. My laptop synchronizes with it every time I dock it, thus providing me with a cheap NAS-like solution for my photos, university lectures and assignments.

Also, since Internet radio is something else I like to dabble in, and my ISP providing me a nice list free from bandwidth charges - and ultimately wishing to purchase a Wi-Fi enabled radio streamer, I decided to enable streaming of music across my house.

You might be asking your self - why am I running a VMWare Server + Linux in order to just stream music held on a Windows™ share? Well there are alternatives, and apps such as Tangerine that will run quite happily on Windows™ itself and serve music - but I do not want to run anything that isn’t necessary on the server itself - I prefer to leave it to virtual machines, as it allows for a much quicker re-deployment, coupled with the fact that I need a few other services to run on the VM for the network.

So here’s a guide on how to get streaming mp3’s with the use of mt-daapd, linux and about 30 minutes of spare time.

I will use names for usernames and server configurations that I have used, change as needed.

On the Windows box

Create a username “buntu” [assign a password]
Create a directory which will host the root of your music. So if it’s present in H:\Backup\Files\Personal\Music, then share and make sure you share Music, thus it will be accessible via \quack\music [quack is the name of the server]
- Make sure the security as well as the sharing settings allow for “buntu” to have ‘read-only’ access
Download + Install VMware Server
Download Ubuntu Desktop/Server [or a distro of your choice - I will only run through Ubuntu, as it’s based on my farourite Debian, only prettier + more user friendly]
- I recommend you download the *alternate* ISO, ubuntu-7.04-alternate-i386.iso as with VMWare server, you can get a “no screens found” error, and the alternate install will merely remove the GUI whilst installing Ubuntu.

If you’re creating the VM - 2 things.

When you’re going through the ‘New Virtual Machine Wizard’ - make sure you select “Use Bridge Networking”, as that way you’re making it easier for your new virtual machine to have access to the current network and thus the network shares on another computer.

Use Bridge Network - Screenshot

Hard drive space allocation - if you’re going to be using this for something other than just streaming, and have other services in mind for this VM - such as perhaps a dedicated development environment, you might consider giving it more than the default 8Gb.

On with it.

Assign the ISO you download as the CD drive.

As I mentioned above,

Proceed through the menus and install Ubuntu, for most - just accept the defaults.

Installing Ubuntu - Screenshot

Optional

In order to have the VM start when the host boots up, select the appropriate options after going to the VM Settings and under the Options tab - you will see - “Startup/Shutdown”, with the respective options being now open to you on the right hand side.

Then

apt-get install smbfs
mkdir /music
mount -t smbfs -o username=buntu //quack/music /music
password:`

The above should go through, if you have all the permissions set up correctly.
Now you have to make sure it is mounted on boot. Edit the /etc/fstab and append the following:

note the space between the source and the mount point

//quack/music /music cifs username=buntu,password=music,user 0 0

A quick check:

mount /music
ls /music

Should return no errors, and you will now have /music mounted each boot.

Time to set up the streaming.

apt-get install mt-daapd
/etc/init.d/mt-daapd start

Now if you installed something with X, launch your browser of choice [firefox], and point to: http://localhost:3689 (Default username:password -> admin:mt-daapd)

alternatively modify the /etc/mt-daapd.conf manually

Here you are able to specify any parameters as well as monitor who’s leeching [scratch] listening to your music.

At the very least, I recommend you modify (under Configuration on the site):

Server Name [as per the file itself - ‘the stuff that comes up in iTunes’]
Admin password [used to log into the the site]
Music Folder, set it to ‘/music’
Hit ‘Save’

Now on the status page, hit ‘Start Scan’ - and you’ll see the little counter for ‘served’ increase quite rapidly. Any iTunes™ instance on the network, will be able to see Quack under shared music.

iTunes Screenshot

In the beginning

Sun, 22 Apr 2007 10:29:24 +0000

After much heart-ache, I have decided to settle down with ~~Word-Press~~ Hugo, hopefully this will be the start of a beautiful, long relationship.