All posts by Plocmstart

The Dell Venue that Destroys Batteries

I have a Dell Venue tablet which has a removable battery.  It’s a good thing the battery is removable, because I’ve now how to replace it once a year over the last two years.

Dell battery label.

Each battery is made of two LiPO cells.  Over time the battery cells begin to puff up.  Since there’s no room for the expanding cells, the battery begins to push on the back of the LCD, causing discolored blotches on the screen.  Removing the battery makes this issue go away, but why are the batteries dying so fast?

Bulging battery.
More bulging battery.

This battery was disassembled, and the monitoring PCB was examined.  I didn’t find anything unusual there.  The on-board circuit more than likely only protects the battery from being discharged too much, and from overheating (given the thermistor attached).

Dell battery monitor circuit.

The voltage on each cell was probed and measured 4.3V.  A LiPo cell is fully charged at 4.2V, showing that the cells are being overcharged.  Overcharging a cell will stress it and can cause the cell to puff up like mine have.

Unfortunately the problem is with the tablet and not necessarily with the batteries or protection circuit, and that means battery 3 will eventually meet the same fate.  I don’t have a solution for this yet except for attempting to contact Dell.  Since it’s out of warranty they probably won’t care, but it’s unfortunate they’ve released a product that prematurely destroys batteries.

“Paris” Prototype Apple Macintosh II Motherboard

I recently acquired this motherboard, primarily because it had NuBus connectors so I can explain to people outside of Apple fanatics circle don’t know what it is these days.  To my surprise, I discovered it is much more special and rare than your average Apple product.

A photo of the top:

Apple Paris Motherboard – Top

And a photo of the bottom:

Apple Paris Motheboard – Bottom

This has a few unique features:

  • It’s reddish-brown, which I’ve never seen in a released Apple product.
  • It doesn’t have much to denote it is an Apple product, other than a few parts on the board with the Apple name.   The board itself doesn’t say “Apple” anywhere.  Looking up the part number online yields nothing.
  • It looks similar to other Macintosh II boards, except in some areas parts have significantly moved around.
  • The copyright (1986) is a year before the release of the Macintosh II.
  • There are a couple debug wires on it.
  • There’s an Apple Computer add-in card in what would normally be a single IC socket.
Apple prototype board

Not being a true Apple fanatic, it took some digging to figure out that this is in fact a “Paris” prototype motherboard for the Macintosh II computer.  In fact, I have yet to find another “Paris” photo online anywhere!

Unfortunately it is missing most of the socketed parts, which were probably taken to build a “released” Macintosh when the final board was available.  Also the ROM chips are missing, but if my theory is correct the ROM was close enough to release that it wasn’t anything that unique.

What will I do with it now?  First I plan on taking some higher-res photos with a camera better than what’s in my phone.  Then I might take the time to try to build it up with parts I can find online.  Another option would be to sell it to a true Apple collector, but for now it makes an interesting conversation piece.

Converting a FreeNAS stripe to mirrored drive

RAIDZ is a powerful tool, and FreeNAS makes it easier to use via the GUI, but there are a number of advanced things you can’t do in the GUI alone.

In this case, I had an expanded RAIDZ-1 volume with a single striped disk (by accident), and I wanted to convert that striped disk to a mirrored one.  You could also use the same steps to convert a single striped disk (yes a single disk is still labeled as striped) to a mirrored disk.

There are a few other sites that provide the commands, but they don’t do a good job of explaining what the commands are doing.  I’ll try to clear that up!

Determine that name of the new drive.  You can do this within the GUI.  Disks are usually named /dev/ada0, /dev/ada1, etc. In my case (using a PERC H310 with Avago firmware) they show up as /dev/da0, /dev/da1, etc.  The newest drive will have the highest number.  You can also use the SMART test to verify the drive by serial number by running:
smartctl -a /dev/<yournewdrive> | more

Once you determine the correct new drive, remove any existing partitions with:
gpart destroy -F /dev/<yournewdrive>

When all partitions are deleted you can then create the ZVOL partitions.  First start by creating the partition table of type gpt with:
gpart create -s gpt /dev/<yournewdrive>

With the partition table created, you can now add the partitions. All drives get 2GB reserved for swap, and the rest can be used for the ZVOL.  First create the 2GB swap partition (partition 1), which starts at an offset of 128 sectors (which is reserved for the partition table):
gpart add -i 1 -b 128 -t freebsd-swap -s 2g /dev/<yournewdrive>

Now you can create the ZVOL partition (partition 2), which is essentially the rest of the disk space:
gpart add -i 2 -t freebsd-zfs /dev/<yournewdrive>

Next check your work:
gpart show /dev/<yournewdrive>

If you did everything correctly, you should end up with a table like this (in my case for /dev/da5, which is a 4TB drive):

[root@nas ~]# gpart show /dev/da5                                               
=>        34  7814037101  da5  GPT  (3.6T)                                      
          34          94       - free -  (47K)                                  
         128     4194304    1  freebsd-swap  (2.0G)                             
     4194432  7809842696    2  freebsd-zfs  (3.6T)                              
  7814037128           7       - free -  (3.5K)    

If you find you’ve done something wrong, you can always destroy the partition table and start over again.

With the drive partitioned, you can now add it to the pool.  This part is tricky, because you need to do it by GPTID, which is a long hexidecimal code.  If possible, do this using an actual SSH session to your server, because you can’t copy and paste (to my knowledge) to the shell available in the web browser.  To determine the GPTID of each disk, run:
zpool status

You will see something like this:

  pool: VOL1                                                                                                                        
 state: ONLINE                                                                                                                      
  scan: scrub repaired 0 in 0h17m with 0 errors on Sun Mar 19 00:17:55 2017                                                         
        NAME                                          STATE     READ WRITE CKSUM                                                    
        VOL1                                          ONLINE       0     0     0                                                    
          gptid/c3b13e97-ef38-11e6-9be5-74867ad1a828  ONLINE       0     0     0                                                    
          gptid/f36a38d3-1744-11e7-8856-0090fa79871a  ONLINE       0     0     0                                                    
errors: No known data errors 

Those gptid/<GPTID> values are the descriptors you’ll need.  Find the one for the existing drive and copy it to a text editor (notepad).  Then run:
glabel status

Here you will see a list of every drive in the system with the gptid/<GPTID> names for each.  Find the one that matches the new disk (it’s listed multiple times for each disk, so make sure you select the GPTID that matches the ZVOL, not the swap partition) and copy that to notepad as well.

Now you will need to add the disk to the zpool:
zpool attach <volumename> /dev/gptid/<existing disk GPTID> /dev/gptid/<new disk GPTID>
where volumename = the name or your zpool volume.

If all went well, the disk will now be added as a mirror, and the system will begin to resilver (copy all data over to create the mirror).  You can check this in a number of places, but one of the easiest is the GUI.  Go to the storage pane, click on the volume, then click on the Volume Status button at the bottom.  You will then see status like this:

Resilver of an array after the mirror was added.

You can also run zpool status again, which will now show the disk in the list and indicate a resilvering status until complete.  The status light will also go critical in the GUI until the resilver is complete, but there is no reason to worry and all your data will be available during this process.

That’s it!  You’ve successfully created a mirrored disk array without having to wipe the original disk and start from scratch.

A summary of commands:

Get drive name: smartctl -a /dev/<yournewdrive> | more
Clear partitions: gpart destroy -F /dev/<yournewdrive>
Create partition table: gpart create -s gpt /dev/<yournewdrive>
Create swap partition: gpart add -i 1 -b 128 -t freebsd-swap -s 2g /dev/<yournewdrive>
Create ZVOL partition: gpart add -i 2 -t freebsd-zfs /dev/<yournewdrive>
Check your work: gpart show /dev/<yournewdrive>
Get GPTID of existing drive: zpool status
Get GPTID of new drive: glabel status
Add disk mirror: zpool attach <volumename> /dev/gptid/<existing disk GPTID> /dev/gptid/<new disk GPTID>
Check for resilver: zpool status

The Dead OBi200

After some thunderstorms came through, my OBi200 VoIP adapter stopped working.  The network end worked fine, but there was no longer any dial tone, and the device status page for the PHONE port no longer showed any information.

I only had the device for 11 months, so I promptly contacted support.  Since it was probably broken due to a surge/overstress event on the phone line, I decided to open it up and take a look.  I wanted to see if there was anything obviously broken that I could just replace and bring it back online, as well as I was curious what parts they had used in the design (and if there actually was any protection on the ports).

OBi200 Board

As expected, there really isn’t much inside.  There are three primary ICs:

  • Marvell MCU which provides the Ethernet interface, system control, config pages, etc.
  • RAM for the Marvell MCU
  • A Silicon Labs Si32260-FM1 ProSLIC telephone interface IC

The rest of the board is power supplies and a few components required by the primary ICs.  For what it’s worth there does appear to be an ESD protection IC on the USB port, but that’s good general practice for USB anyways.

I couldn’t find a full datasheet online for the Si32260-FM1.  The best I found was a couple-page datashort with a block diagram, pinout, and a summary of what the device does.  It essentially is everything necessary to provide a VoIP interface, including phone line voltage generation, DSP to encode and decode analog and FAX data, and a simple SPI interface for digital data transport (in this case to/from the MCU for transfer over Ethernet).

The SI3226x block diagram

Note that the block diagram shows two channels, but the OBi200 only provides one phone channel.  It appears that the second channel is connected and populated, so getting a second phone port is probably just a firmware change.

Unfortunately the datashort available didn’t have a reference circuit vendors usually provide, which more than likely is what Obihai used in this design.  From what I can tell (without a bunch of probing to determine actually connections) there appears to be a simple analog filter on the frontend made of 0805- and 0603-sized components.  The resistors appear to be either thick- or thin-film and the caps all MLCC.  A quick check with my DMM didn’t find anything that was obviously open, short, or different than a neighboring part with a matched circuit shape.

I did not see anything in the way of TVS diodes, spark gaps, or any other component that would provide significant protection from a high-voltage transient event, which is somewhat unfortunate.  Part of this is probably due to the small size, and the other due to there not being an actual ground lug anywhere on the product (the GND of the power port through a wall wart isn’t a true GND).

If I were to redesign this, knowing it’s probably going to connect to a set of phone lines that might be connected to a network of phone cable where lightning could possibly couple in, I would have probably added at least a couple TVS diodes and a GND lug.  Most customers probably wouldn’t connect the GND lug, but it’s better than nothing.

Fortunately, there are surge suppressors for phone lines available,  but of course it’s a separate product that needs to be purchased.  I decided to go with a Tripp-Lite DTEL2 suppressor, which connects between the OBi200 and the phone network in my home.

Tripp Lite DTEL2 Surge Suppressor

In the end, Obihai honored their 12-month warranty, and I sent the broken device back before doing any additional debugging.  I can only hope that adding an external suppressor will avoid another failure in the future.

Setting up CrashPlan using VMWare and FreeNAS

I’ve had CrashPlan on my list of apps to get installed in order to better backup my system data that isn’t already installed on a NAS.  The FreeNAS forums allude to difficulty involved in properly setting up and upgrading CrashPlan as it isn’t extremely straightforward, plus getting clients to connect to a FreeNAS-based CrashPlan instance also seemed hacky (mentions of changing config files, etc).

I gave the server install a try and dealt with a number of missing driver dependencies, issues with bash properly running, and a bunch of other annoying things.  Then I decided to try a different way.

Crashplan has clients for Windows, Linux and Mac.  In addition to FreeNAS, I also have a VMWare server that is already running an instance of Windows that is providing a few services that weren’t available in other OSes.  I decided this might be the easiest way to go.

One downside to this is I didn’t want to store the backup data on the VMWare server.  The volumes on that server aren’t as large, and while it is setup with RAID volumes, they don’t currently have the 2x redundant backup that my volumes on FreeNAS currently take advantage of.

The easiest solution would be to mount a network drive on the FreeNAS server, but for some reason CrashPlan doesn’t allow a network-mapped drive to be a backup destination.

Fortunately we are running this in VMWare, and there’s more than one way to mount a drive.  What I ended up doing was creating another VMWare disk volume that is stored via NFS on the FreeNAS server.  This is then mounted as a “local” hard drive within the Windows VMWare instance.  The disk was created with 250GB space and thin provisioning, which should be more than enough for my needs.  And if I ever run out of space it’s easy to grow drives.

After this I was worried that the disk image might be replicated as a full copy in FreeNAS each time it detects that the file has changed, but it appears that snapshots and replication are able to keep track of differences and only use space required to replicate the changes.

So in summary, to avoid issues with running CrashPlan in a FreeNAS jail:

  • Run a Windows installation in a VM (I used VMWare)
  • Create a second virtual disk using the network storage as the disk location (if not on the same machine)
  • Mount this disk, partition, and format it like any other disk in Windows.
  • In the CrashPlan app within the Windows VM, set the new disk as the default location for CrashPlan backups
  • Enjoy!

How NOT to setup a RAIDZ Volume

I’m relatively new to FreeNAS, which I somehow missed out on for many years.  Now that I found it I’ve moved over to using it as much as possible for my storage needs.

Based on a number articles it seems as though it’s easy to add storage any time you like.  This is true, BUT there are some HUGE caveats to this.  The FreeNAS GUI makes it somewhat hard to make a mistake, but if you think you know what you’re doing you can override these features and actually put your data at risk.

I started out with three drives and created a RAIDZ-1 volume.  For those unaware, this means that if any ONE drive fails then all the data will still be safe.  To be even more safe, it’s recommended to use a RAIDZ-2 configuration and/or backup your data to yet another location.  I decided on the latter, and have backed up my data to a second FreeNAS server (via snapshot replication).

So far I’ve followed generally-accepted good data integrity practices here, but now for my mistake.  I decided to buy another drive of the same size, and I wanted to add that to my server to increase the space available.  The FreeNAS GUI wouldn’t let me add it with the basic Volume Manager utility, so I went to advanced mode and added it to the ZVOL.  What I ended up with was this:

Adding an additional drive to VOL0 resulted in a striped disk configuration, which is bad news here.

The three original disks (da0-da2) maintain their RAIDZ-1 state, but now a portion of the data is shared with the striped disk da3.  The striped disk has NO redundancy.

Note that the cache drive ada0 is a SSD providing a L2ARC to the volume.  Cache is usually a single striped drive, and if the data is lost there then no harm will be done to the system.

What wasn’t explained very well (or I failed to read earlier), is that RAIDZ volumes can be added to, but each group of disks, once configured, remain in the configuration that they were initially created with.  What I had done was EXPANDED the storage available, but STRIPED my original raidz1-0 array with a single disk.  Because the expanded storage is a SINGLE disk, if that one disk fails then the entire volume will be destroyed.

So if any single one of da0-da2 fails then the array continues to operate in a degraded state. But if da3 were to fail, then the entire array goes down!  Kind of kills the point of RAID, right?

The reason for this is that RAIDZ is a high-performance datacenter-quality product.  Datacenters aren’t usually adding single drives, but rather entire arrays of disks when they need to increase storage.  If you decide to use FreeNAS and RAIDZ, then you must keep this in mind.  It does take more planning (and money for additional disks), but that’s the tradeoff of using this tool.

So how do I solve it?  Well right now da3 needs some redundancy.  I ordered another drive of the same size, which will then be a mirror for this disk.  Note that the mirror must be added via the console, not through the GUI.  There is no GUI-based way to convert a striped disk to a mirrored disk, even though the FreeBSD tools support it.

So then I will have a RAIDZ-1 array + expanded storage of a mirrored array, all sharing data of VOL0.  This still has some limitations on redundancy though (assuming disk 5 is named da4):

  • If any one of da0-2 fails, then the array is OK.
  • If any one of da3-4 fails, then the array is OK.
  • If da3 and da4 fail then the array is dead (the mirrored array has completely failed).
  • If any two of da0-2 fails, then the array is dead (the raid-z1 array has completely failed).

The likelihood of this happening is low, plus all my data is still backed up on a second server.

Another option (since all my data is backed up) would be to destroy the array, and rebuild it with all 5 disks (da0-3 + the one that will arrive soon) in a RAIDZ-2 configuration.  That way if up to TWO of ANY of the disks fails then the array would continue to operate.

I haven’t yet decided if I will do this, but for now adding the mirrored disk fixes my mistake.  Don’t do what I did.

Pinewood Derby Axle Polishing

As you’ve probably read elsewhere, most of the speed comes from the reduction of friction, which is all where the wheels rub against the axles.  Do reduce this, you want to spend lots of time polishing them.  Here’s what I found can be done to get a mirror-like finish.

Ideally use a drill press (like shown in my previous post) to hold your axle nail.  If you don’t have a drill press, a corded or cordless drill in a mount or vise of some sort will also work (make sure it’s secure though).

Optionally mill away the middle section where the wheel would touch, leaving only the inner- and outermost areas.  This reduces the surface area the wheel will touch the axle.  I did this with a file mounted in a vise on my drill press.

Use a file and grind away any burrs that are around the nail head.  Also bevel the inside of the nail head if allowed.  For grinding I use a set of jewelry files.

Start with a coarse sand paper and work your way down to finer grits.  I used 400, 800, and 1200-grit.  Cut strips of sandpaper about 1/4″ x 3″.  While the axle is rotating at medium-high speed, pull the sand paper back and forth over the axle.  Do this over the areas where the wheel will touch the axle, including the inside of the nail head.  It might help to have a small piece of wood to push the sandpaper against the nail head.

Compressed graphite sticks used for axle polishing.

1200-grit produces a pretty reasonable result, but you won’t quite achieve a final mirror finish yet.  To get to the final finish, I decided to use compressed graphite sticks.  Again while the nail is spinning, press hard (but not too hard to bend the nail) with the graphite stick to polish and lubricate.  Parts of the graphite stick will break off, and this is OK.  After a few minutes of this you should have a very shiny and pre-lubricated surface for your wheel to run on.

Standard BSA axle before polishing.
Axle after milling, sanding, and polishing.

As you can see, there are still some minor scratches on the polished part.  Next time I might add an additional higher-grit polish before the graphite.

Of course you will want to add additional graphite powder to your assembled wheel, but with a polished axle you will have greatly reduced the friction produced.

Build your own axle lathe

I have quite a few tools, but I don’t yet have a lathe.  In this case, I needed to mill grooves in the axles for my son’s Cub Scouts Pinewood Derby car.

There are plenty of pages out there that talk about using a hand drill to hold the axle and then polish it up, but to reduce friction even more ideally the wheel only makes contact at a few small points on the axle.  Plus if you mill a groove, technically you have somewhere to store more graphite, which could possibly help improve lubrication over many runs (assuming it doesn’t just all fall out).

In my case, I was going for essentially one small point on the inner and outer edge of the wheel.  This way the wheel shouldn’t wobble any more than when the entire axle is present.

Obviously a drill press would work better than a hand drill, since it is more stationary.  For the cutting tools I had a set of jeweler’s files that work well for fine grinding like this.  As for holding the tool I initially started by resting my hands on the drill press table and tried manually grinding down the area I wanted.  I couldn’t get a very consistent result though, given occasionally the tool would get pulled more, causing it to graze the area where the wheel would make contact, ruining the smoothness of this area.

I then decided to make a tool holder of sorts by using a vise and a few blocks of wood (in this case oak) to hold the file in place.  Then I could simply raise and lower the table to the correct height, then slide the vise along the table to grind the axle at the location I wanted without worrying about it running away from me.

Grinding an axle with a file mounted in a vise.

It takes a bit of patience to adjust the table to the correct height, but with some practice I was able to finish the last axle in less than 10 minutes.

The head is also ground back with a triangle file to add a groove at the head as well as an angle to reduce the surface area touching the wheel.  This I could do reliably enough by hand that I didn’t make another fixture for it.

Milling breakout PCBs

Components continue to come in smaller sizes, which makes it harder and harder to prototype with unless you also buy the vendor’s evaluation module or breakout board.  With the right tools, a bit of patience, and a few tips you can build your own basic breakout boards.

The primary tool I use is a 3040T CNC mill. I purchased it on eBay for around $650.  I’ve upgraded it with a better spindle and stepper controllers, but I could get the same results with the original setup for these projects.

I use the free version of Eagle PCB tools to design my breakout boards.  There’s a free CAM tool for taking your designs and outputting machine code.  Here’s where things get interesting though.

When the tool generates the cuts, it wants to place the center of the tool at the edge of the trace, assuming the tool diameter is 0.000″ For a perfect tool, this would be great, but unfortunately you won’t be able to find a tool like this.  You will find tools that have relatively fine tips, but anything below about 0.010″ is going to be unreliable.


Milling with a 15-degree engraving bit.

When creating your PCB, with fine-pitch part you will want the tool to run in the areas between the traces, and not necessarily mill out each trace individually.  I’ve managed to do this by lying about the size of the trace in Eagle.

For example, a 0.008″ trace might need to be 0.020″.  This seems crazy, right?  But in the end, this causes the tool to run in between each trace, rather than cutting away a majority of what you wanted to be that 0.008″ trace.

Breakout board and module that will be connected to it.

The best bits I have found are Kyocera 2-flute 15-degree engraving bits.  Search eBay for the seller “drillman1“.  Their store happens to be a few miles from me, but either way they seem to have the best prices.  You’ll spend $8-15 on a bit which seems a lot compared to the Chinese equivalent you’ll also find online, but every time I’ve tried using one of those the tip literally breaks off within 30 seconds of starting to mill copper due to their brittle carbide composition.  Do yourself a favor and buy a good tool that will last awhile, you’ll save yourself a lot of frustration.

Fine pitch at 30x magnification

Another variable is cutting depth.  Unfortunately boards are not perfectly flat, which means you’ll end up cutting to different depths across the board.  The photo below shows the case where the top was higher than the bottom, so the tops were cut through, while the bottom still has a small amount of copper where it was engraved.  Once the copper gets that thin, it generally will tear off at the engraved spots though, so cleanup is pretty easy.

Boards are never perfectly flat, resulting in undercut areas

There are fancier tools that will account for this, but with what I have I try to find a zero point near the center, so I don’t cut too deep, which would result in cutting traces too thin, but do end up with some edges not cut completely though.  It’s better to err on the side of cutting shallow rather than deep, since you can always take more away, but it’s harder to add something back.

Mach3 Milling Software

X and Y movement rate also matters a lot.  If your CNC machine has some backlash or slosh, you will want to run the cuts slowly in order to allow the bit to keep up with the commanded position. If you run too quickly, you’ll end up with traces not very straight, and worst case accidentally cutting some portions of a trace too thin.

Assembled breakout board for a GPS module. You can see the first attempt below the assembled circuit.

Finally, it still comes down to trial and error.  I usually end up milling a portion of a design and then making some adjustments and starting again.  The most common changes are to the trace width and cutting depth.

With a bit of patience you can mill your own breakout boards.  Others have built fixtures for mounting their PCBs, which allows them to create 2-layer PCBs as well.


Build your own wooden open-frame rack

My “networking” closet was a mess, so I decided it was time to build a rack of sorts to better organize everything.  I could buy a rack, but I decided to build one out of some scrap wood I had laying around instead.  This way I can size it for exactly what I need, plus customizing it just involves screwing stuff into the wood where ever I wanted.

Wood isn’t perfectly straight, so you need to leave some room for warping.  I used to cheap grade 2×4’s for the rear posts and higher grade 2×4’s for the front (I only had 2 laying around).  For the sides and top/bottom I used some 1×6.  The frame opening is 17.5″ wide, and I decided a 20″ depth worked well for the equipment and space I had available.  For putting it all together I used 2″ drywall screws.

My main piece of equipment is a Cisco Catalyst 4503-E chassis with a GBIC fiber card and POE gigabit card. Air flows right to left through this chassis, so having the sides open is nice.  I plan on adding more venting to this as I have time.  Also the fans are pretty loud, so adding vents should help muffle the noise too.

Rackmount ears would probably hold in wood, but I decided to build a makeshift shelf by adding two more 2×4’s along the sides where I wanted to mount the chassis.  This way the chassis just sits on these 2×4’s at each end.

Wire routing involved using some open frame panels with keystone jacks, plus some wire conduits of various types I found recycled at work.  The photo shows what I currently have complete.

Wire routing in open wooden frame rack.

There is a vent loosely attached to the intake side that runs to the ground. This helps somewhat with circulating air from bottom to top of the closet.  Eventually I’m going to add more permanent vents.

There’s also a few other pieces of equipment that will be added once ventilation is improved.