Category Archives: Electronics

Not all HBA adapters are created alike

Moving my backup NAS to a Dell R620, the version I ended up purchasing has three low-profile PCIe slots (riser 1 + riser 3). While I hadn’t spent time considering it, this meant that I could no longer insert my full-height SAS9201-16e adapter into the chassis. I searched for a replacement, and found the SAS9207-8e for a reasonable price.
What I later learned is that I was relying on a feature that the 9207 doesn’t seem to have: redundant SFF-8088 capabilities. When I had installed the 9201, I had run two cables between the adapter and my external JBOD storage (the NetApp DS4243 previously discussed). Doing the same thing with the 9207 results in a lot of conflicts, since instead of detecting that the two cables are connected to the same system, it instead sees two copies of the same system. You can quickly see how FreeNAS would get confused when seeing two copies of the drives, and trying to mount the same drives twice. I had tried updating the 9207 to the latest firmware, but never found a way to enable this same redundancy with this adapter.
So I was stuck using a single cable. Not a big deal, right? Well occasionally I would get communication errors, and once during a lightning storm the single link completely died, resulting in some (minor) data corruption of the JBOD array.
While I can’t guarantee the redundant connection would have saved me in this case, I believe it is a more robust solution. I ended up purchasing a different R620 chassis enclosure (riser 2 + riser 3) and moving the electronics over to this so I can swap back to my 9201-16e. I will continue testing to see if this does in fact improve reliability.
What’s most annoying is that I can’t seem to see this feature listed in either of the product’s documentation. So without trying, it would be hard to know which HBA adapter provides this feature.

Quanta Winterfell RAM stability and RAPL Mode

As I previously posted, I got my hands on a few inexpensive Quanta Winterfell compute nodes.  The systems have been running well except for occasional RAM errors that I’ve been trying to debug.

I’ve been using PC3-10600R of 2GB, 4GB, and 8GB sizes, single and dual rank.  The system would seem to be more stable when RAM was only inserted into the first slot of each bank (with the white tabs), but produce more errors once the second slots were populated.  This wasn’t 100% consistent either though.

In FreeNAS this would appear as read errors on the console.  Under VMWare it would appear as a complete system lockup and/or pink screen of death with a message about memory errors.

Both systems would run more reliably if I were to force the RAM speed to 1066MHz (instead of the default 1333MHz), but this of course reduced system performance and also wasn’t 100% consistent (some RAM combinations would still cause failures).

The BIOS is more of a developer-level BIOS, which means there are lots of settings available to tune all sorts of parameters, including changing RAM voltage and timing.   At first tried increasing the timing values, thinking maybe there wasn’t enough timing margin.  This really didn’t help, so then I reduced the clock rate and had improved but still flaky.

After some further testing I believe I’ve found a solution in the BIOS that results in stable performance.  Longer-term testing will be necessary, but initial results look good.

There is a setting called RAPL Mode.  RAPL stands for Running Average Power Limit, which essentially is a way to save power by limiting how much the RAM gets to use.  There are different modes, which appear to essentially be different algorithms to determine how much power the RAM should be using.  My best guess is that it is poorly implemented on these systems, which results in the RAM being starved of power and then getting corrupted.  Each mode is a newer algorithm which is supposed to result in better performance.  By default the system is set to Mode 1.

On one system I have 56GB of RAM and am running FreeNAS.  In this case I could adjust the RAM speed to 1333MHz and use RAPL Mode 0.  To test I copied data to the disk, which fills the ARC.  After 24hrs there have been no reported errors on the console (typically would start within a few minutes at 1333MHz previously).

On another system I have 112GB of RAM and am running VMWare ESXi.  At RAPL Mode 0 and 1333MHz the system hung immediately after a VM was migrated to it.  At RAPL disabled and 1333MHz the system has been up with no issues.  To test, I ran a 3D EM field solver that consumes 62GB of RAM when active on a task for about 45 minutes.

So in summary, RAPL might save power, but it could corrupt your data.  I will be disabling this feature on these systems and continue monitoring their performance.

Quanta Winterfell FreeNAS Server

I recently acquired what is known as a Quanta Winterfell Open Compute blade.  Quanta seems to make a number of OEM solutions for large companies.  In this case, Open Compute is a standard for designing no-frills high-density server systems, utilized at least by Facebook.  So what I have here could have been processing my posts, likes, etc.

Quanta Winterfell with “cover” off

The no-frills means you get a very basic chassis, which doesn’t technically even have a front, or completely enclose the entire system!  Strange looking, but when it’s sitting in a datacenter by itself, who is going to care as long as it is doing its job.

The blade itself takes a 12V input (actually 12.5V nominal) from a large space connector.  Since I didn’t have the mating part, for now I just jammed some large-gauge wire into the spades and used one of my Agilent System DC power supplies, so I could also monitor current consumption.

Horrible connection for testing (don’t try this at home).
With 2GB RAM, it idles around 50W.

The barebones blade was $90 (free shipping), which included the heatsinks and a 10Gb SFP+ NIC.  The NIC alone runs about $50, so it wasn’t a bad deal overall.

All that was left to add was a CPU, RAM, and a video card.  The system can output the console over a built-in serial port and I believe serial over LAN, but for ease of bringup I opted for the video card for now.

I wanted to see if FreeNAS would boot, so I plugged a bootable USB key into a hidden USB port (there are only 2 total), and used the other port for a keyboard.  By default the hidden USB port is disabled, so after enabling this in the BIOS it booted right up!

FreeNAS boots.

It is actually very quiet too, at least when not heavily loaded.  The fans are large, so there isn’t any of that loud datacenter whirring sound you would attribute to that environment.

Next steps:

  • Replace Agilent supply with HP server supply (modify connector)
  • Add more RAM
  • Test external SAS card and hard drive shelf
  • Get 10Gb adapter up
  • Investigate headless boot (remove video card)
  • Order second CPU

Dell T620 Power Interface Board that won’t power up

I recently acquired a Dell T620 chassis that included everything except a motherboard and power supply.  I had an extra motherboard already, so I installed it only to find that it would not turn on.  The 12V_AUX LED on the motherboard would light up, but when pressing the power button it wouldn’t do anything.

I started debugging by swapping components from another chassis I had, at first thinking it was the frontpanel, switch, or maybe a bad cable.  It turns out that it was the power interface board (PIB) itself.

The board appeared to be in good shape, with no obvious scratches or parts missing.

It was time to get out the microscope and do a closer inspection.  Since there aren’t a lot of parts on the board it didn’t take long to find a suspect problem.  One of the parts appeared to have a solder bridge between two of the pins (6 and 7).

A closer look at the pins:

All it took was removing this solder bridge, and the system then powered up without any further problems.

I have no clue how this would have ever worked in this state, so I’m not sure how it even made it out of Dell’s factory.  It didn’t appear to be reworked based on my experience, so this is a very strange escape.

Either way, I was able to rather quickly find the issue and fix it, saving the need to purchase a replacement.

Building an Outdoor Gate Sensor

With wireless door sensors becoming inexpensive, it would be nice to have one that can work in all elements.  I found one that has the ability to hook up an external dry contact sensor, but it was still only rated for indoor use.  To deal with this, I built an outdoor enclosure to house the wireless receiver, and used an external magnetic reed sensor on the gate.

For the housing, Lowes has an outdoor-rated plastic box by Taymac.  The box comes with three openings and a few fittings and covers.  A multi-size pack of glands was from Amazon.

In my case, I didn’t want to use any of the included holes, and instead wanted to use a separate gland.  The covers have slots for flathead screwdrivers, so a gland wouldn’t make a good seal.  Given the Taymac boxes only come with two covers, I also needed a third cover (also found at Lowes).

I also used a piece of unopen corrugated tubing which covers the sensor wires.  

I then drilled a hole to fit the gland.  

Unfortunately the mounting end of the gland was pretty short and couldn’t be secured with the locking ring, so instead I used epoxy to secure it in place.

The sensor is mounted by drilling a hole through the board close to where the gate hinge is.  Near the hinge, the door won’t swing much relative to the hinge so there’s no need to worry about the wind causing it to send false positives.  

The magnet that activates the switch is mounted to the door.  There are spacers to ensure the magnet is mounted close to the switch when it is closed.

From there, the assembly can be mounted to the fence so wiring and sealing can be completed.  The box is mounted so the gland is higher than the sensor so water wouldn’t flow down into the box if any did manage to get into the tubing.

From there, I attached the sensor to the electronics.  Then I used sealant to cover all the mating interfaces as an extra measure to make sure it is water-tight.  This includes around the tubing/gland interface as well as in the end of the tube where the wires exit.  It looks messy, but it will help seal things.

After the sealant has set, the electronics can be assembled and the door can be tested before placing the cover on the box.

Testing confirms it works well, and doesn’t falsely trigger when I shake the gate.  

Final assembly.  As you can see, it’s next to the wall of my house so I couldn’t hide the box around the corner easily.  It will need to be accessed to change the battery occasionally.  This is mounted behind the entrance, so no one should mess with it, though I might still tack down the cable so it’s not hanging out as much.  

Dell Inspiron with Overstressed Parts

I was given a Dell Inspiron 3451 that wouldn’t turn on.  Some probing around showed that power is getting onto the motherboard, but none of the switchers that power the CPU would turn on.  After some time doing this I noticed that board was getting warm near the HDMI connector, so I then began to focus on closely checking the parts in that area.

What I found was one of the switching power supply controllers had a hole in the package.  This can happen when a part overheats, or has to deal with an electrical overstress event (which also turns into overheating and package cracking/venting).

TPS51225 with a tiny hole in the package.
Circuit with broken IC.

Some more searching found a second suspect part, in this case what appears to be a diode with a pit in the package.

Package with pitting.

From what I can tell, this traces a path back toward the HDMI port itself.

Location of suspect parts.

I haven’t tested all the diodes in that path, but I’m guessing there was an electrical overstress event (ESD or surge) through the HDMI port, which decided to take a path through these parts to GND.

Is this an issue with the design? It’s hard to say for sure, but ideally the energy wouldn’t make its way into major components.  Usually there are protection diodes (possibly all the other parts in the top left) that should provide a fast path to the board’s ground plane, but in this case it didn’t turn out well.

Given the damage done, it’s probably not worth attempting to repair, since I don’t know what other parts might also be damaged and this is a low-cost laptop to begin with.  Best option in this case is to salvage parts to repair other things.

The Dell Venue that Destroys Batteries

I have a Dell Venue tablet which has a removable battery.  It’s a good thing the battery is removable, because I’ve now how to replace it once a year over the last two years.

Dell battery label.

Each battery is made of two LiPO cells.  Over time the battery cells begin to puff up.  Since there’s no room for the expanding cells, the battery begins to push on the back of the LCD, causing discolored blotches on the screen.  Removing the battery makes this issue go away, but why are the batteries dying so fast?

Bulging battery.
More bulging battery.

This battery was disassembled, and the monitoring PCB was examined.  I didn’t find anything unusual there.  The on-board circuit more than likely only protects the battery from being discharged too much, and from overheating (given the thermistor attached).

Dell battery monitor circuit.

The voltage on each cell was probed and measured 4.3V.  A LiPo cell is fully charged at 4.2V, showing that the cells are being overcharged.  Overcharging a cell will stress it and can cause the cell to puff up like mine have.

Unfortunately the problem is with the tablet and not necessarily with the batteries or protection circuit, and that means battery 3 will eventually meet the same fate.  I don’t have a solution for this yet except for attempting to contact Dell.  Since it’s out of warranty they probably won’t care, but it’s unfortunate they’ve released a product that prematurely destroys batteries.

“Paris” Prototype Apple Macintosh II Motherboard

I recently acquired this motherboard, primarily because it had NuBus connectors so I can explain to people outside of Apple fanatics circle don’t know what it is these days.  To my surprise, I discovered it is much more special and rare than your average Apple product.

A photo of the top:

Apple Paris Motherboard – Top

And a photo of the bottom:

Apple Paris Motheboard – Bottom

This has a few unique features:

  • It’s reddish-brown, which I’ve never seen in a released Apple product.
  • It doesn’t have much to denote it is an Apple product, other than a few parts on the board with the Apple name.   The board itself doesn’t say “Apple” anywhere.  Looking up the part number online yields nothing.
  • It looks similar to other Macintosh II boards, except in some areas parts have significantly moved around.
  • The copyright (1986) is a year before the release of the Macintosh II.
  • There are a couple debug wires on it.
  • There’s an Apple Computer add-in card in what would normally be a single IC socket.
Apple prototype board

Not being a true Apple fanatic, it took some digging to figure out that this is in fact a “Paris” prototype motherboard for the Macintosh II computer.  In fact, I have yet to find another “Paris” photo online anywhere!

Unfortunately it is missing most of the socketed parts, which were probably taken to build a “released” Macintosh when the final board was available.  Also the ROM chips are missing, but if my theory is correct the ROM was close enough to release that it wasn’t anything that unique.

What will I do with it now?  First I plan on taking some higher-res photos with a camera better than what’s in my phone.  Then I might take the time to try to build it up with parts I can find online.  Another option would be to sell it to a true Apple collector, but for now it makes an interesting conversation piece.

The Dead OBi200

After some thunderstorms came through, my OBi200 VoIP adapter stopped working.  The network end worked fine, but there was no longer any dial tone, and the device status page for the PHONE port no longer showed any information.

I only had the device for 11 months, so I promptly contacted support.  Since it was probably broken due to a surge/overstress event on the phone line, I decided to open it up and take a look.  I wanted to see if there was anything obviously broken that I could just replace and bring it back online, as well as I was curious what parts they had used in the design (and if there actually was any protection on the ports).

OBi200 Board

As expected, there really isn’t much inside.  There are three primary ICs:

  • Marvell MCU which provides the Ethernet interface, system control, config pages, etc.
  • RAM for the Marvell MCU
  • A Silicon Labs Si32260-FM1 ProSLIC telephone interface IC

The rest of the board is power supplies and a few components required by the primary ICs.  For what it’s worth there does appear to be an ESD protection IC on the USB port, but that’s good general practice for USB anyways.

I couldn’t find a full datasheet online for the Si32260-FM1.  The best I found was a couple-page datashort with a block diagram, pinout, and a summary of what the device does.  It essentially is everything necessary to provide a VoIP interface, including phone line voltage generation, DSP to encode and decode analog and FAX data, and a simple SPI interface for digital data transport (in this case to/from the MCU for transfer over Ethernet).

The SI3226x block diagram

Note that the block diagram shows two channels, but the OBi200 only provides one phone channel.  It appears that the second channel is connected and populated, so getting a second phone port is probably just a firmware change.

Unfortunately the datashort available didn’t have a reference circuit vendors usually provide, which more than likely is what Obihai used in this design.  From what I can tell (without a bunch of probing to determine actually connections) there appears to be a simple analog filter on the frontend made of 0805- and 0603-sized components.  The resistors appear to be either thick- or thin-film and the caps all MLCC.  A quick check with my DMM didn’t find anything that was obviously open, short, or different than a neighboring part with a matched circuit shape.

I did not see anything in the way of TVS diodes, spark gaps, or any other component that would provide significant protection from a high-voltage transient event, which is somewhat unfortunate.  Part of this is probably due to the small size, and the other due to there not being an actual ground lug anywhere on the product (the GND of the power port through a wall wart isn’t a true GND).

If I were to redesign this, knowing it’s probably going to connect to a set of phone lines that might be connected to a network of phone cable where lightning could possibly couple in, I would have probably added at least a couple TVS diodes and a GND lug.  Most customers probably wouldn’t connect the GND lug, but it’s better than nothing.

Fortunately, there are surge suppressors for phone lines available,  but of course it’s a separate product that needs to be purchased.  I decided to go with a Tripp-Lite DTEL2 suppressor, which connects between the OBi200 and the phone network in my home.

Tripp Lite DTEL2 Surge Suppressor

In the end, Obihai honored their 12-month warranty, and I sent the broken device back before doing any additional debugging.  I can only hope that adding an external suppressor will avoid another failure in the future.

Milling breakout PCBs

Components continue to come in smaller sizes, which makes it harder and harder to prototype with unless you also buy the vendor’s evaluation module or breakout board.  With the right tools, a bit of patience, and a few tips you can build your own basic breakout boards.

The primary tool I use is a 3040T CNC mill. I purchased it on eBay for around $650.  I’ve upgraded it with a better spindle and stepper controllers, but I could get the same results with the original setup for these projects.

I use the free version of Eagle PCB tools to design my breakout boards.  There’s a free CAM tool for taking your designs and outputting machine code.  Here’s where things get interesting though.

When the tool generates the cuts, it wants to place the center of the tool at the edge of the trace, assuming the tool diameter is 0.000″ For a perfect tool, this would be great, but unfortunately you won’t be able to find a tool like this.  You will find tools that have relatively fine tips, but anything below about 0.010″ is going to be unreliable.


Milling with a 15-degree engraving bit.

When creating your PCB, with fine-pitch part you will want the tool to run in the areas between the traces, and not necessarily mill out each trace individually.  I’ve managed to do this by lying about the size of the trace in Eagle.

For example, a 0.008″ trace might need to be 0.020″.  This seems crazy, right?  But in the end, this causes the tool to run in between each trace, rather than cutting away a majority of what you wanted to be that 0.008″ trace.

Breakout board and module that will be connected to it.

The best bits I have found are Kyocera 2-flute 15-degree engraving bits.  Search eBay for the seller “drillman1“.  Their store happens to be a few miles from me, but either way they seem to have the best prices.  You’ll spend $8-15 on a bit which seems a lot compared to the Chinese equivalent you’ll also find online, but every time I’ve tried using one of those the tip literally breaks off within 30 seconds of starting to mill copper due to their brittle carbide composition.  Do yourself a favor and buy a good tool that will last awhile, you’ll save yourself a lot of frustration.

Fine pitch at 30x magnification

Another variable is cutting depth.  Unfortunately boards are not perfectly flat, which means you’ll end up cutting to different depths across the board.  The photo below shows the case where the top was higher than the bottom, so the tops were cut through, while the bottom still has a small amount of copper where it was engraved.  Once the copper gets that thin, it generally will tear off at the engraved spots though, so cleanup is pretty easy.

Boards are never perfectly flat, resulting in undercut areas

There are fancier tools that will account for this, but with what I have I try to find a zero point near the center, so I don’t cut too deep, which would result in cutting traces too thin, but do end up with some edges not cut completely though.  It’s better to err on the side of cutting shallow rather than deep, since you can always take more away, but it’s harder to add something back.

Mach3 Milling Software

X and Y movement rate also matters a lot.  If your CNC machine has some backlash or slosh, you will want to run the cuts slowly in order to allow the bit to keep up with the commanded position. If you run too quickly, you’ll end up with traces not very straight, and worst case accidentally cutting some portions of a trace too thin.

Assembled breakout board for a GPS module. You can see the first attempt below the assembled circuit.

Finally, it still comes down to trial and error.  I usually end up milling a portion of a design and then making some adjustments and starting again.  The most common changes are to the trace width and cutting depth.

With a bit of patience you can mill your own breakout boards.  Others have built fixtures for mounting their PCBs, which allows them to create 2-layer PCBs as well.