Back To Front But Functional

While the circuit required for programming an ATmega328P-PU chip (i.e. a standalone Arduino) is fairly simply (regardless of whether you are running at 8MHz or 16MHz) it would still be a pain to have to set everything up on a breadboard each time I wanted to programme a chip. The solution was to build a shield for the Arduino which I could simply slot on whenever I wanted to programme a new ATmega328P-PU.

Now I could have gone all fancy and designed a custom PCB for this project, but for such a simple project (where I wouldn't ever need more than one shield) this seemed overkill. Instead I built the shield out of a budget prototyping shield from oomlout. Obviously I don't want to solder an ATmega328P-PU to the shield as that would defeat the purpose, so I've added a 28pin DIP socket in its place. It took me a while to figure out the best placement of the components and to cut appropriate sized pieces of wire (lots of the connections are actually made simply by soldering the leads together on the underside of the board), although actually soldering everything in place was fairly quick (the advantage of lots of planning time). Most of the components were held in place by bending their leads slightly. The DIP socket, however, kept falling out. Unfortunately one of those occasions was just as I was about to solder it into place, and I managed to put it in back to front :(

Fortunately, the only problem with having the DIP socket back to front is that the notched end of the chip doesn't match up with the notch on the socket, which is why I've added a white paint mark to the socket to remind me to put the chip in the wrong way around. Anyway I now have a functioning shield which I can use to programme an ATmega328P-PU to run at either 8MHz or 16MHz and to upload sketches to it. I'd call that a success.

Double The Speed For Just 35p

In my previous post I explained how I'd eventually managed to figure out the procedure for configuring and programming an ATmega329P-PU micro-controller. The configuration in that post was deliberately designed to be as minimal as possible, which was why I was limited to using the internal 8MHz RC oscillator. While 8MHz is possibly fast enough for a lot of projects, there are occasions where speed is important; a lightening trigger for a camera for example. In these cases running the chip at 16MHz would be preferable. Fortunately adding an external 16MHz crystal to double the speed is both easy and cheap!

Before we get to the hardware though, we need add a new entry to the boards.txt file defining the new setup (see the previous post for details on this and how to upload code etc.).
atmega328p16mhz.name=ATmega328P (16 MHz external crystal)

atmega328p16mhz.upload.protocol=stk500v1
atmega328p16mhz.upload.maximum_size=30720
atmega328p16mhz.upload.speed=9600

atmega328p16mhz.bootloader.low_fuses=0xD7
atmega328p16mhz.bootloader.high_fuses=0xDF
atmega328p16mhz.bootloader.extended_fuses=0x07
atmega328p16mhz.bootloader.unlock_bits=0x3F
atmega328p16mhz.bootloader.lock_bits=0x0F

atmega328p16mhz.build.mcu=atmega328p
atmega328p16mhz.build.f_cpu=16000000L
atmega328p16mhz.build.core=arduino:arduino
atmega328p16mhz.build.variant=standard
Most of this is identical to the definition for the 8MHz version. The two main changes are to the low fuse (to specify that we are using an external full-swing crystal) and to the processing speed (changed from 8000000 to 16000000).

To make this work we need to add three small components to the previous circuits: a 16MHz crystal and two 22pF capacitors. Fortunately these are very cheap components and add just 36p to the cost of the circuit (making a total cost of £5.54 or £3.87 if you ordered parts for ten circuits). You can see how these are wired up in these diagrams (for programming on the left and in use on the right).


Configuring the fuses and uploading sketches to this new circuit is exactly the same procedure as for the 8MHz version, other than ensuring you select the 16MHz version from the boards menu.

There is, however, one important thing to note. Once you have configured the chip to run at 16MHz, you can't then use it without a 16MHz crystal present. You can't even re-programme the fuses to set it back to run at 8MHz. So if you want to go back to using the internal RC oscillator re-programme the fuses with the crystal present and then simplify the circuit.

Honey, I Shrunk The Arduino

As you might be able to tell from recent posts, I've been doing quite a bit of work with an Arduino. I've now got at least one project that I'd like to make a little more permanent, rather than it just being a bunch of components on a breadboard.

The easiest option would probably be to solder the components onto a prototyping shield (for example, these budget models from oomlout). There are two problems with this approach though. Firstly not every component is ideally suited for the form factor of an Arduino shield (for example a 16x2 character LCD takes up most of the prototyping space), and secondly to be any use a shield still needs to be connected to an Arduino. While Arduino's aren't extortionately expensive at just over £20 (depending where you buy them from), that's quite a cost to add to a project.

There are cheaper versions of the Arduino available, such as the Arduino Mini, which would bring the cost down to around £10 and which might be ideal for some projects. If you want to move your project onto a custom PCB though, you probably don't want to have to solder on a second PCB. The solution is to build the Arduino components directly into your circuit rather than relying on a pre-built solution. This should be cheaper than buying an Arduino and allows you to lay out the components in the most efficient way on a custom PCB.

The problem of course is figuring out how to construct an Arduino from the component parts. I spent quite a while trawling the internet trying to find the answers, but couldn't find a single page that collected everything you needed to know in one place, so I'm sure you can guess what the rest of this post is going to cover :)

The first decision you have to make is which Arduino compatible microcontroller you are going to base your circuit around. This will depend on the number of input/output pins you need as well as the size of the compiled sketch you intend to use. Given that I'm prototyping my ideas using an Arduino UNO, I decided to stick with the ATmega328P-PU chip the UNO is based around. This is by far the most expensive component you will need to buy, but you can pick them up for around £3. You will find that a number of places sell them with the Arduino bootloader programmed in, for which they charge a premium. Unless you are simply wanting to replace the chip in an existing Arduino UNO you don't actually need to use the bootloader, so save yourself a few pounds and buy the plain un-programmed chips.

Most of the tutorials on using a ATmega328P-PU expect you to run the chip at 16MHz (the same speed as the UNO) but this requires you to provide an external clock signal using a 16MHz crystal. As components mean costs, I decided I'd be a cheapskate and run the chip just using the internal 8MHz clock. We don't need any other components in order to programme the chip, but we do need to tell the Arduino software how to work with the chip.

Extra boards/chips are made available through the Arduino software by adding them to the boards.txt file in the hardware/arduino/ folder of the Ardunio installation (for full details of the format of this file try this Arduino wiki page). After a fair amount of messing around (starting from this example) I eventually figured out the following definition:
atmega328p8mhz.name=ATmega328P (8 MHz internal clock)

atmega328p8mhz.upload.protocol=stk500v1
atmega328p8mhz.upload.maximum_size=30720
atmega328p8mhz.upload.speed=9600

atmega328p8mhz.bootloader.low_fuses=0xE2
atmega328p8mhz.bootloader.high_fuses=0xDF
atmega328p8mhz.bootloader.extended_fuses=0x07
atmega328p8mhz.bootloader.unlock_bits=0x3F
atmega328p8mhz.bootloader.lock_bits=0x0F

atmega328p8mhz.build.mcu=atmega328p
atmega328p8mhz.build.f_cpu=8000000L
atmega328p8mhz.build.core=arduino:arduino
atmega328p8mhz.build.variant=standard
I'm sure that to most people (me included) that isn't self explanatory, so let's step through it to see what it means.

Firstly, line 1 simply assigns a human readable name to the board definition (this is what will apeartr on the Boards menu). Lines 3-5 state that we will be uploading to the chip using the right protocol to use the Arduino as the ISP (In System Programmer), the maximum sketch size, and the speed at which we will upload to the chip. Unfortunately, at least in version 1.0 of the Arduino software the upload speed seems to be ignored which will come back to bite us later.

Lines 7 to 11 specify the fuse settings. These are essentially used to configure the chip. I used Engbedded's fuse calculator to figure out the correct settings. Note that I'm not using the default fuse settings that the calculator suggests as I've turned off the Divide clock by 8 internally option. I found that otherwise the chip seemed to be running at 1MHz rather than 8MHz (i.e. the blink example sketch while setup to blink at 1 second intervals appeared to be actually blinking at 8 second intervals). I also specified the smallest bootloader space option, given that we aren't going to use one. Lines 13 and 14 specify the chip type and speed and finally lines 15 and 16 specify which internal libraries to compile the sketch against (in this example the standard variant which is the UNO).

So now that the Arduino software knows how to programme the board, we need to turn our existing Arduino UNO into an ISP which we can use to programme the chip. Now there is an example sketch called ArduinoISP bundled with the Arduino software, but try as I might I couldn't get it to work. Instead I'm using a slightly improved version released by Adafruit. Simply download the new version into your sketchbook folder and then upload it to your Arduino UNO the same way you would any normal sketch. Now wire the ATmega382P-PU you want to programme to the Arduino as shown to the left (this is also documented in the comments at the top of the ArduinoISP sketch, which also talks about adding LEDs for feedback; feel free to add these if you wish). Now in the IDE, from the Tools menu set the Programmer to ArduinoISP and the board to ATmega328P (8 MHz internal clock).

Now before we can actually programme the chip we need to set the fuses to the correct values as discussed above. To do this simply choose Burn Bootloader from the Tools menu (note that this is slightly misleading in that we aren't burning a bootloader as we didn't specify one in boards.txt, but this menu item sets the fuses as well as burning the bootloader if specified). Hopefully this will "just work", but for some reason my version of the IDE uses the wrong upload speed and as such this fails. Fortunately it's fairly easy to do manually. The easiest way is to get the IDE to generate the command line for you. To do this open the Preferences dialog from the File menu, and turn on the verbose output during upload. Now try to burn the bootloader again. It will still fail, but the first message will be the command line it is trying to use. On my machine this is:

/usr/share/arduino/hardware/tools/avrdude -C/usr/share/arduino/hardware/tools/avrdude.conf -v -v -v -v -patmega328p -cstk500v1 -P/dev/ttyACM0 -b19200 -e -Ulock:w:0x3F:m -Uefuse:w:0xFF:m -Uhfuse:w:0xD9:m -Ulfuse:w:0x62:m

This command is setting the baud rate to 19200 which is an awful lot faster than both the upload speed we specified in boards.txt and the baud rate specified in the ArduinoISP sketch, both of which use 9600. So simply change the -b19200 to -b9600 and execute the command manually (from a terminal or command prompt etc.). Hopefully that should work correctly. Once you have the fuses set, assuming you don't want to change the way the chip is configured, you won't need to do this step in the future.

Now you are ready to actually upload a sketch to the new chip. I usually test that everything is working using the simple Blink example that comes with the Arduino IDE. As you have already connected the Aruino to pin 13 of the new chip (this doubles as the SCK pin) I usually change the sketch to use pin 8 which is the bottom right pin if you have yours in the same orientation as the above diagram, so that you can see the sketch working without having to change any of the wiring used for programming. Now to upload to the new chip you need to use the "Upload Using Programmer" option from the File menu (making sure that the programmer and board are still set correctly) and not the Upload button on the toolbar. If setting the fuses failed the first time and you had to manually change the baud rate then you will probably run into the same problem here as well, but the solution is the same (run the command from the IDE then copy it out, change the baud rate, and then run it manually). Once it has uploaded successfully you should find your LED is blinking on and off at one second intervals.

Now, as the main aim was to remove the need to dedicate a full Arduino UNO to each project we need to power the chip in some other way than by being connected to the UNO. Fortunately this is also fairly simple as you can see from the setup to the left. This uses just four components; a voltage regulator, two capacitors, and a 9V battery connector.

I've actually found that the supply from a fresh 9V battery is stable enough that you can do without the two capacitors, but as they are stipulated by the datasheet for the voltage regulator I've included them.

Minimal Arduino Part List
PartUnit Cost
ATMEL ATmega328P-PU£3.02
5V Voltage Regulator£0.40
100µF Capacitor£0.151
9V Battery Connector£0.60
These extra components are, as you can see from this table, much cheaper than the main ATmega328P-PU, and adding the two capacitors adds just 30p to the cost. In total the components add up to just £4.32 + VAT for a grand total of £5.19 -- or around half the cost of the Arduino Mini, or a quarter of the cost of an Arduino UNO. The circuit could be made even cheaper if you were willing to buy the components in bulk; for example if you buy 10 ATmega328P-PU then they cost just £1.63 each, so you could make 10 of these circuits for just £3.52 each.

The one thing I haven't tried yet is using any analog inputs with this setup. The datasheet for the ATmega328P-PU says that the AREF pin needs to be connected to power via a low-pass filter to remove noise if the analog to digital converter is being used. I've seen a number of websites that at least suggest this isn't necessary, especially if powering via a battery, rather than an alternating current based power source. I'll figure this out in due course but even if I do need to add extra components they will be priced in the pence so won't add much to the overall cost.

Levelling Out The Wear

I've been playing around with an Arduino since May, but up until now I haven't had any need to use the 1024 bytes of EEPROM embedded in the ATMEGA328 microcontroller at the heart of the Arduino UNO. EEPROM, or to give it its full name Electrically Erasable Programmable Read-Only Memory, is really useful for storing configuration data as any values saved to EEPROM are retained even when the power is removed.

The main problem with EEPROM is that it can quickly wear out. The datasheet for the ATMEGA328 gives an expected life of just 100,000 writes for each 32 byte page. If you are just using the EEPROM to save configuration data, which won't change very often, then this won't be a problem, but if you intend to frequently write to the EEPROM this could quickly become a real issue. To mitigate this problem what we need is a wear levelling algorithm.

In the sketch I'm working on I have just four bytes that I need to store as configuration data. The easiest option would be to simply store them at offsets 0, 1, 2 and 3, but of course this would mean that every time they were changed the same four bytes (and hence the same 32 byte page) would be re-written, while the rest of the EEPROM would remain unused. Clearly this won't result in even wear across the EEPROM.

My solution, which probably isn't optimal, is to simply shift the data along by one byte each time it needs to be written. This completely ignores the idea of 32 byte pages, but if data is being written often enough then hopefully the pages should get fairly evenly used.

Now the main problem of shifting the data along by one byte each time it changes is knowing where to read it back from when the Arduino starts up. The trick here is to employ some form of multi-byte marker that denotes the location of the data block. In this example I'm going to use the four characters "TS01" to stand for Test Sketch version 1. This allows me to increment the version number if I change the format of the stored data in the future, and tells me which sketch the data refers to. This second point is important if you are going to use the EEPROM across different sketches (less important if you dedicate an Arduino to a single project). So enough talking lets start putting this into practice.

Firstly, while the core Arduino software does include an EEPROM library it's fairly rudimentary. Fortunately another Arduino user has contributed the EEPROMex library which offers a whole bunch of useful methods (note that I did have a few problems getting this library to work; you have to make sure that the folder and filenames all share the same capitalization -- EEPROMex, and the header file doesn't need to include the original EEPROM library). This extended library is capable of writing arbitrary structures to EEPROM, so instead of having separate variables for the configuration data it's easier to store them in a custom structure giving us the following code:
#include <EEPROMex.h>

#define CONFIG_VERSION "TS01"

struct ConfigStructure {
    byte a, b, c, d;
    char version[5];
} config = {
    76, 74, true, 15,
    CONFIG_VERSION
}, stored;

int offset = 0;
As you can see this snippet imports the library, defines a constant to hold the multi-byte marker and then creates the custom structure. We also create two instances of the structure config and stored giving the first one some sensible default values for each variable, so that if we don't find any stored settings the sketch can still operate. You'll also notice that I've put the multi-byte marker at the end. This is so that it only gets written if we have successfully written the rest of the data -- I'll come back to this point later.

Now before we get into actually reading and writing the settings I'm going to introduce a utility function for comparing the current and stored settings.
boolean compareSettings() {
  if (config.a != stored.a) return false;
  if (config.b != stored.b) return false;
  if (config.c != stored.c) return false;
  if (config.d != stored.d) return false;
 
  return true; 
}
Ideally I would have liked to actually pass the two instances to the function so that it wasn't a hard coded comparison of the config and stored instances, but there is an issue with passing user defined structures to functions in the version of the Arduino IDE I'm using (it has been fixed in 1.0.2 but I'm still on 1.0.0). Anyway this function simply checks that the two instances hold identical values for each variable.

So now we can turn our attention to reading stored settings out of the EEPROM.
void setup() {
  //search through the EEPROM for a valid config structure
  for ( ; offset < EEPROMSizeUno-sizeof(stored) ; ++offset) {
      
      //read a struct sized block from the EEPROM
      EEPROM.readBlock(offset, stored);

      if (strcmp(stored.version, CONFIG_VERSION) == 0) {
        //if the block has the right version tag then we can
        //assume that the rest of the struct is valid

        //overwrite the default settings with those we have
        //just loaded from the EEPROM
        config = stored;

        //we don't need to look any further so break out of
        //the loop to speed things up a bit
        break;
     }
  }
  
  if (!compareSettings()) {
    //if they haven't changed the settings from the default then
    //there is no reason to waste an EEPROM write in storing them
    //so assume the stored version is the default
    stored = config;  

    //as there wasn't anything in the EEPROM set the offset to -1
    offset = -1;
  }
}
Hopefully the comments make this code easy to understand, but essentially all it does is read a block of data from the EEPROM, see if it is a valid settings object by checking if the version variable is the same as the CONFIG_VERSION constant. If it is then we ensure that the current config structure is the same as the stored structure, otherwise we move on 1 byte and try again (this all happens in lines 3 to 20, ). Once we have searched through the entire EEPROM, either we will have found some settings and config and stored will be equal, or we haven't found any and they will differ. Note that we control the loop using the EEPROMSizeUno constant defined by the EEPROMex library, if you have a different Arduino then you will need to change this. If, when we have searched through the EEPROM, config and stored aren't the same then lines 22 to 30 make them so (so we don't store the default values to EEPROM for no reason) and reset the offset so that when it is incremented to save the settings it moves to the beginning of the address range.

Now in the main part of your sketch when you decided the settings need to change you can simply update the config instance and call a saveSettings function to actually write them to the EEPROM.
void saveSettings() {
  if (!compareSettings()) {
    //we need to store the new settings in EEPROM

    //move on one position from the previous place we stored things
    ++offset;

    //if writing at offset would mean going outside the EEPROM limit then
    //reset the pointer to the beginning of the EEPROM
    if (offset > EEPROMSizeUno-sizeof(config)) offset = 0;

    //do the actual EEPROM writing
    EEPROM.updateBlock(offset, config);

    //stored should now be the same as config
    stored = config;
  }
}
Essentially if the config and stored instances differ then we need to write config to the EEPROM. We increment the offset (the wear levelling part), check we haven't gone outside the valid range, and then write the block (we actually use update so that we don't write bytes that won't actually change) and now we know that config and stored are the same again.

I mentioned earlier that the reason for putting the multi-byte marker at the end of the config structure was so that I knew once it had been written then the rest of the data must have been as well. The problem here is that if writing fails before the start of the marker we will have no way of knowing, other than we will see random configuration data. The easiest option would be to add a different marker at the start. If both markers are present and in the right place then the data between them must be valid. The problem here is that a four character marker takes up more space than the config data I'm saving so having two of them seems like a waste.

The approach I'm taking is to compute a hash of the four config values and store this as the first piece of data. This means that when I read the structure back I can check for the multi-byte end marker and that the data matches the hash. The problem is that the hash function is specific to the number and type of config variables you want to store and so isn't a generic solution but something you would have to adapt to your needs. As an example though here is how I'm currently hashing the four bytes of configuration data.
const int prime = 31;
int result = 1;
result = prime * result + config.a;
result = prime * result + config.b;
result = prime * result + config.c;
result = prime * result + config.d;
This code (which is based on what Eclipse generates for a four byte java structure) generates an integer (two bytes on the Arduino) based upon the four config values. Changing any of the config values changes the result of the hash function. There is also a second reason for using a hash function: data stored in EEPROM can degrade over time and the hash function allows us to check that the data hasn't changed since we stored it.

I know this hasn’t led to a single example sketch you can use, but hopefully the snippets I've presented will come in handy.

Write Once, Debug Everywhere

One of the reasons that I quite enjoy developing software in Java is the promise of WORA or write once, run anywhere. In principle this means that I can develop software on one machine and it should work exactly the same on any other computer, regardless of operating system, as long as a copy of Java is installed. I've been writing Java code for over a decade and in all that time I don't think I've ever personally seen a case where the principal falls down -- and certainly not if the code is restricted to just using the core Java libraries. So this morning, with a deadline fast approaching, I hit the opposite of WORA, WODE; write once, debug everywhere!

I'd written and debugged some code on my PC, and it was working flawlessly. I'd integrated the code into a tomcat hosted web app, and again testing on my own machine showed no problems at all. I uploaded the web app to one of the servers at work (more processing power and a better internet connection for demo purposes) and it instantly crashed! A quick look at the log showed the problem to be a ConcurrentModificationException. I've seen this exception before when I've tried to modify a list (or some other Collection instance) while looping through the list, but in this case a) I couldn't see anything wrong with my code and b) I had a machine where the code was working fine!

After a little debugging it became clear that the exception was being thrown by the following line of code:
items.removeAll(items.subList(0, items.size()/2));
Essentially this should halve the original list, retaining the second half (the items in the list were sorted based on an assigned score and I wanted to throw away the lower scoring items). It's worth noting that the subList call returns a view backed by the original list. In other words changes to one list should be reflected in the other. As the code worked fine on my machine I was more than a little confused.

Given that uploading the full web app to the server at work was a relatively time consuming process I wrote a small test application instead to try and figure out what was happening, including the expected output in the comments.
// create a list of four strings
List<String> items = new ArrayList<String>(
  Arrays.asList(new String[]{"a","b","c","d"}));
System.out.println(items); // [a, b, c, d]

// get a view over the first half of the list
List<String> someItems = items.subList(0, items.size()/2);  
System.out.println(items); // [a, b, c, d]
System.out.println(someItems); // [a, b]

// change the first item in the sub-list to "e"
System.out.println(someItems.set(0, "e")); // a
System.out.println(items); // [e, b, c, d]
System.out.println(someItems); // [e, b]

// change the first item in the original list to "f"
System.out.println(items.set(0,"f")); // e
System.out.println(items); // [f, b, c, d]
System.out.println(someItems); // [f, b]

// remove all items in the sub-list from the original list
System.out.println(items.removeAll(someItems)); // true
System.out.println(items); // [c, d]
System.out.println(someItems); // will probably throw an exception
I then ran this on my local machine, where my original code had been working and got the following result:
mark@mag:~$ java -version
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.4) (6b24-1.11.4-1ubuntu0.12.04.1)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

mark@mag:~$ java CMETest
[a, b, c, d]
[a, b, c, d]
[a, b]
a
[e, b, c, d]
[e, b]
e
[f, b, c, d]
[f, b]
true
[c, d]
Exception in thread "main" java.util.ConcurrentModificationException
  at java.util.ArrayList$SubList.checkForComodification(ArrayList.java:1091)
  at java.util.ArrayList$SubList.listIterator(ArrayList.java:972)
  at java.util.AbstractList.listIterator(AbstractList.java:300)
  at java.util.ArrayList$SubList.iterator(ArrayList.java:968)
  at java.util.AbstractCollection.toString(AbstractCollection.java:431)
  at java.lang.String.valueOf(String.java:2838)
  at java.io.PrintStream.println(PrintStream.java:788)
  at CMETest.main(CMETest.java:24)
So far so good. The output matches what I expected to happen, including the fact that the last line throws an exception. It makes sense to throw an exception at this point as I'm essentially trying to access a list consisting of items which I've deleted. So having checked that my test case behaved as I expected I uploaded this to the server at work and ran it.
mark@derwent:~$ java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

mark@derwent:~$ java CMETest
[a, b, c, d]
[a, b, c, d]
[a, b]
a
[e, b, c, d]
[e, b]
e
[f, b, c, d]
[f, b]
Exception in thread "main" java.util.ConcurrentModificationException
  at java.util.SubList.checkForComodification(AbstractList.java:752)
  at java.util.SubList.listIterator(AbstractList.java:682)
  at java.util.AbstractList.listIterator(AbstractList.java:284)
  at java.util.SubList.iterator(AbstractList.java:678)
  at java.util.AbstractCollection.contains(AbstractCollection.java:82)
  at java.util.AbstractCollection.removeAll(AbstractCollection.java:336)
  at CMETest.main(CMETest.java:22)
Fortunately for my sanity this shorter example failed to behave in the same way on the server as it did on my machine. You can see from the exception that it is failing on line 22 which is the call to removeAll. While that at least proves that the removeAll call is at fault it still leaves the question as to why it works on one machine and fails on another?

Even when the Java language wasn't available under an open source license the source code to the core APIs has always been available. Back in the early days of Java I even used this fact on a number of occasions to submit bug reports to Sun (who originally developed Java before they were bought by Oracle) along with suggested fixes. Unfortunately I haven't been able to track down the source bundles for the exact two versions of Java involved but I did manage to track down the OpenJDK source for version 1.6.0 build 26. This doesn't match either version exactly as I'm running b12 of OpenJDK whereas the server is running buld 26, but the Oracle version rather than OpenJDK. Having looked at the stack trace and this version of the source code I think I can now see why it fails.

The stack trace shows that when I call removeAll on list (which is actually an instance of ArrayList), the implementation in java.util.AbstractCollection is the one that is actually used. This method is a very simple iterator based implementation:
public boolean removeAll(Collection<?> c) {
  boolean modified = false;
  Iterator<?> e = iterator();
  while (e.hasNext()) {
    if (c.contains(e.next())) {
      e.remove();
      modified = true;
    }
  }
  return modified;
}
Now the documentation for the subList method states that
The semantics of the list returned by this method become undefined if the backing list (i.e., this list) is structurally modified in any way other than via the returned list. (Structural modifications are those that change the size of this list, or otherwise perturb it in such a fashion that iterations in progress may yield incorrect results.)
Clearly the call to e.remove() on line 6 structurally modifies the backing list, which will render the sublist, c undefined, which explains why an exception is thrown (probably on the call to contains on line 5). So now I know why it fails on the server, but why does it work on my machine.

I'm guessing that in the version of Java running on my machine, when I call removeAll, rather than using the default implementation in AbstractCollection, it instead uses the ArrayList specific version.
public boolean removeAll(Collection<?> c) {
  return batchRemove(c, false);
}

private boolean batchRemove(Collection<?> c, boolean complement) {
  final Object[] elementData = this.elementData;
  int r = 0, w = 0;
  boolean modified = false;
  try {
    for (; r < size; r++)
      if (c.contains(elementData[r]) == complement)
        elementData[w++] = elementData[r];
  } finally {
    // Preserve behavioral compatibility with AbstractCollection,
    // even if c.contains() throws.
    if (r != size) {
      System.arraycopy(elementData, r,
                       elementData, w,
                       size - r);
      w += size - r;
    }
    if (w != size) {
      for (int i = w; i < size; i++)
        elementData[i] = null;
      modCount += size - w;
      size = w;
      modified = true;
    }
  }
  return modified;
}
As you can see this implementation is slightly more complex, but I'm guessing that it's a lot more efficient. Essentially this implementation modifies the underlying array directly rather than via the public API -- it works by shifting each entry towards the beginning of the array overwriting any entries that need to be removed. Similarly the implementation of contains called from line 11, access the array directly meaning that the entire removeAll call never results in a check for a concurrent modification and hence never throws an exception.

So now I understand why the same code behaves differently on two different machines. Well I almost understand. The missing piece of the puzzle is why the rather naive version of removeAll is used on the server? Without access to the source code of the exact versions of Java involved I can't be certain, but I'm assuming that the ArrayList specific implementation is an OpenJDK efficiency improvement that isn't in the Oracle branded version.

Fortunately my original code is easy to fix, I can achieve the same effect (deleting the items in the first half of the list) using the following line of code which does appear to run anywhere.
items.subList(0, items.size()/2).clear();

Formatted Comments

One thing that has always confused me when using Blogger is why the comment form lacks any real features. When we use Blogger to write a post the dashboard gives us lots of options for formatting text and adding links etc. but when, as a reader of a blog, you want to add a comment you are presented with a simple text box. If I want to add a link or make some text standout I can, but I have to manually enter the required HTML. This isn't too much of a problem for me, but I'm sure there are many people that read blogs and leave comments who aren't comfortable writing HTML. I'm guessing that this explains why I see so many comments that contain URLs which aren't actually links.

So to make my life a little easier, and to enable other people to easily add formatting and links to their comments, I've developed a script that provides buttons for bold, italic and for adding links. Unfortunately I couldn't find a way of adding something to a blogs template to make this work, rather I've developed a Greasemonkey script that, once installed in your browser, will add this functionality automatically to every Blogger hosted blog you visit.

If you are using Firefox then you will need to have the Greasemonkey extension installed. If you use Chrome then (due to an annoying change Google have recently made) you will need to install Tampermonkey, and for Safari you will need to install NinjaKit. The script also works in Chromium (the open source version of Chrome without the Google branding) without requiring you to install an extension first.

Once you have your browser ready you can simply click this link to install my script. If you are using Tampermonkey then, when asked, click OK to install into Tampermonkey rather than Cancel to install natively in Chrome.

Once installed whenever you go to add a comment on a Blogger hosted blog you should find links for bold, italic and to add a link just above the comment box. Hopefully there shouldn't be any bugs as a) it's a fairly simple script and b) thanks to Adrian it's actually undergone some user testing, but if you have any problems or any suggestions for improvements then leave me a comment and I'll see what I can do.

Updated 14th July 2013: Added the link to NinjaKit for getting it to work in Safari.
Updated to v0.6 18th October 2022: Now supports the newer blogger comment form.

An 8-Bit Waterfall

So far my Arduino projects have all needed only a small number of output pins. In a previous post I talked about moving some of the logic off the Arduino and into other integrated circuits in order to free up pins on the Arduino. While this can help, I'm sure there will come a time when I'll need more output pins then the Arduino Uno has to offer. One solution would be to upgrade to an Arduino Mega as it has 54 digital output pins compared to the 14 digital pins on the Arduino Uno. The cost difference of around £15, however, doesn't make this an attractive option. Fortunately there is a much cheaper way of adding extra output pins; 8-bit shift registers.

Whilst you don't need to know the full details of how a shift register works an understanding of the basic principle will help. Essentially data is shifted into the chip one bit at a time using two pins; a data pin specifying the bit value and a clock pin that is triggered to show when the bit has been set. Once all the bits have been passed to the chip a third pin (known as the latch) is triggered at which point each bit is output on a separate pin of the chip. In the case of an 8-bit shift register (the 74HC595N for example) this allows us to control 8 pins (one for each bit) using just three pins on the Arduino (one each for data, clock and latch). In this way we can add five new digital output pins to an Arduino for just 70p (I got five chips for £3.50 from oomlout).

The 74HC595N is in fact such a common way to add extra pins to an Arduino that there is a simple tutorial already available, although I didn't find this before I'd already wired up the circuit you can see to the left. Due to a shortage of space on my breadboard I only bothered to wire LEDs to three of the output pins, but it should be clear how to add LEDs to the other five output pins should you wish to. Also it's worth noting that I haven't needed to add a capacitor to the latch pin (a capacitor would ensure a smooth flow of power, but the Arduino on it's own seems to work fine and I didn't have any spare capacitors lying around). It's important to make sure you pull the enable pin to ground otherwise you'll find the lights flicker uncontrolably, if they come on at all.

Using this circuit I can cycle through all combinations of the three LEDs by writing the numbers 0 to 7 to the data pin. This works because we only need three bits to encode numbers 0 to 7 which will cause the relevant pins to go LOW/HIGH as appropriate. For those of you who aren't so comfortable with binary the following table will help.

IntegerBinary
000000000
100000001
200000010
300000011
400000100
500000101
600000110
700000111

So to turn on the yellow LED we would write 2 to the chip or to turn on the green and orange LEDs we would write 5. The following simple sketch can be used to show that this does indeed work.
//the pins we are using
int latchPin = 2;
int clockPin = 3;
int dataPin = 4;

void setup() {
  //set all the pins used to talk to the chip
  //as output pins so we can write to them
  pinMode(latchPin, OUTPUT);
  pinMode(clockPin, OUTPUT);
  pinMode(dataPin, OUTPUT);
}

void loop() {
  for (int i = 0; i < 8; i++) {
    //for each of the numbers 0 to 7...

    //take the latchPin low so the LEDs don't
    //change while we are writing data
    digitalWrite(latchPin, LOW);

    //shift out the bits
    shiftOut(dataPin, clockPin, MSBFIRST, i);  

    //take the latch pin high so the pins reflect
    //the data we have sent
    digitalWrite(latchPin, HIGH);

    // pause before next value:
    delay(2000);
  }
}
Now while it is possible to control all 8 pins in the same way, sending a number from 0 to 256 to set the 8 pins isn't really an ideal approach (unless there are a small fixed number of output you need at which point you could define a constant for each of them and write those out as necessary). Ideally what we want to be able to do is to simply set each pin as LOW or HIGH without changing the state of any of the other pins. Fortunately this is easy to do using the Arduino bitSet and bitClear functions. For example, take this snippet of an Arduino sketch.
//assume all pins start out LOW
byte state = B00000000;

//set the third pin high, i.e.
//state = B00000100
bitSet(state, 2);

//take the latchPin low so the LEDs don't
//change while we are writing data
digitalWrite(latchPin, LOW);

//shift out the bits
shiftOut(dataPin, clockPin, MSBFIRST, state);  

//take the latch pin high so the pins reflect
//the data we have sent
digitalWrite(latchPin, HIGH);
Most of this code is from the previous sketch, the interesting bit is line 6. Here we set the 2nd bit (which forces it HIGH). Remember that, as in most computer situations, numbering starts from 0, so this sets the third pin HIGH, which given our example circuit would turn on just the orange LED. The bitClear function works in exactly the same way, but pulls the bit LOW rather than HIGH. It's reasonably straightforward to convert these ideas into a couple of functions which make working with a shift register easy. Partly as a learning exercise, but mostly as I think it will be useful, I've actually created a new Arduino library called ShiftRegister so that I can easily include the functionality in any sketch where I need it. You can grab the library from my subversion repository. The library includes the following example sketch which assumes the same circuit we've already looked at.
/**
 * ShiftRegister
 * Copyright (c) Mark A. Greenwood, 2012
 * This work is licensed under the Creative Commons
 * Attribution-NonCommercial-ShareAlike 3.0 Unported License.
 * To view a copy of this license, visit
 * http://creativecommons.org/licenses/by-nc-sa/3.0/.
 **/

#include <ShiftRegister.h>

ShiftRegister sr(4,3,2);

void setup() {
  //there is nothing to do
}

void loop() {
  sr.setPin(0, HIGH);      
  delay(2000);
       
  sr.setPin(1, HIGH);
  delay(2000);
    
  sr.setPin(2, HIGH);      
  delay(2000);
     
  sr.setPin(0, LOW);      
  delay(2000);
      
  sr.setPin(1, LOW);      
  delay(2000);

  sr.setPin(2, LOW);      
  delay(2000);
}
If you run this sketch on the Arduino you'll see that it simply lights up the three LEDs starting with the green one, and then turns them off again in the same order. I've tried to make it easy to configure the libary and as you can see in line 12 you can specify which pins you are using on the Arduino. By default calling setPin with two arguments automatically causes the changes to be sent to the shift register. If you want to send a number of changes at once then you can as this snippet shows.
//set pins 0 and 2 high
sr.setPin(0, HIGH, false);
sr.setPin(2, HIGH, false);

//push the new values to the shift register
sr.sync();
By this point you may be wondering why I called this post "An 8-Bit Waterfall", well the answer lies in one of the pins of the 74HC757N which we haven't looked at yet; the overflow pin.

When you have shifted in 8 bits the register is full. Shifting in a 9th bit causes the first bit you pushed in to be pushed out on the overflow pin. So just like water cascades down a waterfall so data can cascade down a series of shift registers. The circuit on the left shows how we can link two 8-bit shift registers together to give us 13 new digital output pins for just £1.40.

You can see the waterfall in action by running the example ShiftRegister sketch using this new circuit. What you'll notice is that the second set of LEDs is always one step behind the first set. To make actual use of the second register we need a new version of the library which can handle multiple registers. My imagination failed me and so the new library is called ShiftRegisters! You can also grab this version of the library from my subversion repository. This library includes the following example sketch.
/**
 * ShiftRegisters
 * Copyright (c) Mark A. Greenwood, 2012
 * This work is licensed under the Creative Commons
 * Attribution-NonCommercial-ShareAlike 3.0 Unported License.
 * To view a copy of this license, visit
 * http://creativecommons.org/licenses/by-nc-sa/3.0/.
 **/

#include <ShiftRegisters.h>

ShiftRegisters sr(4,3,2,2);

void setup() {
  //there is nothing to do
}

void loop() {
  sr.setPin(0, HIGH);      
  delay(2000);
       
  sr.setPin(1, HIGH);
  delay(2000);
    
  sr.setPin(2, HIGH);      
  delay(2000);

  sr.setPin(8, HIGH);      
  delay(2000);
       
  sr.setPin(9, HIGH);
  delay(2000);
    
  sr.setPin(10, HIGH);      
  delay(2000);
     
  sr.setPin(0, 0, LOW, true);      
  delay(2000);
      
  sr.setPin(0, 1, LOW, true);      
  delay(2000);

  sr.setPin(0, 2, LOW, true);      
  delay(2000);

  sr.setPin(1, 0, LOW, true);      
  delay(2000);
      
  sr.setPin(1, 1, LOW, true);      
  delay(2000);

  sr.setPin(1, 2, LOW, true);      
  delay(2000);
}
This example simply turns the six LEDs on and then off in sequence, but shows that there are two ways of addressing each pin. Firstly we can simply address each pin in turn; so the pins on the first chip are 0 to 7 and on the second 8 to 15, etc. Or we can specify pin 0 to 7 for each chip (numbered 0 and 1). Note that if you use the second approach you have to use the four argument version of the method and specify if you want sync called for you or not (this is because the arguments preclude me producing a three argument version as the compiler gets confused). The second method will be slightly quicker as there is a little less maths involved, although which you find most intuitive is up to you. Note also that line 12 now includes (as the last argument) the number of shift registers we have in sequence. If you don't specify the number then the library assumes there are 32 registers giving you 256 pins in total (or 233 new pins). Given the way the library is written (the pin variable is a byte) 32 registers is also the maximum it can support.

You could use the second library with just one shift register if you wanted to, but I've kept the two versions separate in case code size is an issue as the second version is larger and there is no point including extra code within a sketch if it isn't needed.

So for my initial investment of £3.50 I can add an additional 37 digital output pins to my Arduino Uno. Of course you can't use these pins for input as you could with the extra pins on an Arduino Mega, but you can also add extra input pins in a similar way.

Fritzing: Custom Components

So I've been playing around with a few more electronic circuits over the last week or so and found myself wanting to move some of the simple logic processing off the Arduino. As an example, I wanted to implement the following logic table

InputsOutputs
ABYZ
0000
0101
1111
1011

Now if you look closely at this table then you will see that Y is the same as A, while Z is simply A or B. Now this would be simple to implement using an Arduino, in fact you would just need the following sketch.
void setup() {                
  pinMode(11, OUTPUT); // output Y
  pinMode(12, OUTPUT); // output Z
  
  pinMode(8, INPUT); // input A
  pinMode(9, INPUT); // input B  
}

void loop() {
  digitalWrite(11, digitalRead(8));
  digitalWrite(12, digitalRead(8) | digitalread(9));
}
This could be simplified slightly by not using the Arduino to set Y and just using A directly, but it would still require three pins to read A and B and generate Z. Given the limited number of pins on an Arduino this seems wasteful. Now most logic gates can be built from a combination of transistors, but again they can quickly get complicated, especially if you need more than one logical operation. Fortunately there are a number of cheap integrated circuits that you can buy that implement the standard logical operations (and some not so common). After a quick trip to Maplin on Thursday, 99 pence got me a single chip with four 2-input OR gates: the 74HCT32N from NXP. It was simple to add it to the breadboard and wire it into my circuit instead of using the Arduino to compute the OR.

The only problem arose when I went to record my circuit using Fritzing. Clearly Fritzing can't support every single component in the world, and it didn't have a specific component to represent the 74HCT32N. It does, however, have a number of generic parts which are easy to customize. One of these allows you to represent any integrated circuit by specifying the number of pins etc. The problem is that this generates a breadboard view where the chip is simply labelled IC and a schematic view that just labels the pins on the chip and tells you nothing about the internal workings. Now for a complex chip this makes sense but for something as simple as OR gates it would be helpful if the schematic view included them.
The left hand chip is how Fritzing represented the 74HCT32N when I used the generic IC part, the chip on the right is how I wanted it to look. Now given this is a screenshot I obviously succeeded, and the rest of this post will explain how.

Now you can create parts from scratch, but after an hour or so of trying and getting nowhere, I hit upon a simpler solution.

The first step is to use the parts editor (ignoring the warning about it being bug ridden) to label the pins and properly name the component etc. Once you are happy with those values click to save it as a new part.

The new part will be listed in the "MINE" parts bin. Simply right-click on it and choose "Export Part". This will give you a file with a fzpz extension. This is just a zip file which you need to unpack. You should find that you now have five files: a fzp file which describes the part and four SVG image files for the icon, breadboard, schematic and PCB views.

You can now go ahead and edit the SVG files. Be careful not to edit or alter any of the connectors in the images but otherwise it's safe to make changes. If you are wanting to draw logic gates inside a chip then you can simply grab the appropriate images from Wikipedia which makes life really easy as all you need to add are the connections between the gates and the pins. As well as improving the schematic view I also edited the icon and breadboard views: the breadboard view now gives the full part number and the icon shows OR instead of IC. Note that to avoid any problems try saving to plain SVG rather than any application specific extension.

Once you are happy with the images go back to the part editor and replace the images with the new versions (even though I didn't edit the PCB image I replaced it anyway as a bug in the part editor means it seems to get lost if you don't). Once you are happy you can export the part again if you want to keep a backup copy or share it with other Fritzing users.

I'm going to make any custom components I produce available to anyone who wants them. For now there is just this single chip but they will all appear here.

:), :-), :o), or :]

I've been doing a lot of different things at work recently but one of them got me thinking about emoticons, or as I think most people call them, smileys.

Smileys (as I'll refer to them throughout) often convey emotive opinions. For example, if you have just been given a gift you might use :) in an e-mail or tweet saying thank you. On the other hand if your flight has been delayed you might well use :( to show your displeasure. It's fairly easy to build two lists of smileys, one positive and one negative, containing all the possible variations of each smiley. These lists can then be used as one feature when trying to classify text and such lists can be found in most opinion mining systems which attempt to label a piece of text as positive, negative or neutral.

It turns out though, that in some cases a sad smiley can actually have a positive meaning. This came to my attention when participating in the profiling task of RepLab 2012. The motivation behind the profiling task is brand management. For example, imagine you are the PR company responsible for a company like Apple. In an ideal world you would read every document ever written about the company and try and address any negative opinions you discover. Of course we don't live in an ideal world and the profiling task aimed to encourage development of systems that could do this task automatically. The task used tweets as the documents, and the aim was to first determine if a tweet was relevant to a given entity (i.e. is it talking about Apple the computer manufacturer rather than the fruit or record label) and then to determine it's polarity, i.e. does the tweet have positive or negative implications for the company's reputation. On the face of it, you would think that the second step could be performed by a standard opinion mining system. Unfortunately I don't think that is necessarily the case.

The example given in the task description makes it quite clear that a sad sounding tweet can actually have a positive effect on a brand: R.I.P. Michael Jackson. We'll miss you. I could easily imagine this tweet being extended with the addition of a sad smiley. So a sad smiley could easily occur in both negative and positive tweets.

As I was intending to use a machine learning algorithm to build a classifier to learn polarity this dual use of smileys (and other words in general) didn't bother me too much as if I could find enough training data then hopefully the algorithm would sort out the contradictions. The problem was that I didn't have that much training data (400 tweets for each of 6 entities). Unfortunately there are many ways to write the same smiley and this variation means that the algorithm might see each version just once and would therefore be unable to draw any strong conclusions from it's presence.

The solution of course is to normalize each smiley to a given form. So for example, I would normalize all the smileys in the title of this post to :) and then feed that to the machine learning algorithm. In GATE this was easy to achieve using a gazetteer with an additional feature to store the normalized version. The gazetteer I've built covers all of the western style (i.e. viewed from the side) smileys I could find as well as the relevant Unicode symbols. To save anyone else the hassle of having to build such a gazetteer I've made it available for anyone who is interested (note when loading it into GATE use a space as the feature separator).

Of course, after the work of assembling the gazetteer it didn't actually make any difference to the performance of my polarity classifier. It turns out that in all the training data I had there weren't many smileys! Given that I'm going to be doing more work in this area over the coming months, I'm hoping that it will eventually turn out to be useful.

This does all lead to a question though -- while there are lots of ways of writing the same smiley do most people use the simplest form, i.e. :) instead of any of the three character versions?

An Arduino Powered (Scale) Speed Trap

After a break of around two decades I've recently started building a model railway. One of the issues I've faced is trying to work out how fast I should be running the trains so that their speed reflects reality given the scale at which they are modelled. I'm guessing the details won't interest everyone reading this post but if you are interested then I've blogged about this from the model railway side on one of my other blogs. Suffice it to say that what I needed was to be able to measure the time it took for a model locomotive to travel a certain distance. Now I could have used a ruler and a stop watch but this would never be very accurate. So being me I turned to a more computer oriented solution: an Arduino powered speed trap!

The brief I set myself was simple:
  • two switches to measure the time taken to travel a given distance
  • a green LED to signify the train was travelling below a given speed limit
  • a red LED to signify the speed limit was being broken
  • full speed details passed back to the PC for display
The main question was what form of switches should I use. Physical switches were out as I would never be able to accurately place them on the track in a way that any locomotive would be able to trigger them without incident. A reed switch would be easy to use and to hide on the layout, but would mean adding a magnet to each locomotive which seemed a bit daft. An infrared beam across the track would also work, but hiding it on the layout would be difficult. In the end the only sensible idea I could come up with was using a light dependent resistor (LDR) and watching for a sudden change in resistance as the light was blocked by the moving locomotive.

So on my way home on Thursday I called in at my local Maplin store and bought two of the smallest and flattest LDRs they had (specifically the 1.8k-4.5k version).

Now while I knew that the more light you shine on an LDR the less resistance it has I wasn't sure of the best way of making use of this information in conjunction with the Arduino. Fortunately the web is awash with information and tutorials and I quickly came across the solution.

So with my two LDRs, two LEDs and a bunch of resistors I knocked together the following (note that both views were generated at the same time using Fritzing, I really am impressed by this application).


As you can probably see this is a little more complicated than it needs to be as it uses two resistors for each LED, but this was the best I could manage with the resistors I had.

Of course the hardware is only half of the solution. Without appropriate microcode the Arduino isn't going to do anything useful. Fortunately I'm better at writing software than I am at designing circuits so this half was easier.

The code is (fairly) straightforward. Essentially it's a state machine that (ignoring invalid inputs) follows the following steps:
  1. wait until sensor 1 is triggered
  2. when sensor 1 is triggered record the time
  3. wait until sensor 2 is triggered
  4. determine the time difference between the two sensors being triggered and use this to calculate the speed of the locomotive
  5. wait until both sensors have returned to normal then return to step 1
This is easy to implement and the full Arduino sketch is as follows:

/**
 * ScaleSpeed
 * Copyright (c) Mark A. Greenwood, 2012
 * This work is licensed under the Creative Commons
 * Attribution-NonCommercial-ShareAlike 3.0 Unported License.
 * To view a copy of this license, visit
 * http://creativecommons.org/licenses/by-nc-sa/3.0/. 
 **/

//keep the sketch size down by only compiling debug code into the
//binary when debugging is actually turned on
#define DEBUG 0

//all possible states of the state machine
const byte TRACK_SECTION_CLEAR = 0;
const byte ENGINE_ENTERING_SECTION = 1;
const byte ENGINE_LEAVING_SECTION = 2;

//the current state machine state
byte state = TRACK_SECTION_CLEAR;

//the analog pins used for each sensor
const byte SENSOR_1 = 0;
const byte SENSOR_2 = 1;

//the digital pins for the signalling LEDs
const byte GREEN_LIGHT = 13;
const byte RED_LIGHT = 12;

//the threshold values for each sensor
int sensor1 = 1024;
int sensor2 = 1024;

//intermediate steps to calcualte the scale distance we are measuring
const float scale = 76; //this is OO gauge
const float distance = 74; //measured in mm
const float scaleKilometer = 1000000.0/scale;

//This is the only value we actually need to do the calculation
//We could do this calculation on the computer and pass it across
//or store it in EEPROM. it all depends if we expect to always be
//attached to a computer or if we want to run standalone etc.
const float scaleDistance = distance/scaleKilometer;

//the track speed limit in mph
const float speedLimit = 15;

//the time (in milliseconds from the Arduino starting up that the
//first sensor was last triggered
unsigned long time;

void setup(void) {
  //enable output on the digital pins
  pinMode(GREEN_LIGHT, OUTPUT);
  pinMode(RED_LIGHT, OUTPUT);

  //turn on both LEDs to show we are calibrating the sensors
  digitalWrite(GREEN_LIGHT, HIGH);
  digitalWrite(RED_LIGHT, HIGH);
  
  //configure serial communication
  Serial.begin(9600);
  
  //let the user know we are calibrating the sensors
  Serial.print("Callibrating...");
  
  while (millis() < 5000) {
    //for the first five seconds check and store the lowest light
    //level seen on each sensor
    sensor1 = min(sensor1, analogRead(SENSOR_1));
    sensor2 = min(sensor2, analogRead(SENSOR_2));
  }
  
  //the cut off level for triggering the state machine
  //is half the resistance seen during calibration
  sensor1 = sensor1/2;
  sensor2 = sensor2/2;  
  
  //we have now finished callibration so tell the user...
  Serial.println(" Done");
  
  //... and set the signalling to green (i.e. we haven't yet seen
  //anything break the speed limit!
  digitalWrite(GREEN_LIGHT, HIGH);
  digitalWrite(RED_LIGHT, LOW);
}

void loop(void) {
    
  if (state == TRACK_SECTION_CLEAR) {
    //last time we checked the track was clear
    
    if (analogRead(SENSOR_1) < sensor1) {
      //but now the first sensor has been triggered so...
      
      //store the time at which the sensor was triggered
      time = millis();

      //advance into the next state
      state = ENGINE_ENTERING_SECTION;
      
      #if (DEBUG)
        Serial.println("Train entering measured distance");
      #endif

    }
  }
  else if (state == ENGINE_ENTERING_SECTION) {
    //the last time we checked the first sensor had triggered but
    //the second was yet to trigger
    
    if (analogRead(SENSOR_2) < sensor2) {
      //but now the second sensor has triggered as well so...
      
      //get the difference in ms between the two sensors triggering
      unsigned long diff = (millis() - time);

      //calculate scale speed in kph
      //3600000 is number of milliseconds in an hour
      float kph = scaleDistance*(3600000.0/(float)diff);

      //convert kph to mph
      float mph = kph*0.621371;
      
      //report the time and speed to the user
      Serial.print("Speed Trap Record: ");
      Serial.print(diff);
      Serial.print("ms ");
      Serial.print(kph);
      Serial.print("kph ");
      Serial.print(mph);
      Serial.println("mph");
      
      if (mph > speedLimit) {
        //if the speed we calculated was above the speed limit
        //then turn off the green LED and turn on the red one
        digitalWrite(GREEN_LIGHT, LOW);
        digitalWrite(RED_LIGHT, HIGH);
      }
      else {
       //if the speed we calculated was not above the speed limit
       //then turn off the red LED and turn on the green one
       digitalWrite(GREEN_LIGHT, HIGH); 
       digitalWrite(RED_LIGHT, LOW);
      }
      
      //move into the next state
      state = ENGINE_LEAVING_SECTION;
    }
  }
  else if (state = ENGINE_LEAVING_SECTION) {
    //last time we checked both sensors had triggered but both
    //had yet to reset back to normal

    if (analogRead(SENSOR_1) > sensor1 && analogRead(SENSOR_2) > sensor2) {
      //both sensots are now clear so...
      
      //move back to the first state ready for next time
      state = TRACK_SECTION_CLEAR; 
      
      #if (DEBUG)
        Serial.println("Train is clear of measured distance");
      #endif
    }
  }
}

Now that might look like a lot of code but if you remove the code comments there isn't really that much going on. Everything up to line 50 just creates some constants and variables ready for the speed calculations. The setup loop in lines 52 to 86 calibrates the two LDRs: we look at the sensors for five seconds and record the lowest light level we see and set the threshold for triggering at half this value. The main loop method then simply implements the state machine we outlined above using a set of if...else statements.

So the main question is does it work? Testing by simply using my hands to cover the sensors suggested that everything worked well. The last thing to do was to actual time a locomotive. As you can see from this photo it was easy to attach to the track for testing and I can report that it works really well.

There are many ways in which the code and hardware could be improved. Configuring the distance, speed limit and scale without re-compiling would be useful, as would using a small LCD or multi-segment LED to display the speed, but for now, at least, I'll leave those as exercises for the interested reader.

Pixel Level Precision

I've heard a number of people claim recently that Blogger doesn't give you much control over the size of images you can add to your post. Essentially they give you small, medium, large, x-large and original size. In theory this gives you full control as you could resize your images outside of blogger and then display them at original size. The problem with this of course is that you might not know what size would work best until you have uploaded your image. The solution is to switch to the HTML view where, if you know what you are doing, you can have full control over the size of your image right down to the pixel.

So for those of you who would like more control but have never messed around with HTML before I'll walk you through the steps using an image about dead pixels.

I chose to upload the image to be displayed at a small size and to the left. Blogger generated the following HTML to achieve this (note Blogger puts it all on one line but I've separated it out to make it easier to read and refer to):

<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgznpxj-ACnxUWDExnR4EzI4kGQ-JuD2TzBjYMWega3VV51nQZPVQ50jILnN727oC58SDzIIdlTPclNODNu8oLG_OP1V2wM0YJ1FUaxF6JAW9wp3iDcEXgl4eEZcX8oVI780PbsW1yqCec/s1600/dead-pixels.png" imageanchor="1" style="clear:left; float:left;margin-right:1em; margin-bottom:1em">
<img border="0" height="150" width="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgznpxj-ACnxUWDExnR4EzI4kGQ-JuD2TzBjYMWega3VV51nQZPVQ50jILnN727oC58SDzIIdlTPclNODNu8oLG_OP1V2wM0YJ1FUaxF6JAW9wp3iDcEXgl4eEZcX8oVI780PbsW1yqCec/s200/dead-pixels.png" />
</a>
</div>

Essentially Blogger has put the image (line 3) inside a link (starts on line 2 ends on line 4), so that when you click on the image you get a larger version, which in turn is placed inside it's own section (a div tag starting on line 1 and ending on line 5). Fortunately we can ignore everything but the image on line 3.

If you look at line 3 you will see that the width and height of the image are explicitly set to 200 pixels and 150 pixels respectively. Now if you change one of these values you will obviously need to change the other one to ensure that your image maintains the same ratio of width/height. Fortunately you don't actually need to do any maths! I'm not sure why Blogger explicitly sets both dimensions of an image because by default all browsers maintain the aspect ratio of an image when only the width or the height is set. So depending how you want to specify the size you need to delete one of the values and change the other. I usually find it easiest to delete the height and alter the width.

So let's change remove the height and change the width to 400 pixels to give us:

<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgznpxj-ACnxUWDExnR4EzI4kGQ-JuD2TzBjYMWega3VV51nQZPVQ50jILnN727oC58SDzIIdlTPclNODNu8oLG_OP1V2wM0YJ1FUaxF6JAW9wp3iDcEXgl4eEZcX8oVI780PbsW1yqCec/s1600/dead-pixels.png" imageanchor="1" style="clear:left; float:left;margin-right:1em; margin-bottom:1em">
<img border="0" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgznpxj-ACnxUWDExnR4EzI4kGQ-JuD2TzBjYMWega3VV51nQZPVQ50jILnN727oC58SDzIIdlTPclNODNu8oLG_OP1V2wM0YJ1FUaxF6JAW9wp3iDcEXgl4eEZcX8oVI780PbsW1yqCec/s200/dead-pixels.png" />
</a>
</div>

As you can see the image is now larger and still has the right aspect ratio. Success? Not quite yet. What you might not be able to see overly well in this example is that the new larger image isn't actually as sharp as it should be.

To try and make sure that your blog loads as quickly as possible Blogger tries to keep down the amount of data it needs to transfer to your computer. One of the ways it does this is by resizing the images before it sends them to you. If you look again at line 3 you will notice that the end of the image URL looks like /s200/dead-pixels.png. The 200 in the URL tells Blogger which size version of the image to send, in this case 200 pixels along the longest edge. So you now have a 200 pixel wide image being scaled up to 400 pixels wide by the browser which is why it doesn't look as sharp as it should do. Fortunately it is easy to change the URL to give us an appropriate size image.

I'm not sure of the full range of valid sizes but I do know that you can specify any sizes up to 1600 in multiples of 200. So for this example we can specify that we want a 400 pixel wide image:

<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgznpxj-ACnxUWDExnR4EzI4kGQ-JuD2TzBjYMWega3VV51nQZPVQ50jILnN727oC58SDzIIdlTPclNODNu8oLG_OP1V2wM0YJ1FUaxF6JAW9wp3iDcEXgl4eEZcX8oVI780PbsW1yqCec/s1600/dead-pixels.png" imageanchor="1" style="clear:left; float:left;margin-right:1em; margin-bottom:1em">
<img border="0" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgznpxj-ACnxUWDExnR4EzI4kGQ-JuD2TzBjYMWega3VV51nQZPVQ50jILnN727oC58SDzIIdlTPclNODNu8oLG_OP1V2wM0YJ1FUaxF6JAW9wp3iDcEXgl4eEZcX8oVI780PbsW1yqCec/s400/dead-pixels.png" />
</a>
</div>

As you can see changing the URL to ask for a more appropriately sized image gives us a much sharper photo as the browser doesn't need to stretch it to the requested size.

The trick of course is to choose an image size that balances the trade off between sharpness and download speeds. I always go for a URL pointing to the smallest image that is the same size or larger than I want to display. This ensures that the browser never has to stretch the image and crushing it downwards doesn't have quite such a dramatic effect on sharpness.

Hopefully all that made sense and will allow those of you who aren't too comfortable editing HTML to have better control over the images you use in your posts. If there was anything that wasn't clear or didn't make sense leave me a comment and I'll try and help.

Fritzing

Over the last few days I've been playing around with my Arduino and have finally got as far as interfacing it with other electronic components rather than just writing software to run on it. Whilst it is easy to store the software side of multiple Arduino projects safely I wasn't entirely sure of the best way of recording the hardware setup -- basically I want to be able to take a snapshot of a project so that at some later date I can recreate it, either because I messed something up or because I'm playing with multiple projects at the same time and reusing components. It turns out that the solution is obvious: Fritzing.

If you look at any of the Arduino tutorial pages (e.g. the basic blink demo) you can see that it contains both a visual representation of the project and a traditional circuit diagram. I'd always assumed that these were produced separately but I was wrong.

These diagrams are all drawn using Fritzing. Fritzing supports three views of a project: breadboard, schematic and PCB. The breadboard view shows a drawing of your project that is almost a photograph of the real thing. The schematic view gives a traditional circuit diagram, and the PCB view allows you to convert a prototype into a PCB that could be manufactured. As far as possible, changes in one view are reflected in the other views.

This means it is easy to document a project simply be recreating it in the breadboard view and allowing Fritzing to generate the circuit diagram for you.

On top of all that the people behind Fritzing have also produced Fritzing Fab: a cheap way of printing custom PCBs. So you can easily prototype an idea with an Arduino, record the prototype in Fritzing, and then generate a permanent version by printing and populating the PCB. I haven't got as far as needing to print a PCB yet, but given how integrated the steps are it could turn out to be really useful. If nothing else expect any future Arduino related posts to include Fritzing generated images.

Message Passing

As I mentioned in a previous post, I've recently bought an Arduino to play with; both as a way of expanding into possibly doing some experimenting with electronics and also as I don't want to risk damaging my Raspberry Pi by playing with it's GPIO pins.

Now while it is fun to make lights blink on and off, and with the right electronics there are lots of things you can do which don't require the Arduino to be connected to a PC (other than to upload the code to it) I'm more interested (at least at the moment) on controlling the Arduino and any connected electronic components from within software running on my computer. This allows me to move some of the logic etc. off the Arduino, and means I can interface with it using any language I like; or at least any language that can talk to the USB port. The Arduino is programmed using C or C++ both of which are languages which I have only slight experience of (I now remember just how much I hate pointers) and so I can write the computer side of things in Java where I'm a lot more comfortable.

Given how I wanted to use the Arduino, one of the first main issues I've come across is finding a sensible way of passing messages backwards and forwards. I can easily read and write text to the USB port (actually a sequence of bytes but for this description saying text will suffice), the problem I had was figuring out a sensible way of converting the text into commands that I could easily interpret on the Arduino. There are a few libraries available for doing this (CmdMessenger for example) and a number of complex state machine based approaches but they all seemed either overly complex or over engineered for what I wanted. As I'm going to be in control of the code running on both the Arduino and the PC I can cut down on some of the error handling, and I will also know the full syntax of all possible commands. This allows me to reduce the code down to just the functional aspects I'll need. While it would certainly have been quicker and easier to use an existing library I a) wouldn't have (re)learnt as much as I did and b) it would have resulted in a larger code size. This second point is important as the ATmega328 PIC at the heart of the Arduino only has 32K of RAM. In comparison to modern computers this is almost nothing; although it is twice as much as the Acorn Electron on which I learnt to program and the same as the BBC Model B. My point is, that with such limited memory available (to hold both the program and the data) using a library with lots of functions I'll never need will reduce the amount of space I have available for no good reason. So let's dive in and I'll explain the approach I'm using.

The first part of any message passing system has to involve deciding on the format of the messages. For this example I'm assuming that all messages that we want to pass to the Arduino meet the following restrictions:
  • the message type is encoded as a single character
  • the command and any parameters are separated by the : character
  • each message is terminated by the new line character
  • each message can be at most 64 characters in length
So, as an example, we could have a command that takes two numbers adds them together and returns the result. Given that the command is doing addition we could use A to represent the message type. This would then give us the simple message A:3:5 to send (with a new line at the end) to the Arduino which should respond with 8. We could do something similar for subtraction, S:8:3 giving a response of 5.

So now that we know what a message will look like we need code to read one in:
//The maximum length (in characters) of a command
const int MAX_LEN = 64;

//the buffer that holds incoming data
char buffer[MAX_LEN+1];

//the offset in the buffer where the next byte should be placed
int offset = 0;

//true if there is a new command to be processed
boolean cmdReady = false;

/**
 * A SerialEvent occurs whenever a new data comes in the
 * hardware serial RX, and this method is called between
 * successive calls of loop()
 */
void serialEvent() {
  while (Serial.available() && !cmdReady) {
    //if there are bytes available and there isn't a completed
    //command waiting to be processed...
    
    //get the new byte:    
    char inChar = (char)Serial.read();

    //if the incoming character is a newline, set a flag
    //so the main loop can do something about it:
    if (inChar == '\n') {
      //a new line char signifies the end of the command so...
      
      //null terminate the string
      buffer[offset] = NULL;

      //reset the offset ready for next time
      offset = 0;

      //flag up that there is a command ready to process
      cmdReady = true;
    } else {
      //add the character to the end of the command
      buffer[offset] = inChar;

      //move the offset on ready for the next character
      ++offset;
    }
  }
}
Lines 1 to 11 define the maximum message size, create a buffer to hold a message, and set some flags for recording how much message we have read in so far and if we have read in a complete command. The method defined in the rest of the snippet is called frequently when running on the Arduino (between successive invocations of the main loop), and it is responsible for reading in the actual message. Firstly it checks to see if there is any data to read and even if there is it only continues if there isn't a completed command waiting to be processed (the loop condition on line 19). It then reads in the next available character. If the character is the new line character then we have read in a complete message, so we NULL terminate the message (line 32; this is a C thing so that we know where in the buffer the message finishes), reset the offset counter ready for the next message (line 35), and then flag that there is a command waiting to be processed (line 38). If it wasn't a new line character then we add it to the message we are building up (line 41) and then record where we should put the next character (line 44).

Now that we can read in a message we need to actually do something with it. Chances are that you will want to do this in the main loop method, probably with code that fits the following template:
void loop() {
  
  if (cmdReady) {
    //there is a command waiting to be processed...

    //so process it in here
  }

  //do other stuff that has to be done no matter what 
}
There are probably many ways you could set about processing each message but what follows is an example showing an implementation of the add message described above.
//The set of commands we understand
const char ADD = 'A';

//the set of token separators that split the cmd into sections
const char separator_tokens[] = { ':' };

void loop() {
  
  if (cmdReady) {
    //there is a command waiting to be processed...
     
    //get the actual command (i.e. the first token)
    char* current = strtok(buffer,separator_tokens);
    
    //switch based on the command
    switch (*current) {
        
      case ADD:
      {
        //get and convert to ints the two numbers to add
        int x = atoi(strtok(NULL,separator_tokens));
        int y = atoi(strtok(NULL,separator_tokens));
        
        //do the addition
        int z = x+y;
        
        //return the result
        Serial.println(z);
        
        break;
      }
        
      default:
        //this only happens if the command is unknown
        //which should never happen!
        Serial.print("Unknown Command!");
    }
           
    //we have processed the command
    cmdReady = false;
  }
  
  //do other stuff that has to be done no matter what 
}
So let's work through this to make it clear what's happening. Firstly, while the message types are encoded as single characters it's nicer to be able to refer to them by name, and so line 2 maps message type A to a constant called ADD. We also know that the sections of a command are separated by a colon, so we record this in line 5; note that we could have multiple separators if we wanted to, but the most important thing to remember is not to use a character you ever want to use in a message. Once we know we have a command to process the first thing we need to know is the type of the command. To do this, on line 16, we get the first section of the message (everything before the first colon) using the strtok method. Essentially this splits the message into sections, given the separator character, and returns the first section. Now as we are using single characters for the message type we can simply switch based upon the character (line 19). If we were using multiple characters for the message type then we would need to use a set of if-then-else statements instead to check the type. Once we know what the message type is we can then process the arguments (if it has any). In this example you can see that lines 21-34 deal with the ADD message. You don't really need to understand the example, other than to see that we can retrieve the arguments by calling strtok again (note that we pass in NULL instead of buffer to essentially tell the method to carry on from where it was last time). Once we have dealt with the current message we can get ready for the next one by resetting the cmdReady flag (line 43).

I've put together a slightly longer example which also includes messages for subtraction and echoing arguments back to the PC and which shows how the snippets I've presented go together to produce the full Arduino sketch. Hopefully someone else will find this useful, but please do let me know if I've missed something obvious or if there is a simpler way of achieving a similar result.