netzstaub

beatz & funkz

Monday, August 18, 2008

Sysex bootloader for AVR

I build a lot of small MIDI devices, and using a programmer often is not an option because the programming port is used for MIDI itself, or to connect other circuitry, and toggling jumpers is just not an option. Also, it is nice to have the option to have users of my hardware update the firmware themselves. The logical solution was thus to use MIDI itself to update the firmware. Download the C sourcecode of the bootloader here: bootloader.c

A bootloader for the AVR family resides in the upper regions of flash memory. A fuse can be programmed to tell the AVR to boot not from address 0x0000, but from the start address of the bootloader. Thus, the bootloader gets called before any application code, and is able to reprogram the device before executing code. This has two advantages for us: the obvious one of being able to program the device over MIDI, and the good thing to have a way to upgrade firmware if a bug is found in the code. The size of the bootloader portion can be programmed into the fuses as well. The bootloader can be 2048 words big (not possible on the atmega8 I’m using), 1024 words big (that’s 2kB), 512 words, 256 words or 128 words big. I settled on the 512 words size, cause that just about fits my MIDI bootloader code. You can set the fuse bits using your atmel programmer. I use avrdude and the following command (out of the Makefile):

init:
#       enable watchdog, external crystal
# 1024 words bootloader
	avrdude -p m8 -P usb -c usbasp -U hfuse:w:0xd8:m -U lfuse:w:0xcf:m
init512:
# 512 words bootloader
	avrdude -p m8 -P usb -c usbasp -U hfuse:w:0xda:m -U lfuse:w:0xcf:m

The memory taken up by the bootloader is called “No-Read-While-Write” memory, while the lower memory used for storing the program flash is called “Read-While-Write” memory. What this means is that while writing to the RWW section, the CPU can read from the NRWW section.

Finally, we have to tell the linker the correct start address for the code, which I do by passing a flag to gcc:

bootloader.elf: bootloader.o 
	$(CC) $(CLDFLAGS) -Wl,--section-start=.text=0x1C00 -o $@ $+

0x1c00 is the start adress of the code here (0x1c00 = 0x2000 – 0x400).

Reprogramming the flash memory of the AVR is done page-wise. Each page is 64 bytes big. First a page needs to be erased, and then written all at once. avr-libc provides a few helper function to reprogam the flash. You have to be careful while reprogramming not to overwrite the bootloader itself. Here is the code doing the actual programming in my MIDI bootloader, extracted from the function write_block:

  uint8_t sreg = SREG;
  cli();
  boot_page_erase(sysex_address);
  boot_spm_busy_wait();
  uint16_t address = sysex_address;
  for (i = 0; i < 64; i+=2) {
    uint16_t tmp = sysex_data[i] | (sysex_data[i + 1] << 8);
    boot_page_fill(address, tmp);
    address += 2;
  }
  boot_page_write(sysex_address);
  boot_spm_busy_wait();
  boot_rww_enable();
  SREG = sreg;

First we disable interrupts, the erase the page at sysex_address, which is the address where we want to write. Waiting for the page to be erased is done by calling the boot_spm_busy_wait function. Then we fill the write buffer using the function boot_page_fill, and write the page using boot_page_write. Finally, we reenable reading from the RWW section calling boot_rww_enable. You can see in the loop above that boot_page_fill takes a word as argument, so we create 16 bit values out of the data in the sysex buffer (LSB first). Now that we know how to flash a page, let's see how the communication is handled.

On the microcontroller side, MIDI is just a serial bus. The MIDI input and output are connected to the TX and RX pins of the microcontroller. So why not just use a serial bootloader? That's because MIDI imposes certain restrictions on the data that can be sent over it. Bigger data messages have to be encapsulated in "System Exclusive" messages. These are messages that begin with 0xF0 and end with 0xF7. Due to the inband-signaling of MIDI using the MSB of each byte, we can only send 7-bit data in a sysex message. Thus we have to encode 8-bit data into 7-bit data as described in this previous blog post.

We saw that writing to the flash is done page-wise, and that each page is 64 bytes big. It is pretty straightforward then to transmit 64 bytes of program data over sysex, and to flash them into a new page. Should flashing go wrong, the bootloader is still there to restart the whole process. However, we want to avoid flashing invalid data. Thus, we use a simple checksumming code in the sysex communication to increase the reliability of the sent data. We XOR each byte together and store that as a checksum at the end of the sysex data. This checksumming is done in write_block:

  uint8_t checksum = 0;
  uint8_t i;
  for (i = 3; i < sysex_cnt - 1; i++) {
    checksum ^= data[i];
  }

  uint8_t length = data[4];
  uint16_t sysex_address = make_word(5, 4);

  if (sysex_address >= APP_END) {
    return 0;
  }
  
  uint8_t cnt = 0, recvd = 0;
  uint8_t bits = 0;
  for (cnt = 0; cnt < (sysex_cnt - 9); cnt++) {
    if ((cnt % 8) == 0) {
      bits = data[9 + cnt];
    } else {
      sysex_data[recvd++] = data[9 + cnt] | ((bits & 1) << 7);
      bits >>= 1;
    }
    if (recvd >= length)
      break;
  }

  uint8_t check = data[sysex_cnt - 1];
  checksum &= 0x7f;

  if ((checksum != check) || (recvd != 64)) {
    return 0;
  }

We can see the checksumming in the first few lines. We get the length of the page out of the data packet, and create a 16 bit address out of 4 sysex bytes with make_word. Finally, we decode the sysex data as described in the previous blog post. We check both the checksum and the length of the received data before writing the block.

We have however another major problem to take care of. Writing to the flash is not fast, and we don't have enough memory to store a whole firmware in flash. We do actually just store the current page in memory. We have to find a way to throttle the sending of the firmware over MIDI to allow the device to write to its flash. This is done in a very simple way: each block writing is acknowledged with an ACK message over MIDI. The sender won't write another firmware block to MIDI before receiving the ACK message.

This was basically the whole MIDI part of the bootloader, for more detail check the bootloader.c file. handle_midi is a very simple state machine taking care of recognizing sysex messages and ignoring the rest.

We have a few other issues to take care of first. How does the bootloader enter the real program once it has written it to flash memory. Actually, it's very easy, we just have to jump to the adress 0x0000. In C, we can declare a function pointer to be 0x0000.

void (*jump_to_app)(void) = 0x0000;

There is a special trick regarding interrupts. The interrupt vector table is normally stored at the beginning of the flash memory. But that is the RWW memory, while our code is executing in NRWW. There is a flag in GICR to move the interrupt table to the bootloader section, and we set this flag at the beginning of our main function:

  /* move interrupts to bootloader section */
  GICR = _BV(IVCE) | _BV(IVSEL);

This means that after moving to the "real" application code, we'll have to restore the interrupts to their normal location (out of main.c of the controller):

  GICR = _BV(IVCE);
  GICR = 0;

Another issue is that we don't want to always reprogram our flash before executing our software. We want to execute the stored flash except when explicitly asking to reprogram (or when the firmware has become broken). At the start of our device, we check if a button is pressed. If it is, we stay in the bootloader, else, we call the main program. Additionally, we want to signal the device that it is waiting for a reflash while running our "main" application. We do this by writing a special byte into EEPROM and then calling the bootloader. Thus, at the beginning of the bootloader, we have the following code:

  if (eeprom_read_word(START_MAIN_APP_ADDR) == 1 && IS_BIT_SET8(PINB, PB4)) {
    jump_to_main_program();
  }

If the flag is set and the button is up, we jump to the main program. Else, we stay in the bootloader and wait for a new firmware over sysex. Repressing the button will start the main software anyway (the following contraption waits for the button to go up, then down again):

    if (!IS_BIT_SET8(PINB, PB4)) {
      button = 1;
    } else {
      if (button) {
	jump_to_main_program();
      }
    }

There is one last thing we need to take care off, and that is checking if the firmware is valid. The flashing device sends a final checksum byte over MIDI that is the checksum of the whole firmware in memory. This checksum is written to the eeprom, and checked before jumping to the main program. If the checksum is correct, the firmware is valid. Else, we stay in the bootloader and wait for a new firmware:

uint8_t check_firmware_checksum(void) {
  uint16_t len = eeprom_read_word(FIRMWARE_LENGTH_ADDR);
  uint16_t firm_checksum = eeprom_read_word(FIRMWARE_CHECKSUM_ADDR);
  uint16_t i;
  uint16_t checksum = 0;
  
  for (i = 0; i < len; i++) {
    checksum += pgm_read_byte(i);
  }

  if ((checksum & 0x3FFF) == firm_checksum)
    return 1;
  else
    return 0;
}

uint8_t jump_to_main_program(void) {
  if (check_firmware_checksum()) {
    jump_to_app();
    return 1;
  } else {
    return 0;
  }
}

As you can see, the code for checksumming is very straightforware, and we just use pgm_read_byte to read from the flash memory. The length of the firmware is also stored in the eeprom.

That's how my MIDI bootloader works! 🙂

posted by manuel at 1:58 pm  

3 Comments »

  1. That’s a very nice post – it seems you took care of all the problems I thought were associated with a bootloader. Enlightening. 🙂

    Keep up the posts on the AVR platform, every single one is a joy to read!

    Comment by Felix — August 18, 2008 @ 4:35 pm

  2. Indeed, I enjoyed your post also. It gives me a chance to daydream about having enough spare time to do stuff like this… 😉

    Can I ask how many is “a lot of MIDI devices”? Are they perhaps small runs for people you know, or do you sell them (or both)? What sort of things do you make?

    Cheers,

    John :^P

    Comment by John Pallister — August 19, 2008 @ 4:05 am

  3. Good stuff. Thanks and greetings!

    Comment by Cookbook — October 6, 2008 @ 7:57 pm

RSS feed for comments on this post. TrackBack URI

Leave a comment

Powered by WordPress