so I have 5 board that uses an ATmega 2560 that I designed. They are all wired properly and initially were able to communicate with the AtmelStudio studio using an AVRISP mk2. The first board was able to flash, have the lock bit set and the fuses set, after which it would still be able to communicate with the programmer. The second board was initally able to flash the program, and have the lock bit set, but after setting the fuses I got the error:
"Failed to enter programming mode. ispEnterProgMode: Error status received: Got 0xc0, expected 0x00 (Command has failed to execute on the tool)
Unable to enter programming mode. Verify device selection, interface settings, target power, security bit, and connections to the target device."
I was unable to even read the device signature. I thought maybe it was faulty chip, and since I had 3 other boards to work with I just ignored it. When programming the 3rd board, I went through the same procedure and the same error came up. But the 4th board worked when doing the same thing.
I'm still new to the AVR scene and would appreciate any help to get the broken 2 boards to work. I know its not something that's wrong with the crystal (16MHz), or the programmer, or even the ISP clock (125kHz). & its not something with the wiring. I tried erasing the faulty chips but an unable to do so and kept get the same error. Is there a way to make the chips reset to stock, or just to be able to reestablish communications with the chip.
The procedure was as follows: 1) flashed the chip 2) set lock bit to "0xCF" 3) set fuses to "EXTENDED 0xFD", "HIGH 0xD8", "LOW 0xFF" 4) flash chip again and receive error.
LOW fuse 0xFF means CKSEL3:0 bits are 0b1111. That means the low power crystal oscillator is selected (please refer 10.4 on page 40 of the datasheet).
Low power oscillator could be unstable when driving 16 MHz crystal, and unable to drive a ceramic resonator more than 10 MHz. It may be very sensitive to the schematic implementation and noise. Instead of it, it is better to use full-swing oscillator (low fuse byte 0xF7). Check schematic implementation, type of the resonator and capacitance on XTAL pins.
To restore ISP connectivity, you can unsolder the resonator, and apply about 1 MHz square wave on the XTAL1 pin (refer to 30.8 on page 339 of the datasheet).