Machine won't boot if the USB is plugged in, but I want to install a new OS from live USB?

I've found an old laptop and I want to install Elementary OS (a Linux distribution) on it. The laptop is Lenovo S10e with Windows XP on it.
It boots fine if there are no USB drives plugged in, but when I plug in a live USB to boot from it freezes at the first screen with Lenovo logo and does nothing no matter what key I press. Is there a way to fix this or at least some workaround. I should also note that it's a small laptop with no CD drive so bootable CD is not an option.
Things I've tried:
- Removing everything from boot order except for the USB drive
- Putting the HDD first, USB second and then trying to manually override boot device
- Removing everything from boot order except for the HDD and plugging in the USB to see if it makes any difference (still won't boot from HDD)

If it helps this is some info:
CPU: Intel Atom CPU N270 1.60GHz
Memory: 1GB
OS: Windows XP SP 3

The only thing I can guess is that however you set up the live/install iso on the USB is not quite correct to actually be bootable. What method did you use to put the iso on USB? Have you successfully booted the USB device on any other computer?

I have never used Elementary OS, but I think it is a stripped down version of Ubuntu (which I use). I was going to suggest Lubuntu which is a lightweight version or Ubuntu, but I wonder if the following from Lubuntu website has something to do with your USB boot not working (usb-creator is something in Ubuntu variations to write a bootable live/install iso to USB):

"Due to changes in syslinux, it is not currently possible to use usb-creator from 14.04 and earlier releases to write USB images for 15.04; we believe that it is also not possible to use usb-creator from a 15.04 system to write USB images for earlier releases. For now the workaround is to use a matching release of Ubuntu to write the images, but we intend to issue updates soon to work around this incompatibility. 1325801 (Lubuntu suggested workaround is to use OBI which is not affected by this problem)."

https://help.ubuntu.com/community/OBI

Try plugging the USB immediately after the screen where it freezes. A Windows XP computer probably doesn't have secure boot, but look in the BIOS settings and turn it off if it's there.

Do you have the required driver for that USB device?

Yes, that happens a lot. No sense arguing with the BIOS. You won't win. Try another USB device instead.

You will have to access the bios of the laptop and see if it is set to boot from the usb. Some older laptops can't boot from usb, but can boot from cd/dvd. If your laptop can boot from cd, you can either burn an.ISO file for the OS and install that way or burn an.ISO of "Plop boot loader" or another boot mechanism app. Your boot order is what you are looking for when you access the bios. If however the laptop won't boot, you will have to use another machine to help see what is wrong with the laptop. You can diagnose almost anything yourself.