Do I have to remove the derailleur batteries to charge them?

Yes, you have to remove the batteries from the derailleurs to charge them. 

See the SRAM RED eTap System Overview video for a tutorial on removal, installation, charging of the batteries, and the indicator LEDs.