Well, you don't ever have to. It's not like it's a requirement. However, all of those worms are available in very small sizes, so it's likely that your chameleon could handle them even now.
You can offer a variety of feeders from the first week of age. no waiting required.
You might find silkworms, superworms and butterworms easier to get in the right (smaller) sizes than hornworms.