Trusted Enclaves, a Foundation for Network Security, Fall to Physical Attacks

The point of the enclave is so that you can have a provider operate the hardware without access to any unencrypted data. That's why they go through the trouble of encrypting the data on the wires in the first place. So it wouldn't be a physical intrusion of the building or access to the machine as they are operators and should have it. What they shouldn't have is access to unencrypted data.