Ah, yes, if it tries to boot and finds the assigned interface missing then you will see problems. I would not normally expect a panic though.
An Ethernet connected external modem will give far better results pretty much every time.
Indeed, I've never tried it and only recently became aware you could connect directly like that. I could see how it might be possible though.