SG-2100 dropping LAN connections with EM7305 Cellular WAN

why

@stephenw10
Thanks for getting back to me so quick on a Sunday..

Q: What version of U-Boot BIOS is your SG-2100 running? (Just want to confirm I am on the right version)

Q: Is there a separate BIOS for SG-2100 or is it rolled into the "special" pfsense builds that customers need to request for Netgate hardware?

Was going to call your distributor tomorrow to see if I can switch out my SG unit for a new one and test further. Is that a pointless exercise given there is something being explored by devs?

Thanks for your support

Andy

stephenw10

There is only one uboot version that should ever have been on a prodution device as far as I know:

Vendor: U-Boot
Version: 2018.03-devel-1.2.0ROGUE2-01.00.00.02+
Release Date: Fri Feb 7 2020

There were some earlier versions on prototypes etc and may be updates at some point but there are not currently.
If there was an update it would likely be rolled into a pfSense update. If that was not practical it might be released as a package but that's just speculation.

Given that I was able to replicate it I would hold on swapping it out until we have some better idea of the cause.

Did you try running the EM7305 in the external enclosure? Did the switch still fail?

Steve

why

@stephenw10

Running EM7305 in external M.2/USB adapter connected to SG-2100 appears to be working without causing etherswitch to freeze. Well,... at least I am comfortably past 10 mins without it freezing which is good. Will update later this afternoon to confirm that its a workable temporary fix, but I would like an integrated solution for deployment.

Can you elaborate on work Devs are looking at and potential timeline for feedback on whether its likely to be fixed within 2 weeks?

Thanks Steve

stephenw10

I don't really have anything further at this point. I doubt there will be any updates here within two weeks. Even if the cause is found quickly a solution would be near impossible in that time frame.

Any further data points here can only help though.

Steve

why

@stephenw10

Ok thanks for clarification Steve. .... so looks like we are talking months for a fix then. Weeks to get confirmation of an issue/fix schedule and then months to roll out in an official update? Is 6 months to resolution a reasonable assumption?

Couple of weeks ago you posted the following ... " I tested it with an em7345, though that is a different connection type".

Can you expand on that comment please.

Does the EM7345 not use PPP as a connection?
If I was to get a EM7345 card is it likely I can achieve a working integrated solution with the SG-2100 that I have or is it likely to be impacted by the issue being investigated?

Thanks Andy

stephenw10

The EM7345 card appears as a USB Ethernet device with a separate AT port for connecting only. Potentially giving greater throughput.

Dec 7 20:31:08 	kernel 		ugen0.2: <Sierra Wireless Inc. Sierra Wireless EM7345 4G LTE> at usbus0
Dec 7 20:31:08 	kernel 		cdce0 on uhub0
Dec 7 20:31:08 	kernel 		cdce0: <Sierra Wireless EM7345 4G LTE NCM> on usbus0
Dec 7 20:31:08 	kernel 		umodem0 on uhub0
Dec 7 20:31:08 	kernel 		ue0: <USB Ethernet> on cdce0
Dec 7 20:31:08 	kernel 		ue0: Ethernet address: ff:ff:ff:ff:ff:ff
Dec 7 20:31:08 	kernel 		umodem0: <Sierra Wireless EM7345 4G LTE> on usbus0
Dec 7 20:31:08 	kernel 		umodem0: data interface 3, has no CM over data, has break

However I have no reason to think it would behave any differently with the switch.
The EM7305 did not show that behaviour initially either, it took some hours. So it's entirely possible the EM7345 would also have done given long enough. It isn't substantially different.
Similar power draw, similar emissions etc. You saw the same thing with an EM7455?

Steve

why

@stephenw10

OK Thanks.

With regards to the question in your last sentence, .... yes, the switch locked up both with EM7305 and EM7455. For me, it locks up within 10 mins with both modules.

stephenw10

Ok, good to know. I'll add that to our internal report.

Steve

stephenw10

Do you know what LTE Band(s) you were using when you saw this?

We have been unable to replicate this on one device but it may be using different LTE bands, we are investigating.

Steve

why

@stephenw10

Hi Steve Apologies for delay in responding.

Interesting comment. I am on B3 (with Three UK).

If you saw an issue (albeit it took longer than me) and we're on same freq/carrier, then it's an avenue to explore.

If US dev's are not seeing anything, could be because they are not waiting long enough, or because the issue is freq band dependent (long shot but worth exploring with Sierra Wireless).

Using AT!GSTATUS? command I can see that carrier aggregation is not set up. Was it set up with your EM7455 to get the rates you reported earlier in the thread?

Thanks Andy

stephenw10

Ah, same as me. Also Three in the UK. Though it's hard to tell it seems to move about a bit when the ppp link is not up and you can't interrogate it when it is.

I never set up anything additionally so if aggregation is nor enabled by default I wouldn't have been using it.
I'm not sure which speeds you're referring to but they would have been with the 7305 almost certainly.

It's interesting when I first tested this it took some hours to trigger the switch failure. I reset it and it triggered again overnight. However now it has been up for days and seems OK.
It's definitely on the edge of whatever it is.

Steve

why

@stephenw10

I think CA is allocated by tower based on bands you allow on the client. I believe that if your modem is set to band group 09 (LTE ALL), then the tower cell should enable carrier aggregation based on other metrics that are dynamic and ISP dependent. (...... but would welcome being corrected if this is not correct).

with AT!GSTATUS? command I see that my LTE CA state is "NOT ASSIGNED" I also see a bunch of signal quality and power measurements which might be useful to share.

I suspect mine are worse that yours and would be interested to note what you see.

RSRP = -118 dBm [ (-44dBm = good) <=> (-140dBm=Bad)]
RSRQ = -14.5 dB [ (-3dB=good) <=> (-19.5dB=Bad)]
SINR = 2 dB

RSRP = average received power from ref signal
RSRQ = indication of received signal quality
SINR = Signal to Noise Ratio

Overall I have felt that I am on outer edge of cell coverage and it is reflected in signal metrics above (UL/DL speeds and link metrics leaning to "Bad" end of scale). Given your higher data rates from EM7305 tests, I would be interested to know what your power figures are when seeing performance similar to what you reported in thread above ( 21.5Mbps DL / 26.3Mbps UL).

If your wireless performance is much better than mine reported above, it hints that "maybe" there is some correlation between quality of wireless link and stability of SoC firmware controlling embedded switch. ..... yeah a stretch I know, but thats where I am at the moment as I cannot use SG2100 with an integrated modem like I bought it for.

Thoughts?

Is it worth trying to switch it out for a 3100 to test it?

stephenw10

I'm in London so signal strength is usually good. I've made no real effort to position it for better signal and it's surrounded by other firewalls etc. I see (last week):

!GSTATUS: 
Current Time:  1550848		Temperature: 44
Bootup Time:   0		Mode:        ONLINE         
System mode:   LTE        	PS state:    Attached     
LTE band:      B3     		LTE bw:      15 MHz  
LTE Rx chan:   1392		LTE Tx chan: 19392
EMM state:     Registered     	Normal Service 
RRC state:     RRC Connected  
IMS reg state: No Srv  		
RSSI (dBm):    -71		Tx Power:    0
RSRP (dBm):    -95		TAC:         048C (1164)
RSRQ (dB):     -7		Cell ID:     000E1F02 (925442)
SINR (dB):     14.2

That's the EM7305 in an SG-2100 with random antennas!

I don't see anything listing CA state.

Steve