WG x750e - automatic speed adjustment: mbmon going crazy

bigramon

I'm not too sure why you set register 0x295 to 0x50 once 0x296 has been properly set to 0x01 ?

Nevertheless, I could confirm that executing these four commands returned the same value than mbmon, both when CPU is idle and supercharged with burnP6.
I restarted my automated script and will patiently wait for it to go crazy again :P

Thanks for your detailed explanation and time!

Damien

stephenw10

@bigramon:

I'm not too sure why you set register 0x295 to 0x50 once 0x296 has been properly set to 0x01 ?

Because in order to read the actual temperature value, which is at register 50 on Bank1, you need to point the index register, 0x295, at it. It was previously still pointed to 0x4e, the bank select register.

The first time I read through the data sheet and tried to comprehend how this all worked my mind nearly melted. :D It still took a while the second time around. One thing to be aware of is that in order to save processing time/instructions many of the registers automatically increment the index register. This helps if you want to read in a series of values, say 0x20 to 0x2f, you just set the index to 0x20 and then read the data register. That automatically increments the index by 1 so now you just read the data register again and it will be the value of 0x21 etc. This means that if you set the Bank to 1, set the index to 0x50 and read the value at the data register you cannot just read the data again to see it has changed, the index register will now be pointing at 0x51. You have to set the index to 0x50 every time.

It's interesting that mbmon expects to find the bank select register at 0 and when it's not it has a hard time, perhaps a clue? Something else intereacting with the SuperIO chip is leaving it in a state that confuses mbmon perhaps.

As an aside I notice your values for sensor 3 are different from mine. is 67C a common reading? My own shows mostly 127 with occasional 0 and even more occasional 2.0. Have you changed the CPU? Interestingly reconfiguring it as a thermistor instead of a thermal diode produced much more reasonable values though still clearly not real temperatures.

Steve

bigramon

Ah, it's (somewhat) clearer now :)

I did not have a look at the mbmon code yet so I do not know why it behaves like this.
I can't wait though to see whether your trick with the registers will still work when mbmon doesn't anymore !

I indeed invested ~~massively~~ ($10 including shipping) to replace the original Celeron 1.3GHz:
Intel(R) Pentium(R) M processor 2.00GHz
Current: 225 MHz, Max: 1500 MHz

More horsepower and less electricity/heat ;D

Damien

bigramon

I added some detailed logging to understand what is going one and after 11 days, mbmon was only sporadically returning incoherent values (around once a day whereas measures are made every 10 seconds).
Maybe the multiple register manipulations helped ???

At this stage, I'd say the automatic fan speed adjustment is working !

Thanks Steve for your input ;D

Damien

stephenw10

@bigramon:

Maybe the multiple register manipulations helped ???

Plausible but sort of disappointing. Nothing to say it won't happen again or start doing something different.

Steve

bigramon

Well, time will tell. At some point, I will reboot the x750e and with the level of logging I have in place now, it won't escape me.

Damien

stephenw10

I added the temperature reading to WGXepc so you can try that if mbmon starts acting up again. Though you'd have to parse it's output for the value in your script somehow.
http://forum.pfsense.org/index.php/topic,32013.msg346092.html#msg346092

Steve

bigramon

This time, mbmon started producing bogus data (CPU temp around 9°C) after 24 days of uptime.
Unfortunately, the new version of WGXepc provides the same measure.

Let me know if you want me to try something else, otherwise, I'll just reboot the machine.

Damien

kejianshi

So, is auto fan speed control daemon showing up in the next release as an option click-to-enable?

stephenw10

This script is specifically for the Watchguard firebox X-e boxes so it's very unlikely to be in a pfSense realease. Even if it were more generic it probably wouldn't ever be included as more than a package. Any script that can control the fan speeds in a box has the potential to cause damage by overheating the CPU. Usually in something with thermal fan control, like a laptop, the control is handled directly by the SuperIO chip such that it will continue to cool correctly even if the OS crashes. Unfortunately the superio chip in the X-e box doesn't support this.

@bigramon
So reading the temperature with WGXepc gives the same value as mbmon. That implies the SuperIO chip is actually reporting the wrong value. Why could that be? It could be the temperature offset register has been set some how or that it's reading the wrong register for some reason. We could try investigating that but it will be quite involved. You could try using WGXepc in your script instead. Since it only reads one register (where as mbmon reads all registers everytime it's run) it may make a difference.

Steve

bigramon

Unfortunately, it did not work much better. After 2 weeks, I started getting temperatures of 255°C.
When I tried mbmon from the command line, it reported an error stating it could not access the hardware.
Everything went back to normal after a reboot.

Damien

stephenw10

Ah, so what were you using in the script that didn't work? I forget where we left off. ::)

Steve

bigramon

Something gets wrong after a short while:

WGXepc does not properly set the fan speed anymore
WGXepc does not report the proper temperature anymore
mbmon does not execute properly anymore.


$ /usr/local/bin/WGXepc -f 50
Found Firebox X-E
Fanspeed set to 50

$ /usr/local/bin/WGXepc -f
Found Firebox X-E
Fanspeed is ff

$ /usr/local/bin/WGXepc -t
Found Firebox X-E
SuperIO sensor 2 reads:
255

$ /usr/local/bin/mbmon -I -i -c1 -T2
No ISA-IO HWM available!!
InitMBInfo: Unknown error: 0

At this stage, I guess the best option would be to modify the WGXepc source code to allow it to keep executing in the background and set the fan speed depending on the temperature that is read.

Damien

stephenw10

Ok, but was your script using mbmon or WGXepc to read the temperature? The difference between them is that mbmon reads reads a whole load of values every time it's run even if you only need one. To get all those values requires setting the SuperIO chip in various modes. It's possible that under certain conditions mbmon leaves the superio chip in some error state. It may be possible to determine what the error state is and recover from it or to avoid it in the first place.

Steve

bigramon

My script was using only WGXepc.

Damien

stephenw10

Damn!
Well the only thing to do then is try to find out why the SuperIO chip is no longer responding usefully. Quite how to do that though…. ::)

Steve

sthames42

Did you guys ever resolve this? I have exactly the same problem. Running mbmon returns reasonable temps for a while and suddenly starts reporting this:

Temp.= 255.0, 0.0, 0.0; Rot.= 0, 0, 0
Vcore = 4.08, 4.08; Volt. = 4.08, 6.85, 15.50, 6.07, 5.11

When I cancel mbmon and run it again, I get:

ioctl(smb0:open): No such file or directory
No Hardware Monitor found!!
InitMBInfo: Bad file descriptor

WGXepc -t reports good temps and then nothing but 255.
Only way to fix this is to reboot.
I'm assuming the SuperIO chip got itself hosed somehow. Is there any way to reset or reboot the chip?

This is on two X-Core 550e boxes and the default SL6N7 Banias chips have been replaced with SL7EP Dothan chips according to https://doc.pfsense.org/index.php/PfSense_on_Watchguard_Firebox#Further_Enhancements_3.
All dip switches set correctly. No other changes (powerd/speedstep) were made.

stephenw10

No, or at least I haven't heard from anyone who did. However reading back through the data sheet there appear to be a number of possible things we could try. Since the chip is just giving results that are registers full of all 1s we don't know if it's actually returning anything or if we're even talking to it properly. Though it would seem likely we are because under mbmon the voltage readings continue to come back as reasonable numbers.
As a rather extreme option it looks like there is a register that can re-initialise the chip, back to it's power on defaults. However I've no way of knowing what registers are configured by the BIOS at boot so the results could be…. unpredictable. ;)

Steve

sthames42

Thanks for responding, Steve. I posted a new topic because this one was old and I wasn't sure it was still active. I'm happy to stay in this one.

This has happened on two boxes but I have another that works fine. I suppose it's possible that two of the Dothan chips I put in these boxes are creating this problem but it seem unlikely it would be two. I'm going to replace the replacement in one of them when another chip and wait for results.

Is the data sheet you spoke of available electronically? Where can I get it?

stephenw10

Yes, it's available in many places such as here.

Interacting with the chip manually for test purposes is a PITA. ::) It involves writing many individual registers to read one value. For example:
https://forum.pfsense.org/index.php?topic=43574.msg261279#msg261279
The reset register will not require so much though. It would be interesting to install superiotool to see what has stopped and what is still readable when the chip enters it's uncooperative state. We may get a clue.

Steve