Noticed that my Smartermail IIS logs had been getting huge these days so went to investigate.
It seems to be something to do with the Bandwidth Counting Scheduled task getting stuck in an infinite loop.
When I run it, the SCP Server keeps hitting the url “/services/svcDomainAdmin.asmx” continuously over and over (we’re talking 3gb a day of logs just showing that!)
Sm 15.7 (not yet upgraded to 16)
@Ben – may be due to SM API as you’re running SM v12
I suggest taking a look through the SQL logs to see what is happening
I’m on SM 16 now 🙁
For those packages (Email Only) I don’t even have bandwidth logging enabled and it *still* freezes on them. I’m wondering if this is a separate issue to the original one.
Is there any way to run the Bandwidth cron as a CLI and get verbose output?
17 and same issue.
Would really be useful if we could get a CLI output – is it possible?
Can you please provide any SolidCP Auditlog and Windows Eventlog errors?
We do use the GetDomainStatistics API call from Smartermail so you could try run that directly for the domain to see if it errors (More details on the call at https://mail.smartertools.com/Services/svcDomainAdmin.asmx?op=AddDomain ).
Nothing in either of those logs 🙁 (the audit log is just the one under root right – Default.aspx?pid=AuditLog&UserID=1)
So there is no way to get a verbose log of what a task is actually doing?
It works for random packages, but not others – with nothing seemingly linking the working ones together.
OMG, finally, I have found the issue!
It’s due to the “BandwidthUpdated” field being NULL for a package in the dbo.Packages table.
So in the calculate bandwidth procedure, SolidCP will count up from the last updated date until today, hitting Smartermail each time to get the data.
If the field is NULL however (which is the default for new packages) – then 0000-01-01 will be used as the start date!?! So the counter goes into a loop thousands of times!!!
A quick fix to get the process running again for now is to run the following SQL command:
SET BandwidthUpdated = ‘2019-01-01’
WHERE BandwidthUpdated is null
But I assume that any new packages added will also have the same issue?
I can confirm new packages do have a NULL BandwidthUpdated value but if i run the bandwidth task this completes and the value is filled.
The domains you are seeing with NULL and causing an issue are these older domains? I’m wondering if part of calling the Smartermail API its returning alot of data and either Smartermail or SolidCP is failing with the call.
Could you try make a new hosting space with Smartmail to see if there is an issue with the report when it next runs and if the value is filled?
I’ll check that later with a new package.
However it doesn’t explain the original issue (which is the SM being hit in a loop).
I traced all of the calls being made, and as I wrote above, the issue is the SOAP request to svcDomainAdmin.asmx is passing 0000-00-01 as its first item (I assume that is NULL) and then going up from there, 0000-00-02, 0000-00-03, etc, etc – hence the seemingly endless loop.
The bandwidth task on our test environment takes 6minutes on first run when the value is null. This is reasonable and once i ran it a second time after confirming the value is filled it took around the same amount of time.
I wonder if this is the call on the Smartermail API which times out due to a large amount of data it may have to collected.
It would be interesting to see if this is the case and we can look at reporting this to Smartermail.
That still wouldn’t explain why the “since” value starts at 0000-00-00. (if you capture all the requests, you can see this in the SOAP call)
I was looking through the source code and I can’t find anything which actually checks the “since” date is valid.
6 minutes to go from 0000-00-01 up to 2019-03-12 seems about right.
- Views9222 times
- Answers27 answers