Out of memory error

Avatar
Jürgen Jürgenson
  • updated

Hey,

Asking for help or info.

Ive been getting this Out of memory error lately with 4.6.20 and 4.6.26 servers. 
First i noticed it with a old server that was still on 4.4.4 version but got updated without my knowledge. After the update it crashed every 3 days giving a out of memory errors in the log.


Today it happened again, when our 4.6.20 Bos server was acting strange - it wouldn't update button presses on a control panel - So i did a server reload from bos config to see if it would help. But after the reload everything was gone. It loaded up a new empty config all themes and devices deleted. 

With a error:

Devices : 27.01.2020 10:32:39 : Error : Server : Error loading file C:\Program Files (x86)\ComfortClick\bOS Server\Building\Configuration.bin. Exception of type System.OutOfMemoryException was thrown. at System.Collections.Generic.Dictionary`2.Resize(Int32 newSize, Boolean forceNewHashCodes)
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at System.Runtime.Serialization.SerializationInfo.AddValueInternal(String name, Object value, Type type)
at System.Runtime.Serialization.SerializationInfo.AddValue(String name, Object value, Type type)
at System.Runtime.Serialization.SerializationInfo.AddValue(String name, Object value)
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ParseMember(ParseRecord pr)
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Parse(ParseRecord pr)
at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadMemberPrimitiveUnTyped()
at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at BOSCommon.Helpers.Serialization.ReliableSerialization.Load(IServer server, String filename, BinaryFormatter bf)

Next reload wont help and the error is gone with the next reload as if it wouldn't even load the building folder.

Im really hoping that comfortclicks guys can restore my building folder because there was a lot of updates and no backups from today. 

Regards,

Guy with a headache 

Replies 8
Avatar
0
Jürgen Jürgenson
Quote from Ricardo Pinto

Damn...

How big are those projects? Are you using single server for massive tons of data?

I always think about what should be the limit of resources used on a project. There has to be a limit somewhere, not only depending on the hardware, but also on the .NET level.

My projects are usually smaller, and have a dedicated bOS for each one, so I have never had any problems with memory. But if I had to use bOS for some kind of serviced apartments, probably I would think about some kind of load-balanced scheme.

Even though, I'm very careful with some stuff, like not using transparency, etc. At least the projects feel lighter, and the menus are faster.

About your issues, I'm afraid that only bOS team can help with that.

Nevertheless, I think that bOS team should clarify what would be the feasible limit for each device (like, a Jigsaw obviously will have less power than a Grinder, and so on).

This kind of issues, led me to backup everytime I change stuff.

Best regards

Project 1 has 180 apartments running on one server, project 2 has 19 apartments also running on one server. Both crash tho the bigger building was OK for a long time so i was hoping that the beta fixed this outofmemory issue. Smaller server crashes every week, its unable to stop or restart the bOS service and only a full server restart helps.

But i managed to restore it after all. I did a full restart and then opened only bOS and started to restore it, i saw a memory bump of ~2gb when it was restoring it. This doesnt look much as the total usage was 4,2gb and when it finished it fell down to 2,5gb.
Server has total of 16gb ram... max ive seen it use was 13gb and most of it was ETS whos a huge memory hog. So maybe it was ETS's fault i dont know... On the other smaller bulding i only run bOS on it, not ETS. 

I would love to do it so every apartment has a separate server/jigsaw but there is always the money question. Everybody wants the best but are willing to pay for the worst. And i have little to say about that, im just a monkey who obeys orders.  

Avatar
0
Ricardo Pinto

Damn...

How big are those projects? Are you using single server for massive tons of data?

I always think about what should be the limit of resources used on a project. There has to be a limit somewhere, not only depending on the hardware, but also on the .NET level.

My projects are usually smaller, and have a dedicated bOS for each one, so I have never had any problems with memory. But if I had to use bOS for some kind of serviced apartments, probably I would think about some kind of load-balanced scheme.

Even though, I'm very careful with some stuff, like not using transparency, etc. At least the projects feel lighter, and the menus are faster.

About your issues, I'm afraid that only bOS team can help with that.

Nevertheless, I think that bOS team should clarify what would be the feasible limit for each device (like, a Jigsaw obviously will have less power than a Grinder, and so on).

This kind of issues, led me to backup everytime I change stuff.

Best regards

Avatar
0
Jürgen Jürgenson
Quote from Jürgen Jürgenson

So today the 4.6.30 version after a bOS reload managed to get that outofmemory error again and deleted all my work again. Last backup was done 3.04.20 so ive lost 2 weeks of work. 

Am i the only unlucky one? As i now have 2 buildings that acting the same way. After a week or so it either crashed and needed a server restart because stopping the service fails.
Other building is on 4.6.33 beta and it too crashes once a week.

Little stressed out as its a lot of work i need to redo.

Now i tried to restore the config. Made a restore from the old backup and then started to add the missing devices again and just before i finished i saw another outofmemory error. 


Devices : 17.04.2020 16:31:53 : Error : Server : Error saving file C:\Program Files (x86)\ComfortClick\bOS Server\Building\Configuration.bin. Exception of type System.OutOfMemoryException was thrown. at System.Runtime.Serialization.ObjectIDGenerator.Rehash()
at System.Runtime.Serialization.ObjectIDGenerator.GetId(Object obj, Boolean& firstTime)
at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.InternalGetId(Object obj, Boolean assignUniqueIdToValueType, Type type, Boolean& isNew)
at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Schedule(Object obj, Boolean assignUniqueIdToValueType, Type type, WriteObjectInfo objectInfo)
at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteMembers(NameInfo memberNameInfo, NameInfo memberTypeNameInfo, Object memberData, WriteObjectInfo objectInfo, NameInfo typeNameInfo, WriteObjectInfo memberObjectInfo)
at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteMemberSetup(WriteObjectInfo objectInfo, NameInfo memberNameInfo, NameInfo typeNameInfo, String memberName, Type memberType, Object memberData, WriteObjectInfo memberObjectInfo)
at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Write(WriteObjectInfo objectInfo, NameInfo memberNameInfo, NameInfo typeNameInfo, String memberNames, Type memberTypes, Object memberData, WriteObjectInfo memberObjectInfos)
at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Write(WriteObjectInfo objectInfo, NameInfo memberNameInfo, NameInfo typeNameInfo)
at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Serialize(Object graph, Header inHeaders, __BinaryWriter serWriter, Boolean fCheck)
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize(Stream serializationStream, Object graph, Header headers, Boolean fCheck)
at BOSCommon.Helpers.Serialization.ReliableSerialization.Save(IServer server, Object serObject, String destination)

I also made a backup before the reload AND that backup is now corrupted as it does not restore from it. I still get an empty config when the server loads in. So yeah, its a big problem to us right now.

Avatar
0
Jürgen Jürgenson

So today the 4.6.30 version after a bOS reload managed to get that outofmemory error again and deleted all my work again. Last backup was done 3.04.20 so ive lost 2 weeks of work. 

Am i the only unlucky one? As i now have 2 buildings that acting the same way. After a week or so it either crashed and needed a server restart because stopping the service fails.
Other building is on 4.6.33 beta and it too crashes once a week.

Little stressed out as its a lot of work i need to redo.

Avatar
0
Jürgen Jürgenson

Side note: I have two more building that are running fine on bOS 4.6.26 version, Both buildings have 16 apartments. Only heating control with KNX and ventilation control with Modbus. And water, heating and electricity meters witch values are collected and logged by bOS thru mbus to modbus gateway. 

And as Comfortclicks support told me today it seems that ive hit a bOS server limit with my projects(so it got that out of memory error). They are working on a fix. Smaller projects should work OK with 4.6.26.

Avatar
0
Ricardo Pinto

Ah, ok. I was intrigued about the update, because I also have some installations with 4.4.4, and I was afraid that it could happen.

I also had a similar problem with one local office demo, when updating to a newer version, I faced lot's of errors. Unfortunately I had to recover an old backup and redo some stuff.

Best of luck for the project recovery.

Best regards




Avatar
0
Jürgen Jürgenson
Quote from Ricardo Pinto


Hey mate, any news on that one?

Were you able to recover anything?

By the way, I'm seriously intrigued on the supposed "auto update" of a 4.4.4 installation. As far as I know, bOS never updates by itself.

Are your sure that no one messed around? I had serious problems with an update as well, had to recover a big part of the configuration.

Best regards

Hey, im not saying it was auto updated, it was updated just without my knowledge. Our intern was changing some color i think thats how it happened. Nobody is saying who it was and i was unable to get the ip who did it. 

About the recovery - full restore does not work, no matter how old the backup is. I was able to recover the project two times manually node by node importing them back, but the config still "crashed" and when it comes back up its an empty just like new install. I've made pictures and reports to comfortclick and ive opened a ticket. Comfortclick is working on a fix for my issues. As this config crashing is happening on two different buildings.

Avatar
0
Ricardo Pinto


Hey mate, any news on that one?

Were you able to recover anything?

By the way, I'm seriously intrigued on the supposed "auto update" of a 4.4.4 installation. As far as I know, bOS never updates by itself.

Are your sure that no one messed around? I had serious problems with an update as well, had to recover a big part of the configuration.

Best regards


Customer support portal powered by UserEcho