Please enable JavaScript.

Coggle requires JavaScript to display documents.

Comms Lost Problem - Coggle Diagram

- - - - FW changes
        
        New reconnection behavior introduce in Hudson release (v21.12 something something)
        
        Adds a timer and raises an error when the timer is reached
        
        Tried to increase the timer to 10s instead of 5s. Did not work
        
        Might try a 3 minutes.
        
        What happens if we disable the whole timer?
        
        We might lose the auto-reconnection after e-stop is pushed, but maybe that is not a big problem as it was the case before Hudson.
        
        If the print continues after a long gap in communication, we might be resuming the gcode while having lost teh reference position
        
        We could revert to an earlier version and see if the problem was happening back then to pinpoint the problem in time
        
        Larger scale code-review might be needed. I poke check the code where I taught might be related, but a larger review might help finding some problematic code.
      - Comms.py file hosting the error.
        
        Python script jamming
        
        Python script not having priority
        
        Heartbeat not properly being updated
        
        Heartbeat timeout too restrictive
      - Mobius service jamming up
      - Bad gcode management
        
        Print farm is saying prints fails around 8h-10h of print jobs. Can this be seen on pattern in the logs?
      - Controller firmware problem, hosting the other half of the communication
        
        Controller might be having a memory management problem, kind-of memory leak or something like this. I want to monitor this.
    - - BalenaOS and Supervisor (we have no control over this)
        
        A service or process might take up all CPU or memory, jamming the communication (which is a python script)
        
        Monitor the mobius container for services, process, CPU usge and memory/. As well as DMESG and JOURNALCTL
        
        Monitor the Host OS for services, process, CPU usage and memory. As well as DMESG and JOURNALCTL
        
        Could we increase the python script priority so it's not jammed?
    - - Tried to DDos the printers with no success. The network seems solid and the performances of the RPis are not affected
  - - - Are the RPi at the proper FW version?
      - Are the RPi at the proper HW revision?
      - Do RPi have the same CPU chip?
        
        All verified. The whole internal fleet is running the same HW.
      - RPi GPIO pins failing
        
        Confirmed by RPi that 5V on gpio pins is now bad on RPi4 (difference with RPi3)
        
        Controllino is sending 5V on the TX pin. This is high for the RPi. But why does the RX pin also fails? And it was also the case with the Azteeg.
      - RPi get throttled?
        
        Checked temperature, looks clean. throttling starts at 80, and hard throttle at 85. We are around 55-60.
        
        There are some low voltage pi that are throttled, but there doesn't seem to have a correlation with anything,.
      - Does comms loss only affect RPi4?
        
        One instance of lost comm on a RPi3, out of many, many on RPi4. The internal fleet only runs RPi4 though. Only 6-7 customers complained, all RPi4
      - RPi were hard to get and maybe we used a bad reseller that gave us bad Pi
        
        Some lemons were catch before getting into a machine. All the internal machines are running the same HW revision chips, so seem all good here. + Every internal machine had the problem at least once... we can't have only lemons
      - bad power supply making RPi reboot
        
        Currently being tested on H52023-02-10: Currently being tested on H5
      - SD Card corruption
        
        We can diagnose SD card with the following tool which will give the read/write speed of the SD card:
        bash dd if=/dev/zero of=./speedTestFile bs=20M count=5 oflag=direct
    - - Can the Controllino age and starts jamming?
        
        Check age of Controllinos / machines having the issue
      - bad power supply making Controllino reboot
        
        2023-02-10: Currently being tested on H5
    - - Hydro-Quebec power outtage
      - Building power supply/neightbors afftecting us, etc.
        
        All of our machines should be affected at the same time if this was the case. Badly situated customers should be complaining a lot more.
      - Astrological event like solar flare
- - - - Timing of the warning/errors happening
        
        Trends/Patterns from the mobius logs
      - Possibility to add more information as we want.
        
        I have added the pulses (which is the value from which we calculate the timeout) as well as controller messages (need to add gcode messages)