6 Replies Latest reply on Mar 16, 2009 1:07 PM by Truffles

    503 service unavailable when connecting to the gsb




      Every X weeks I got a problem when I try to connect to the management gateway (https://address/gsb)

      I put my credential and after got the page 503 service unavailable.


      The only way I can do to have my MG responding is the do a hard reset.


      I had a look on the log but I don't know too much how I can handle this.

      I have the verbosity of the log messages set to Normal and now I just updated to Verbose. so I hope I will be able to know why my LDMG hang or crash time to time.


      Did you already face to this kind of issue ?

      How can I resolve it so I can avoid to drive every then to my Data Center to reset the appliance.


      Thanks for your help.


        • 1. Re: 503 service unavailable when connecting to the gsb
          mrspike SSMMVPGroup

          I had this same issue, it also kept me from adding systems and I strongly suspect it was not allowing off network systems to be patched, etc.


          I worked with my TAM on this, and the quick solution is to rebuild your gateway, will take maybe an hour.


          When rebuilding I found if I installed the opennssl-0.9.8i patch that I could no longer change the IP on the gateway (I rebuilt it on a different network), it looked like it changed, but if I went back to the network tab it was wrong.


          I rebuilt it again, without installing that patch and all is good now.


          My TAM talked with some folks in engineering and it looks like that patch is 'bad' and they are supposed to work on it.


          It may have been that patch that broke mine and your gateway to begin with.


          Also, I found that if I went into reports > systems logs I had not logs and if I went to reports > system test that I had one failure, dbo logger.





          • 2. Re: 503 service unavailable when connecting to the gsb
            Truffles SupportEmployee



            The openssl patch does cause a problem on the Gateway. For some reason during the patch install it modifies the group and permissions on the ETH folders so that the IP address etc cannot be changed there after. Support has this issue escalated to development to be fixed but the manual work around is easy and envolves changing these settings on the appropriate folders. Here are the details below:


            From within the /etc/sysconfig/network-devices/ directory:


            Fix the Group Ownership of the ifconfig folders with this command:
            sudo chgrp webedit ifconfig.*


            Fix the Permissions of the ifconfig folders with this command:
            sudo chmod 775 ifconfig.*
            However, this issue shouldn't cause the 503 error message that you're referencing. The Management Gateway is a proprietary build of Linux but it still performs like Linux and should rarely if ever need to be rebooted. If this problem persists then I would recommend going to the machine itself and see if you can access the console. There has been 1-2 cases that I've seen where the system basically runs out of memory especially on the /dev/sda3 share (which is the root and should never run out). Upon reboot it clears the memory and allows it to function for a period of time. If this is happening then I recommend contacting support. The cases that I've heard of were fixed by rebuilding a portion of MySQL on the machine as it appears to be corrupted. This may also explain why a complete rebuild also corrects the problem.
            I hope this helps
            1 of 1 people found this helpful
            • 3. Re: 503 service unavailable when connecting to the gsb



              The problem should come from the memory space on the /dev/sda3 share who runs out of memory.

              How could I rebuild or get MySQL get fixed ?


              Many thanks for your help

              I highly appreciated.


              • 4. Re: 503 service unavailable when connecting to the gsb
                Truffles SupportEmployee



                I assume from your last response that your "sda3" partition does appear to be filling up? If so let me know in this posting. I wasn't the one that rebuilt MySQL in the case that I mentioned so I'll need to research further if your issue appears to be the same.




                • 5. Re: 503 service unavailable when connecting to the gsb



                  Yes my "sda3" was full.

                  After applying the corrective action to fix the group ownership and fix the permission for the OpenSSL I rebooted the applicance and some free space was there.

                  The "sda3" partition is a 1.5G and after the restart I have 70% used on this partition.

                  What is stored on the "sda3" partition ?


                  Many thanks


                  • 6. Re: 503 service unavailable when connecting to the gsb
                    Truffles SupportEmployee



                    I apologize for not responding until now. The SDA3 partition is basically the root (/) partition in Linux. The issue that I saw involved MySQL and the log tables were corrupted. This resulted in MySQL failing to release inodes and resulted in the drive appearing to be full when it wasn't. (A test of creating a file on the partition worked) Thus a reboot clears the problem. The instructions that we went through to correct this has a chance of clearing the logs. In fact the final fix actually does clear the logs. It was basically a two step process to attempt to fix MySQL.


                    1- Attempt to repair and optimize MySQL. If this works then there shouldn't be a need for step 2 etc. In the case that I worked on the corruption was too severe.

                    2- Stop the MySQL daemon. Move the corrupted Syslog Tables and Index files. Restart the MySQL daemon.

                    3- Syslog will then need to be dropped and recreated


                    I don't have the exact instructions for performing these actions but I hope to have at least the first step soon. If you don't mind rebuilding the Gateway then it would be a quicker resolution at this time. A normal reimage on a Gateway Appliance device only takes a few minutes and then it will need to be reconfigured.