Locating A Memory Leak?

Support & Help Requests
1

I have 64Gb RAM on UM server, which isn't a lot these days, but it's still sufficient that I rarely find myself with memory problems (on the machine, that is. My own aging memory is a different story). But I do experience them.

Since it's a server, it runs 24x7x365 except when it needs a reboot due to a system upgrade. But 99.9% percent of the time I access it from a laptop over RDP. And I've noticed that after a period of days, my connections get slower, responsiveness get jerky, noticeable buffering of keyboard input occurs, and at times my connection seems frozen, as if I'm locked out.

So, where's the culprit? I suspect it's either a browser (I keep two open - Firefox and Brave), the xrpd program, network congestion, or something else. I'm less inclined to think of network issues, as my other devices and computers continue to run fine.

I run Webmin and a few tabs (like the one I'm using for this site) in Brave, and Firefox for my other web work. I'm a lazy sort, as quitting a browser after every time I use it seems like a waste of time, but I'm also aware that browsers can consume a lot of memory. Oddly enough, Webmin never notifies me that I'm running low on memory, even though it seems like that's often the case. (I stand corrected on this last comment; I just checked and Webmin reports that I'm using 80% of real memory - which is higher than I expected).

The reason I included xrdp is that sometimes I can clear up my connection difficulties by connecting via ssh and issuing sudo systemctl restart xrdp. I recently received an upgrade to xrdp and now I wonder if somehow a regression wasn't introduced?

What is the best way to isolate runaway memory? I use bashtop at times, but that introduces its own memory overhead (it's more informative thatn top, in my opinion, and friendlier to read than htop). And the two highest consumers are, of course, Brave and Firefox. Should I just close them periodically and hope their memory and cpu consumption get reclaimed by the system?

2

This is a really hard question/situation.

First, degradation of xrdp connection is not necessarily caused by memory leak on server. Frankly, I've stumbled similar ssh connection problems regularly and never managed to find the root cause despite having industrial monitoring system installed. The most probable reason guessed was over-swapping.

IMHO it takes as OS, as per process continuous monitoring to approach somewhat definite conclusions.

3 Likes
3

Try connecting directly to the server when your browsers are running slowly/erratically and if they work normally then it's likely "xrpd" causing the problem. Another option might be to run "x2goserver" instead of using RDP or "AnyDesk" if you don't mind using proprietary software.

1 Like
4

Well, all things point to xrdp as the cause. My less-than scientific testing has shown this:

  • Firefox was the biggest consumer of memory. So I shut it down.
  • Brave Browser was the next. So I shut it down.
  • Checking free -h, cat /proc/meminfo and ps aux --sort=-%mem reveals high memory consumption still.
  • ps aux | grep -v grep | grep xrdp | wc -l showed over 79,000 "sleeping" processs

So, I stopped xrdp via systemctl and then did a sudo killall xrdp. When I restarted xrdp the number of processes had dropped, but still registered over 100. I then started (and stopped) Firefox. The xrdp processes climbed to over 500 and continued up into the 700s. I'm running Brave to write this and now the xrdp processes have climbed to over 2,000. It doesn't really seem what processes I run on the box, xrdp seems to be eating all the memory. The classic definition of a memory leak.

My next step is to review the xrdp settings. I made some modifications based on some articles I read, but the program itself received an update recently. Sadly, when I terminate xrdp I lose connection to all the running programs I had and have to do a lot of cleanup the next time I connect.

It's stuff like this that I think frighten people away from Linux.

1 Like
5

Do you think some other solution could be more fortunate? E.g. some of VNC servers for Ubuntu are listed here VNC/Servers - Community Help Wiki

1 Like
6

I edited my /etc/xrdp/xrdp.ini file and changed fork=true to fork=false. This is a toggle that allows for multiple RDP sessions and causes the program to fork upon every new connection. Since I don't need multiple connections (I tried that a while back, to mixed results), I figured it wouldn't likely cause the situation to get worse. So far, it seems to have worked -- I have a robust RDP connection and I'm not seeing the thousands of processes in ps aux | grep xrdp. I'll let it run this way for a while, because memory leaks take a while to appear, and I've only been running it this way for two hours. But it feels pretty smooth right now!

5 Likes
7

This other discussion looks at the "port=" value to encourage XRDP to reuse piggy-back multiple follow-on sessions through a single pre-existing port; essentially limiting each remote user to consuming ONLY a single port!


This discussion references "MaxSessions" as a controlling parameter to limit the count limit on the number of sessions/processes spawned. Unless you expect every user to be able to open at least one session, and not knowing your user environment, you might consider limiting the count to the number of max real simultaneous users.

3 Likes
8

It's now been 5.5 hours since I made my change to the xrdp.ini file and I'm still not seeing any newly-spawned xrdp processes, and thing are running more smoothly now than they have in a while. I did get an SSL-related error, probably due to my fiddling around making myself a CA, but I now know how to generate the necessary files and where to put them, so I can address that when I have more time. At present, things are running swimmingly!

7 Likes
9

I might be late to help but have you tried running smem:

This should show you which processes are holding on to memory over time and therefore make you skip the guesswork of finding culprits

3 Likes
10

Thanks! I don't have smem on my server, but I may just add it so it can be part of my troubleshooting "toolbox."

2 Likes
11

Another great tool is hardinfo2.Been using this for a while now,though I'm just an end user on a single machine.

5 Likes
12

I had the old "hardinfo" but this one seems to have a little extra polish to it. :slight_smile:

Thank you, Norm.

3 Likes
13

Yeah, same here. I note that hardinfo2 replaces hardinfo (they can't coexist). It also places itself in the System Tools portion of Main Menu. Nice!

3 Likes
14

Reminder you can install things with --no-install-recommends. smem brings in a few python dependencies just for graphical output. If you don't need the graphical output, don't install them and it'll still run fine on command line!

3 Likes