1 Topic by rleyden (edited by rleyden 2010-01-26 08:42:43)

  • Registered: 2009-12-02
  • Posts: 34

Topic: Chumby webcam server up- Memory leaks?

I have my Chumby 1 set up as a webcam server.  It's surprisingly responsive on my local network.  I only have 56k DSL service so it appears slowly when accessed over the internet.  Please take a look.
http://ahwahnee.dyndns.org:81/

I'm using the webcam as a security cam, so don't expect much activity.  Clicking on the archive link will give a list of captured images sorted by date.  Clicking on the file name will bring up the image.  I found the perl script, list-pics.pl, to run the archive page on the net.(http://www.inwap.com/mybin/list-pics.html).  It was nice that it ran without modification on the Chumby although I found that I had to put it in lighty's cgi-bin directory.  The author notes, "It is possible to use this program on a web server that only allows CGI scripts to be executed out of the /cgi-bin/ directory. Such usage is deprecated, and not documented here."  (I love the way linux people talk.).

Also I was pretty happy that fbwrite and imgtool could put dates on the jpg images after reading all the compaints about how small the print is using fbwrite.  It looks fine when viewed on a web browser.

Ok, enough good news.  The Chumby stops capturing images after a while (30 min to a couple hours), apparently, unable to allocate memory.  I tried being more gentle, slowing the capture rate and decreasing the image resolution.   This seemed to make little or no difference.  I also tried switching from uvvcapture to ffmpeg also with no difference. 
After I added 126 M of swap memory, the time until capturing stopped only increased a little bit.

This seems strange to me because the memory usage doesn't seem to change much while my script is running.  The script captures 3 images a minute, displays one image, fbwrites the date, the captures the images back to a file.  Here's the memory status after running a the script a few minutes.

chumby:/psp# free
              total         used         free       shared      buffers
  Mem:        59780        33296        26484            0         2424
 Swap:       262136            0       262136
Total:       321916        33296       288620

After running 30 minutes and capturing failed (script still running).

chumby:/psp# free
              total         used         free       shared      buffers
  Mem:        59780        37920        21860            0         2900
 Swap:       262136           32       262104
Total:       321916        37952       283964

The used memory increases by 5 mb but only a tiny amount of swap is being used.

Here is part of dmesg output  (the entire output is on the web archive as dmesg.txt).

.....
[ 8572.260000] ffmpeg: page allocation failure. order:5, mode:0x0
[ 8572.270000] [<c03365a8>] (dump_stack+0x0/0x14) from [<c007bc34>] (__alloc_pages_internal+0x39c/0x3bc)
[ 8572.280000] [<c007b898>] (__alloc_pages_internal+0x0/0x3bc) from [<c002e400>] (__dma_alloc+0x160/0x414)
[ 8572.290000] [<c002e2a0>] (__dma_alloc+0x0/0x414) from [<c002e740>] (dma_alloc_coherent+0x58/0x64)
[ 8572.300000] [<c002e6e8>] (dma_alloc_coherent+0x0/0x64) from [<bf046ba0>] (hcd_buffer_alloc+0x98/0xa0 [usbcore])
[ 8572.310000]  r7:0001e000 r6:c395a050 r5:00000001 r4:000000d0
[ 8572.320000] [<bf046b08>] (hcd_buffer_alloc+0x0/0xa0 [usbcore]) from [<bf03b400>] (usb_buffer_alloc+0x24/0x30 [usbcore])
[ 8572.330000]  r5:00000001 r4:c395ae68
[ 8572.330000] [<bf03b3dc>] (usb_buffer_alloc+0x0/0x30 [usbcore]) from [<bf131af4>] (uvc_alloc_urb_buffers+0x50/0x88 [uvcvideo])
[ 8572.340000] [<bf131aa4>] (uvc_alloc_urb_buffers+0x0/0x88 [uvcvideo]) from [<bf131c80>] (uvc_init_video+0x154/0x378 [uvcvideo])
[ 8572.350000]  r7:c395a050 r6:00000028 r5:00000c00 r4:00000000
[ 8572.360000] [<bf131b2c>] (uvc_init_video+0x0/0x378 [uvcvideo]) from [<bf131f30>] (uvc_video_enable+0x8c/0x90 [uvcvideo])
[ 8572.370000] [<bf131ea4>] (uvc_video_enable+0x0/0x90 [uvcvideo]) from [<bf1314e0>] (__uvc_v4l2_do_ioctl+0xcd0/0xf2c [uvcvideo])
[ 8572.380000]  r5:40045612 r4:00000001
[ 8572.390000] [<bf130810>] (__uvc_v4l2_do_ioctl+0x0/0xf2c [uvcvideo]) from [<bf131758>] (uvc_v4l2_do_ioctl+0x1c/0x20 [uvcvideo])
[ 8572.400000] [<bf13173c>] (uvc_v4l2_do_ioctl+0x0/0x20 [uvcvideo]) from [<bf0c8b64>] (video_usercopy+0x1a8/0x2b0 [videodev])
[ 8572.410000] [<bf0c89bc>] (video_usercopy+0x0/0x2b0 [videodev]) from [<bf1304e8>] (uvc_v4l2_ioctl+0x54/0x68 [uvcvideo])
[ 8572.420000] [<bf130494>] (uvc_v4l2_ioctl+0x0/0x68 [uvcvideo]) from [<c00ae9f0>] (vfs_ioctl+0x78/0x94)
[ 8572.430000]  r7:00000003 r6:40045612 r5:bef6882c r4:c24fa280
[ 8572.440000] [<c00ae978>] (vfs_ioctl+0x0/0x94) from [<c00aefa8>] (do_vfs_ioctl+0x4a0/0x4f0)
[ 8572.450000]  r7:00000003 r6:40045612 r5:c24fa280 r4:00000003
[ 8572.450000] [<c00aeb08>] (do_vfs_ioctl+0x0/0x4f0) from [<c00af050>] (sys_ioctl+0x58/0x7c)
[ 8572.460000] [<c00aeff8>] (sys_ioctl+0x0/0x7c) from [<c0027e40>] (ret_fast_syscall+0x0/0x2c)
[ 8572.470000]  r8:c0027fe8 r7:00000036 r6:bef68784 r5:0087e320 r4:00000020
[ 8572.480000] Mem-info:
[ 8572.480000] Normal per-cpu:
[ 8572.480000] CPU    0: hi:   18, btch:   3 usd:  17
[ 8572.490000] Active_anon:207 active_file:2177 inactive_anon:784
[ 8572.490000]  inactive_file:2665 unevictable:395 dirty:5 writeback:0 unstable:0
[ 8572.490000]  free:794 slab:1420 mapped:6024 pagetables:97 bounce:0
[ 8572.510000] Normal free:3176kB min:1016kB low:1268kB high:1524kB active_anon:828kB inactive_anon:3136kB active_file:8708kB inactive_file:10660kB unevictable:1580kB present:65024kB pages_scanned:0 all_unreclaimable? no
[ 8572.520000] lowmem_reserve[]: 0 0
[ 8572.530000] Normal: 214*4kB 74*8kB 26*16kB 17*32kB 10*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3176kB
[ 8572.540000] 5410 total pagecache pages
[ 8572.540000] 50 pages in swap cache
[ 8572.550000] Swap cache stats: add 52, delete 2, find 2/2
[ 8572.550000] Free swap  = 261936kB
[ 8572.550000] Total swap = 262136kB
[ 8572.560000] 16384 pages of RAM
[ 8572.570000] 932 free pages
[ 8572.570000] 2353 reserved pages
[ 8572.570000] 1027 slab pages
[ 8572.570000] 10587 pages shared
[ 8572.580000] 50 pages swap cached
[ 8572.580000] fsl-ehci fsl-ehci: dma_pool_free buffer-32, ffd72000/40a80000 (bad dma)
[ 8582.970000] fsl-ehci fsl-ehci: dma_pool_free buffer-32, ffd92000/436e0000 (bad dma)
[ 8582.980000] fsl-ehci fsl-ehci: dma_pool_free buffer-32, ffdb0000/437c0000 (bad dma)
[ 8582.980000] fsl-ehci fsl-ehci: dma_pool_free buffer-32, ffdce000/43660000 (bad dma)

I've changed the script to reboot every hour at :59 and not much functionality is lost.  I imagine this approach to programing is "depreciated" by the linux experts.

  • Duane
  • Duane
  • Blue Octy
  • Offline
  • From: San Diego, CA
  • Registered: 2006-08-24
  • Posts: 8,197

Re: Chumby webcam server up- Memory leaks?

Where are you saving the images to (ie /tmp, /mnt/usb, etc)?

Duane's Website

  • From: Austin, TX
  • Registered: 2009-05-07
  • Posts: 471

Re: Chumby webcam server up- Memory leaks?

I'm wondering about the mapping of the space for the webcam.  It's possible that since that has to be physical memory addresses, there's some missing unmap or some fragmentation happening that keeps system memory allocated.  It's failing when allocating pages for DMA from the USB controller to the frame buffer which is needed to pull out the image data.

unwiredben's Website

  • Registered: 2009-12-02
  • Posts: 34

Re: Chumby webcam server up- Memory leaks?

Duane wrote:

Where are you saving the images to (ie /tmp, /mnt/usb, etc)?

Right now I'm saving the images to /mnt/storage/lighty/cgi-bin.  I initially had everything on a 16 Gb flash drive (/mnt/usb) but   some of the errors I was getting seemed to indicate problems writing or reading that drive.  Probably this was just a symptom of low memory.  I thought putting everything within the Chumby would make debugging easier.  I still have a small flash drive with debugchumby starting lighty in /mnt/storage.

In the long run, I'm worried about flash memory wear out so I have a powered USB harddrive waiting in the wings.  Alternatively, I could archive images on a remote server.

  • Duane
  • Duane
  • Blue Octy
  • Offline
  • From: San Diego, CA
  • Registered: 2006-08-24
  • Posts: 8,197

Re: Chumby webcam server up- Memory leaks?

When it runs out of memory, what do you get when you do a "du -s /tmp" command?

Duane's Website

  • Registered: 2009-12-02
  • Posts: 34

Re: Chumby webcam server up- Memory leaks?

Duane wrote:

When it runs out of memory, what do you get when you do a "du -s /tmp" command?

Conveniently, it's out of memory right now.  I forgot to uncomment by reboot command this morning.

chumby:~# du -s /tmp
616     /tmp
  • Duane
  • Duane
  • Blue Octy
  • Offline
  • From: San Diego, CA
  • Registered: 2006-08-24
  • Posts: 8,197

Re: Chumby webcam server up- Memory leaks?

Hmmm, that seems OK...

What's the output of "ps"?

Duane's Website

  • Registered: 2009-12-02
  • Posts: 34

Re: Chumby webcam server up- Memory leaks?

Duane wrote:

Hmmm, that seems OK...

What's the output of "ps"?

In case it's not obvious, psp/ff.sh is my script.  I'm not sure why it's shown twice.  It was called by cron.  I also copied ff.sh in case you would like that too.  It reflects a messy history of approaches and debugging.

chumby:/tmp# ps
  PID  Uid        RSS    VSZ Stat Command
    1 root        528   2908 SW  init
    2 root                   SW< [kthreadd]
    3 root                   SW< [ksoftirqd/0]
    4 root                   SW< [events/0]
    5 root                   SW< [khelper]
   10 root                   SW< [suspend]
  141 root                   SW< [kblockd/0]
  152 root                   SW< [kseriod]
  156 root                   SW< [kmmcd]
  185 root                   SW  [pdflush]
  186 root                   SW  [pdflush]
  187 root                   SW< [kswapd0]
  230 root                   SW< [aio/0]
  248 root                   SW< [unionfs_siod/0]
  249 root                   SW< [nfsiod]
  255 root                   SW< [cifsoplockd]
  256 root                   SW< [cifsdnotifyd]
  357 root                   DW< [freeze-thaw]
  360 root                   SW< [kondemand/0]
  373 root                   SW< [rpciod/0]
  378 root                   SW< [mmcqd]
  387 root                   SW< [kjournald]
  397 root        264   2908 SW  init
  422 root       1580   1612 SW  watchdog
  425 root                   SW< [kjournald]
  437 root                   SW< [kjournald]
  445 root        560   1944 SW< udevd --daemon
  928 root                   SW< [ksuspend_usbd]
  939 root                   SW< [khubd]
 1099 root                   SW< [scsi_eh_0]
 1100 root                   SW< [usb-storage]
 1154 root                   SW< [phy1]
 1202 root        676   3108 SW  /sbin/syslogd -C200
 1309 root       1128  12408 SW  chumbradiod
 1370 root        380   1544 SW  acceld
 1372 root        284   1588 SW  /usr/bin/cpid -d
 1408 root        880   3276 SW  /sbin/sshd
 1442 root        788   2768 SW  wpa_supplicant -iwlan0 -Dwext -c/tmp/wpa.wlan0.conf -C/var/run/wpa_supplicant -B -W
 1444 root        356   1584 SW  wpa_cli -B -p/var/run/wpa_supplicant -a/usr/chumby/scripts/start_network
 1552 root        724   3228 SW  /usr/sbin/crond -c /etc/cron/crontabs -S
 1564 root       1224  21024 SW  chumbhowld
 1590 root       1060  11496 SW  mDNSResponder
 1606 root        924   2416 SW  /mnt/storage/lighty/sbin/lighttpd -f /mnt/storage/lighty/lighttpd.conf
 1642 root        652   2916 SW  mDNSPublish piggy _http._tcp 80
 1646 root        392   3492 SW< /bin/chumbalarmd
 5322 root        548   2908 SW  /bin/sh -c /psp/usr/ff.sh
 5324 root       1052   2224 SW  /bin/bash /psp/usr/ff.sh
 7319 root       1760   6104 RW  sshd: root@pts/0
 7333 root        856   3232 SW  -ash
 9584 root        384   2776 SW  sleep 15
 9604 root                   Z   [watchdog]
 9605 root        772   3232 RW  ps

ff.sh

chumby:/psp/usr# cat ff.sh
#!/bin/bash
. /etc/profile
/usr/chumby/scripts/stop_control_panel
/psp/makeswap
for ((i=100;i<=155;i+=1)); do
/psp/ffmpeg -f video4linux2 -s vga -vframes 1 -i /dev/video0   -f image2 -vcodec mjpeg /mnt/storage/lighty/cgi-bin/"$(date +"%H-%M").jpg"
sleep 10
/psp/ffmpeg -f video4linux2 -s vga -vframes 1 -i /dev/video0   -f image2 -vcodec mjpeg /mnt/storage/lighty/html/cam1.jpg
sleep 10
/psp/ffmpeg -f video4linux2 -s qvga -vframes 1 -i /dev/video0   -f image2 -vcodec mjpeg /mnt/storage/lighty/html/c320.jpg
sleep 10
imgtool --mode=draw /mnt/storage/lighty/html/c320.jpg
fbwrite --pos=20,20 --color=255,255,255 $(date +"%b_%d_%H:%M:%S")
imgtool --mode=cap /mnt/storage/lighty/html/cam1t.jpg
sleep 15
echo $i , $(date) > /mnt/storage/lighty/cgi-bin/loop_count.txt
done
imgtool --mode=draw /psp/usr/clock.jpg
#/usr/chumby/scripts/start_control_panel
  • From: Austin, TX
  • Registered: 2009-05-07
  • Posts: 471

Re: Chumby webcam server up- Memory leaks?

Any idea why chumbhowld is using so much memory?   I just checked /proc/NNN/status for it on my C1 and got this output (edited for just the memory stuff)

Name:   chumbhowld
State:  S (sleeping)
VmPeak:    21024 kB
VmSize:    21024 kB
VmLck:         0 kB
VmHWM:      1280 kB
VmRSS:      1280 kB
VmData:    17668 kB
VmStk:        84 kB
VmExe:        24 kB
VmLib:      3000 kB
VmPTE:        14 kB

It's possible that much of that data is allocated but not used.  Looking at /proc/NNN/maps, there are two 8MB r/w regions mapped outside of the heap/stack that seems to be the bulk of usage, so maybe this is a phantom problem.

As far as I can tell, howl and the mDNSResponder are useful for finding the Chumby on your local network without needing to know it's IP address, but nothing is actually using that capability right now.

unwiredben's Website

  • Registered: 2009-12-02
  • Posts: 34

Re: Chumby webcam server up- Memory leaks?

unwiredben wrote:

Any idea why chumbhowld is using so much memory?  .

Good idea.  I killed chumbhowld, the webpage still loaded.  Capturing seemed to continue a little longer but eventually the memory still ran out.

I just noticed that if I repeatedly enter "free" I can catch the memory state where ffmpeg is loaded.  (there must be a better way).  I can watch the free memory dribble away 

chumby:/psp/usr# free
              total         used         free       shared      buffers
  Mem:        59780        54064         5716            0         3756
 Swap:       262136            0       262136
Total:       321916        54064       267852
chumby:/psp/usr# free
              total         used         free       shared      buffers
  Mem:        59780        55664         4116            0         3908
 Swap:       262136            0       262136
Total:       321916        55664       266252

1:10
chumby:/psp/usr# free
              total         used         free       shared      buffers
  Mem:        59780        56720         3060            0         4136
 Swap:       262136            0       262136
Total:       321916        56720       265196

1:12
chumby:/psp/usr# free
              total         used         free       shared      buffers
  Mem:        59780        58232         1548            0         4256
 Swap:       262136            0       262136
Total:       321916        58232       263684
1:15
chumby:/psp/usr# free
              total         used         free       shared      buffers
  Mem:        59780        37192        22588            0         4436
 Swap:       262136          104       262032
Total:       321916        37296       284620

Capturing stopped!

I wonder if I called ffmpeg in a continuous capture mode whether it would stay loaded and avoid the memory leak.  I'll give it a try.

  • Registered: 2009-12-02
  • Posts: 34

Re: Chumby webcam server up- Memory leaks?

I rewrote my script with uvccapture in the continuous capture mode (-t) and copied files instead of recapturing them to reduce captures to 1/min.  I could more easily see what was happening with the memory because it wasn't jumping around.
At first, it looked like the same story, the memory was smoothly leaking away at 75 kb/min.  The capture rate is about 1 image/min with a daytime file size of about 15kb,  The free memory went from 25652 to about 2000 over 2.5 hours and then stopped going down.  After 18 hours, 

chumby:~# free
              total         used         free       shared      buffers
  Mem:        59780        57964         1816            0        25976
 Swap:       262136          296       261840
Total:       321916        58260       263656

Buffers increased from an initial 4056 and swap useage is increaing very slowly.  This looks unstable to me but only having to reboot once a day is helpful.

In my largely uninformed opinion, this memory leak is outside the direct control of Chumby Inc.  Doing a Google search on "Linux 2.6.16 v4l uvcvideo memory leak" gives more than 3000 hits.
Here's one just for flavor 

o Herton Ronaldo Krzesinski 
- Updated uvcvideo to svn r158
- Add new sync quirk to uvcvideo and use it (needed by Syntek
174f:5212 webcam), from Claudio S. Matsuoka 
- Prevent uvcvideo to alloc too much memory in usb_buffer_alloc,
lowering UVC_MAX_ISO_PACKETS. In machines with not much memory +
webcam that uses isoc transfers, several calls to it can cause
memory fragmentation or requesting too much memory resulting in an
Page Allocation Failure (OOM). UVC_MAX_ISO_PACKETS is arbitrary, see
http://article.gmane.org/gmane.linux.drivers.uvc.devel/1956
- Updated to 2.6.24-rc6-git5

My guess is that there's less than one chnace in 3000 this applys in my case.  Most disturbing to me was that there were so many patches to individual webcams (How many are out there?  1000's ?)

  • From: Austin, TX
  • Registered: 2009-05-07
  • Posts: 471

Re: Chumby webcam server up- Memory leaks?

That matches up with my expectations... it's a leak in the webcam driver.  Too bad!

unwiredben's Website